puppetdb hangs and I see no error messages
PuppetDB keeps hanging and I find no reason why.
puppetserver 2.2.1-1puppetlabs1 running on server1 puppetdb 3.2.3-1puppetlabs1 running on server2 postgres 9.4.5 running on server2 but in a docker container. Approx 8000 puppet agents.
PuppetDB keeps hanging. From the PuppetServer and command curl, I see that the server is not responding:
2016-09-05T09:46:34.722956777Z 2016-09-05 09:46:34,722 WARN [qtp888237476-279] [puppet-server] Puppet Error connecting to server2 on 8081 at route /pdb/cmd/v1?checksum=1e341887631e1749199fc32ef3c2f8bac3a7d7a3, error message received was ''. Failing over to the next PuppetDB url in the 'server_urls' list
$ curl --tlsv1 --cacert /etc/puppetlabs/puppetdb/ssl/ca.pem --cert /etc/puppetlabs/puppetdb/ssl/public.pem --key /etc/puppetlabs/puppetdb/ssl/private.pem https://server2:8081/pdb/meta/v1/version PuppetDB is currently down. Try again later.
In Postgresql logs, I see only the messages "WARNING: there is already a transaction in progress". But I've searched and found that I should ignore this message in this version of puppetdb.
In /var/log/puppetlabs/puppetdb/puppetdb.log, I see only this:
2016-09-05 01:24:01,294 INFO [p.p.c.services] Starting sweep of stale reports (threshold: 14 days) 2016-09-05 01:24:01,319 INFO [p.p.c.services] Finished sweep of stale reports (threshold: 14 days) 2016-09-05 01:24:01,320 INFO [p.p.c.services] Starting database garbage collection 2016-09-05 01:24:03,295 INFO [p.p.c.services] Finished database garbage collection
And if I restart the postgres container and puppetdb, It will spend a few minutes in maintenance mode, and then It works again correctly for 20 minutes or so.
Do you know how I could troubleshoot this? Thanks for your help.