PuppetDB outage makes Puppet Servers hang
Where I work, we're using PuppetDB 2.3.8 and Puppet Servers 1.2.0, and are facing an issue that occurs during PuppetDB outages. Note that PuppetDB outages are not at all frequent for us, but it happened twice (not because of PuppetDB itself but for other reasons) and our entire fleet of servers was completely unable to run Puppet.
When PuppetDB is down and a Puppet Agent on a random server runs Puppet, the Puppet Servers hang right after outputting the following in the logs:
2017-02-22 16:27:04,504 INFO [puppet-server] Puppet Caching node for agent_hostname
Consequentially, the Puppet agents also hang waiting for the Puppet servers. After restoring PuppetDB, I'm forced to restart the Puppet Servers, after which they eventually output the following to the log:
2017-02-22 16:32:16,144 ERROR [puppet-server] Puppet Failed to submit 'replace facts' command for agent-hostname to PuppetDB at puppetdb-hostname: Error while waiting for http response
Based on my interpretation of the Puppet documentation, I expected the Puppet Servers to at least hard fail due to lack of response from PuppetDB, but not block eternally. It's as if the Puppet Servers don't timeout trying to reach PuppetDB (setting serverurltimeout to a lower value than the default made no difference either). I also attempted to set
storeconfigs to false (since we have that set to true) hoping it would make any difference as the documentation states soft fail works if set to true and if
storeconfigs is set to false. Unfortunately, it made no difference.
What are we missing? There's certainly a misconfiguration on our end. That or we ran into a bug of some sort. I can't seem to find anybody else complaining about this. Any ideas?
Any help is appreciated.