PuppetDB outage makes Puppet Servers hang

asked 2017-02-22 19:44:47 -0600

Hi,

Where I work, we're using PuppetDB 2.3.8 and Puppet Servers 1.2.0, and are facing an issue that occurs during PuppetDB outages. Note that PuppetDB outages are not at all frequent for us, but it happened twice (not because of PuppetDB itself but for other reasons) and our entire fleet of servers was completely unable to run Puppet.

When PuppetDB is down and a Puppet Agent on a random server runs Puppet, the Puppet Servers hang right after outputting the following in the logs:

2017-02-22 16:27:04,504 INFO [puppet-server] Puppet Caching node for agent_hostname

Consequentially, the Puppet agents also hang waiting for the Puppet servers. After restoring PuppetDB, I'm forced to restart the Puppet Servers, after which they eventually output the following to the log:

2017-02-22 16:32:16,144 ERROR [puppet-server] Puppet Failed to submit 'replace facts' command for agent-hostname to PuppetDB at puppetdb-hostname: Error while waiting for http response

Based on my interpretation of the Puppet documentation, I expected the Puppet Servers to at least hard fail due to lack of response from PuppetDB, but not block eternally. It's as if the Puppet Servers don't timeout trying to reach PuppetDB (setting serverurltimeout to a lower value than the default made no difference either). I also attempted to set storeconfigs to false (since we have that set to true) hoping it would make any difference as the documentation states soft fail works if set to true and if storeconfigs is set to false. Unfortunately, it made no difference.

What are we missing? There's certainly a misconfiguration on our end. That or we ran into a bug of some sort. I can't seem to find anybody else complaining about this. Any ideas?

Any help is appreciated.

Thank you.

Best Regards,

Pedro

edit retag flag offensive close merge delete

Comments

1

There are some settings in Puppet Server which should apply here: connect-timeout-milliseconds (defaults to 2 minutes) and idle-timeout-milliseconds (defaults to 20 minutes). See https://docs.puppet.com/puppetserver/2.7/config_file_puppetserver.html#examples. You may want to lower these.

camlow325 gravatar imagecamlow325 ( 2017-02-25 09:06:24 -0600 )edit
1

Unfortunately, the server_url_timeout in PuppetDB has no real effect when running in Puppet Server. https://tickets.puppetlabs.com/browse/SERVER-1666 was filed to investigate this further.

camlow325 gravatar imagecamlow325 ( 2017-02-25 09:09:46 -0600 )edit

@camlow325 the first comment was super helpful! I just tried reducing those values and the Puppet Servers timed out while trying to reach PuppetDB as expected ! As a consequence, the Puppet Agent was able to run Puppet successfully despite PuppetDB being down! Thank you so much!! :))

Pedrorochagoncalves gravatar imagePedrorochagoncalves ( 2017-03-01 22:01:00 -0600 )edit