Ask Your Question
0

What could be the cause for my node hanging during some Puppet runs?

asked 2017-04-27 09:09:38 -0500

href gravatar image

updated 2017-04-28 08:38:23 -0500

I have a deployment of about 60 Puppet nodes. Most of the nodes run on either Ubuntu 16.04 or 14.04.

For a few days now two of those nodes have been acting up. Roughly one out of two times the puppet agent will randomly hang when connecting to the puppetmaster. On both nodes this is the output I receive when running sudo puppet agent --test --debug:

Debug: Applying settings catalog for sections main, agent, ssl
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Failed to load library 'selinux' for feature 'selinux'
Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppetlabs/puppet]{:path=>"/etc/puppetlabs/puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'codedir': 'File[/etc/puppetlabs/code]{:path=>"/etc/puppetlabs/code", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Facter: searching for custom fact "operatingsystem".
Debug: Facter: searching for operatingsystem.rb in /opt/puppetlabs/puppet/cache/lib/facter.
Debug: Facter: searching for operatingsystem.rb in /opt/puppetlabs/puppet/cache/facts.
Debug: Facter: searching for operatingsystem.rb in /opt/puppetlabs/puppet/cache/lib/facter.
Debug: Facter: searching for operatingsystem.rb in /opt/puppetlabs/puppet/cache/facts.
Debug: Facter: fact "facterversion" has resolved to "3.6.3".
Debug: Facter: fact "aio_agent_version" has resolved to "1.10.0".
Debug: Facter: searching "/opt/puppetlabs/facter/facts.d" for external facts.
Debug: Facter: skipping external facts for "/etc/facter/facts.d": No such file or directory
Debug: Facter: skipping external facts for "/etc/puppetlabs/facter/facts.d": No such file or directory
Debug: Facter: no external facts were found.
Debug: Facter: resolving operating system facts.
Debug: Facter: resolving kernel facts.
Debug: Facter: fact "kernel" has resolved to "Linux".
Debug: Facter: fact "kernelrelease" has resolved to "4.9.15-x86_64-linode81".
Debug: Facter: fact "kernelmajversion" has resolved to "4.9".
Debug: Facter: fact "kernelversion" has resolved to "4.9.15".
Debug: Facter: executing command: /usr/bin/lsb_release -a
Debug: Facter: Distributor ID:  Ubuntu
Debug: Facter: Description: Ubuntu 16.04.2 LTS
Debug: Facter: Release: 16.04
Debug: Facter: Codename:    xenial
Debug: Facter: No LSB modules are available.
Debug: Facter: process exited with status code 0.
Debug: Facter: fact "osfamily" has resolved to "Debian".
Debug: Facter: fact "operatingsystemmajrelease" has resolved to "16.04".
Debug: Facter: fact "operatingsystemrelease" has resolved to "16.04".
Debug: Facter: fact "hardwaremodel" has resolved to "x86_64".
Debug: Facter: fact "architecture" has resolved to "amd64".
Debug: Facter: fact "lsbdistid" has resolved to "Ubuntu".
Debug: Facter: fact "lsbdistcodename" has resolved to "xenial".
Debug: Facter: fact "lsbdistdescription" has resolved to "Ubuntu 16.04.2 LTS".
Debug: Facter ...
(more)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-05-04 04:01:06 -0500

href gravatar image

I worked around my issue by adding an HTTP read timeout to all my nodes:

cat /etc/puppetlabs/puppet/puppet.conf

...
[agent]
http_read_timeout = 30

Occasionally the nodes still trip up but it happens much less frequently now, which leads me to assume that the HTTP connections somehow never got terminated and would, if not cause, then certainly increase the problem.

Not really something that counts as a solution, but certainly better than nothing and maybe helpful to others.

edit flag offensive delete link more

Comments

I had run into a similar problem, except it was a misconfigured proxy. In my case, once I had the appropriate proxy/no_proxy configuration passed to puppet (via /etc/sysconfig/puppet) my problem went away

DarylW gravatar imageDarylW ( 2017-05-04 10:12:42 -0500 )edit

I could only set http_proxy in puppet.conf, but NOT no_proxy, which I needed in my setup

DarylW gravatar imageDarylW ( 2017-05-04 10:13:21 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

Stats

Asked: 2017-04-27 09:09:38 -0500

Seen: 80 times

Last updated: May 04