Ask Your Question
1

corrupted crl files after disk full on server

asked 2017-07-17 11:38:35 -0600

praines gravatar image

The root disk on my puppet master server filled up and now I cannot get puppet to run properly.

First when tryin to run puppetmaster it would immediately exit with "Headers too long". After searching Google the issue most people pointed to was zero length files in /var/lib/puppet/ssl. I found ssl/crl.pem and ssl/ca/ca_crl.pem at zero length and removed theml

Then puppet started but nothing can connect to it, not even itself. I get the following errors:

Jul 17 12:07:32 spacewalk puppet-agent[9736]: Reopening log files
Jul 17 12:07:32 spacewalk puppet-agent[9736]: Starting Puppet client version 3.8.7
Jul 17 12:07:33 spacewalk puppet-agent[9743]: Unable to fetch my node definition, but the agent run will continue:
Jul 17 12:07:33 spacewalk puppet-agent[9743]: Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/facts.d]) Failed to generate additional resources using 'eval_generate': Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/facts.d]) Could not evaluate: Could not retrieve file metadata for puppet://spacewalk.nmr.mgh.harvard.edu/pluginfacts: Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve file metadata for puppet://spacewalk.nmr.mgh.harvard.edu/plugins: Connection refused - connect(2)

I then regenerated ca_crl.pem by doing a revoke on a node. Still did not work. I then copied crl.pem from a client node to the master and now it seems to be working.

My question is what is the proper way of handling this situation? Is there any other consequence I need to worry about.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-08-09 23:47:56 -0600

joshc gravatar image

This is due to https://tickets.puppetlabs.com/browse.... Puppet does not atomically write to its CRL, which can cause the file to be corrupted if it runs out of disk space or if two revoke commands happen concurrently.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2017-07-17 11:38:35 -0600

Seen: 46 times

Last updated: Aug 09