corrupted crl files after disk full on server

asked 2017-07-17 11:38:35 -0500

The root disk on my puppet master server filled up and now I cannot get puppet to run properly.

First when tryin to run puppetmaster it would immediately exit with "Headers too long". After searching Google the issue most people pointed to was zero length files in /var/lib/puppet/ssl. I found ssl/crl.pem and ssl/ca/ca_crl.pem at zero length and removed theml

Then puppet started but nothing can connect to it, not even itself. I get the following errors:

Jul 17 12:07:32 spacewalk puppet-agent[9736]: Reopening log files
Jul 17 12:07:32 spacewalk puppet-agent[9736]: Starting Puppet client version 3.8.7
Jul 17 12:07:33 spacewalk puppet-agent[9743]: Unable to fetch my node definition, but the agent run will continue:
Jul 17 12:07:33 spacewalk puppet-agent[9743]: Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/facts.d]) Failed to generate additional resources using 'eval_generate': Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/facts.d]) Could not evaluate: Could not retrieve file metadata for puppet://spacewalk.nmr.mgh.harvard.edu/pluginfacts: Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': Connection refused - connect(2)
Jul 17 12:07:33 spacewalk puppet-agent[9743]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve file metadata for puppet://spacewalk.nmr.mgh.harvard.edu/plugins: Connection refused - connect(2)

I then regenerated ca_crl.pem by doing a revoke on a node. Still did not work. I then copied crl.pem from a client node to the master and now it seems to be working.

My question is what is the proper way of handling this situation? Is there any other consequence I need to worry about.

edit retag flag offensive close merge delete