Ask Your Question
0

while executing puppet agent -t getting error

asked 2018-02-23 04:11:20 -0500

gyanendra.ojha gravatar image

updated 2018-02-26 12:49:26 -0500

DarylW gravatar image

while executing puppet agent -t getting error

Could not request certificate: The certificate retrieved from the master does not match the agent's private key. Certificate fingerprint: 74:D2:CF:0B:3A:1F:4D:E3:34:49:85:95:F1:C2:0B:FD:91:A7:6C:00:71:37:1C:4C:5B:45:0E:80:01:C1:A1:E7 To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate. On the master: puppet cert clean qa-ccios-524.use1.greeintl.net On the agent: rm -f /var/lib/puppet/ssl/certs/qa-ccios-524.use1.greeintl.net.pem puppet agent -t
edit retag flag offensive close merge delete

Comments

i want permanent fix for it , i am creating puppet agent instance on aws daily so even if i get it fixed once i get it b again after the server is newly created

gyanendra.ojha gravatar imagegyanendra.ojha ( 2018-02-23 04:14:28 -0500 )edit

3 Answers

Sort by ยป oldest newest most voted
0

answered 2018-02-23 11:40:59 -0500

reidmv gravatar image

updated 2018-02-23 15:40:02 -0500

There are two things that could be contributing to this problem.

  1. If your certificate names are re-used frequently and aren't cleaned up when instances deprovision (as will be the case in EC2 if you use the default certname), you could hit this periodically.
  2. There is this bug in Puppet. Depending on how you provision agents, a race condition can cause the private key to be corrupted on the agent after it has submitted its CSR. This will cause the error you're reporting.

To avoid #1, it's recommended in EC2 that the instance ID of the node be incorporated in the certname, among other node lifecycle management recommendations. See this whitepaper for more details.

To avoid #2 (the bug), it's important to make sure Puppet isn't invoked twice in quick succession during or immediately after install. This can easily happen for example if you have a provisioning script that both installs the agent, and runs puppet agent -t.

In Puppet Enterprise, you can avoid this by either of these options:

  1. Don't call puppet agent -t directly in your user-data script or provisioner. Let the service run by itself.
  2. Make sure the service isn't started when you install the package. After installation run puppet agent -t, and then start the service. E.g.

    instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
    
    curl -k https://master.example.com:8140/packages/current/install.bash | sudo bash -s -- \
      main:certname="$instance_id" \
      extension_requests:pp_instance_id="$instance_id" \
      --puppet-service-ensure stopped
    
    /opt/puppetlabs/bin/puppet agent -t
    /opt/puppetlabs/bin/puppet resource service puppet ensure=running
    

    This just makes sure the service doesn't try to run in the background at the same time you run puppet agent -t in the foreground. More information on these Puppet Enterprise agent installation options.

edit flag offensive delete link more

Comments

Thanks buddy will try out ur suggesstions and let you know :)

gyanendra.ojha gravatar imagegyanendra.ojha ( 2018-02-23 12:00:23 -0500 )edit
0

answered 2018-02-23 09:03:34 -0500

DarylW gravatar image

So, you have a persistent master, but rotating nodes, and your node hostnames get reused? If the master already has a cert, it will decline a new cert from the same 'certname', which defaults to the hostname.

I know of two solutions to this problem 1) You can write a custom cleanup job that runs on a cron (every X mins) that does an ec2 describe instances, parses out the list of hostnames that are still in your environment, and remove hosts using the puppet cert clean $node for each 'offending' node, which will then 'free up' that hostname to be used for a new cert.

We ran into that problem because we were using the aws ip based hostnames, and they would eventually roll over (ip-1-2-3-4.region) and get reused.

2) Another option is to set up nodes with permission to delete their own cert, and as a part of spinning down your boxes, you can have them connect to the puppet master, via rest, and revoke their own cert. That will do the equivalent of the above, but on a node by node basis.

Some information is found in this discussion https://groups.google.com/forum/#!top... That talks about using the REST API to deactivate a node, but I'm not sure on what is required to give your individual node the right permission to do that on the master... If I can track down those steps, I'll update this post

edit flag offensive delete link more

Comments

Hi, Thanks for suggesstion. i will explain you the complete process. i am getting this error everytime the AWS instance is recreated . now even if i am deleting its old certificates from master manually still getting the same issue

gyanendra.ojha gravatar imagegyanendra.ojha ( 2018-02-23 09:07:31 -0500 )edit
0

answered 2018-02-24 02:14:12 -0500

gyanendra.ojha gravatar image

updated 2018-02-26 12:50:06 -0500

DarylW gravatar image

Hi Buddy,

getting below error now, i tried as you said like giving --certname instance_id and error is changed

[[0;32mInfo: Creating a new SSL key for i-0a3329aeef95d69ec^[[0m
^[[mNotice: Using less secure serialization of reports and query parameters for compatibility^[[0m
^[[mNotice: with older puppet master. To remove this notice, please upgrade your master(s) ^[[0m
^[[mNotice: to Puppet 3.3 or newer.^[[0m
^[[mNotice: See http://links.puppetlabs.com/deprecate_yaml_on_network for more information.^[[0m
^[[0;32mInfo: Caching certificate for ca^[[0m
^[[0;32mInfo: csr_attributes file loading from /etc/puppet/csr_attributes.yaml^[[0m
^[[0;32mInfo: Creating a new SSL certificate request for i-0a3329aeef95d69ec^[[0m
^[[0;32mInfo: Certificate Request fingerprint (SHA256): B9:8E:23:62:CC:CE:C1:B7:E5:D6:E2:1C:52:65:2B:99:82:83:70:70:2C:16:6A:16:E5:3B:D2:F9:AE:43:3B:C9^[[0m
^[[0;32mInfo: Caching certificate for i-0a3329aeef95d69ec^[[0m
^[[1;31mError: Could not request certificate: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [certificate signature failure for /CN=puppet-blue-i-09f61d3ab59e69ad3.int.funzio.com]^[[0m
edit flag offensive delete link more

Comments

1

From that error it sounds like you're using a very, very old version of Puppet. The current Puppet release is 5.4. Puppet 3.8 was the last release of the 3.x series, and was EOL'd 2016-12-31. I would highly recommend upgrading to a recent release. It'll be much more capable and stable.

reidmv gravatar imagereidmv ( 2018-02-26 11:01:12 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2018-02-23 04:11:20 -0500

Seen: 88 times

Last updated: Feb 26