Ask Your Question

Puppet CA Shared Certificate Guide: Scalable Puppet?

asked 2016-09-09 08:24:22 -0500

Ivan Arjune gravatar image

updated 2016-09-19 20:39:23 -0500

This setup was so simple to get off the ground and running.
Only a couple of configs and it works great, at least in my lab it does.
So the question is, where's the gotcha here. Will it work in production. Why do i need to centralize the ca if all the masters can sign certs.

If i'm right this should be a high availability installation of puppet that scales easily, i'm usually wrong so do hurt me too bad.

Testing Environment

puppet version: 4.6.2
puppetserver version: 2.5.0
Hostnames: puppetserver-01.vm, puppetserver-02.vm
DNS SRV record: puppetserver.vm

Install packages


Shared Cert. Creation. The SSL Directory

In puppet.conf add:
dns_alt_names = puppet, puppetserver.vm

Then run 
# puppet cert generate --allow-dns-alt-names puppetserver.vm

Remove dns-alt-names from puppet.conf

We dont need this any more.  Its purpose was just to create the shared cert.

Point the webserver to the certificates

Add the following to /etc/puppetlabs/puppetserver/conf.d/webserver.conf
webserver: {
  access-log-config: /etc/puppetlabs/puppetserver/request-logging.xml
  client-auth: want
  ssl-port: 8140
  ssl-cert : /etc/puppetlabs/puppet/ssl/certs/puppetserver.vm.pem
  ssl-key : /etc/puppetlabs/puppet/ssl/private_keys/puppetserver.vm.pem
  ssl-ca-cert : /etc/puppetlabs/puppet/ssl/certs/ca.pem
  ssl-cert-chain : /etc/puppetlabs/puppet/ssl/certs/ca.pem
  ssl-crl-path : /etc/puppetlabs/puppet/ssl/crl.pem

Optional Stuff

Setup hiera.yaml
Copy puppet code
Setup autosign.conf
Disable IPv6
Firewall rule for 8140

Start the server

service puppetserver start

Tar up the ssl directory.

When deploying the next puppet master/ca server you'll need this.

Launch your next puppetserver/ca node

-extract the ssl directory
-set the permissions for the ssl directory
-modify the webserver.conf
-start the service.


With all your puppet servers behind a proxy, you just point the clients to this address. Now they can have their certs signed by any ca and all ca's will verify the cert just fine.

edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted

answered 2016-10-18 15:21:32 -0500

Ivan Arjune gravatar image

Thanks everyone for your interest in this topic and helping me understand the follies in my ways. I see now how this is not the right way to go about the problem due to technical design limitations. We have decided to go with a single CA and deal with a provisioning outage, at least until the downtime becomes unacceptable. Thanks again everyone.

edit flag offensive delete link more

answered 2016-09-12 01:35:10 -0500

On which of the masters are you going to sign the client certificate signing request? You have two options: 1. you use one dedicated for signing and copy the ssl dir over to all puppet servers. 2. use a central ca which is responsible for signing, cleaning and revoking certificates.

edit flag offensive delete link more


What sorta problems can be expected if all masters sign new certs. When reprovisioning a node we can run a disturbed command to clean the certs. How does puppet go about cleaning certs when you wipe or reprovision? Isn't it still a manual process?

Ivan Arjune gravatar imageIvan Arjune ( 2016-09-12 07:10:19 -0500 )edit

The problem that usually comes up when multiple active CAs are signing certs is that every certificate needs a unique serial number. Puppet stores the next serial to be issued in /etc/puppetlabs/puppet/ssl/ca/serial. Keeping this file properly synced and locked in an active/active CA setup is hard.

csharpsteen gravatar imagecsharpsteen ( 2016-09-12 10:01:50 -0500 )edit

Is this more of a security concern than functionality? Will clients fail compilation for any particular reason? When a client requests a certificate it is signed only once and that serial should be unique. Is there a possibility that the same serial will exist for another node on another ca?

Ivan Arjune gravatar imageIvan Arjune ( 2016-09-19 08:28:03 -0500 )edit

The usual time that serial number collision causes an issue is when an agent node is de-commissioned (or re-keyed) and it's certificate is cleaned. Revocation is done by serial number --- which means that the certificates of other nodes can become collateral damage.

csharpsteen gravatar imagecsharpsteen ( 2016-10-18 12:13:28 -0500 )edit

answered 2016-09-20 08:59:59 -0500

Jeremiah Powell gravatar image

updated 2016-09-20 09:17:47 -0500

These are not supported configurations. I have used them in the past and they have serious issues. Why is rather complex.

In short, to have a distributed SSL Certificate Authority (CA), where multiple servers sign and revoke certificates for a single CA, all signing systems need to coordinate with each other. Almost any CA commercial system worth purchasing has either built-in support for this or some outside mechanism to handle the coordination. Puppet, and Puppet Enterprise, does not.

A Certificate Authority is just an X.509 document, private-key/public-key pair and a Certificate Revocation List(CRL). The private key is used to sign other X.509 documents. The private key also signs the CRL document.

Puppet agents authenticate to a master using a signed client SSL certificate. This is an X.509 document containing something that identifies the client but was signed with the private-key from the CA. That certificate has a serial number generated by the signing process.

SSL serial numbers are not Globally Unique Identifiers but simple ordered integers. Two different servers signing for the same CAs are guaranteed to have large numbers of conflicting serial numbers. A single CA will duplicate serial numbers if the CRL or existing lists of certificates is missing.

These serial numbers have to remain unique for the private key and SSL certificate from the CA used to generate the signature. This is because of the Certificate Revocation List (CRL). The CRL is just a list of serial numbers revoked by the CA. The CA signed the list so the clients and servers trusting the CA know list list is valid.

If you have two different client certificates with the same serial number signed by the same CA then a valid Certificate Revocation List (CRL) cannot be created for that CA. Furthermore any collection of certificates that contain the duplicate certificates cannot uniquely identify the certificate anymore.

This latter duplicate is a serious problem with Puppet Enterprise when Puppet Masters are reinstalled or replaced. Clients like web browsers see a certificate for with a different signature or a duplicate serial number and generate errors instead of connecting the the server.

To have multiple signing servers you will need to coordinate the signing and revoking of certificates between servers. This can be done by locking systems to prevent concurrent update. It can also be done by some transaction system with the next valid serial number and CRLs exchanged between all signing systems.

Today puppet cert lacks any of this logic. The CRL is shuttled around by the Puppet Agents on the Puppet Masters during their runs. This cause service downtime as CRL updates restart Puppet Masters. At best you can backup the CRL, keys and CA to another master but you have to stick to signing consistently on just one master.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2016-09-08 20:40:27 -0500

Seen: 167 times

Last updated: Oct 18 '16