Ask Your Question

puppet infrastructure scaling: Best practice?

asked 2013-08-27 05:53:21 -0600

mgk gravatar image


We have been using puppet for several years now to manage ~ 100 nodes over a global network we directly control.

This has served us well up until now with a single master with a few sacrifices.

Is there any documentation or guides on best practices for scaling puppet? Puppet is becoming more and more key to our services.

The idea or design that I have started to investigate is a several puppet masters deployed. Two in the UK serving Europe, 1 in LA for LA/AUS/NZ. These servers would anycast the puppet master IP. Behind the scenes git ... (more)

edit retag flag offensive close merge delete


DB performance over that wide an area would be my first concern. My second concern would be keeping the masters in sync (orchestrating git pulls across environments, and then auditing ...(more)

Ancillas gravatar imageAncillas ( 2013-08-28 03:47:38 -0600 )edit

5 Answers

Sort by ยป oldest newest most voted

answered 2013-08-29 13:13:39 -0600

nibalizer gravatar image

Scaling Puppet is essentially a giant flow chart. The first split is: are you using Puppet Open Source or Puppet Enterprise.

If you're using PE, the Puppet Labs people have a bunch of tricks you can dialogue with them about how to scale.

If you're using Puppet Open Source, then there are a multitude of technologies you can use to scale Puppet.

There are multiple layers of a Puppet installation. This link explains each layer and provides some insight into the challenges behind each.

Roughly, it says that there ... (more)

edit flag offensive delete link more


Thanks this break down is exactly the sort of thing I was hoping for. I was planning on using fabric instead of mcollective but will look into this. Thanks

mgk gravatar imagemgk ( 2013-08-30 06:57:15 -0600 )edit

If this answer is satisfactory, you can press the checkmark to 'accept' the answer. It helps other people who land here from search engines know that there is a solution ...(more)

nibalizer gravatar imagenibalizer ( 2013-08-30 12:20:55 -0600 )edit

Enjoy the internet points, thanks.

mgk gravatar imagemgk ( 2013-08-30 15:20:33 -0600 )edit

answered 2013-08-28 09:53:47 -0600

aharden gravatar image

The book "Pro Puppet" by James Turnbull and Jeffrey McCune (Amazon link) addresses scalability in Chapter 4. Recommendations include:

  • Running the Puppet Master with Apache and Passenger
  • Load-balancing Multiple Puppet Masters (addressing HTTP as well as the Puppet CA)

They go into a lot of detail. I'm still pretty new to Puppet, but it seems like this would address a lot of your questions.

edit flag offensive delete link more

answered 2013-08-28 14:51:20 -0600

banjer gravatar image

updated 2013-08-28 14:52:31 -0600

I'm approaching 100 puppet agents myself, so I've started some research. I've read several articles about using a masterless puppet setup when you have many nodes in your environment (200+ in the examples I came across).

For my setup, I like the idea of load balancing multiple puppet masters with some sort of proxy (haproxy, perhaps?). Round robin DNS or DNS srv records (newly supported in Puppet 3.x) will probably come into play here as well.

I like the notion of a decentralized environment using masterless puppet, but AFAIK I'd have to give up Foreman ... (more)

edit flag offensive delete link more


I agree with you. I like the idea of masterless Puppet because it aligns the deploy process of modules/environments with application releases, but it's hard to give up ...(more)

Ancillas gravatar imageAncillas ( 2013-08-29 22:49:51 -0600 )edit

answered 2013-08-28 11:47:44 -0600

mgk gravatar image

Hi All

Thanks for your answer.
Ancillas, I was planning on following setup similar to this guide for the CA Your comment re DB performance is a valid one. One we will need to investigate. I was considering a hot spare but perhaps some sort of clustered setup would be best. Will need to look into that. Thanks for flagging up the git deployment side, I had considered this to be one of the better options here. We've not had issues thus far with other projects where this is used ... (more)

edit flag offensive delete link more



This is definitely an area where theory and practicality meet together in different ways based on need. I'd love to read a blog post once you come up with ...(more)

Ancillas gravatar imageAncillas ( 2013-08-29 14:30:00 -0600 )edit

Also, I'm rarely a fan of hot spares. Having multiple active servers tests the redundancy constantly, while also distributing load, at the cost of increased complexity.

Ancillas gravatar imageAncillas ( 2013-08-29 23:12:16 -0600 )edit

Blog post sounds like a good idea, will look into it on our company site.

mgk gravatar imagemgk ( 2013-08-30 06:57:39 -0600 )edit

answered 2013-08-28 10:46:36 -0600

LeCouz gravatar image

On the paper, it is great. But all this depend of your network.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2013-08-27 05:53:21 -0600

Seen: 1,466 times

Last updated: Aug 29 '13