profiling code manager

asked 2018-04-10 14:18:35 -0500

sirinek gravatar image

Is there a way to get information from code manager/file sync about how long the various actions are taking? My code deploys are now taking over 2 minutes to complete. They used to be 60-90 seconds with PE 2017.2.2 prior to upgrading to 2017.3.5 and turning on package inventory, and were even faster when we first started using code manager with PE2017.1.1

I am currently running PE 2017.3.5 on RHEL 7.3

bsirinek@puppet-mom [PROD] $ time puppet code deploy production --wait
Found 1 environments.
[
  {
    "deploy-signature": "ee76cfd596ee6b45f23074ce8baeee6e02eeee08",
    "environment": "production",
    "file-sync": {
      "code-commit": "ec6e7fc7b9d0bddeeee1ba74bd1ad1885e6a3eee",
      "environment-commit": "8e0e0e3e1eaefe7e2eee1e1e6e6eeece8e8e6e2e"
    },
    "id": 239,
    "status": "complete"
  }
]

real    2m19.550s
user    0m0.703s
sys     0m0.058s

The production branch Puppetfile specifies 32 modules to get from the forge, and 77 local modules to get from our Bitbucket/git server, which is in the same datacenter as the Puppet MoM.

There are 94 puppet environments in the $CODE_DIR/environments directory (we moved this to /var/opt/puppetlabs/code/environments, since /etc/puppetlabs/code is on the / filesystem and kept filling it up)

On top of this, we have 52 puppet compile masters in several different locations. With the old r10k, we were given output over how long it took to deploy code to each, but with Code Manager/File Sync, I am not sure what the holdup may be. We haven't been adding new modules or compile managers recently. The number of branches is starting to creep up (we have a lot of systems engineers submitting hiera data updates, testing on their branch first) We do only deploy one branch at a time though.

I realize this is a pretty large environment, but I'd like to be able to see if there's something I can do to speed this up, or if I have to just accept that deploys will take a couple minutes. But knowing what is taking most of the time would be a good start.

I'm guessing it could be:

  • Delay in grabbing modules from puppet forge. Does code manager re-download all the forge modules on a deploy?
  • Some puppet compile masters may be slow, either because they are far away or they are having performance issues.
  • It takes a long time to pull the latest updates for everything from our local git server.

Any pointers in how to get additional information would be great.

Thanks

Bill

edit retag flag offensive close merge delete

Comments

> Delay in grabbing modules from puppet forge. Does code manager re-download all the forge modules on a deploy? It depends on how you define the module in your Puppetfile. For instance, using a git id tag, it doesn't call out at all if it's the same. If it's a git tag, it'll make a meta req... etc

frizop gravatar imagefrizop ( 2018-04-15 15:51:34 -0500 )edit

I specify version numbers for all of them... such as: mod 'puppetlabs/accounts', '1.3.0'

sirinek gravatar imagesirinek ( 2018-05-09 12:56:20 -0500 )edit