Ask Your Question
0

How to diagnosing hopeless performance

asked 2015-03-18 13:51:48 -0500

RobertII gravatar image

We have about a dozen CentOS 6 servers being managed by puppet 3.7.4.

Recently, performance on some of the has collapsed to hopeless levels: whereas puppet used to take 5 - 10 minutes to run, processing essentially the same manifest is now taking around 40 minutes. Which is getting silly.

In struggling to diagnose this, I have tried cutting down the content of the node. And discovered that an EMPTY NODE DEFINITION is now taking something over 3 minutes. During this time, the CPU is 100% busy, with Puppet the only process running.

On a closely comparable machine, the "empty node" takes approximately 8 seconds...

How do I diagnose this?

Robert.

P.S. Here is the complete output of puppet apply with "--evaltrace"

Info: Loading facts
Info: Loading facts
Notice: Compiled catalog for XXXX.XXXXXXXXXXX.XX.XX in environment production in 1.71 seconds
Info: Applying configuration version '1426703726'
Info: /Schedule[daily]: Starting to evaluate the resource
Info: /Schedule[daily]: Evaluated in 0.00 seconds
Info: /Schedule[monthly]: Starting to evaluate the resource
Info: /Schedule[monthly]: Evaluated in 0.00 seconds
Info: /Schedule[hourly]: Starting to evaluate the resource
Info: /Schedule[hourly]: Evaluated in 0.00 seconds
Info: Stage[main]: Starting to evaluate the resource
Info: Stage[main]: Evaluated in 0.00 seconds
Info: Class[Main]: Starting to evaluate the resource
Info: Class[Main]: Evaluated in 0.00 seconds
Info: Node[app1]: Starting to evaluate the resource
Info: Node[app1]: Evaluated in 0.00 seconds
Info: Node[app1]: Starting to evaluate the resource
Info: Node[app1]: Evaluated in 0.00 seconds
Info: Class[Settings]: Starting to evaluate the resource
Info: Class[Settings]: Evaluated in 0.00 seconds
Info: Class[Settings]: Starting to evaluate the resource
Info: Class[Settings]: Evaluated in 0.00 seconds
Info: /Schedule[never]: Starting to evaluate the resource
Info: /Schedule[never]: Evaluated in 0.00 seconds
Info: /Filebucket[puppet]: Starting to evaluate the resource
Info: /Filebucket[puppet]: Evaluated in 0.00 seconds
Info: /Schedule[weekly]: Starting to evaluate the resource
Info: /Schedule[weekly]: Evaluated in 0.00 seconds
Info: /Schedule[puppet]: Starting to evaluate the resource
Info: /Schedule[puppet]: Evaluated in 0.00 seconds
Info: Class[Main]: Starting to evaluate the resource
Info: Class[Main]: Evaluated in 0.00 seconds
Info: Stage[main]: Starting to evaluate the resource
Info: Stage[main]: Evaluated in 0.00 seconds
Notice: Finished catalog run in 238.48 seconds
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2015-03-21 14:27:03 -0500

RobertII gravatar image

updated 2015-03-21 14:28:15 -0500

I've added he answer to my related/follow-up question

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

Stats

Asked: 2015-03-18 13:51:48 -0500

Seen: 204 times

Last updated: Mar 21 '15