Puppetmaster Caching catalog
Puppetmaster 3.7.4/Centos 7, using Hiera as my ENC, puppetdb for storedconfigs, 500+ nodes, all seems great, agents are humming along with no apparent issue except the agent on the puppetmaster itself. It runs for sometimes as long as 4 minutes, CPU pegged at 100%, and using over a 1 GB of memory.
I'm guessing I have something wrong in my puppet.conf on the master but I'm not sure where to start looking. Can someone point me in the right direction.
I thought it was the tidy resource in my puppetmaster's node manifest which I use to cleanup old reports but I disabled that and still see the same behavior.
When I run with debug I get 3 lengthy pauses:
....... Info: Applying configuration version '1429657329'
....... Debug: Finishing transaction 25792720 Debug: Loaded state in 68.51 seconds
and then ...........
Debug: Creating default schedules Debug: Loaded state in 92.10 seconds
and lastly ..............
Debug: Storing state Debug: Stored state in 75.01 seconds
........ Notice: Finished catalog run in 175.18 seconds
PuppetDB and Pupptboard however report that the run took just over 10 seconds:
Hostname Configuration version Start time End time puppet.nym1.placeiq.net 1429657329 Apr 21 2015 - 19:04:56 Apr 21 2015 - 19:05:08
More info, /var/lib/puppet/state/state.yaml is about 65 MB
it's full of all the report files that have been tidied, so maybe using tidy to maintain the report is a bad idea?
PuppetDB claiming 10 seconds is strange though.
Yep, that huge state file was the reason ..... but why is that happening, why is it still tracking resources that no longer exist, ie. file that tidy removed a long time ago.