Ask Your Question
0

Problems with large numbers of Git files created by filesync module

asked 2016-06-10 05:23:49 -0500

gd gravatar image

Recently our dev puppet master ran out of inodes - the culprit was the git files used internally by the filesync module (in turn used by Code Manager). We had over a million files in /opt/puppetlabs/server/data/puppetserver/filesync and these used up all the inodes, and these had reached this number over a period of about 2 weeks. Unsurprisingly this caused a few things to break, so I emptied the directory and restarted everything to get back up and running.

Background: This is a shared server pulling code from Git via code manager, used by multiple teams. In the absence of a Git Hook to trigger the code manager deployment, we have a script running regularly to run the "puppet-code deploy" commands. At the time of the inode problem it was running every minute (so code pushed to Git would almost instantly appear on the host).

I have now moved the cron script to every 15 mins but I'm just delaying the inevitable. And if I remove them while the server is up, puppet server crashes.

Is there a way to prevent filesync from creating so many files, or a process I can use to clear them periodically without causing problems with code manager & puppet server?

Version here is 2016.1

edit retag flag offensive close merge delete

Comments

Out of curiosity - is there a reason you can't use a git hook?

cprice404 gravatar imagecprice404 ( 2016-06-10 14:56:53 -0500 )edit

A git hook would still only slow the ineveitable, correct? If you have teams actively pushing to repos? A cleanup process is probably still warrented

DarylW gravatar imageDarylW ( 2016-06-10 22:39:30 -0500 )edit

There isn't really a reason we can't use a git hook - it is just that our Git environment is managed by another team, including the deployment of Git hooks. so the request is in but it will take a little while. But yes, DarylW is right that we do still need a means to tidy it up.

gd gravatar imagegd ( 2016-06-12 20:40:50 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2016-06-13 09:57:27 -0500

Kevin Corcoran gravatar image

First of all - as cprice404 pointed out - running 'puppet code deploy ...' every minute is a really bad idea. Not only is it going to create a whole bunch of unnecessary data, it's also going to cause puppet server to do a tremendous amount of unnecessary work, meaning that the process will consume more CPU and RAM than it really needs to, and limiting the number of agents to which it can serve catalogs before becoming overloaded.

That said, you're right that this shouldn't happen. We are going to look into improvements that we can make but for now you're going to be somewhat on your own. My suggestion would be to try running 'git gc' in the following directories:

  • /opt/puppetlabs/server/data/puppetserver/filesync/storage/puppet-code.git
  • /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code.git

There are likely additional gains to be made by running 'git gc' in environment-specific Git directories, such as /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code/production.git.

'git gc' can definitely delete a lot of unnecessary files, so this can likely solve your problem. However, I would not recommend doing this while puppet server is running, as I would be worried about the 'git gc' commands causing problems with file sync's Git operations which might occur concurrently if the pe-puppetserver process is running.

edit flag offensive delete link more

Comments

1

Thanks Kevin - I've done some testing and this seems to work well, so I'll look to incorporate this into our maintenance process. Once we have the proper hook in place the filesync activity will of course diminish significantly so this will do nicely - thanks for the help.

gd gravatar imagegd ( 2016-06-14 21:48:16 -0500 )edit

glad to hear it!

Kevin Corcoran gravatar imageKevin Corcoran ( 2016-06-21 10:39:06 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2016-06-10 05:23:49 -0500

Seen: 87 times

Last updated: Jun 13 '16