Ask Your Question

Ensuring idempotency using exec?

asked 2015-07-03 22:54:49 -0500

R gravatar image

I have about 20 python scripts which I am invoking via exec. My question is how to ensure idempotency? One way I could think of is to touch a different file every time the one script is executed. But that would require us to maintain 20 different files. I was looking at "refreshonly" in exec, where the triggers only if there is an event (notify or subscribe). I am not clear on this refreshonly option. Could somebody shed some light on this?

Thanks in advance

edit retag flag offensive close merge delete


It's hard to know what the best way to solve your problem might be without knowing what the 20 python scripts do.

Alex Harvey gravatar imageAlex Harvey ( 2015-07-04 00:31:19 -0500 )edit

As a workaround, let each script create a file. Test for the existence of this file before running the script (unless, onlyif). Of course, other commands in each script might fail silently then.

sinned gravatar imagesinned ( 2015-07-07 03:14:26 -0500 )edit

@ sinned Thanks a lot yes that is exactly what I was thinking, but the drawback is I will have to maintain 20 different files. Do you know what the refreshonly option in exec does? I am not too clear from reading the doc.

R gravatar imageR ( 2015-07-07 19:17:03 -0500 )edit

2 Answers

Sort by » oldest newest most voted

answered 2015-07-08 14:56:56 -0500

cbarbour gravatar image

The best way to ensure idempotency with exec is to invert your approach. Rather than focusing on how exec should invoke your script, focus on what this script is intended to accomplish. Test for that using the onlyif, unless, creates, and refreshonly parameters.

Focusing on the test rather than the command to run is declarative approach to writing exec statements.

For example, if your script is responsible for managing security settings, start by writing unless tests for those settings. Remember that you can pass an array to unless (all conditions must pass.) In this case, your test declares that the security settings are required, and execs the script to correct them if they are not set correctly.

If your script is responsible for installing an application, use an unless test that checks to see what version of the application is currently installed. If the application is not installed, or is the wrong version, the installer will then be invoked. In this case, your exec resource declares that a specific version of the application should be installed, and runs the installer if it's not correct.

I generally recommend refreshonly in cases where your script is actually refreshing something. Refreshonly can be unreliable; if the exec resource fails, there's no guarantee it will re-attempt the next time Puppet runs. If you do use refreshonly, be sure that a notification event will always be generated until the refresh succeeds.

I hope this helps.

edit flag offensive delete link more


Thanks a lot it surely does help :-)

R gravatar imageR ( 2015-07-09 22:02:06 -0500 )edit

answered 2015-07-09 09:22:06 -0500

helge gravatar image

updated 2015-07-09 09:25:21 -0500

cbarbour says it all - but let me add from experience:

IMHO managing Exec idempotency is hard with Puppet. In the long run with code I write myself I always found it much easier to make the script idempotent then to do it in Puppet.

Puppet Doc states:

Any command in an exec resource must be able to run multiple times without causing harm — that is, it must be idempotent.

I read this the way the script itself should take care of itself. A not perfect but common practice is to write these pid files inside your command. If you know the PID path, you can even use native creates => $pidfile to skip the run.

If this is too cumbersome or else not desirable, next thing I would recommend is refreshonly => true and have some other more easily manageable resource trigger the Exec with notify => Exec[$myexec].

Only as a last resort do the testing with onlyif and unless since it most certainly breaks platform independence (witch might be irrelevant for you, but if not you will end up writing a lot more puppet code):

exec {$mybin:
  command => "${mbybin} --myparam",
  unless  => "/usr/bin/ps aux | /usr/bin/grep \"${mybin}\" | /usr/bin/grep -v grep"

Hint: if you really must have your Exec run and there might be another instance running but doing some other thing. have a look at tries => and try_sleep.

edit flag offensive delete link more


Every exec resource should have a condition, otherwise Puppet will report change even if no change took place. Regarding locking; Puppet itself has a lockfile, and it blocks on exec statements, so locking isn't super critical.

cbarbour gravatar imagecbarbour ( 2015-07-09 15:38:58 -0500 )edit

@helge, @cbarbour, thanks a lot.

R gravatar imageR ( 2015-07-09 22:04:22 -0500 )edit

cbarbour, certainly. I like my manifests to do 'apply' only if necessary; so I mostly use a conditional around the Exec or 'refreshonly' . But this is also a philosophy question of your code. IHMO the exec should fail or do nothing rather then apply the same thing twice.

helge gravatar imagehelge ( 2015-07-10 03:43:49 -0500 )edit

Conditional logic breaks Puppet's reporting system. Personally, I try to avoid wrapping resources in conditional logic; unless the resource simply is inapplicable to the target host or platform.

cbarbour gravatar imagecbarbour ( 2015-07-10 17:35:03 -0500 )edit

Care to elaborate about 'braking Puppet's reporting system'? Currently I favor conditionals mostly in conjunction with (custom) facts to apply execs if facilities like 'creates' do not work. (I use them manly to work around some puppet bugs / missing functions, try install rubygem-git package git)

helge gravatar imagehelge ( 2015-07-11 05:13:34 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2015-07-03 22:54:49 -0500

Seen: 1,255 times

Last updated: Jul 09 '15