Ask Your Question
0

Node availability dependent behaviour

asked 2014-11-03 08:50:52 -0500

Glueon gravatar image

Each agent using exported resources adds itself to the cluster as a frontend or a backend and then I use this information to build dns zone and nginx configs.

In case agent goes down I want to exclude its exported resources therefore its IP will disappear from nginx and dns zone.

What is the best solution?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-11-03 12:51:19 -0500

cbarbour gravatar image

updated 2014-11-03 13:22:19 -0500

Exported resources are generally best for failure tolerant services. For example, they are great for haproxy backend servers and nagios host entries. The managed system it's self can detect and handle failures.

For services that need more responsive pool management, your best bet is to look at a service discovery tool such as Consul, or SmartStack.

Edit: The original poster asks:

I thought there is a possibility to detect if node failed while collecting exported resources. So for example if node fails i detect that using nagios and and run puppet apply using mcollective. While the puppet run agents collect only exported resources received from online nodes.

It can, but there are a bunch of caveats. See the PuppetDB documentation: de-commissioning nodes using a TTL.

Puppet exports resources during catalog compilation, and it does not assume that the node is down simply because it hasn't checked in. After all, it's possible that Puppet has simply been disabled on the node, and that the node is still up.

In order to purge the resources from PuppetDB, you must explicitly deactivate the node or allow the TTL of the node to expire.

Short TTLs run the risk that healthy nodes may be purged if for some reason they haven't checked in for a while, or if for some reason your puppet infrastructure becomes partitioned in such a way that the nodes can't check in.

Another factor is that simply un-exporting a resource isn't enough to remove that resource from DNS and haproxy. Unless you explicitly tell puppet to purge all un-managed resources, an undefined resource is neither present nor absent; it is simply not-managed. That means that Puppet will allow the configuration for your failed node to exist, even if it's not exported, unless you explicitly use the resources metatype to purge that node's configuration from other nodes who are realizing your exported resource.

Purging nodes runs the risk that your entire pool may be purged if for some reason PuppetDB fails without disabling your Puppet Masters.

So, to accomplish what you're asking, you need to do the following:

  1. Setup Nagios or a TTL to purge deactivated nodes.
  2. Configure your services to purge unmanaged resources.

While Puppet can absolutely do what you're asking, there are tools designed to solve exactly this problem. Those tools are more robust and much more responsive.

edit flag offensive delete link more

Comments

I thought there is a possibility to detect if node failed while collecting exported resources. So for example if node fails i detect that using nagios and and run puppet apply using mcollective. While the puppet run agents collect only exported resources received from online nodes.

Glueon gravatar imageGlueon ( 2014-11-03 13:02:59 -0500 )edit

Is is possible to use mcollective for such tasks? As far as I know it can check whether node is up or down. So maybe it can purge a given exported resource and force puppet run on each node to update? I checked consul and do not understand how can I both manage nginx.conf with puppet and consul.

Glueon gravatar imageGlueon ( 2014-11-03 16:03:13 -0500 )edit

Look into consul template. Have puppet write the template, and use Consul to render the template into a final config. https://hashicorp.com/blog/introducing-consul-template.html

cbarbour gravatar imagecbarbour ( 2014-11-03 16:56:58 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2014-11-03 08:50:52 -0500

Seen: 382 times

Last updated: Nov 03 '14