PuppetDB in production: best practise for reliability and integrity
I'm planning to use PuppetDB and the exported resources to automatically define firewall rules over the infrastructure. Firewall rules are very important ans sensitive. Everything should be reliable and integer.
I've the following easy scenario:
A group of nodes "MariaDB" and a group of nodes "webserver". The webservers should have access to the MariaDB nodes (always, otherwise my site will fail..) If I add a new webserver node, the MariaDB iptables should be updated. (yes, it doesn't happen immediately)
reliability: puppetdb went down and puppet is running on the nodes. Which caused the puppet run to fail and the firewall rules are staying in the old state. => solution: adding a second / third puppetdb for redundancy
integrity what happen if the exported webserver IP resources list is empty due a faulty puppet change? or what happen if someone flushes the postgresql tables by accident?
how can I avoid that not every MariaDB node is purging the webserver iptables afterwards? (sure, I could check if the resource is empty and fail...)
how do you improve your reliability and integrity?