SOME agents _not_ copying module's facts.d. Most work
I would understand this problem better if it affected all clients equally but it doesn't.
A long time after changes in the master (24h+), to make sure there are no caches, after several "puppet agent -t" runs, in short this is what I get :
[root@host1 ~]# find /var/lib/puppet/ -name discover-hdfs-partitions.* /var/lib/puppet/facts.d/discover-hdfs-partitions.sh [root@host2 ~]# find /var/lib/puppet/ -name discover-hdfs-partitions.* [root@host2 ~]#
As it helps debugging, here's the first puppet run (after completely wiping the agent, reinstalling RPM, revoking certificate and starting from scratch) :
As you see the fact file is pretty much ignored and not copied.
Both agents have the same exact version :
And the master is the same. (very close if not the same 3.7 release)
Any clues on how to debug this further? I have close to 300 nodes and most of them are fine. Few are presenting this but is enough to create the mess.
For completeness, this is where the custom fact sits on the module :
And healthy nodes will use the fact for a notify among other things :
Notice: /Stage[main]/Hdfs::Datanode/Notify[Found the following HDFS data mountpoints: /data1/hadoop/data,/data2/hadoop/data]/message: defined 'message' as 'Found the following HDFS data mountpoints: /data1/hadoop/data,/data2/hadoop/data'