Ask Your Question

Multiple backends for hiera's hierarchy

asked 2014-12-10 09:11:08 -0600

skirk gravatar image

updated 2015-02-03 09:20:05 -0600

Currently at we have several Puppet masters running via passenger with hiera v 1.2.1. Our hiera.yaml located at /etc/puppet/hiera.yaml and linked to /etc/ for cli usage. Our hiera config in use right now is defined as:

Our puppet masters are running puppet 3.7.1 whereas our agents run a variety of versions with puppet 2.7.19 being the most used. In this question, the machine I am working with is running puppet 2.7.19.

I want to introduce an additional datadir that hiera can use besides the one we have in git, to be populated dynamically by other event driven scripts within our infrastructure. I intend on maintaining our standard of keeping all files in that directory applicable to their respective nodes as <>

At first I attempted to use multiple JSON data dirs and had no success and it seems as though this is not possible yet: Feature #13954 Support multiple back-ends of the same type

Updated to try using the following as per the example given here by @cbarbour and also some tidbits I have gleamed in the past week of reading and testing things:

I have the following file /usr/test_backend/ defined as :

I also updated hiera to 1.3.4 and tried specifying the environment and fqdn variables which seemed to work when testing this using hiera cli:

hiera test_string ::environment=hiera_sk --debug

DEBUG: Wed Dec 17 14:02:49 -0500 2014: Hiera JSON backend starting
DEBUG: Wed Dec 17 14:02:49 -0500 2014: Looking up test_string in JSON backend
DEBUG: Wed Dec 17 14:02:49 -0500 2014: Looking for data source usr/test_backend/

However when I do a puppet sync from my test vm I get a 400 ERROR:

puppet agent --verbose --no-daemonize --onetime ---environment hiera_sk

notice: Ignoring --listen on onetime run
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item release_package in any Hiera data file and no default supplied at /etc/puppet/environments/hiera_sk/modules/test/manifests/test_manifest.pp:9 on node
info: Not using expired catalog for from cache; expired at Fri Dec 16 16:20:22 -0500 2014
notice: Using cached catalog
err: Could not retrieve catalog; skipping run

I have the following the manifest (/etc/puppet/environments/hierask/modules/test/manifests/testmanifest.pp) :

Some debug statements and replies to helpful commenters below:

puppet apply -e 'alert(hiera("test_string"))' --verbose --debug :

Debug: Runtime environment: ruby_version=1.8.7, run_mode=user, puppet_version=3.7.1
Debug: Loading external facts from /var/lib/puppet/facts.d
Info: Loading facts
Debug: Loading facts from /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Debug: Loading facts from /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Debug: Loading ...
edit retag flag offensive close merge delete


For testing purposes, try this on the command line on your puppetmaster: puppet apply -e 'alert(hiera("test_key")) --verbose --debug

cbarbour gravatar imagecbarbour ( 2014-12-19 15:29:27 -0600 )edit

Ensure that test_key actually appears in your hierarchy of course. You can easily query arbitrary keys this way. From your hiera test_string output, it looks like Hiera is working normally. Have you confirmed that puppet is using the right hiera.conf file, and that 'test_string' exists in Hiera?

cbarbour gravatar imagecbarbour ( 2014-12-19 15:30:33 -0600 )edit

@cbarbour the output from running `puppet apply -e 'alert(hiera("test_key")) --verbose --debug` added above

skirk gravatar imageskirk ( 2014-12-19 16:37:03 -0600 )edit

3 Answers

Sort by ยป oldest newest most voted

answered 2014-12-10 17:28:16 -0600

cbarbour gravatar image

To the best of my knowledge, you cannot define the same backend mutiple times, and you cannot specify multiple : :datadir keys for the same JSON or YAML back end.

The normal solution is to shift the datadir to the highest common root, and shift the rest of the path to your hierarchy. E.g.

  - json
  :datadir: /
  - etc/puppet/environments/%{environment}/hiera/node/%{::fqdn}
  - etc/puppet/environments/%{environment}/hiera/node/%{fqdn}
  - etc/puppet/environments/%{environment}/hiera/common
  - usr/test/yaml_backend/node/%{::fqdn}
  - usr/test/yaml_backend/node/%{fqdn}
  - usr/test/yaml_backend/common
edit flag offensive delete link more


@cbarbour I do not fully understand what you mean by not specifying multiple datadir keys... are you saying that the same data dir would have to be used but that I could then have yaml or json files contained in it?

skirk gravatar imageskirk ( 2014-12-10 20:59:11 -0600 )edit

You cannot list multiple :datadir: keys (each with a different directory value) under the :json: key. The rest of his answer describes how to do what you want, and your top-level datadir can be located wherever you like. The rest of the subdirs under it can be version controlled or not as needed.

GregLarkin gravatar imageGregLarkin ( 2014-12-10 21:37:07 -0600 )edit

I tried implementing this as a test, included results in initial question above

skirk gravatar imageskirk ( 2014-12-11 08:18:54 -0600 )edit

Hiera works the same whether or not you're using passenger. Try using the hiera command line tool to lookup <hiera-variable>. Make sure you supply the 'fqdn' and 'environment' variables, and use the --debug option. Post the output.

cbarbour gravatar imagecbarbour ( 2014-12-15 19:59:13 -0600 )edit

thanks for the followup @cbarbour I have ran hiera with debug and included the output above.

skirk gravatar imageskirk ( 2014-12-17 13:09:35 -0600 )edit

answered 2014-12-11 01:29:32 -0600

spend gravatar image

is... :merge_behavior: 'deeper' what you are after?

edit flag offensive delete link more


no I do not think so, in this use case the various fqdn files each contain unique variables, so I would expect that the first found instance of a variable would be returned.

skirk gravatar imageskirk ( 2014-12-19 10:46:38 -0600 )edit

answered 2014-12-17 21:03:58 -0600

GregLarkin gravatar image

Let's backtrack a bit to the original question. Why can't you store version-controlled and non-version-controlled files in the same datadir hierarchy that you started with? You can easily set up .gitignore files to avoid messages about untracked files in your git working directory.

I think that might be easier to manage than what you're trying to do now, unless there's some reason why my suggestion won't work.

edit flag offensive delete link more


the use case we have right now is to have fqdn named specific json files in git which we checkout on a per environment basis. This allows us to easily rollback or test out many changes as once with a given puppetmaster.

skirk gravatar imageskirk ( 2014-12-18 09:05:11 -0600 )edit

Yes, but what about the untracked files that you said are created by an out-of-band process? Why can't they live in the same checked-out directories and added to the .gitignore files in them? Setting your datadir to / seems like a bad idea to me with potential security implications as well.

GregLarkin gravatar imageGregLarkin ( 2015-01-06 23:21:11 -0600 )edit

@GregLarkin our puppet masters maintain a git checkout for each "environment" that gets pulled automatically whenever something new gets committed. The out-of-band process will be producing variables that are node specific , hence my desire to be able to use them directly from hiera

skirk gravatar imageskirk ( 2015-01-07 09:23:08 -0600 )edit

@skirk I still don't understand why you can't merge your hierarchy directories together so they can be rooted at a lower level than "/". You could create the OOB files in a directory that only contains that type, for organizational purposes. I think this scheme would make things more understandable.

GregLarkin gravatar imageGregLarkin ( 2015-01-09 16:18:44 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2014-12-10 09:11:08 -0600

Seen: 2,157 times

Last updated: Feb 03 '15