Ask Your Question
0

hiera lookup failure

asked 2016-06-19 23:42:40 -0500

cm01 gravatar image

updated 2016-07-10 23:58:02 -0500

Given this /etc/puppetlabs/code/hiera.yaml

---

:backends: 
  - json

:hierarchy: 
  - "%{hostname}.%{nt_environment}.%{nt_location}"
  - "%{nt_type}.%{nt_environment}.%{nt_location}"
  - "%{nt_type}.%{nt_environment}"
  - "%{nt_environment}.%{nt_location}"
  - "%{nt_environment}"
  - "%{nt_location}"
  - world
  - default

:logger: console # debug

:json:
    :datadir: '/etc/puppetlabs/code/environments/%{::environment}/hieradata' # this is the default anyway

and there exists /etc/puppetlabs/code/environments/prod/hieradata/sac.json, this "puppet lookup --debug icinga_servers --environment prod" produces

Debug: hiera(): Looking for data source puppetmaster.ops
Debug: hiera(): Looking for data source ops.sac
Debug: hiera(): Looking for data source ops
Debug: hiera(): Looking for data source sac
Debug: hiera(): Looking for data source world
Debug: hiera(): Looking for data source default

(& lots more of course).
Why can't it find the /etc/puppetlabs/code/environments/prod/hieradata/sac.json file which contains the key I'm looking for ?
Puppet server 2.4

23-06-2016:
Here's an extend debug o/p

Debug: Performing a hiera indirector lookup of icinga_servers with options {:variables=>Scope(Class[main]), :merge=>#<Puppet::Pops::FirstFoundStrategy:0x00000001024e18 @options_t=#<Puppet::Pops::Types::PStructType:0x00000000fa4150 @elements=[#<Puppet::Pops::Types::PStructElement:0x00000000fa4218 @key_type=#<Puppet::Pops::Types::POptionalType:0x00000000fa4268 @type=#<Puppet::Pops::Types::PStringType:0x00000000fa42b8 @size_type=nil, @values=["strategy"]>>, @value_type=#<Puppet::Pops::Types::POptionalType:0x00000000fa4448 @type=#<Puppet::Pops::Types::PPatternType:0x00000000fa44e8 @patterns=[#<Puppet::Pops::Types::PRegexpType:0x00000000fa46f0 @pattern="first", @regexp=/first/>]>>>]>, @options={}>}
Debug: hiera(): Looking up icinga_servers in JSON backend
Debug: hiera(): Looking for data source puppet4test101.ops.sac
Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/prod/hieradata/puppet4test101.ops.sac.json, skipping
Debug: hiera(): Looking for data source puppet4test.ops.sac
Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/prod/hieradata/puppet4test.ops.sac.json, skipping
Debug: hiera(): Looking for data source puppet4test.ops
Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/prod/hieradata/puppet4test.ops.json, skipping
Debug: hiera(): Looking for data source ops.sac
Debug: hiera(): Looking for data source ops
Debug: hiera(): Looking for data source sac

HTH. As above, the file /etc/puppetlabs/code/environments/prod/hieradata/sac.json, exists and is readable. nt_location shows as 'sac' on the node by using 'facter -p'.
It seems Hiera just doesn't see that file ...
Any ideas?

Chris

2016-07-05
jsonlint validates ok.

hiera  -d icinga_servers -c /etc/puppetlabs/puppet/hiera.yaml nt_location=sac ::environment=prod

DEBUG: 2016-07-05 01:38:52 +0000: Looking for data source .sac
DEBUG: 2016-07-05 01:38:52 +0000: Cannot find datafile /etc/puppetlabs/code/environments/prod/hieradata/.sac.json, skipping
DEBUG: 2016-07-05 01:38:52 +0000: Looking for data source sac
DEBUG: 2016-07-05 01:38:52 +0000: Looking for data source world

(Shortened o/p)
Its looking in the right place, but doesn't match :(

@Alex: can't tarball it; its a prod system (confidential). OTOH, the hiera.yaml is as above, the sac.json (location as above) snippet would be

{
   "env" : {
      "datacenter" : "DC",
      "dhcpserver" : "dhcp101.x.x.x",
      "icinga_servers" : [
         "icinga101.x.x.x"
      ],
    },
}

Its working on the v3 server (this is v4), so its either a v4 issue or I've missed something in the setup.
HTH

edit retag flag offensive close merge delete

Comments

Did you add the json file "sac.json" in :hierarchy: tab hiera.yaml file.

Chutki gravatar imageChutki ( 2016-06-20 00:53:43 -0500 )edit

He's really asking, dose nt_location or nt_environment fact resolve to sac?

DarylW gravatar imageDarylW ( 2016-06-20 14:38:33 -0500 )edit

Good qn: tbh I'm not sure how to check that. Never had this type of prob with hiera and I copied everything from a working 3.2 system, so it should(?) work.
Something different on v4? Maybe I missed something?

cm01 gravatar imagecm01 ( 2016-06-20 23:55:34 -0500 )edit

Cont: so I ran 'facter -p' on the node & those values are filled correctly.
It seems like hiera is checking the right dir, looking for eg nt_location ie 'sac', but it doesn't find it, even though sac.json is there... as per above comment. Its really odd.

cm01 gravatar imagecm01 ( 2016-06-21 01:05:15 -0500 )edit

So as you can see from above, it prefixes each keyname with the dir and appends '.json', as it should BUT only if keyname/data src has a numeric digit in it! Is this a bug in puppet??

cm01 gravatar imagecm01 ( 2016-06-23 00:43:51 -0500 )edit

1 Answer

Sort by » oldest newest most voted
1

answered 2016-07-13 04:42:14 -0500

updated 2016-07-13 19:24:09 -0500

I set this up and it works fine so hopefully my working config will help you to understand what you've done wrong in your broken config.

$ cat spec/fixtures/hiera/hiera.yaml 
---
:backends: 
  - json

:hierarchy: 
  - "%{hostname}.%{nt_environment}.%{nt_location}"
  - "%{nt_type}.%{nt_environment}.%{nt_location}"
  - "%{nt_type}.%{nt_environment}"
  - "%{nt_environment}.%{nt_location}"
  - "%{nt_environment}"
  - "%{nt_location}"
  - world
  - default

:logger: console # debug

:json:
  :datadir: 'spec/fixtures/%{::environment}/hieradata'

The JSON file:

$ tree spec/fixtures/prod/
spec/fixtures/prod/
└── hieradata
    └── sac.json

1 directory, 1 file

Its content:

$ cat spec/fixtures/prod/hieradata/sac.json 
{
  "icinga_servers": [
    "icinga101.x.x.x"
  ]
}

Note that your snippet is:

{
   "env" : {
      "datacenter" : "DC",
      "dhcpserver" : "dhcp101.x.x.x",
      "icinga_servers" : [
         "icinga101.x.x.x"
      ],
    },
}

To see that data you would need to look up the env key, not icinga_servers. In your snippet icinga_servers is a Hash key, not a key that can be looked up directly.

And to demonstrate:

$ bundle exec hiera -d icinga_servers -c spec/fixtures/hiera/hiera.yaml nt_location=sac ::environment=prod
DEBUG: 2016-07-13 19:35:34 +1000: Hiera JSON backend starting
DEBUG: 2016-07-13 19:35:34 +1000: Looking up icinga_servers in JSON backend
DEBUG: 2016-07-13 19:35:34 +1000: Looking for data source ..sac
DEBUG: 2016-07-13 19:35:34 +1000: Cannot find datafile spec/fixtures/prod/hieradata/..sac.json, skipping
DEBUG: 2016-07-13 19:35:34 +1000: Looking for data source ..sac
DEBUG: 2016-07-13 19:35:34 +1000: Cannot find datafile spec/fixtures/prod/hieradata/..sac.json, skipping
DEBUG: 2016-07-13 19:35:34 +1000: Looking for data source .
DEBUG: 2016-07-13 19:35:34 +1000: Cannot find datafile spec/fixtures/prod/hieradata/..json, skipping
DEBUG: 2016-07-13 19:35:34 +1000: Looking for data source .sac
DEBUG: 2016-07-13 19:35:34 +1000: Cannot find datafile spec/fixtures/prod/hieradata/.sac.json, skipping
DEBUG: 2016-07-13 19:35:34 +1000: Looking for data source sac
["icinga101.x.x.x"]

To get it to work using puppet lookup I had to do this:

Because I can't find anyway to specify the hiera.yaml on the command line, I created a symlink to hiera.yaml:

$ ls -l ~/.puppetlabs/etc/puppet/hiera.yaml 
lrwxr-xr-x  1 alexharvey  staff  56 14 Jul 10:05 /Users/alexharvey/.puppetlabs/etc/puppet/hiera.yaml@ -> /Users/alexharvey/git/foo/spec/fixtures/hiera/hiera.yaml

Because the $::environment has a special meaning in the context of directory environments I treated $::environment as a fact.

Created a facts file:

$ cat facts.yaml 
---
nt_environment: sac
environment: prod

Then looked up as follows:

$ bundle exec puppet lookup icinga_servers --facts facts.yaml 
---
- icinga101.x.x.x

So a bit clunky, I assume more features are on the way for lookup, which is quite new.

edit flag offensive delete link more

Comments

Alex, thanks - you get an upvote. :) So my basic problem was Hiera can only specify/search top-level keys :( We have a lot of multi-level json files which I'm going to have to search by hand before I can run a test to check a value - shame.

cm01 gravatar imagecm01 ( 2016-07-13 17:58:36 -0500 )edit

For bonus pts, can you make the 'puppet lookup' cmd work as well? See my attempts above. I tried again after reading this hiera soln, but no joy.

cm01 gravatar imagecm01 ( 2016-07-13 18:00:18 -0500 )edit

Sure, updated.

Alex Harvey gravatar imageAlex Harvey ( 2016-07-13 19:24:29 -0500 )edit

ok - thx for that. Definitely clunky; I'm surprised they haven't got it all on the cmd line already.

cm01 gravatar imagecm01 ( 2016-07-13 21:04:47 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2016-06-19 23:42:40 -0500

Seen: 270 times

Last updated: Jul 13 '16