Ask Your Question
0

Why hiera() function returning flattened hash as a string, while both lookup() and hiera from command-line return hash ?

asked 2015-01-05 08:35:09 -0500

galser gravatar image

updated 2015-01-05 13:44:51 -0500

Running Puppet Open Source 3.7.3 under Linux 64 bit with ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]. In a different scenarios - master-less with Vagrant, and with a real master of the same version running under Ruby Passenger inside Apache.

The question if more of curiosity.

Manual (function reference) says :

hiera() Performs a standard priority lookup and returns the most specific value for a given key. The returned value can be data of any type (strings, arrays, or hashes).

Okay, so I am creating some values for databases access credentials, and testing them from command line :

   # hiera staging
    {"databases"=>
      {"uml"=>
        {"dbname"=>"uml", "host"=>"stg_db_01", "user"=>"muser", "port"=>8234},

Looks good, let's try this in code :

# puppet apply -e "notice hiera('staging')"
Notice: Scope(Class[main]): databasesstg_testhoststg_db_01port9092userstg_usr_01dbnameuml01hoststagingport9432 
..... ( skipeed as it long ) 
Notice: Compiled catalog for localhost.XXXX.local in environment production in 0.02 seconds
Notice: Finished catalog run in 0.35 seconds

Hmm.. not something that I expected, according to manual. Here - hiera() function returned flattened hash as one huge string.

3rd attempt - using lookup() :

# puppet apply --parser future -e 'notice lookup(staging)'

Notice: Scope(Class[main]): {databases => {uml => {user => muser, port => 8234, dbname => uml, host => stg_db_01}, stg_test => {user =>  
( ..skippped .. )
Notice: Compiled catalog for localhost.XXX.local in environment production in 0.29 seconds
Notice: Finished catalog run in 0.37 seconds

Wov, that's much better. Notice that the order of hash elements is different, but that is expected behavior, nothing new to me.

The question is why here : https://docs.puppetlabs.com/references/latest/function.html#hiera or here https://docs.puppetlabs.com/hiera/1/puppet.html#hiera-lookup-functions nothing written about recommended usage of lookup() function?

Note : And still in all manuals hiera() function is used, never even mentioned about lookup(). Yes, I do understand about automatic parameters lookup, and in the situation I am describing - the source of Hiera data is generated by third-party script, so I have a little of influence on format right now, and can't use parametrized class/definition right away. I've only managed to caught a glimpse in Google Groups that hiera() function is deprecated, and maybe better to use lookup()

UPDATE : Valid YAML file, for those who want to check original data :

---
staging:
  databases:
    uml:
      host: stg_db_01
      port: 8234
      dbname: uml
      user: muser
    stg_test:
      host: stg_db_01
      port: 9092
      dbname: stg_test
      user: stg_usr_01
    stg_match:
      host: stg_db_02
      port: 9432
      dbname: stg_match
      user: stmatch
edit retag flag offensive close merge delete

Comments

Which Hiera backend are you using? To the best of my knowledge, the hieradata presented above is not valid YAML or JSON.

cbarbour gravatar imagecbarbour ( 2015-01-05 13:07:50 -0500 )edit

Hmmm. You started on a wrong note :-). Did a posted any source YAML ? You make superstitions, with no grounds. I did not presented any hieradata,. Specially for You I am adding original VALID YAML .

galser gravatar imagegalser ( 2015-01-05 13:43:36 -0500 )edit

notice() renders data structures to a string. You cannot make any assumptions about how Puppet handles data structures based on the output of notice.

cbarbour gravatar imagecbarbour ( 2015-01-05 14:33:17 -0500 )edit

Well , then it is still not clear why in one case notice (and notify) outputs everything as flattened string, and in another case - supposedly the same data displayed as a normal hash. It is absolutely fine seen from my screenpastes.

galser gravatar imagegalser ( 2015-01-05 15:28:17 -0500 )edit

See my updated answer. Enabling the future parser is what produces the results your seeing. If you don't want to waste time on this sort of stuff, I'd advise you not to use experimental features such as the future parser.

cbarbour gravatar imagecbarbour ( 2015-01-05 16:22:38 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted
0

answered 2015-01-05 15:46:32 -0500

galser gravatar image

updated 2015-01-05 15:48:30 -0500

Well. I do have a partial answer to my own question. A lot of time lost.

So, the answer is with --parser future -> results of hiera() and lookup() are identical.

Without --parser future - I cannot use lookup, but then results of hiera() - looks like flattened hash - string withou any spaces in output. And I can't access it a hash either - produces an error.

In short - hiera() behaves differently with and without future parsing. Not pleasant experience.

edit flag offensive delete link more
0

answered 2015-01-05 13:15:28 -0500

cbarbour gravatar image

updated 2015-01-05 16:21:14 -0500

To answer part of your question:

I don't see a specific deprecation warning for hiera(). However, lookup() does seem set to supersede it. The benefit of lookup() is that it uses the Puppet data terminus interface rather than calling Hiera explicitly. While hiera() is the only data binding I'm aware of at the moment, it would make your code more portable if another data binding was released in the future, or if you wished to write your own data binding terminus.

lookup() also provides a much more flexible interface, and support for future parser style block syntax.

The hiera() call does not flatten arrays into strings; if hiera() returns a hash or array data type, hiera() will return a hash or an array to Puppet. The benefit of hiera_hash() and hiera_array() is that they will merge multiple hash or array data structures if they are found at different levels of your hierarchy.

Edit:

To answer the second part of your question: hiera() doesn't flatten data structures. The flattening you see is coming from notice not from hiera. Notice attempts to render the data structure provided into a string. When passed something other than a string, it's behavior is not defined.

lookup() isn't referred to in much of the documentation because it's an extremely new function call. hiera() works all the way back to the Puppet 2.7 days. Lookup is only available in 3.3.0 or later, and only with the future parser enabled. The future parser is still marked as experimental as of 3.7.

Edit 2:

To answer this question

Well. I do have a partial answer to my own question. A lot of time lost.

So, the answer is with --parser future -> results of hiera() and lookup() are identical.

Without --parser future - I cannot use lookup, but then results of hiera() - looks like flattened hash - string withou any spaces in output. And I can't access it a hash either - produces an error.

In short - hiera() behaves differently with and without future parsing. Not pleasant experience.

No. Puppet's handling of data structures, especially the way they are stringified for display is affected by the future parser. It's pretty trivial to see that the behavior of notify changes depending on whether or not the future parser is enabled.

$ cat test.pp

$hash = {
  foo => [ 'foobar', 'foobaz' ],
  bar => [ 'barbar', 'barbaz' ],
  baz => [ 'bazbar', 'bazbaz' ],
}

notice($hash)

$ puppet apply ./test.pp
Notice: Scope(Class[main]): barbarbarbarbazbazbazbarbazbazfoofoobarfoobaz
Notice: Compiled catalog for nc1-aq1-mapps-app-1003.corp.apple.com in environment production in 0.02 seconds
Notice: Finished catalog run in 0.04 seconds

$ puppet apply --parser future ./test.pp
Notice: Scope(Class[main]): {foo => [foobar, foobaz], baz => [bazbar, bazbaz], bar => [barbar, barbaz]}
Notice: Compiled catalog for nc1-aq1-mapps-app-1003.corp.apple.com in environment production in 0.31 seconds
Notice: Finished catalog run in 0.01 seconds

Again, the behavior of notify() when passed a complex data structure is not defined. You should not use it to analyze Puppet ... (more)

edit flag offensive delete link more

Comments

I do understand that also. That's why I am not using hiera_hash(). For example, for me - using hiera_hash() in this source of data means to merge values across different environments - staging, beta, development and do on. Though, I am thinking of this behaviour as inconsistent.

galser gravatar imagegalser ( 2015-01-05 13:38:41 -0500 )edit

So, in addition - that is very strange that is **do flattening** for me. And You can that clear see in my original post.

galser gravatar imagegalser ( 2015-01-05 13:51:10 -0500 )edit

Okay, thanks for clarification, good to know. >> Again, the behavior of notify() when passed a complex data structure is not defined. You should not use it to analyze Puppet's handling of complex data structures. Sounds like a mess to me though. But probably OK with an experimental feature. Thank U

galser gravatar imagegalser ( 2015-01-06 03:03:57 -0500 )edit

Notify is mostly there to create log entries on your Puppet Master or standalone node. It looks like steps are being made to improve it with the future parser, which is nice. :)

cbarbour gravatar imagecbarbour ( 2015-01-06 12:59:57 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2015-01-05 08:35:09 -0500

Seen: 1,297 times

Last updated: Jan 05 '15