Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

I was able to resolve this by looking into puppetdb in the ENC script for "agent_specified_environment"

curl -X GET http://puppetdb001.example.com:8080/pdb/query/v4/facts --data-urlencode 'query=["=", "name", "agent_specified_environment"]'

Inside a ruby script, it would be like this:

require 'net/http'
require 'uri'
require 'json'
uri = URI.parse("http://puppetdb001.example.com:8080/pdb/query/v4/facts")
request = Net::HTTP::Post.new(uri)
request.content_type = "application/json"
request.body = JSON.dump({
  "query" => [ "=", "name", "agent_specified_environment" ]
})
req_options = {
  # use_ssl: uri.scheme == "https",
}
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
  http.request(request)
end

response_env = JSON.parse(response.body)[0]["value"]
print response_env

I found this value is created with the correct environment, and is available before the ENC runs so it's true.

I was able to resolve this by looking into puppetdb in the ENC script for "agent_specified_environment"

curl -X GET http://puppetdb001.example.com:8080/pdb/query/v4/facts --data-urlencode 'query=["=", 'query=["and", ["=", "name", "agent_specified_environment"]'
"agent_specified_environment"], ["=", ["fact", "clientcert"], "puppettest001.example.com"]]'

Inside a ruby script, it would be like this:

require 'net/http'
require 'uri'
require 'json'
uri 'json'uri = URI.parse("http://puppetdb001.example.com:8080/pdb/query/v4/facts")
request = Net::HTTP::Post.new(uri)
request.content_type = "application/json"
Net::HTTP::Get.new(uri)
request.body = JSON.dump({
"query=[\"and\",
 "query" => [ "=", "name", "agent_specified_environment" ]
})
[\"=\", \"name\", \"agent_specified_environment\"],
  [\"=\", [\"fact\", \"clientcert\"], \"puppettest001.example.com\"]
  ]"

req_options = {
  #  use_ssl: uri.scheme == "https",
}
 response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
  http.request(request)
end

response_env = JSON.parse(response.body)[0]["value"]
print response_env

I found this value is created with the correct environment, and is available before the ENC runs so it's true.

----- In response to Henrik's comment above about the purpose of this: Maybe I'm using the wrong term, and it should be "node_terminus". It looks like this: node_terminus = exec external_nodes = $confdir/environments/production/bin/enc.rb

I know systems like Foreman want to define the environment of a server, and this can't be changed in puppet.conf or on command line. I do testing manually on servers using --environment so I want to be able to have the enc script inside each environment for testing. The enc script is only for providing more info about the node and helping hiera, it's not to tell the node about classes or environments. Node decides environment, and roles/profiles modules define classes.

Since I can only define a single path to the external_nodes in puppet.conf, I have that one find the "agent_specified_environment" fact in the puppetdb, then use that to include the enc_local.rb script inside that environment directory for classifying nodes. This gives facts like "role" and "location" which can be used as paths in hiera. I get a few other things from a mysql db which is only really possible in a ruby script. If I define these in site.pp or a module, it's too late since hiera.yaml doesn't recognize those, and I can't access the mysql db from .pp files or hiera.

An example of hiera.yaml to give you an idea:

---
:backends:
  - yaml

:yaml:
  :datadir: /etc/puppet/environments/%{::environment}/hieradata

:hierarchy:
  - "fqdn/%{::fqdn}"
  - "role/%{::role_name}/app_%{::app_version}"
  - "role/%{::role_name}"
  - "location/%{::location}/app_%{::app_provider}"
  - "location/%{::location}"
  - "group/%{::group}"
  - "os/%{::operatingsystem}/%{::operatingsystemmajrelease}"
  - "os/%{::operatingsystem}"
  - common

The %role_name is created by enc.rb using regex based on hostname, %app_version and %app_provider are from the database that controls our application, %location is based on the domainname. So you can see that there are a lot of hiera locations that depend on the enc script for these paths.

I wanted to do this a long time ago, I actually had a discussion with you on tickets.puppetlabs.com about it here: https://tickets.puppetlabs.com/browse/PUP-4642

And you were the one that mentioned to me that agent_specified_environment was coming, and I should make a wrapper script.

It seems I have a different workflow in my environment than you do, but it actually works really well for me. Every part of our environment exists as pure code inside git branches for easy branching and testing (a tool like foreman can't give me this). If you're interested in learning more about how I have it set up, send me a message to "Colin" on the slack channel and I can tell you more.

---- Official answer that solved my issue

I was able to resolve this by looking into puppetdb in the ENC script for "agent_specified_environment"

curl -X GET http://puppetdb001.example.com:8080/pdb/query/v4/facts --data-urlencode 'query=["and", ["=", "name", "agent_specified_environment"], ["=", ["fact", "clientcert"], "puppettest001.example.com"]]'

Inside a ruby script, it would be like this:

require 'net/http'
require 'uri'
require 'json'uri = URI.parse("http://puppetdb001.example.com:8080/pdb/query/v4/facts")
request = Net::HTTP::Get.new(uri)
request.body = "query=[\"and\",
  [\"=\", \"name\", \"agent_specified_environment\"],
  [\"=\", [\"fact\", \"clientcert\"], \"puppettest001.example.com\"]
  ]"

req_options = {
  #  use_ssl: uri.scheme == "https",
}

response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
  http.request(request)
end

response_env = JSON.parse(response.body)[0]["value"]
print response_env

I found this value is created with the correct environment, and is available before the ENC runs so it's true.

----- In response to Henrik's comment above about the purpose of this: Maybe I'm using the wrong term, and it should be "node_terminus". It looks like this: this:

node_terminus         = exec
 external_nodes        = $confdir/environments/production/bin/enc.rb

$confdir/environments/production/bin/enc.rb

I know systems like Foreman want to define the environment of a server, and this can't be changed in puppet.conf or on command line. I do testing manually on servers using --environment so I want to be able to have the enc script inside each environment for testing. The enc script is only for providing more info about the node and helping hiera, it's not to tell the node about classes or environments. Node decides environment, and roles/profiles modules define classes.

Since I can only define a single path to the external_nodes in puppet.conf, I have that one find the "agent_specified_environment" fact in the puppetdb, then use that to include the enc_local.rb script inside that environment directory for classifying nodes. This gives facts like "role" and "location" which can be used as paths in hiera. I get a few other things from a mysql db which is only really possible in a ruby script. If I define these in site.pp or a module, it's too late since hiera.yaml doesn't recognize those, and I can't access the mysql db from .pp files or hiera.

An example of hiera.yaml to give you an idea:

---
:backends:
  - yaml

:yaml:
  :datadir: /etc/puppet/environments/%{::environment}/hieradata

:hierarchy:
  - "fqdn/%{::fqdn}"
  - "role/%{::role_name}/app_%{::app_version}"
  - "role/%{::role_name}"
  - "location/%{::location}/app_%{::app_provider}"
  - "location/%{::location}"
  - "group/%{::group}"
  - "os/%{::operatingsystem}/%{::operatingsystemmajrelease}"
  - "os/%{::operatingsystem}"
  - common

The %role_name is created by enc.rb using regex based on hostname, %app_version and %app_provider are from the database that controls our application, %location is based on the domainname. So you can see that there are a lot of hiera locations that depend on the enc script for these paths.

I wanted to do this a long time ago, I actually had a discussion with you on tickets.puppetlabs.com about it here: https://tickets.puppetlabs.com/browse/PUP-4642

And you were the one that mentioned to me that agent_specified_environment was coming, and I should make a wrapper script.

It seems I have a different workflow in my environment than you do, but it actually works really well for me. Every part of our environment exists as pure code inside git branches for easy branching and testing (a tool like foreman can't give me this). If you're interested in learning more about how I have it set up, send me a message to "Colin" on the slack channel and I can tell you more.

---- Official answer that solved my issue

I was able to resolve this by looking into puppetdb in the ENC script for "agent_specified_environment"

curl -X GET http://puppetdb001.example.com:8080/pdb/query/v4/facts --data-urlencode 'query=["and", ["=", "name", "agent_specified_environment"], ["=", ["fact", "clientcert"], "puppettest001.example.com"]]'

Inside a ruby script, it would be like this:

require 'net/http'
require 'uri'
require 'json'uri = URI.parse("http://puppetdb001.example.com:8080/pdb/query/v4/facts")
request = Net::HTTP::Get.new(uri)
request.body = "query=[\"and\",
  [\"=\", \"name\", \"agent_specified_environment\"],
  [\"=\", [\"fact\", \"clientcert\"], \"puppettest001.example.com\"]
  ]"

req_options = {
  #  use_ssl: uri.scheme == "https",
}

response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
  http.request(request)
end

response_env = JSON.parse(response.body)[0]["value"]
print response_env

I found this value is created with the correct environment, and is available before the ENC runs so it's true.