Ask Your Question
0

Puppet runs failing - Error 400 on SERVER

asked 2014-11-18 16:45:46 -0500

updated 2014-11-19 12:41:30 -0500

Suddenly, all agents in our development environment are failing with an obscure ruby error:

File[/var/opt/lib/pe-puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 400 on SERVER: expected next element in array...
  • We have 3 Puppet environments, (Development, Testing, Production)
  • All nodes in Dev environment are failing runs
  • Nodes in Prod and Test run fine
  • There have been no changes on the master that I know of that should have caused this

Our site.pp and nodes.pp are the same across the board.

Here's the error when running an agent:


sudo /usr/local/bin/puppet agent -t
info: Caching certificate for apibuild01-sc9.virginam.com
info: Retrieving plugin
info: Caching certificate_revocation_list for ca
err: /File[/var/opt/lib/pe-puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 400 on SERVER: expected next element in array at '
  "dependencies": ['!
err: /File[/var/opt/lib/pe-puppet/lib]: Could not evaluate: Error 400 on SERVER: expected next element in array at '
  "dependencies": ['! Could not retrieve file metadata for puppet://puppet.virginam.com/plugins: Error 400 on SERVER: expected next element in array at '
  "dependencies": ['!
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment development: expected next element in array at '
  "dependencies": ['! at /etc/puppetlabs/environments/development/manifests/site.pp:49 on node apibuild01-sc9.virginam.com
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Our config on the master:


## Environments
[main]
    vardir = /var/opt/lib/pe-puppet
    logdir = /var/log/pe-puppet
    rundir = /var/run/pe-puppet
    modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
    manifest = /etc/puppetlabs/puppet/manifests/site.pp
    user = pe-puppet
    group = pe-puppet
    archive_files = true
    archive_file_server = admin03-sc9.virginam.com
#    pluginsync = true

[development]
    modulepath = /etc/puppetlabs/environments/development/modules:/opt/puppet/share/puppet/environments/development/modules
    manifest = /etc/puppetlabs/environments/development/manifests/site.pp

[testing]
    modulepath = /etc/puppetlabs/environments/testing/modules:/opt/puppet/share/puppet/environments/testing/modules
    manifest = /etc/puppetlabs/environments/testing/manifests/site.pp

[master]
    certname = admin03-sc9.virginam.com
    dns_alt_names = admin03-sc9,admin03-sc9.virginam.com,puppet,puppet.virginam.com
    ca_name = 'Puppet CA generated on admin03-sc9.virginam.com at Mon Oct 22 11:20:52 -0700 2012'
    facts_terminus = inventory_active_record
    dbadapter = mysql
    dbname = console_inventory_service
    dbuser = console
    dbpassword = *********
    dbserver = localhost
    reports = https
    reporturl = https://localhost:443/reports/upload
    node_terminus = exec
    external_nodes = /etc/puppetlabs/puppet-dashboard/external_node
    ssl_client_header = SSL_CLIENT_S_DN
    ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]
    certname = admin03-sc9.virginam.com
    server = admin03-sc9.virginam.com
    report = true
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
    graph = true
    pluginsync = true

Our config on the agent:


[main]
    vardir = /var/opt/lib/pe-puppet
    logdir = /var/log/pe-puppet
    rundir = /var/run/pe-puppet
    modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
    user = pe-puppet
    group = pe-puppet
    archivefiles = true
    archivefile_server = puppet.virginam.com

[agent] certname = apibuild01-sc9.virginam.com server = puppet.virginam.com report = true classfile = $vardir/classes.txt localconfig = $vardir/localconfig graph = true pluginsync = true environment = development

And here is our site.pp

filebucket { 'main': server => 'admin03-sc9.virginam.com', # path => false, }

File { backup => 'main' } stage { 'b4main' : before ...

(more)
edit retag flag offensive close merge delete

3 Answers

Sort by ยป oldest newest most voted
0

answered 2014-11-23 10:48:11 -0500

nvpuppet gravatar image

I have some theories but would like to see the output of an agent run with --debug. Pastebin might be a better place to dump the output than on this site due to the size if the log.

edit flag offensive delete link more

Comments

Output of debug mode http://pastebin.com/frf8jitR

Cornellio gravatar imageCornellio ( 2014-11-24 19:19:46 -0500 )edit
0

answered 2014-11-20 12:59:41 -0500

Yeayu gravatar image

updated 2014-11-20 13:04:59 -0500

Are you using environment.conf?

edit flag offensive delete link more

Comments

No, I setup the environments in puppet.conf on the master.

Cornellio gravatar imageCornellio ( 2014-11-20 13:55:12 -0500 )edit

What do you have in site.pp line 49? Can you paste it?

Yeayu gravatar imageYeayu ( 2014-11-20 14:29:18 -0500 )edit

On line 49: import "templates" and this manifest defines a class that all nodes get. It basically just creates user accounts.

Cornellio gravatar imageCornellio ( 2014-11-20 15:30:55 -0500 )edit

I would like to see the whole site.pp to understand whats its really doing. Apart from that, can you run puppet agent -t -d? What happens if you comment the line 49 out? can you post here the outcomes? does the template class work being invoked on its own?

Yeayu gravatar imageYeayu ( 2014-11-21 01:49:56 -0500 )edit

I ran the agent with debug. If I comment out line 49, I get this, yet line 51 has nothing. Seems to indicate a bogus error re. site.pp. "dependencies": ['! at /etc/puppetlabs/environments/development/manifests/site.pp:51 on node apibuild01-sc9.virginam.com

Cornellio gravatar imageCornellio ( 2014-11-24 19:42:30 -0500 )edit
0

answered 2014-11-21 09:28:58 -0500

rnelson0 gravatar image

I think you have multiple errors, so this may be difficult to track the root cause (bad syntax can cause cascading parser errors that all resolve by fixing a quote, for instance). Here's another:

  "dependencies": ['! Could not retrieve file metadata for puppet://puppet.virginam.com/plugins: Error 400 on SERVER: expected next element in array at '

This looks like it could be an error with the metadata.json for the plugins module, particularly the dependencies key, or it could be an issue with a file that is sourced from that module. It's another place to look, and with three errors on "dependencies" it might be where I would start. Correcting the syntax in metadata.json might take care of this.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2014-11-18 16:45:46 -0500

Seen: 2,555 times

Last updated: Nov 23 '14