Ask Your Question
2

Passenger fails on RHEL/CentOS 7

asked 2014-08-15 11:17:45 -0500

Mojo gravatar image

I've been beating my head against RHEL 7 (CentOS 7) trying to set up a puppet master. I'm using IPA and NSS for certificate management, and those problems appear to be solved.

I can run the puppet master daemon (Webrick default) and connect from an agent successfully, but passenger throws 500 exceptions and a diagnostic page.

I tried changing the config.ru to a simple "hello world" web app, and that runs successfully, so the hook through Apache to Passenger seems successful.

I suspected problems with systemd and the fact that passenger makes heavy use of the /tmp directory. I tracked down the httpd service file and set PrivateTmp=false, and confirmed that passenger stopped using the private /tmp directory. This had no effect on the fail.

Here is the stack trace reported on the passenger 500 page:

exit (SystemExit)
  /usr/share/ruby/vendor_ruby/puppet/util.rb:493:in `exit'
  /usr/share/ruby/vendor_ruby/puppet/util.rb:493:in `rescue in exit_on_fail'
  /usr/share/ruby/vendor_ruby/puppet/util.rb:479:in `exit_on_fail'
  /usr/share/ruby/vendor_ruby/puppet/application.rb:369:in `run'
  /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
  /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
  config.ru:35:in `block in <main>'
  /usr/local/share/gems/gems/rack-1.5.2/lib/rack/builder.rb:55:in `instance_eval'
  /usr/local/share/gems/gems/rack-1.5.2/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/local/share/gems/gems/passenger-4.0.48/helper-scripts/rack-preloader.rb:112:in `eval'
  /usr/local/share/gems/gems/passenger-4.0.48/helper-scripts/rack-preloader.rb:112:in `preload_app'
  /usr/local/share/gems/gems/passenger-4.0.48/helper-scripts/rack-preloader.rb:158:in `<module:App>'
  /usr/local/share/gems/gems/passenger-4.0.48/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/local/share/gems/gems/passenger-4.0.48/helper-scripts/rack-preloader.rb:28:in `<main>'

I'd love some suggestions for troubleshooting this problem. My fallback is to ditch RHEL 7 for 6.5.

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
2

answered 2014-08-16 02:18:32 -0500

domcleal gravatar image

updated 2014-08-16 02:29:42 -0500

Stack traces of this sort indicate a failure to start the Puppet application, usually when Puppet is trying to initialise settings and manage internal file permissions.

You need to check for syslog entries from the Puppet master, as it will log which operation failed. On RHEL 7 you should be able to search like this:

journalctl SYSLOG_IDENTIFIER=puppet-master

(On other operating systems, grep puppet-master /var/log/{messages,syslog})

Look for entries similar to:

puppet-master[10644]: failed to set mode 0 on /etc/puppet/autosign.conf: Operation not permitted - /etc/puppet/autosign.conf
puppet-master[10644]: (/File[/etc/puppet/autosign.conf]/mode) change from 0000 to 0664 failed: failed to set mode 0 on /etc/puppet/autosign.conf: Operation not permitted - /etc/puppet/autosign.conf
puppet-master[10644]: Could not prepare for execution: Got 1 failure(s) while initializing: File[/etc/puppet/autosign.conf]: change from 0000 to 0664 failed: failed to set mode 0 on /etc/puppet/autosign.conf: Operation not permitted - /etc/puppet/autosign.conf

In this example the permission on autosign.conf is incorrect, and Puppet's trying to fix it - though it doesn't have rights to do so. File permissions are the usual startup problem, as the Puppet master runs as the unprivileged 'puppet' user. You can disable the permissions management feature by setting manage_internal_file_permissions = false (docs).

In a similar vein, check for SELinux AVC denials if the server is in the default enforcing mode, particularly if the logs are permission-related. Though if you've installed Passenger from a gem, I'm unsure which confined domain it would be running in - perhaps httpd_t, which probably isn't designed for running a Puppet master in.

  • To find AVCs, check /var/log/audit/audit.log (if auditd is on) or /var/log/messages (if not). When setroubleshootd is on, easy to read messages are printed to /var/log/messages.
edit flag offensive delete link more

Comments

Ah ... it's journalctl that tripped me up. Moving to systemd and journalctl is throwing all my work patterns off. This is probably the issue: "(/File[/var/log/puppet/masterhttp.log]) Could not stat; permission denied" and the directory is owned by puppet. Should be owned by apache?

Mojo gravatar imageMojo ( 2014-08-18 10:31:16 -0500 )edit

No, it should be owned by Puppet, but it may indicate a problem with the Passenger/rack config if the master app isn't running as the "puppet" user. Passenger usually setuids based on the ownership of config.ru, so check that's chowned to "puppet". Also check parent dirs are accessible to the user.

domcleal gravatar imagedomcleal ( 2014-08-18 10:49:30 -0500 )edit

Yes, the whole /var/lib/puppet tree needs to be owned by apache to get puppet master to run under passenger. Thank you!

Mojo gravatar imageMojo ( 2014-08-18 11:18:31 -0500 )edit
0

answered 2017-06-10 08:25:59 -0500

Alireza Nasri gravatar image

I have the same error but my server doesn't have this directory tree /var/lib/puppet, instead it has /opt/puppetlabs/puppet/lib

what I did was, chown apache.apache /opt/puppetlabs/puppet/lib -R but it doesn't fix the problem, I still have these whole errors App 3339 stdout: App 3339 stderr: 2017-06-10 App 3339 stderr: App 3339 stderr: 17:52:59.138942 App 3339 stderr: App 3339 stderr: WARN App 3339 stderr: App 3339 stderr: App 3339 stderr: puppetlabs.facter App 3339 stderr: - App 3339 stderr: symbol rbcFixnum was not found in .: facts requiring Ruby will not be resolved. App 3339 stderr: App 3339 stderr: Warning: Setting 'alwayscachefeatures' is App 3339 stderr: deprecated and has been replaced by 'alwaysretryplugins'. App 3339 stderr: (at /opt/puppetlabs/puppet/lib/ruby/vendorruby/puppet/defaults.rb:319:in block in <module:Puppet>') App 3339 stderr: App 3339 stderr: Error: Could not initialize global default settings: undefined methodsearch' for Facter:Module App 3339 stderr: App 3339 stdout: Error ID: 79174119 Error details saved to: /tmp/passenger-error-Vewhwd.html Message from application: exit (SystemExit) /opt/puppetlabs/puppet/lib/ruby/vendorruby/puppet/util.rb:556:in exit' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:556:inrescue in exitonfail' /opt/puppetlabs/puppet/lib/ruby/vendorruby/puppet/util.rb:542:in exit_on_fail' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:66:inexecute' config.ru:43:in block in <main>' /usr/local/rvm/gems/ruby-2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:55:ininstance_eval' /usr/local/rvm/gems/ruby-2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:55:in initialize' config.ru:1:innew' config.ru:1:in <main>' /usr/local/rvm/gems/ruby-2.4.0/gems/passenger-5.1.4/src/helper-scripts/rack-preloader.rb:110:ineval' /usr/local/rvm/gems/ruby-2.4.0/gems/passenger-5.1.4/src/helper-scripts/rack-preloader.rb:110:in preload_app' /usr/local/rvm/gems/ruby-2.4.0/gems/passenger-5.1.4/src/helper-scripts/rack-preloader.rb:156:in<module:app>' /usr/local/rvm/gems/ruby-2.4.0/gems/passenger-5.1.4/src/helper-scripts/rack-preloader.rb:30:in <module:PhusionPassenger>' /usr/local/rvm/gems/ruby-2.4.0/gems/passenger-5.1.4/src/helper-scripts/rack-preloader.rb:29:in<main>'

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2014-08-15 11:17:45 -0500

Seen: 1,944 times

Last updated: Jun 10