Failed to start pe-puppetserver service

asked 2016-04-05 05:05:41 -0500

Raf gravatar image

I have Puppet Enterprise 2015.3.1 installed on a CentOS 7 machine (4 cores/7GB). On a server restart, all of the services come up successfully apart from the main pe-puppetserver service. It is not until a number failed manual service start attempts and an hour or so later a manual start of the service works. I have doubled the java heap size to 4GB and doubled the service timeout, neither of which made any difference.

Below, is a snippet of the puppetserver.log:

2016-04-05 10:15:01,138 INFO  [main] [o.e.j.u.log] Logging initialized @119708ms
2016-04-05 10:15:19,424 INFO  [main] [p.t.s.w.jetty9-service] Initializing web server(s).
2016-04-05 10:15:20,027 INFO  [main] [p.t.s.s.status-service] Registering status callback function for pe-puppet-profiler service
2016-04-05 10:15:20,205 INFO  [main] [p.s.j.jruby-puppet-service] Initializing the JRuby service
2016-04-05 10:15:20,478 INFO  [main] [p.t.s.s.status-service] Registering status callback function for pe-jruby-metrics service
2016-04-05 10:16:10,356 INFO  [clojure-agent-send-pool-0] [puppet-server] Puppet Puppet settings initialized; run mode: master
2016-04-05 10:16:14,804 INFO  [clojure-agent-send-pool-0] [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 1 of 3
2016-04-05 10:16:15,005 INFO  [main] [p.s.c.puppet-server-config-core] Not overriding webserver settings with values from core Puppet
2016-04-05 10:16:15,486 WARN  [main] [o.e.j.s.h.ContextHandler] Empty contextPath
2016-04-05 10:16:15,495 INFO  [main] [p.e.s.f.file-sync-client-service] Initializing file sync client service
2016-04-05 10:16:15,525 INFO  [main] [p.e.s.f.file-sync-client-service] Registering file sync client HTTP API
2016-04-05 10:16:15,525 INFO  [main] [p.t.s.s.status-service] Registering status callback function for file-sync-client-service service
2016-04-05 10:16:27,958 INFO  [clojure-agent-send-pool-0] [puppet-server] Puppet Puppet settings initialized; run mode: master
2016-04-05 10:16:29,817 INFO  [clojure-agent-send-pool-0] [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 2 of 3
2016-04-05 10:16:43,017 INFO  [clojure-agent-send-pool-0] [puppet-server] Puppet Puppet settings initialized; run mode: master
2016-04-05 10:16:44,115 INFO  [clojure-agent-send-pool-0] [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 3 of 3
2016-04-05 10:17:43,018 INFO  [Thread-1] [p.t.internal] Shutting down due to JVM shutdown hook.
2016-04-05 10:17:43,034 INFO  [Thread-1] [p.t.internal] Beginning shutdown sequence
2016-04-05 10:17:43,036 INFO  [Thread-1] [p.e.s.f.file-sync-client-service] Stopping file sync client service
2016-04-05 10:17:43,059 INFO  [Thread-1] [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopping metrics sampler job
2016-04-05 10:17:43,077 INFO  [Thread-1] [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopped metrics sampler job
2016-04-05 10:17:43,158 INFO  [Thread-1] [p.t.s.w.jetty9-service] Shutting down web server(s).
2016-04-05 10:17:43,177 INFO  [Thread-1] [p.t.internal] Finished shutdown sequence

Any help will be appreciated.

Cheers

Raf

edit retag flag offensive close merge delete

Comments

Did you migrate to the new auth.conf format ?
Check the files:
/etc/puppetlabs/puppetserver/conf.d/auth.conf
/etc/puppetlabs/puppetserver/bootstrap.conf
You log doesn't show CA setup as well as file destinations.

FranzCC gravatar imageFranzCC ( 2016-04-05 07:00:07 -0500 )edit

You might try turning up the log level in `logback.xml` to something higher, like DEBUG, to see if you get any more info?

cprice404 gravatar imagecprice404 ( 2016-04-05 11:31:20 -0500 )edit

Also, normally when you see `Shutting down due to JVM shutdown hook.`, that means that something outside of the process told it to shutdown. This could be the service init script (due to timeout that you mentioned), or a puppet agent run on the same machine?

cprice404 gravatar imagecprice404 ( 2016-04-05 11:31:42 -0500 )edit

Just to mention, I can run puppet server outside of the service fine, so I believe like mentioned it is something outside of the process that is causing the shutdown. I have increased the log level to DEBUG but still not seeing anything that would indicate why this is happening.

Raf gravatar imageRaf ( 2016-04-06 10:31:18 -0500 )edit

Then you have to trace start initscript startup with strace and debug.
Something like: strace -f -s 65536 service pe-puppetserver start 2> /tmp/pe-puppetserver.debug
Since Java is involved, the trace can get very big in size !

FranzCC gravatar imageFranzCC ( 2016-04-08 00:18:25 -0500 )edit