Ask Your Question
0

File Sync service failing on PE 2016.1

asked 2016-05-19 22:13:28 -0500

mike.r gravatar image

puppet 4.4.2 , Centos7

trying to start pe-puppetserver

cant trace any changes done to the master that would break file sync. The error msg doesnt give any detail as to whats Not Found

2016-05-19 19:41:31,048 INFO  [Thread-2] [p.t.internal] Shutting down due to JVM shutdown hook.
2016-05-19 19:41:31,054 INFO  [Thread-2] [p.t.internal] Beginning shutdown sequence
2016-05-19 19:41:31,078 INFO  [async-dispatch-3] [p.e.s.f.file-sync-client-service] Stopping file sync client service
2016-05-19 19:41:31,079 INFO  [async-dispatch-3] [p.e.s.f.file-sync-client-service] Waiting for scheduled jobs to complete
2016-05-19 19:41:34,759 ERROR [clojure-agent-send-off-pool-0] [p.e.file-sync-errors] File Sync failure during sync or fetch phase: Unexpected response status 404 for latest commits.  Body: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /file-sync/v1/latest-commits. Reason:
<pre>    Not Found</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

2016-05-19 19:41:34,760 INFO  [async-dispatch-3] [p.e.s.f.file-sync-client-service] scheduled jobs completed, closing HTTP client
2016-05-19 19:41:34,791 INFO  [async-dispatch-3] [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopping metrics sampler job
2016-05-19 19:41:34,798 INFO  [async-dispatch-3] [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopped metrics sampler job
2016-05-19 19:41:34,815 INFO  [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-agents] Flush request received; creating new JRuby pool.
2016-05-19 19:41:34,826 INFO  [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-agents] Replacing old JRuby pool with new instance.
2016-05-19 19:41:34,827 INFO  [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-agents] Swapped JRuby pools, beginning cleanup of old pool.
2016-05-19 19:41:34,858 INFO  [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 1.
2016-05-19 19:41:34,868 INFO  [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 2.
2016-05-19 19:41:34,877 INFO  [clojure-agent-send-pool-1] [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 3.
2016-05-19 19:41:34,924 INFO  [async-dispatch-3] [p.t.s.w.jetty9-service] Shutting down web server(s).
2016-05-19 19:41:34,933 INFO  [async-dispatch-3] [p.t.s.w.jetty9-core] Shutting down web server.
2016-05-19 19:41:34,945 INFO  [async-dispatch-3] [o.e.j.s.ServerConnector] Stopped ServerConnector@36f0d428{SSL-HTTP/1.1}{0.0.0.0:8170}
2016-05-19 19:41:34,953 INFO  [async-dispatch-3] [p.t.s.w.jetty9-core] Web server shutdown
2016-05-19 19:41:34,955 INFO  [async-dispatch-3] [p.t.s.w.jetty9-core] Shutting down web server.
2016-05-19 19:41:34,959 INFO  [async-dispatch-3] [o.e.j.s.ServerConnector] Stopped ServerConnector@1062c82{SSL-HTTP/1.1}{0.0.0.0:8140}
2016-05-19 19:41:34,961 INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@49997f4a{/file-sync,null,UNAVAILABLE}
2016-05-19 19:41:34,963 INFO  [async-dispatch-3] [o.e.j.s.h.ContextHandler] Stopped o.e.j.s.h.ContextHandler@5d813871{/status,null,UNAVAILABLE}
2016-05-19 19:41:34 ...
(more)
edit retag flag offensive close merge delete

3 Answers

Sort by ยป oldest newest most voted
0

answered 2016-05-20 01:23:46 -0500

FranzCC gravatar image

updated 2016-05-20 01:25:56 -0500

Hi, Do you use a multi master setup ?
For a single instance installation, you don't need file-sync and think it won't be enabled.
See:
Otherwise you should file a support ticket !
[https://docs.puppet.com/pe/2016.1/cmg...]

I use a single instance setup and therefore the file /etc/puppetlabs/puppetserver/conf.d/web-routes.conf
looks like this:

 web-router-service: {
    # These two should not be modified because the Puppet 4.x agent expects them to
    # be mounted at these specific paths
    "puppetlabs.enterprise.services.master.master-service/pe-master-service": "/puppet"

    # Puppet 3.x compatibility service
    "puppetlabs.services.legacy-routes.legacy-routes-service/legacy-routes-service": ""

    # This controls the mount point for the CA service.
    "puppetlabs.services.ca.certificate-authority-service/certificate-authority-service": "/puppet-ca"

    # This controls the mount point for the reverse-proxy CA service.  When
    # support for Puppet 3.x CA APIs can be dropped, the following mount point
    # should be changed to "/puppet-ca" to make the match more restrictive and
    # consistent with the mount point for the 'certificate-authority-service'.
    # "puppetlabs.enterprise.services.reverse-proxy.reverse-proxy-ca-service/reverse-proxy-ca-service": ""

    # This controls the mount point for the puppet admin API.
    "puppetlabs.services.puppet-admin.puppet-admin-service/puppet-admin-service": "/puppet-admin-api"
    "puppetlabs.enterprise.services.reverse-proxy.reverse-proxy-ca-service/reverse-proxy-ca-service" : ""
    "puppetlabs.trapperkeeper.services.status.status-service/status-service" : "/status"
}

Hope that helps

Rgds. Franz

edit flag offensive delete link more
0

answered 2016-05-20 08:19:48 -0500

Kevin Corcoran gravatar image

Mike,

In the error you posted:

<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /file-sync/v1/latest-commits. Reason:
<pre>    Not Found</pre></p>

The thing that is "not found" is the /file-sync/v1/latest-commits HTTP endpoint on the Master of Masters. This is likely because file sync is not enabled/running on the MoM. That's a bit confusing, as it shouldn't really be possible to disable file sync on the MoM only (and not also the compile masters). I believe there is a file_sync_enabled parameter on puppet_enterprise::profile::master - can you please verify that it is set to true? During an agent run, that classification, when applied on the MoM, should ensure that file sync is enabled and running with puppet server. So trying to agent run on the MoM after checking that parameter might be worthwhile - if there are errors, that might explain this.

edit flag offensive delete link more
0

answered 2016-05-20 11:37:53 -0500

mike.r gravatar image

updated 2016-05-20 11:38:43 -0500

Im running this on a single master, my File Sync enabled was set to True,

heres the output of Debug services, it shows file sync enabled,

file-sync-client-service":{"serviceversion":"0.1.11","servicestatusversion":1,"detaillevel":"debug","state":"running","status":{"lastcheckin":null,"lastsuccessfulsynctime":null,"lastsync":null,"

[root@puppetmaster3 code-staging]# curl -k https://localhost:8140/status/v1/services?level=debug
{"pe-puppet-profiler":{"service_version":"2016.1.1017","service_status_version":1,"detail_level":"debug","state":"running","status":{"experimental":{"function-metrics":[],"resource-metrics":[],"catalog-metrics":[],"inline-metrics":[]}}},"pe-jruby-metrics":{"service_version":"2016.1.1017","service_status_version":1,"detail_level":"debug","state":"running","status":{"experimental":{"jruby-pool-lock-status":{"current-state":":not-in-use","last-change-time":"2016-05-20T16:28:39.393Z"},"metrics":{"average-lock-wait-time":0,"num-free-jrubies":2,"borrow-count":0,"average-requested-jrubies":0.0,"borrow-timeout-count":0,"return-count":0,"borrow-retry-count":0,"borrowed-instances":[],"average-borrow-time":0,"num-jrubies":3,"requested-count":0,"average-lock-held-time":0,"requested-instances":[],"average-free-jrubies":1.0,"num-pool-locks":0,"average-wait-time":0}}}},**"file-sync-client-service":{"service_version":"0.1.11","service_status_version":1,"detail_level":"debug","state":"running","status":{"last_check_in":null,"last_successful_sync_time":null,"last_sync":null,"current_activity":"idle**","repos":{},"experimental":{"metrics":{"num-fetches":0,"average-fetch-time":0,"num-clones":0,"average-clone-time":0,"num-syncs":0,"average-sync-time":0,"num-sync-clean-checks":0,"average-sync-clean-check-time":0}},"timestamp":"2016-05-20T16:29:54.739Z"}},"pe-master":{"service_version":"2016.1.1017","service_status_version":1,"detail_level":"debug","state":"running","status":{"experimental":{"http-metrics":[{"route-id":"puppet-v3-static_file_content-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-file_content-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-environments","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-file_metadata-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-experimental","count":0,"mean":0,"aggregate":0},{"route-id":"other","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-status-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-experimental-metrics-dashboard-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-report-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-experimental-dashboard_html","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-resource_types-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-node-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-catalog-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-resource_type-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-file_metadatas-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-file_bucket_file-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-experimental-js-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"total","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-environment_classes-/*/","count":0,"mean":0,"aggregate":0},{"route-id":"puppet-v3-environment-/*/","count":0,"mean":0,"aggregate":0}]}}},"status-service":{"service_version":"0.3.1","service_status_version":1,"detail_level":"debug","state":"running","status":{"experimental":{"jvm-metrics":{"heap-memory":{"committed":2070937600,"init":2147483648,"max":2070937600,"used":526754088},"non-heap-memory":{"committed":276922368,"init":2555904,"max":-1,"used":183908392},"up-time-ms":210490,"start-time-ms":1463761584557}}}}}[root@puppetmaster3 code-staging]# systemctl status pe-puppetserver

when attempting to start Puppet Server, it just crashes. Also I cant get into the Console to check the File Sync Enabled property, thats where it was set originally, since Console nginx service isnt running if pe-puppetserver isnt started.

2016-05-20 12:33:07,466 INFO  [async-dispatch-2] [p.s.l.legacy-routes-service] The legacy routing service has successfully started and is now ready to handle requests
2016-05-20 12:33:07,583 INFO  [async-dispatch-2] [p.e.s.f.file-sync-client-service] Stopping file sync client service
2016-05-20 12:33:07,633 INFO  [async-dispatch-2] [p.e.s.j.pe-jruby-metrics-service] PE JRuby Metrics Service: stopping metrics sampler job
2016-05-20 ...
(more)
edit flag offensive delete link more

Comments

Any chance this host is classified as a compile master instead of a monolithic? It's missing the server component of file sync - `file-sync-storage-service`. I highly recommend opening a support ticket at this point. I tried to write a longer response, but there's a character limit on comments

Kevin Corcoran gravatar imageKevin Corcoran ( 2016-05-23 07:49:45 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2016-05-19 22:13:28 -0500

Seen: 221 times

Last updated: May 20 '16