Ask Your Question
0

Foreman will not start after accidental Puppet upgrade, claims port 443 is already in use.

asked 2017-08-15 14:22:19 -0600

John R gravatar image

Summary: We accidentally upgraded puppetserver-2.4.0 to 2.7.2 without preparation. This broke everything. We downgraded the package and Puppet will now start, but it cannot speak to Foreman. Foreman will not start, it insists that port 443 is already in use.

Background: The person who set all this up is no longer with the company and left NO documentation of any sort. I am very new to Puppet myself. I know my way around Linux but I know next to nothing about Puppet.

The issue: We had a working Puppet 2.4.0 server, on a CentOS 7 host, with Foreman. An admin noticed that no security updates were running on the puppet server and ran "yum -y update" and walked away. This upgraded Puppet, among other things. The upgraded puppet would not start. I downgraded puppet to 2.4.0 again and it starts, but the logs show only items like this:

2017-08-15 15:05:40,090 ERROR [qtp274685303-70] [puppetserver] Puppet Server Error: Could not find node 'workstation7.company.lan'; cannot compile
2017-08-15 15:05:40,792 WARN  [qtp274685303-68] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: Could not send facts to Foreman: Connection refused - connect(2)

Foreman itself is set to run under httpd, but httpd will not start with the error

Aug 15 14:58:24 puppetserver.company.lan httpd[8571]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443
Aug 15 14:58:24 puppetserver.company.lan systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE

There's nothing running on port 443. netstat -nat shows nothing on 443. lsof -iTCP:443 shows nothing on 443. fuser -v 443/tcp shows nothing. ps aux |grep httpd shows nothing, ps aux |grep foreman shows

foreman+  3080  0.0  0.3 296676 25496 ?        Sl   14:44   0:00 ruby /usr/share/foreman-proxy/bin/smart-proxy

I know nothing about maintaining Foreman or puppet. I did not set up this server and I have zero documentation from the person who did. Can anyone help me find where to even start?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

answered 2017-08-19 21:22:55 -0600

waveclaw gravatar image

The puppet master process (2.x) or puppetserver process (3.x) wants to run on port 8140. The Foreman wants to proxy both 443 and 8140 connections. Puppet has no component that will listen on port 443. If you still have the original server, what was listening on that port? (lsof -i -P |grep 8140 then run pgrep on that PID.)

Puppet Enterprise (PE) ships with the Puppet Console that will setup a different web service on port 443. That will conflict with the websever for The Foreman.

Butif you are using Puppet Enterprise then there is no reason to use The Foreman. The Foreman is an "alternative" dashboard for Puppet.

With PE already have a full easy-to-use web-base interface and a provisioning system (razor) with Puppet Enterprise. The exception to this is if you are running a Satellite or Katello server. Those also integrate RedHat subscription management and the Pulp content management system into The Foreman.

It is possible to make conflicting ports work well with manual installations. However, it is a better idea to use the carefully integrated puppet installtion setup by the Foreman. This will not only properly handle port conflicts but also feed the classification system in Puppet with data from The Foreman.

Since you upgraded to Puppet 4 from an early 2.x release, does any of your code work?

If the code base was simple user -> package -> file -> service pattern code then it is likely to still work.

edit flag offensive delete link more

Comments

The original install was not Puppet Enterprise. I could fire up the old VM (after disconnecting it's NIC) and check 8140, but all the error messages were very clear that 443 was the problem, not 8140.

John R gravatar imageJohn R ( 2017-08-21 10:52:10 -0600 )edit

The puppet scripts still work. They were all very simple: Ensure package installed (with yum). Ensure symlink created. Ensure NFS mount created and mounted with the right flags, etc. We didn't have any problems with just applying them to the new client version. Thanks!

John R gravatar imageJohn R ( 2017-08-21 10:54:03 -0600 )edit
0

answered 2017-08-18 12:08:25 -0600

John R gravatar image

So it turns out: We weren't able to fix this. A bunch of digging showed that the person who set this up originally had built a working Puppet install, then grafted Foreman on top of it, and this is a classic case of the steps of troubleshooting leading to "that should never have been working".

So our "fix", such as it is, was to rebuild a clean Puppet server. Bare CentOS 7 install, install Foreman first and let it handle the puppet install itself, pull in our pre-existing classes and build scripts, sign all the new certificates.

We needed to update the puppet CLIENT version too, but that's a simple task and can be done easily on each machine. From there we re-add the machines to classes and we're up again, this time with a more recent version of puppet server (4.10.6) AND a clear set of documentation on what we did to get from bare metal to here.

It's a lot more work than just fixing the old server would have been, but we couldn't figure out what was wrong with the old server.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

Stats

Asked: 2017-08-15 14:22:19 -0600

Seen: 65 times

Last updated: Aug 19