Ask Your Question
0

Standard mono-install of 2019.0 PE fails connecting Postgresql with default config

asked 2019-01-02 09:15:21 -0600

bvdveen gravatar image

Im trying to install PE 2019.0 on my CentOS7 box with the most basic configuration (only console pwd set) using ./puppet-enterprise-installer

PuppetDB fails to connect to PE-PostgresQL with the following error logged in /var/log/puppetlabs/puppetdb/puppetdb.log:

2019-01-02T15:30:48.551+01:00 DEBUG [c.z.h.p.HikariPool] PDBMigrationsPool - Cannot acquire connection from data source
org.postgresql.util.PSQLException: The server does not support SSL.
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:379)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:160)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:117)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:123)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:375)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:204)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:459)
        at com.zaxxer.hikari.pool.HikariPool.access$200(HikariPool.java:70)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:682)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Host configuration prior to setup

  • Added master.mydomain.com host to 127.0.0.1 in /etc/hosts
  • Confiirmed that static hostname is set correctly (hostnamectl and puppet config print certname both report master.mydomain.com)
  • master.mydomain.com resolves to IPV4 127.0.0.1

Logs

  • After install fails (as it cannot connect to master.mydomain.com:8140) running puppet agent -t reports

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find facts from PuppetDB at master.mydomain.com:8140: Failed to execute '/pdb/query/v4/nodes/master.mydomain.com/facts' on at least 1 of the following 'server_urls': https://master.mydomain.com:8081

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=981b004ac170e70b951e4fb7c659f0466eb14b87&version=5&certname=master.mydomain.com&command=replace_facts&producer-timestamp=1546439936' on at least 1 of the following 'server_urls': https://master.mydomain.com:8081
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
  • /var/log/puppetlabs/postgresql/postgresql-Wed.log

2019-01-02 10:51:25.939 CET [db:,sess:5c2c880b.7d87,pid:32135,vtid:,tid:0] LOG:  received fast shutdown request
2019-01-02 10:51:25.939 CET [db:,sess:5c2c880b.7d87,pid:32135,vtid:,tid:0] LOG:  aborting any active transactions
2019-01-02 10:51:25.939 CET [db:,sess:5c2c880b.7d8e,pid:32142,vtid:1/0,tid:0] LOG:  autovacuum launcher shutting down ...
(more)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-01-07 02:47:29 -0600

bvdveen gravatar image

After quite a lot of time spent I figured out the problem. I found it strange that a standard install of Puppet Enterprise with default settings wouldn't work on my instance. This led me to believe that it had to be caused by software that was already installed on the system.

The box im running is used multiple purpose. I had Rancher as well as docker installed for our CI/CD environment and our docker registry hosting.

The error that PostgreSQL did give a slight hint into what was going on (albeit confusing). Rancher 1.6 is known to do a lot of network configuration for you. Handy when it is the only thing that you are running on that box but in the end this was found to be the culprit. Rancher had several container stacks deployed that also included an instance of PostgreSQL. One of them running on the default port (5432). The default iptables would not show any open ports, which holds true. The port itself is not open. Only after running iptables -t nat I saw that all packets on my localhost:5432 were prerouted and D-natted to my rancher containers. Since the Rancher Postgres container is only running isolated on the box no SSL was required (for now), thus the reason that Postgres was not configured to handle SSL.

To remedy, disable the Rancher networking to handle it for you and do it manually (this is quite a chore and cumbersome to maintain and should only be done by advanced users). Another option is to catch the prerouting of the D-NAT before it gets sent to the CATTLE_OUTPUT (Rancher) and add an ACCEPT rule for this specific case.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2019-01-02 09:06:04 -0600

Seen: 112 times

Last updated: Jan 07