Ok I got it working by following the suggestion made by @ken. I had to do a few extra steps to get it working.
First off, on my puppetdb server, the puppet.conf stayed like this:
$ cat /etc/puppetlabs/puppet/puppet.conf
dns_alt_names = ip-10-0-101-39.eu-west-1.compute.internal
certname = puppetdb_preprod_i-075d0a8f
server = my_puppet_server.com
On my puppetdb server, I deleted my ssl certificates:
$ puppet config print ssldir
$ rm -rf /etc/puppetlabs/puppet/ssl
On the puppetmaster, I then deleted the puppetdb's certificates:
$ puppet cert clean puppetdb_preprod_i-075d0a8f
Next I did a puppet run on my puppetdb:
$ puppet agent -t
Info: Creating a new SSL key for puppetdb_preprod_i-075d0a8f
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for puppetdb_preprod_i-075d0a8f
Info: Certificate Request fingerprint (SHA256): B5:2F:2A:B6:F0:92:09:6D:8C:F7:DE:89:F2:A5:07:51:DA:31:DA:27:E9:AA:85:1C:BF:66:EC:64:EA:D0:25:F3
Error: Could not request certificate: Error 400 on SERVER: CSR 'puppetdb_preprod_i-075d0a8f' contains subject alternative names (DNS:ip-10-0-101-39.eu-west-1.compute.internal, DNS:puppetdb_preprod_i-075d0a8f), which are disallowed. Use `puppet cert --allow-dns-alt-names sign puppetdb_preprod_i-075d0a8f` to sign this request.
Since I am using the dnsaltname setting, I had to follow the above suggestion and run the 'puppet cert ....' command on the puppetmaster. However might be able to suppress this error message happening in the first place by running the following on the puppetmaster beforehand:
$ puppet ca sign --allow-dns-alt-names
I then did a puppet run on the puppetdb server again and this time it worked.
I then ran the puppet ssl-setup command on the puppetdb server, and got the following warning messages:
$ puppetdb ssl-setup
PEM files in /etc/puppetlabs/puppetdb/ssl already exists, checking integrity.
Warning: /etc/puppetlabs/puppetdb/ssl/private.pem does not match the file used by Puppet (/etc/puppetlabs/puppet/ssl/private_keys/puppetdb_preprod_i-075d0a8f.pem)
Warning: /etc/puppetlabs/puppetdb/ssl/public.pem does not match the file used by Puppet (/etc/puppetlabs/puppet/ssl/certs/puppetdb_preprod_i-075d0a8f.pem)
Setting ssl-host in /etc/puppetlabs/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-port in /etc/puppetlabs/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-key in /etc/puppetlabs/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-cert in /etc/puppetlabs/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-ca-cert in /etc/puppetlabs/puppetdb/conf.d/jetty.ini already correct.
To get rid of the warning messages, I moved these files to my tmp folder (alternatively you could also just delete them):
$ mv /etc/puppetlabs/puppetdb/ssl/private.pem /tmp/private.pem
$ mv /etc/puppetlabs/puppetdb/ssl/public.pem /tmp/public.pem
Then I ran the ssl-setup command again on the puppetdb server:
$ puppetdb ssl-setup
PEM files in /etc/puppetlabs/puppetdb/ssl are missing, we will move them into place for you
Copying files: /etc/puppetlabs/puppet/ssl/certs/ca.pem ...