MoM keeps changing auth.conf authorization rules

asked 2016-05-02 07:18:43 -0500

Wh33ly gravatar image

Hello all,

I have a nice challenge considering file-sync between the MoM and Compile Masters, file sync works perfect, except we have a little re-occuring change in file-sync. We have the following setup 1 MoM and 2 load balanced Compile masters. The nodes use the loadbalanced compile masters for their communication, and the MoM is the CA.

The problem we encounter is that the MoM puppet run frequently changes the authorization rules below:

/Stage[main]/Puppet_enterprise::Master::File_sync/Pe_puppet_authorization::Rule[puppetlabs file sync api]/Pe_puppet_authorization_hocon_rule[rule-puppetlabs file sync api]/ensure
/Stage[main]/Puppet_enterprise::Master::File_sync/Pe_puppet_authorization::Rule[puppetlabs file sync repo]/Pe_puppet_authorization_hocon_rule[rule-puppetlabs file sync repo]/ensure

When diving into this I noticed it has something to do with the order in which the puppet rules in /etc/puppetlabs/puppetserver/conf.d/auth.conf are re-created. For example it looks like this, sometimes this stays a few runs unchanged, but sometimes it changes or keeps changing several runs.

Before run:

{
            "allow" : [
                "MoMaster.domain.com",
                "CompileMaster01.domain.com",
                "CompileMaster02.domain.com"
            ],
            "match-request" : {
                "path" : "/file-sync/v1/",
                "query-params" : {},
                "type" : "path"
            },
            "name" : "puppetlabs file sync api",
            "sort-order" : 500
        }
    ,
        {
            "allow" : [
                "MoMaster.domain.com",
                *"CompileMaster01.domain.com",
                "CompileMaster02.domain.com"*
            ],
            "match-request" : {
                "path" : "/file-sync-git/",
                "query-params" : {},
                "type" : "path"
            },
            "name" : "puppetlabs file sync repo",
            "sort-order" : 500
        }

After puppet run and file-sync changed:

{            "allow" : [
                "MoMaster.domain.com",
                "CompileMaster02.domain.com",
                "CompileMaster01.domain.com"
            ],
            "match-request" : {
                "path" : "/file-sync/v1/",
                "query-params" : {},
                "type" : "path"
            },
            "name" : "puppetlabs file sync api",
            "sort-order" : 500
        }
    ,
        {
            "allow" : [
                "MoMaster.domain.com",
                *"CompileMaster02.domain.com",
                "CompileMaster01.domain.com"*
            ],
            "match-request" : {
                "path" : "/file-sync-git/",
                "query-params" : {},
                "type" : "path"
            },
            "name" : "puppetlabs file sync repo",
            "sort-order" : 500
        }

As you can see the order from the "Allowed (whitelisted) nodes is changed" looking at the ruby code in : /opt/puppetlabs/puppet/modules/puppetenterprise/manifests/master/filesync.pp you can see that the "Allow list array" is build (depending if Compile Master or Master) with 2 variables. The local host and a list of whitelisted "Master queried from the Database" it seems that we can find the issue here, if I test the DB query it sometimes returns : CompileMaster01,CompileMaster02 and other times in the opposite order.

Here a part of the file_sync.pp which creates the authorization rules

$current_certs = pe_union([$certname], $whitelisted_certnames)
    if $get_whitelist_masters_from_puppetdb {
      $masters_in_puppetdb = pe_puppetdbquery_nodes(
        'Class["puppet_enterprise::profile::master"]')
    }
    else {
      $masters_in_puppetdb = []
    }
    $master_certs = pe_unique(pe_union($current_certs, $masters_in_puppetdb))

    pe_puppet_authorization::rule { 'puppetlabs file sync api':
      match_request_path   => '/file-sync/v1/',
      match_request_type   => 'path',
      allow                => $master_certs,
      sort_order           => 500,
    }

    pe_puppet_authorization::rule { 'puppetlabs file sync repo':
      match_request_path   => '/file-sync-git/',
      match_request_type   => 'path',
      allow                => $master_certs,
      sort_order           => 500,

I think this can be easily solved by using a "Sort function" here, this way the results are sorted in order and not in probably "last update" or so. I think it is a sort of timing issue where sometimes on compile master is last updated in the database which causes the opposite result.

Searching on the net, didn't help me and I don't believe I'm the only one who encounters this issue, these are basic puppet components causing this issue. So what's next, I looking for ... (more)

edit retag flag offensive close merge delete