Ask Your Question
0

Migrating from Puppet 3.x to Puppet 4 at scale.

asked 2017-08-01 17:04:56 -0500

ramindk gravatar image

I have a very large and very old code base. 500k lines of manifests code. The code base also goes back to 2007. What are the steps to migrate this codebase to Puppet 4?

edit retag flag offensive close merge delete

4 Answers

Sort by ยป oldest newest most voted
2

answered 2017-08-01 17:29:24 -0500

ramindk gravatar image

updated 2017-08-16 00:53:52 -0500

Hi self. You are in for a world of hurt.

work in progress

Consider this a live blog

Links I found useful

Planned upgrade sequence

Current plan is to do in order

  • fix code and change all master to 3.8.7 with future parser enabled
  • Test code against puppetserver 1.2. Assume another round of code fixes will be required
  • Upgrade all masters to 1.2
  • Start testing puppet-agent 1.0 against puppetserver 1.2
  • Fix code, assume we have dependencies on agent behavior
  • Upgrade all agents to puppet-agent 1.0
  • Upgrade all agents to puppet-agent 1.1, 1.2, 1.3, 1.4. Assume catalog will apply differently at some point so upgrade slow.
  • Start testing against puppetserver 2.7.2. Assume more code to fix
  • Upgrade all servers to puppetserver 2.7.2
  • Upgrade all agent to 1.10
  • Cake.

Upgrade Notes

Upgrade to Puppet 3.8.7

Do any puppet 3.x upgrades now. This will make any future work simpler. You can run puppet 2.7 agents against your Puppet 3 masters and even against 3.8 with the future parser enabled. However you do not want to manage differences in agents and masters.

We update to 3.8.7 because it has the best future parser support before going to puppetserver.

Upgrade to Facter 2.4.6

Standardize on one version of facter. At some point you'll need to turn off stringified facts and best to have a single facter version.

puppet-lint

I recommend starting with puppet-lint 2.2.x if you haven't been linting your code. The best way to run puppet-lint is with the --only-check and --fix flags. This allows you to test one set of changes at time. As you individually test checks you can add them to your commit hooks.

puppet-lint --only-checks=trailing_whitespace --fix modules/
puppet-lint --only-checks=hard_tabs --fix modules/
puppet-lint --only-checks=trailing_whitespace,hard_tabs --fail-on-warnings modules/

The order you run these checks in can be important. In a large codebase that has never been linted I suggest the following progression. For example it's easier to check for vars when your quoting is correct. --fix will get you 95% of the way there, but you will see corner cases you'll need to manually edit.

--only-checks=trailing_whitespace,hard_tabs,duplicate_params,double_quoted_strings,\ unquoted_file_mode,only_variable_string,variables_not_enclosed,\
single_quote_string_with_variables,variable_contains_dash,ensure_not_symlink_target,\
unquoted_resource_title,relative_classname_inclusion,file_mode,resource_reference_without_title_capital,leading_zero

puppet-syntax

https://github.com/voxpupuli/puppet-s...

You can run this across an entire codebase with the future parser turned on. Defaults to the Puppet that is installed locally. puppet-syntax is great for catching syntax mistakes, but will not catch data mistakes such as regex-ing a int, numeric operations on strings, etc.

Data types will be the majority of your problems in my experience.

A word on running 3.8.x with the future parser

It's a terrible idea

If you can spin up a new Puppet 4 infra ... (more)

edit flag offensive delete link more
1

answered 2017-08-02 07:24:33 -0500

slk gravatar image

I found that using rspec-puppet helped a lot with making code changing for a migration like this. This will also help you trim the fat from your current code base. When that is done run the tests against puppet 4 and adjust accordingly.

edit flag offensive delete link more
1

answered 2017-08-09 04:52:34 -0500

Henrik Lindberg gravatar image

Puppet catalog-preview is a very useful tool when migrating 3 -> 4. Here is an excellent presentation that shows you what it can do for you: https://www.youtube.com/watch?v=sHlQ0...

edit flag offensive delete link more

Comments

The presentation says it is PE only, but that is has been released as open source (free).

Henrik Lindberg gravatar imageHenrik Lindberg ( 2017-08-09 04:53:51 -0500 )edit

Go here https://forge.puppet.com/puppetlabs/catalog_preview to get it (also where the documentation is)

Henrik Lindberg gravatar imageHenrik Lindberg ( 2017-08-09 04:58:15 -0500 )edit

Thanks for the heads up. I had heard of the tool, but did not realize it was open source now.

ramindk gravatar imageramindk ( 2017-08-09 14:16:57 -0500 )edit
0

answered 2017-08-16 11:45:10 -0500

Jeremiah Powell gravatar image

updated 2017-08-16 14:53:07 -0500

Much of moving code from Puppet DSL 2 or 3 to Puppet 'full programming language" 4 is almost script-able. If you do native type development and lots of custom facts then you are going to need a lot more help than the following list.

The following psuedo-checklist applies to Puppet Enterprise, Open Source or whatever mix you run. If you are going from 2 to 3 you might just go straight to 4 with these changes.

  • Drop Modulefile for metadata.json
    • Remove checksums as these are now calculated separately during puppet module build
    • Use the new proprietary keyword for non-F/OSS licenses (you do have a LICENSE file, right?)
  • Replace environment and type as these are keywords
  • Replace import withinclude
  • Remove or replace code that used search()
  • Qualify namespaces on classes: include foo becomes include ::foo (puppet-lint will complain loudly)
  • Add the @ symbol to variables in templates that previously did not or that used scope() tricks
  • Update your dependencies
    • Update .fixtures.yaml if you use rspec-puppet
    • Update dependencies in metadata.json
    • Include both minimal and maximal versions (e.g. "version_requirement": ">=a.b.c <x.y.z")
  • Replace any if ($string) tests that depended upon empty strings being False
    • These are now True
    • The nil keyword from Ruby becomes the string 'nil' in Puppet
  • Replace any code that depended on an undef or empty title for a resource as this is now a compiler error
  • Check class parameters
    • Start parameters with letters or underscores
    • Use lowercase letters, numbers and underscores only
    • Add missing documentation in older puppet-doc format for puppet-lint
    • Optionally add newer puppet-strings format documentation
  • Remove any purge options for the cron resource
  • Quote all octal permissions on file resources, this type's API changed in 4 but it won't hurt in 3 clean this up

Others are more subjective and depend on your intended design and site policies

  • Replace complicated anchor use with contain
  • Replace facts with $::facts[] references
  • Consider replacing or adding Puppet Templates in place of ERB
  • Check that all use of file has specific owner, group and mode parameters
  • Stop using the params pattern
    • Drop params.pp, defaults.pp or whatever you called it
    • Use hiera data-in-modules in Puppet 4 (hiera 3 and later)
    • Feel free to form complex hiera trees inside the module for defaults
    • Cut down the amount of complex conditional code since hiera lookup can do it
    • Laugh at the people who told you to switch to Ansible since your configuration is all YAML now anyway
  • Replace all validate_x calls with types (depends on what puppetlabs-stdlib version you want to use)
  • Rework manifests to not use class inheritance (composition and specialization is still okay)

Do you use rSpec? Rubocop? puppet-lint?

  • Check any Gemfile or Rakefile for tasks impacted by Puppet 5's release
    • The semantic-puppet changes will generate warnings for you when bundle install or bundle update is run
    • This affects you if you're using Puppet 2, 3, 4 or 5
  • Use is_expected.to style test matchers in rSpec
    • The ...
(more)
edit flag offensive delete link more

Comments

The comment that "the nil keyword from Ruby becomes the string 'nil' in Puppet" is not correct - where did you get that from?

Henrik Lindberg gravatar imageHenrik Lindberg ( 2017-08-18 03:12:23 -0500 )edit

This nil promotion to string is quite correct if you actually test it (Puppet 5.1.0, rspec-puppet 2.6.8): manifests/init.pp: ``` test ($testparam) { notify {$testparam: }}} ``` spec:/classes/test_spec.pp: ``` let(:params){{:testparam=nil}} ... it { is_expected.to contain_notify('nil') } ```

waveclaw gravatar imagewaveclaw ( 2017-08-19 21:02:56 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

5 followers

Stats

Asked: 2017-08-01 17:04:56 -0500

Seen: 201 times

Last updated: Aug 16