As in the previous installment, you need to have already completed a few steps before arriving at this post. You should have already completed a “split installation” (Documented here). Also, your load balancer needs to be configured and running. The procedure for this portion can be found here. If you’ve completed all these portions, you are now ready to configure and install the compilers themselves. If this is you, read on!

---

Once your Load Balancer and split install are in place and functioning, we need to add more compilers to the serving infrastructure. For the purposes of this tutorial, we will install two additional catalog compilers, register them with the currently existing master. Then, we will direct them to look to the “MoM” or the “Master of Masters” as the CA certificate authority. Further, we will install two agent nodes and connect them to the infrastructure.

You will need to install two compiler nodes and two agent nodes according to the following specifications.

Once these four nodes are in place, we can connect the compilers to the Master of Masters (MoM) and then the agents to the “master” as they see it. Remember, that for our purposes, these nodes are named:

• compile1.example.com
• compile2.example.com
• agent1.example.com
• agent2.example.com

Installing the Compilers

SSH to the first compiler master (compile1.example.com for this post’s purposes) and install the Puppet agent as follows:

curl -k https://master.example.com:8140/packages/current/install.bash | bash -s main:dns_alt_names=compile1.example.com,compile.example.com,compile1,compile

What this does is simple. When this compiler goes behind the load balancer (compile.example.com), traffic may get directed to this node. When the request is made, the agent node will be asking for “compile.example.com” but this node’s name is “compile1.example.com“. The additional options at the end of the curl line are to tell the agent that when it installs, it should be aware of both names, and when speaking to the MoM the first time to request its cert, to represent all the comma delimited names listed at the end of the above command.

Next, SSH to your master node and accept the agent cert request as follows to allow for these names on the MoM you just set up in the previous step.

ssh master.example.com
puppet cert --allow-dns-alt-names sign compile1.example.com

NOTE THAT YOU CANNOT ACCEPT THIS CERT FROM THE CONSOLE. ALT_DNS IS NOT SUPPORTED FROM THE GUI

Finally, run the puppet agent on the first compiler (compile1.example.com) to configure the node:

puppet agent -t

Once the agent run is complete, you need to classify the catalog compiler in the console to make it ready for service.

Classify the Compiler

In the Puppet Enterprise Console:

Choose Nodes | Classification | PE Master

Add compile1.example.com and pin the node to the classification group and commit the change.

BE SURE TO COMPLETE THE NEXT STEPS IN ORDER AS FOLLOWS OR YOU WILL HAVE A BAD TIME

First: SSH to compile1.example.com and run puppet agent -t

Second: SSH to puppetdb.example.com and run puppet agent -t

Third: SSH to console.example.com and run puppet agent -t

Fourth: SSH to master.example.com and run puppet agent -t

BE SURE TO ALLOW EACH RUN TO COMPLETE FULLY BEFORE MOVING ON TO THE NEXT ONE

For All Subsequent Compile Node Installations

Follow the above instructions completed for compile1.example.com for all subsequent compiler installations. This means that if you add compilers six months or a year from now, go back to the previous procedure and duplicate it with the new node name precisely as you did above. To recap:

1. Install the agent as above with the alt_dns switches
2. Accept the cert on the master with the alt_dns switches
3. Classify the compiler in the console
4. Run the Puppet agent in the above specified order, allowing each one to complete fully before moving on.

Configure Future Agent Installations to Point to the Load Balancer by Default

In the Puppet Enterprise Console, you must configure the system to point all future agent installations to the load balancer by default so you do not have to continue to make modifications and customizations after each agent install. To do so, perform the following steps:

1. In the Puppet Enterprise Console, choose: Nodes | Classification | PE Master
2. Select the “Classes” tab.
3. Choose the “pe_repo” class.
4. Under the parameters drop-down, choose “master” and set the text box to the name of your load balancer or VIP (in our case, “compile.example.com”)
5. Commit the changes.

Point the New Compilers at the Master (MoM) for CA Authority

Create a new classification group called “PE CA pe_repo Override”

• Go to Nodes | Classification in the Puppet Enterprise Console
• Create a New Group
• Name the new group “PE CA pe_repo Override“
• From the “Parent Name” drop-down, choose the “PE Master” group.
• Click “Add Group”.

1. Select your new group and pin master.example.com to the new group and click “Commit One Change“
2. Select the “Classes” tab.
3. Add “pe_repo” class.
4. From the parameter drop-down, select “Master“.
5. Enter the name of the MoM in the text box. (in this example, master.example.com)
6. Click “Add Parameter” and then “Commit 2 Changes”.

Test New Agents

The two agents you created at the beginning of the article are now able to be tested with this new group of compilers.

1. Make sure agent1.example.com and agent2.example.com have been installed according to the system requirements covered in this series.
2. Install the Puppet agent on each of these nodes, but this time instead of pointing at the MoM, point to your Load balancer vip like so:

curl -k https://compile.example.com:8140/packages/current/install.bash | bash

In the PE Console, accept the new certificate request for Agent1. SSH to agent1.example.com and run puppet agent -t. Finally, repeat this process for agent2.example.com.

If you have completed all the above steps properly, the agents will reach out to the compile.example.com VIP and be ferried off to one of the catalog compilers. Regardless of which one, since we accepted all the alternate DNS names when creating the connection between them and the MoM, they will respond for compile.example.com, and deliver back to the agent the required information, catalog, etc. as Puppet would do under normal circumstances.

Conclusion

As you can see, we needed the Load Balancer in place to install the catalog compilers. We also needed all the DNS alt-naming to be in place so the load balancer could send traffic to either catalog compiler as needed, and still have it answer for the VIP name. Finally, we needed to refer requests to their appropriate destinations and also classify the new compilers as such with the MoM, and set up appropriate referral of certificate requests from the compilers back to the CA Master, which is the MoM.

The serving infrastructure is almost done, all we have left to do is to scale MCollective with an ActiveMQ Hub & Spoke, and remove that responsibility from the MoM. We will also install a GitLab server to hold our Control Repo and associated Roles & Profiles, and we will configure the Code Manager.

You should have completed a split install before beginning this section. You can find the Split Installation documentation at Puppet’s Website, or the first installment of this tutorial here. If you try and begin here, you might find yourself lost.

Note also that the “Additional Compilers” docs comes in two parts–One to install the Load Balancer and one to install the compilers.

First, Some Philosophy

The Puppet Enterprise documentation circa PE 2015.3.2 had some “issues”. Let me actually preface that, though. Puppet Labs’ documentation is by far some of the most voluminous and in many respects most complete vendor documentation out there. I don’t mean to disparage their work AT ALL. When it comes to the fact they even have documentation at this level, they’re the “bees knees”.

However, I’ve always written documentation to fit the “grandma rule”. My grandmother was a little 4 foot nothing Cajun woman with English as her second language. She never used the first computer, still had a rotary phone when she passed away, and remained suspicious of anything technical. She was, however, a voracious reader, keenly intelligent, and understood considerably more than you’d expect on first glance. She also was a stickler for puncutation, grammar, and the like. In short, if my grandma couldn’t read the documentation and follow a step-by-step process to install Puppet successfully, then its just either too complex, poorly formatted or unclear and needs to be simplified.

This causes a problem, of course. There are technologists out there that would become annoyed at repetition, verbosity around “understood” things, and spelling out each and every step along the way… even painfully. However, I feel it is the only proper way to document something. My rules are simple.

• Leave nothing to question
• Be as verbose and clear as possible
• Make sure everything is in order, step-by-step

By following this simple guideline, I feel I’m doing more of a service to the reader than if I presumed on their level of sophistication with Puppet, Linux/UNIX, Windows, research capability, Google-foo or whatever.

So let’s dive in, shall we?

HAProxy

Seemingly counterintuitive, now that we’ve done a split install, I want to next install the HAProxy we will use as a Load Balancer on the additional compilers. By installing this first, we can utilize Puppet to install the HAProxy, and manage them automatically rather than doing a lot of ad-hoc work.

Also, by doing the proxy first, the prerequisites are satisfied in their proper order, the Load Balancer exists before configuring additional compilers (to be able to utilize the dns_alt_names for the load balancer along with the compilers) and to have the GitLab in place and hosting the control_repo before turning on and configuring Code Manager.

Hardware

In the initial hardware list, I included a node called “Compile Master”. This node looked like:

This node may seem like overkill, but disk and memory are cheap. If you are scaling at this level, its better to not have to reinstall your Load balancer later. Keep in mind, you don’t have to use HAProxy and can use a corporate Load Balancer here, but its configuration is outside the scope of this tutorial.

Once you’ve provisioned the load balancer, ssh to the node as the root user, and use the “frictionless installer” to add your Puppet agent.

curl -k https://master.example.com:8140/packages/current/install.bash | bash

When the client is fully installed, retrieve the Enterprise Console from your browser, and navigate to Nodes | Classification | Unsigned Certificates and select “Accept All”. Finally, ssh to the instance as the root user and run puppet agent -t to finish the setup.

Configure the Load Balancer

At this point, the node is provisioned and you have a Puppet agent running on it, but you have as of yet not configured the HAProxy Load Balancer for use in the environment. The load balancer will be necessary to have in place prior to adding compile masters to your existing split installation. The following instructions guide you through setting up the HAProxy load balancer.

1. SSH to the Puppet Master as root. (master.example.com in our list)
2. Install the HAPRoxy Forge Module on the master

puppet module install puppetlabs-haproxy

leave your root console open while performing steps 3-6

3. Retrieve the Enterprise Console in your browser
4. Select Nodes | Classification
5. Create a New Classification Group called “Load Balancer“
6. Select the new group from the list and pin the node “compiler.example.com” into the new group.
7. In your open SSH session to master.example.com, create the profiles module to hold the configuration for HAProxy

cd /etc/puppetlabs/code/environments/production/modules

mkdir -p profiles/manifests

cd profiles/manifests

8. Once you have changed to the profiles/manifests directory, create the loadbalancer.pp manifest.
9. Follow the documentation here to configure HAProxy. When complete, the loadbalancer.pp manifest should resemble the following with IPs corrected for your particular instance:

# Load Balancer Profile
class profiles::loadbalancer {

  class { 'haproxy': }

  # Main Proxy Listener
  haproxy::listen { 'compiler.example.com':
    collect_exported => false,
    ipaddress        => $::ipaddress,
    ports            => '8140',
  }

  # First Load balanced Compile Master
  haproxy::balancermember { 'compiler1.example.com':
    listening_service => 'compiler.example.com',
    server_names      => 'compiler1.example.com',
    ipaddress         => '10.0.1.24',
    ports             => '8140',
    options           => 'check',
  }

  # Second Load Balanced Compile Master
  haproxy::balancermember { 'compiler2.example.com':
    listening_service => 'compiler.example.com',
    server_names      => 'compiler2.example.com',
    ipaddress         => '10.0.1.25',
    ports             => '8140',
    options           => 'check',
  }
}

Once you have created this profile, retrieve the Puppet Enterprise Console in your browser and navigate to Nodes | Classification | Load Balancer.

1. Selet the Classes tab.
2. Click the “refresh” button so the console will pick up your new loadbalancer.pp profile to classify your node with.
3. Under the “Add new Class” heading, select profiles::loadbalancer from the list that drops down.
4. Click “Add Class”.
5. Select “Commit 1 Change” at the bottom right of the page.
6. SSH back into compiler.example.com and run puppet agent -t to configure the Load Balancer.

Your Load Balancer is now prepared to balance traffic to two catalog compilers (catalog1.example.com and catalog2.example.com) as listed in the above configuration.

###Notes

I noted when putting together the loadbalancer.pp profile above that I had previously used some REALLY ODD ip addresses in the balancer config. Why? For the life of me I cannot recall. The original file looked like so:

# Load Balancer Profile
class profiles::loadbalancer {

  class { 'haproxy': }

  # Main Proxy Listener
  haproxy::listen { 'compiler.example.com':
    collect_exported => false,
    ipaddress        => $::ipaddress,
    ports            => '8140',
  }

  # First Load balanced Compile Master
  haproxy::balancermember { 'compiler1.example.com':
    listening_service => 'compiler.example.com',
    server_names      => 'compiler1.example.com',
    ipaddress         => '10.0.1.24',
    ports             => '8140',
    options           => 'check',
  }

  # Second Load Balanced Compile Master
  haproxy::balancermember { 'compiler2.example.com':
    listening_service => 'compiler.example.com',
    server_names      => 'compiler2.example.com',
    ipaddress         => '10.0.1.25',
    ports             => '8140',
    options           => 'check',
  }
}

In my original implementation I set the ipaddres fields with some odd IP addresses. For info around how to fill those but ,the documentation gives some hints:

ipaddresses: Optional. Specifies the IP address used to contact the balancermember service. Valid options: a string or an array. If you pass an array, it must contain the same number of elements as the array you pass to the server_names parameter. For each pair of entries in the ipaddresses and server_names arrays, Puppet creates server entries in haproxy.cfg targeting each port specified in the ports parameter. Default: the value of the $::ipaddress fact.

Since I was originally setting these up in Digital Ocean, I used the IP space 159.203.x.x which belongs to Digital Ocean. I am guessing these were the hard IPs on the instances I stood up. Since the documentation above states these are optional, you have two options here. Either leave those lines out of your config altogether, or manually set them to the IP Address of the instance you’re using. Try each and do which works for you.

Conclusion

Your HAProxy Load balancer is now complete and ready to take traffic to the additional catalog compiler nodes. In installment IV, we’ll begin to add in more components along the way to a fully developed LEI of Puppet Enterprise.

Installing Puppet Enterprise has been made remarkably easier as time has gone on. The efforts of Puppet Labs (I still can’t get used to simply ‘Puppet’) to make the installation as seamless and powerful as possible with the simplest of interfaces has been highly successful.

Many changes have occurred over time to include changing from answer files to a HOCON formatted pe.conf file containing the various configuration elements you may need to stand up an instance. I somewhat preferred the simple nature of the original answer files, but I can see the sense in moving to HOCON moving forward.

Obtain puppet

Needless to say, you’re going to need the Puppet Enterprise package to install from. Unlike Puppet Community, the entire installer is provided as a tarball rather than repo based installations via package management, and requires a little bit of UNIX-y knowhow to get it started, as the Puppet Enterprise Server is only installable on Linux.

When you navigate to the Puppet Download page, you may be required to sign up for a free account if you haven’t already. The opening download page is found here.

You will be presented with a launch page that contains a “Download” button. Click the button, and one of two things will happen. Either you will be directed to a “Thank You” page or a page to sign up for an account. As you can see, the “Thank You” page means you already have an account and are signed in whereas the signup page is self-explanatory. Sign up for an account, and retry the download link.

Once you’ve made it to the “Thank You” page, there are three tabs containing “Puppet Enterprise Masters”, “Puppet Enterprise Agents”, and “Puppet Enterprise Client Tools”. As of this writing, the only supported Puppet Master platforms are RedHat 6 & 7, Ubuntu 12.04, 14.04, and 16.04, as well as SLES 11 and 12.

If you had intentions of running the Puppet Master server on any other platform, here is where you realign your expectations. 🙂 I have heard that people have hacked the server to run on other platforms, but since we’re dealing with Puppet Enterprise, why would you break support and eliminate warranty? Pick one of the three and download the tarball for your appropriate platform.

NOTE: If You need legacy versions of PE, you can download those here.

Installation

For the purposes of this scenario, we will be installing the Puppet Infrastructure for fictional super-mega huge company “example.com”. I am going to trust you have worked out the DNS/Host file naming structure, and can resolve everything everywhere. If you cannot, don’t comment on the post, as I will make fun of you publicly… you deserve it.

My assumed setup will be:

Automated

The Puppet Enterprise Installer is a GUI web-browser based installer. Puppet has gone through the process of giving you a nice frontend to your installation, and making it dead-easy to perform a monolithic as well as split installation. For our purposes, though, we will be doing a “split” installation.

Stand up 3 Nodes with the specifications from the first article in the series as follows:

In my experience, I’ve found it much easier to exchange root keys between all three of the above nodes to allow the installer to do all it needs to do on each node. You can, however, decide to set the root password to something temporary to hand to the installer as well (and many people opt for this) and then return root’s password to your site default. In any event, all the machines should be able to resolve themselves and each other by name and root should be able to freely ssh between them either via shared keys (easiest) or password.

Transfer the package to the Puppet Master node:

scp -rp puppet-enterprise-2015.3.2-el-7-x86_64.tar.gz root@master.example.com:/root/

Once the package is on the destination machine, you should connect to the machine to work with the package on-box:

ssh root@master.example.com

which places you in the root user’s home directory where you copied the package.

Extract the Package

tar -zxvf puppet-enterprise-2015.3.2-el-7-x86_64.tar.gz
cd puppet-enterprise-2015.3.2-el-7-x86_64

Run the Installer

./puppet-enterprise-installer

You will receive a text prompt that states:

??Install packages and guided install [Y,n]

Simply press “Y” or the [Enter] key and the GUI portion of the installation will begin.

GUI Installer

Once you have started the Installation, the Puppet Enterprise Installer will perform some preparatory steps and then launch an installation interface on your master node on port 3000. To access this interface, you can bring it up in the web browser of your choice at:

https://master.example.com:3000

Navigate to this interface in your Internet browser. When you first arrive at the GUI installer, simply click the “Let’s Get Started” button. On the next page, The Puppet Enterprise Installer will present you with a GUI questionnaire to fill out regarding your environment. The following is that process in order by section.

Puppet Master Component

2. Enter the name of your Master in the Puppet Master FQDN text box. (e.g. master.example.com) 3. Enter all appropriate names for your master in the **Puppe## PuppetDB Component Application orchestration” check box.
3. Enter the hostname of your PuppetDB Node in the PuppetDB Hostname text box. (e.g. puppetdb.example.com)
4. Change no other selections under the remainder of the items for this section.

PE Console Component

1. Enter the hostname of your Puppet Console in the Console Hostname text box. (e.g. console.example.com)
2. Change no other selections under the remainder of the items for this section.

Database Support

No changes are needed to is section. Simply leave “Install PostgreSQL on the PuppetDB host for me” selected.

Console ‘admin’ User

Enter the password you would like to use for the Puppet Enterprise console once your installation is complete in the final text box.

Final Considerations

After completing the final section, click the “Submit” button, and the Puppet Enterprise Installer will present you with a confirmation page for you to review before commencing the installation based on the configuration elements you just provided to the installer.

If everything is to your satisfaction, click the “Continue” button and the Installation progress summary will continue to update you as to the progress of the installation. If you would like to see logging “as it happens”, you can click the “log view” button to see that in real time. If you would like to switcAfter what is roughly 10-15 minutes of installation and configuration, the installer will have completed all its work, and you will be presented with a button at the bottom of the progress screen you have been viewing that says: “Start Using Puppet Enterprise”. Click that button, and the installer will redirect you to the PE Console login screen. Enter the admin credentials you created earlier, and you are ready to begin working with the console as needed.

In my former life as a consultant, I had to install all manner of configurations of Puppet for clients. Some were small and some were large, but none were VERY large. One of the big things I was finding back then was there just wasn’t a lot of publicly available information regarding doing a full install and scaling it large.

So, I took some “research time” on my own and started to build out the configurations according to Puppet Labs’ (at the time… now just “Puppet”) documentation. The problem I was having was that the docs wouldn’t ever lead me to a successful install following a chronolgical set of steps. I had to click into subpages, jump over to sub-sub configurations, and then jump back to the main docs to follow yet another trail down until I reached the end…lather, rinse, repeat.

Some Caveats..

First, this is probably no longer a good “HOWTO” unless you’re installing an older Puppet Enterprise. It was created between 2015.2 and 2016.x, and likely has some amount of artifacting related to those versions.

Second, I’m going by docs I’ve recorded for my own use. I wrote these as mentioned above through prototyping, tearing it down, starting again, and literally doing the entire install over and over until it worked “as advertised”. A lot of this was really just ordering things the right way, and finding documentaiton for various pieces online at Puppet’s documentation site as well as blogs, conversations, and plain old trial and error. I certainly can’t warrant anything to anybody for any reason. As with most open source/creative commons assets, “it works for me, hope it works for you, and if it doesn’t, sorry about that.”

Finally, I hope to use this as the springboard to start brain-dumping all my old notes, conversations, ideas, and other prototyping I did in my home lab. There’s still a fair amount of documentation I cannot use or touch because they belong to my former employer or Puppet Labs, so some things may be less than clear and usually because I’m dancing around an NDA, noncompete, or just plain being a nice guy. If I inadvertently reveal something I shouldn’t, chances are it could disappear without a trace, but I’ll still make a note that I removed something, and try and replace whatever it is with published docs.

In short: I want to help the community, but I’m walking a tightrope here, so please be kind.

Format

I hope to start easy with a decision making process for installing Puppet, how to choose a method, think about scale, and will likely have quite an opinionated view at times. Once PE is installed, we’ll add compilers, scale postgres, etc. but for starters, I hope to just have the following:

PE Master (MoM)

Puppet DB

PE Console

HA Proxy Node for Compilers

Two Catalog Compilers

One ActiveMQ Hub

Two ActiveMQ Spokes

Two Agent Nodes for testing

I know that’s quite a number of nodes to get started with, but this after all a large environment infrastructure, and we want to scale big.

Required Nodes

To put together all the required components for a good large installation, I’ve settled on the below specs. You can change those as you see fit, but note that some of the disk space requirements and related were due to Puppet’s documented requirements at the time. YMMV, of course, but this is what I consider to be a base level installation if you intend on scaling into the multiple tens of thousands of nodes. Be sure that if you’re going to size this down that you’re still meeting Puppet’s needs in regards to memory, cores, and disk. (for a current listing of Puppet’s requirements, you can look here for more information.)

In addition, you’ll need to be aware of firewall requirements for such an installation. Puppet has documentation regarding firewall configurations and needed ports at their website here, but I’ll insert the image and recount the requirements here.

In short:

This is a close approximation to what you need to know. Detailed charts found in the above links, and a “point-by-point” port and use list is available to review.

In short, I’ve found it easiest to have all PE components on the same VLAN with no restrictions between them. If you are going to have a local firewall turned up on each node, you’ll need to manage all the above communications as you see fit, but for the serving infrastructure (if in a secure environment, of course) you can likely drop host firewalls in favor of corporate ones. In short, make it as easy on yourself as you see fit while balancing that toward your corporate security policy.

Finally, make sure DNS and NTP are all ready to go. I can’t tell you the number of times I’ve had major issues trying to get all this working, and NTP was off, or DNS didn’t propagate as expected (it’s always DNS, right?) or some other similar seemingly unrelated piece was not restarted or some such. Just make sure that all nodes resolve to their respective FQDN from all nodes. Obviously, the easiest way to do this is to simply put them all in DNS. You can manage the host files manually, but why would you want to do that?

If you’re at this point and all ready to go, look to the next entry to get started.