Introduction

The name of this portion of our tutorial was difficult to determine.  This is another set of configurations, but we will also be scaling Puppet to handle production quality traffic, be an external node classifier (ENC), have a backend database, employ an enterprise class web server, turn up a console…  there’s a lot to do.  We are indeed configuring the backend, but also scaling Puppet to handle your environment…hence the name.

This portion assumes you’ve followed all previous tutorials from I-IV, have your certs signed and are complete and ready to go with Puppet “as-is”, you simply have not installed any of the following add-ons.  As mentioned last time, you could begin to write manifests and modules right now, using Puppet “as-is”, never utilizing any of the other features.  However, the “out-of-the-box” configuration of Puppet is not ready for enterprise use.  Perfect for a small development environment…perhaps up to 25 hosts or so, the Puppet server as installed includes a small WEBRick server (ruby-based) and is not intended to handle large site traffic profiles.

To make Puppet “enterprise-ready”, we need to do a few things.

1. Install the Puppet Dashboard
2. Install the Puppet DB
3. Install passenger + Apache modules
4. Install MySQL

The dashboard gives you a GUI configuration mechanism as well as an external node classifier.  The PuppetDB is a centralized config storage mechanism for all node facts and configurations.  Passenger+Apache is the piece that replaces Puppet’s WEBRick server, and MySQL holds the database for the dashboard.

This will be the longest portion of the tutorial series, and all the separate & individual pieces will be interdependent, requiring us to do all the work first with configuration testing at the end.  Let’s get started.

Packages

Early in tutorial I, I had you install a number of packages.  Had you stopped after installment IV, you’d have had no need for a few of them, but I wanted all the packages to be on your system as prerequisites to avoid later installation headaches.  However, we do want to get the EPEL package set onto the system to add one prerequisite (and it’s a darned fine repo to have, should you need it for other things)

To install EPEL, run the following command:

_**sudo rpm -ivh http://mirrors.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm**_

The “Extra packages for Enterprise Linux” (EPEL) set is an important addition to any server set.  Especially to install the package prerequisites we need.

Installing Packages

Passenger: First, we will be installing Passenger.  The passenger packages are in their own repository, not hosted at Puppet Labs.  First, import the GPG key:

_**sudo rpm –import http://passenger.stealthymonkeys.com/RPM-GPG-KEY-stealthymonkeys.asc**_

And then install the passenger release repo:

_**sudo yum -y install http://passenger.stealthymonkeys.com/rhel/6/passenger-release.noarch.rpm**_

Finally, install the Passenger Apache module to tie everything together:

sudo yum -y install mod_passenger

Congratulations.  The groundwork for Passenger is now installed.

Dashboard Since we have done so much preparatory work, the dashboard install is quite simple:

sudo yum -y install puppet-dashboard

Simple.

PuppetDB PuppetDB is installed a little differently, using Puppet itself to get and install the package:

sudo puppet resource package puppetdb ensure=latest

This procedure takes a bit of time, but when complete, the PuppetDB is now installed.

MySQL MySQL is installed via the usual yum repos, but we will also turn it on and have it ready for use as well as create our users and remove unneeded and unsecured accounts for the system.

sudo yum -y install mysql-server sudo /sbin/chkconfig mysqld on sudo /sbin/service mysqld start

Let’s set the database root user’s password:

****/usr/bin/mysqladmin -u root password ‘’ /usr/bin/mysqladmin -u root -h password ‘’ mysql -u root -p

A few words here, for those of you unfamiliar with MySQL.  We are setting the root user’s password to be able to administrate the database.  The simplest way to set up this initial security is using the “mysqladmin” tool provided by MySQL.  Note that when I use <> in these above, this is where your site-specific information comes into play.  For \<FQDN>, for my example purposes I would replace this with puppet.example.com.  The password setting & changes, then, would look like so:

/usr/bin/mysqladmin -u root password ‘puppet’ /usr/bin/mysqladmin -u root -h puppet.example.com password ‘puppet’ /usr/bin/mysql -u root -p

I just wanted to clarify this for you in the event my use of <> and ‘ ’ above caused any confusion.

Configuring MySQL

Once you run the above commands, MySQL will prompt you for the password you just set.  Enter that password, and you will find yourself at a mysql prompt that looks like so:

mysql>

What this means is you have now logged into the MySQL database, and are ready to set it up for use.  Following I will list out all the commands you need to run in a set.  Note that these commands are each entered on a line and you press “” at the end of the line to enter the next command.  There is no output from MySQL when you enter these, so I’ll enumerate them all together here for your convenience.

mysql> create database dashboard character set utf8;
mysql> create user ‘dashboard’@‘localhost’ identified by ‘my_password’;
mysql> create user ‘dashboard’@‘’ identified by ‘my_password’;
mysql> grant all privileges on dashboard.* to ‘dashboard’@‘localhost’;
mysql> grant all privileges on dashboard.* to ‘dashboard’@‘’;
mysql> drop user ‘’@‘localhost’;
mysql> drop user ‘’@‘’;
mysql> drop database test;
mysql> flush privileges;
mysql> exit

As before, replace with the fully qualified hostname for your server and ‘my_password’ with the password you wish to set for the dashboard user.  A few notes:

1. First, we created the dashboard database
2. Next, we created the dashboard user for connecting from the localhost name
3. Next, we created the dashboard user for connecting from the server FQDN
4. The next two, we grant the dashboard user rights to the whole database from either location
5. The following two lines delete the user ‘’ from the server (a null user w/o a password)
6. Finally we drop the “test” database, flush all our privilege tables (to take effect immediately) and exit MySQL.

The final steps in getting MySQL configured for production use is to tweak the settings in the database by editing the /etc/my.cnf file and restarting the database.  Open the /etc/my.cnf file and add a new line at the end of the file:

max_allowed_packet = 32M

Save the file and then run

sudo /sbin/service mysqld restart

for the changes to take effect.

Passenger The final piece is to get the appropriate passenger gems and the Apache module installed to handle Puppet Agent requests to the server.  Luckily, our previous prerequisite installs have made this easy for us.  First:

sudo gem install rack passenger

When this is done, install the Apache module, following the prompts as follows:

****sudo passenger-install-apache2-module Press Press At the end of the installation, Press 

If we’ve done everything right up until this point, you should not need to supply any extra information, packages, or configuration, and only need to continue to press as listed above to complete the installation.

Now comes the time to configure the various pieces…

Configuration

Passenger First we need a number of directories and files to exist around the system, so let’s put those in place by using the following commands:

sudo mkdir -p /usr/share/puppet/rack/puppetmasterd
sudo mkdir /usr/share/puppet/rack/puppetmasterd/public
sudo mkdir /usr/share/puppet/rack/puppetmasterd/tmp
sudo cp /usr/share/puppet/ext/rack/config.ru /usr/share/puppet/rack/puppetmasterd
sudo chown puppet:puppet /usr/share/puppet/rack/puppetmasterd/config.ru
sudo chown puppet-dashboard:puppet-dashboard /usr/lib/ruby/gems/1.8/gems/passenger-4.0.37/buildout/agents/PassengerWatchdog

Next, we need to configure the Puppet Dashboard to connect to its database, and setup the tablespaces for use:

cd /usr/share/puppet-dashboard/config
edit database.yml
Remove the last stanza of this file that refers to the “test” database we removed above.
For the Production and Development database stanzas, change the “database:” line to read “dashboard” and the password line to contain your dashboard password so that it appears as follows:

database: dashboard
username: dashboard
password: 

Next, prepare the database for use as follows:

cd /usr/share/puppet-dashboard
__sudo rake gems:refresh_specs
rake RAILS_ENV=production db:migrate

(Even though we’ve reference the production and development databases in the database config above, we’ll only be working in the production database in this tutorial)  

At this point, we should be ready to test the Dashboard configuration to ensure we’re still on the right track.  Top do so, run the following:

cd /usr/share/puppet-dashboard
sudo ./script/server -e production

Now, attempt to connect to the dashboard via web browser by pulling up the server at the following address:  http://:3000.  If the dashboard displays correctly in your browser, we’re ready to continue.

Press CTRL-C to exit the server.
Configure Puppet for Dashboard

While we have already configured the dashboard itself, we have not told Puppet the dashboard exists.  To do so, edit the /etc/puppet/puppet.conf file and add the following.

In the [master] section of the puppet.conf, add the following lines:

Reporting
reports = store,http
_**reporturl = http://localhost:3000 /usr/share/puppet-dashboard/bin/external_node

Exit the puppet.conf file, saving your changes and set permissions for following files like so:

sudo chown -R puppet-dashboard:puppet-dashboard /usr/share/puppet-dashboard
sudo /sbin/chkconfig puppet-dashboard-workers on
sudo /sbin/service puppet-dashboard-workers start

Apache

Next, we need to configure the Apache web server to process requests being made by Puppet agents in your environment and hand them off to the Puppet server.  To do so, we need to create two files in the /etc/httpd/conf.d location.  The passenger installation will have already created a passenger.conf there.  Just remove it before creating the following two files.

/etc/httpd/conf.d/dashboard.conf

NOTE: The example file here is long lost.

And like it:

/etc/httpd/conf.d/passenger.conf

NOTE: The example file here is long lost.

 Starting Everything Up for Testing & Operation

Once these configuration files are in place, it’s time to test Apache’s handoff to Puppet and to make a special SELinux module to allow Passenger handoffs to the various needed places in the filesystem.

First, make sure the puppetmaster process has been stopped:

**/sbin/service puppetmaster stop
/sbin/chckonfig puppetmaster off_

This assumes you’ve run the procedures in the previous tutorials, including (especially) the certificate signing and exchange between master and agent.  If you’ve done this, Passenger now has all the certs it needs to handle requests on behalf of Puppet, and no longer needs the Puppet server running.

Next, test the configuration, that you’ve made no typos:

sudo /sbin/service httpd configtest

If no errors are displayed, then at least the syntax of your Apache configs are correct.  Now, to generate SELinux entries in the audit log to build a custom Passenger SELinux module, you need to start Apache:

sudo /sbin/service httpd start

Turn off SELinux temporarily:

sudo setenforce 0

Restart Apache to generate the log entries:

sudo /sbin/service httpd restart

Test the Puppet dashboard in a browser by going to:

_**

Puppet IV – Signing Certificates

February 27, 2014 | comments

By now you should have 1 Server installation completed and 1 Client installation completed.  The only thing you’re missing now is to link the two together.  The following client side configuration will be performed on ALL existing systems you have, to connect them to the server and to pull their configuration catalog down and apply the changes to the local host.

Certificates

By default, all communications in Puppet goes over SSL channels.  You can configure Puppet to use its own certificate signing mechanism or to use an external signing authority (outside the scope of this tutorial).  The signing process does two things.  First, it authorizes your client to communicate with the Puppet Master and receive configurations and keeps all that traffic secure from prying eyes.

Node Lists

Of course, the master needs a way to enumerate what hosts will be connecting, and how to apply what configuration where.  We’ll look at node classification both on and off the host as well.

Server Configuration

Here we go.

To get the server and client talking, there is a small amount of configuration that has to happen.  It essentially tells the server who will be calling, and tells the caller the server’s “phone number”.  Very short and simple.  So, let’s do that first.

Configs

Two files require editing on the server to prepare for your first Puppet run.  They are:

/etc/puppet/manifests/site.pp /etc/puppet/manifests/nodes.pp

As of this writing, the OSS installation does not place either file, so you will have to create them.  Make their content as follows:

site.pp

import ‘nodes.pp’ $puppetserver = ‘--FQDN--’

These two lines lay the foundation for the server’s self-declaration as authority on configuration.

The first line: “import nodes.pp” tells Puppet to look for the nodes.pp file you are about to create in the next step for node definitions (and thus their individual configurations) and the next line indicates this puppet servers’ complete name.  This line is very important, as configurations throughout the Puppet software product and the client node software continually refer back to this variable.  This setting takes the form of the fully-qualified-domain-name of the server.  It is recommended you use “puppet” as the name of the server in your infrastructure.

So if your site’s name is “bob.com”, this line would be:

$puppetserver = 'puppet.bob.com'

While any hostname should do, I might mention one caveat you should be aware of should you decide to use something other than ‘puppet.x.x’ formatting.  In the event your Puppet server cannot see or read the $puppetserver value, the internals of Puppet cause it to default to “puppet”.  To avoid any strange outage-based silliness, I would recommend sticking with the puppet name.  While this may cut cross-grain with your particular host naming system, there is no reason why the root name of the system cannot be puppet.domain.com and you CNAME your super-special host naming to that.

Why not the other way around?  I’ve seen in the wild situations where Puppet (especially as it concerns certificate signing) take the output of gethostbyname() for signing over a designated CNAME, even when you set this variable to the appropriate value.  Rather than deal with the mess, I recommend the former.

Close the site.pp file.

Next, you’ll want to configure your nodes.pp file.  Now, keep in mind that while we are specifying our nodes manually in this file for the purposes of this tutorial, we will be doing all manner of really cool things with node designations in the future.  So, for now humor me, and I assure you we’ll step into the wonderful world of the ENC, Dashboard and other goodies all too soon.  Right now, we’re just trying to get Puppet up, connected, and talking.

nodes.pp

  node '--FQDN--' { }

Here we are specifying our very first node, letting the master know it’s allowed to connect, and eventually putting a number of things in-between the curly braces {} specifying the configuration for (or pointers to the configuration for) our client node.   Just as above, the “FQDN” piece in between the single tics should be replaced with the fully qualified domain name of the client host you wish to manage.  As above, in our fictitious domain of “bob.com”, we will simply call this host “puppetclient.bob.com”, thus making the content of the nodes.pp file:

  node 'puppetclient.bob.com' { }

For now, there’s no need to  add configuration.  This is all we need to get started.  Exit from the nodes.pp file, saving it into the aforementioned spot of “/etc/puppet/manifests/nodes.pp”

Here ends the server-side configuration for this portion of our tutorial.  On to the client configuration.

Client Configuration

For the client configuration, a base-level file has already been provided for you, and only needs some customization.

The client configuration file is found at /etc/puppet/puppet.conf on the client (as well as the server).  The puppet.conf configuration file is an “ini-style” configuration file with [bracketed] headings and “key = value” line by line variables.

For our initial configuration we will be adding a couple lines.  First of all, at the end of  the [main] section, add the following line:

server = FQDN of the server

or, as above

server = puppet.bob.com

Next, in the  [agent] section, the following three lines (for our fictitious “bob.com” domain)  should be added at the end:

report = true server = puppet.bob.com certname = puppetclient.bob.com

The server line appears to me to be somewhat redundant, but I’ve always placed it in both locations, and cannot comment as to whether it is NOT needed.  The certname, FYI, is the FQDN of the client host you are configuring to connect to the server.

The purpose of the “certname” variable should be explained here a bit.  This is the name you placed in the “nodes.pp” file, and is also the hostname that is passed to the master in the key signing process.  You want all locations in which you’re specifying your node name to agree (puppetclient.bob.com in our example), so it’s highly important to ensure consistency throughout.

Ready to Connect

Ok, let’s recap…

We’ve set our site.pp and nodes.pp files in place on the server, specifying the server name and ensuring there is a FQDN node designation in the nodes.pp.  Next up, we’ll make our first connection to the server and test the connection.

In our last installment “Puppet I – Server Installation”, we covered all the basic necessities of installing Puppet Open Source server.  Of course, a server in and of itself is no good unless we have a client on another host to connect to it.  This article will similarly lead you step-by-step through the process of installing the client.

Puppet Client

Installation of our client is not so very much different than the initial server install.  Both come from the same Yum repository, and both need the same support libraries and files.

As expected, a client will need the OS installed (again, we’re working on the assumption of RedHat Enterprise Linux, but CentOS, SecurityLinux, etc. should do fine.

Once your operating system is completely installed, just as before we’ll need to follow the same steps of getting RedHat registered, the RHEL Yum repo enabled, the Puppet Labs repo configured, the correct support repos added, and finally, the client installed.

Enabling the RHEL Yum Repo

Just as we did before, we register our RedHat installation via “subscription manager” from RedHat.

sudo subscription-manager register sudo subscription-manager auto-attach sudo subscription-manager attach –auto sudo subscription-manager refresh

As I mentioned before, RedHat’s tool isn’t 100% effective 100% of the time, and some level of redundancy in the commands above ensures all the appropriate bits get twiddled.

Installing the Puppet Labs Software Repository

Again, as before, we simply use RPM to get the repository installed (the below command is all on one line):

**_sudo rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-10.noarch.rpm_**

Enabling Additional Packages

Once again, the “optional” RedHat packages and the “development” PupeptLabs repositories have to be enabled.  You can either use the Yum Config Manager or edit the repo files directly, your choice.  Yum Configuration Manager commands are as follows:

sudo yum -y install yum-utils sudo yum-config-manager –enable rhel-6-server-optional-rpms sudo yum-config-manager –enable puppetlabs-devel

You should note that the “enable” command above actually has two hyphens in front of it instead of one long one.  WordPress concatenates those.

Get Current!

Now is the time to get RHEL completely up to date through the usual means:

yum -y update

Install the Client

yum -y install puppet

If all goes well, this will complete the client config on your client host and you are now ready to configure the server, configure the client, and then connect them via SSL as provided by Puppet Labs.

In our next edition, we’ll be doing just that.

Ok, so you’re interested in this whole Puppet thing, and you want to get a full install in your test environment, or on a set of VMs to work and play with the product.  PuppetLabs has a ton of really cool documentation, and you should certainly avail yourself of it, but I wanted to give you a quick “cookbook” style set of instructions to get you rolling.

Puppet Master

There is a lot of documentation surrounding the Puppet Master installation at Puppet Labs, but I find a bit of obfuscation when trying to parse through the huge pile of meat that is there, when all we’re looking for is the best of it… you know, the bacon.  🙂

Puppet Code Repository & Package Source

Puppet Labs provides a myriad of repositories and methods by which to install Puppet.  I tend to stick with the main PuppetLabs repository whenever possible.  I also tend to like RedHat derivatives, so I’ll be talking about RHEL here, but the same applies to CentOS, Scientific Linux, and other RHEL “children” in the ecosystem.  In short, if you’re using something else other than RHEL, you may find yourself hunting packages and repos that contain packages you’ll need that the repositories I list here have by default.  As with any web-based documentation, YMMV.

Installing the Puppet Master

First, you’ll need a functioning Installation of RHEL upon which to install and configure Puppet.  I like to include only the “Base Install” and take the “Defaults” when doing a “special purpose” server, as it doesn’t muddy the waters and it gives you the cleanest slate possible upon which to build your Puppet platform.

Enabling the RHEL Yum Repo

As I’m sure you’re aware, to use RedHat’s repositories, you have to register and have active a subscription with RedHat.  To do a simple connection/attachment to the RedHat repositories, I’ve found a mixture of success with the Subscription Manager.  However, by performing the following commands (some are redundant), I’ve been able to get the configuration working with 100% reliability:

sudo subscription-manager register sudo subscription-manager auto-attach sudo subscription-manager attach –auto sudo subscription-manager refresh

Obviously, this is a RedHat topic, and not a Puppet one, so if you’re having issues with this process, contact Redhat for help.  The crux of this, however, is to just register your instance.

Installing the Puppet Labs Software Repository

Since I’m working with the current 6.10 release of Puppet, the package you need to install can be added by running:

_**sudo rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-10.noarch.rpm**_

This will install the current 6.10 release of the Puppet repo from which we can continue with the Puppet Master installation.

Enabling Additional Packages

Next, we need to have the “optional” set of packages from RedHat and the development packages from Puppet Labs.  There are two ways to do this.  First, you can edit each repo file in /etc/yum.repos.d (red hat.repo and puppet labs.repo respectively) and change the “enable” flag on each repository from “0” to “1”, or you can simply run the provided RedHat Yum configuration manager tool to do it for you as follows.

sudo yum -y install yum-utils sudo yum-config-manager –enable rhel-6-server-optional-rpms sudo yum-config-manager –enable puppetlabs-devel

Get Current!

Bring your current host completely up to date by querying the repositories and running the yum updater:

sudo yum -y update

As I’m sure you’re aware, depending on the updates released since the current major version of your repositories, the update time can vary from very short to quite some time.  Just be patient and get up to date before proceeding.

Install the Puppet Prerequisite Packages

This step name is a bit of a misnomer in that I am including a ton of packages that aren’t necessary for a “base” install of Puppet Server, but they do support later installations you may wish to do (like PuppetDB, Dashboard, Passenger, etc.).  I just wanted to give a complete list so there’s no after-installation required just to get moving with the next component.  Simply install the following via Yum:

sudo yum -y install gcc libcurl-devel gcc-c++ openssl openssl-devel ruby-devel httpd httpd-devel  mod_ssl policycoreutils-python apr-devel apr-utils-devel

Prep Your Firewall

You may or may not be using the IPTables Firewall, but you’ll want the following rule in place to support the Puppet Master port.  Simply insert the following line into your /etc/sysconfig/iptables file.

-A INPUT -m state –state NEW -m tcp -p tcp –dport 8140 -j ACCEPT

Note that if you intend to install the Puppet Dashboard as well, you can add that port here too.

-A INPUT -m state –state NEW -m tcp -p tcp –dport 3000 -j ACCEPT

Make sure you place your firewall rules in the appropriate place in the file.  While outside the scope of this article, IPTables has an order and a format you should follow.  You can find out more here.

Lastly, restart your firewall to pick up the new rules:

sudo service iptables reload

Install Puppet Server

After all the above work, if you’ve had no repository or package errors, your system should be ready to install Puppet Server.  Simply run the following:

yum -y install puppet-server

Conclusion

And that should be it.  All packages installed, base RedHat updated with all the latest Ruby & RedHat goodness, and the bad Puppet Server installed and ready for action.

Next time:  Installing the Puppet Client.