I’m speaking tonight at the Atlanta Perl Mongers meeting at the Central Cox building in Dunwoody at 7PM on Puppet Configuration management. Please do feel free to drop by and say hello. If you cannot make it, however, here’s my slide deck all compressed down and PDF-ified for your viewing pleasure.
I hope to see you there!
I guess you Atlanta folk will get that one…
Howdy all, and a hearty hello. I’m back from my exile to the land of single shiny glowing buttons and happiness, rainbows, and OSX, and have returned to Atlanta with SysAdmin bells on. I’ve done a lot over the last couple years (much I cannot talk about), but have developed quite an affinity for the Puppet configuration Management system. So much so, in fact, that I’ve found a need to start a whole new category right here on the ol’ blog to accommodate.
Not in a long time have I liked something “new” that has come along to SysAdmin work… The moniker “Devops” still gives me hives, but I will endeavor to navigate the puppet-y waters without uttering the vile mantra but wading into the wonderful waters of automation and managed infrastructure.
In short, “Puppet” in and of itself is quite a feat of engineering. A series of platform tools to enforce a single “blessed” configuration upon your infrastructure in a consistent, premeditated way from a central location, eliminating the need for all manner of scripting and “loop lists” to do individual functions on each system, one at a time. No no!
Puppet enforces policies you write in a uniform, secure way, rather quickly and scales rather well for most environments. For the larger environments that cannot be scaled to, Puppet will utilize other “workhorse” industry standard products as modular replacements of various pieces of the Puppet system, to broaden and expand its abilities.
Puppet comes in two flavors, the commercial “Enterprise” system where everything is packaged together and all gets installed and configured at load time in an “automagical” sort of way so you don’t have any need to post-configure, but can just begin writing configurations immediately.
The “Open Source” Puppet has most of the same pieces, although named somewhat differently, and you can build a very solid, full-featured system of your own without the price tag. All you need is time. :)
In the coming sections, I will take apart Puppet for those of you wishing to know more, and give a rudimentary survey of all the different things it can do. Hope you come along!
I think it is important to make a note about all of the people and places that have helped me ferret out a lot of this configuration. I don’t find this stuff in a vacuum (like anything in the Open Source world), and find it quite important to give a nod where a nod is due.
First, to Turner Broadcasting who gave me my first shot at doing this in a relatively small environment. There was a ton of time spent on the phone, in online forums, reading documentation, and the like. Everything sort of proceeded from there. I tried to get such a thing into the backend over at The Weather Channel, but the climate wasn’t right, and there was a great deal of change going on. Trying to implement such a thing there just wouldn’t have worked out very well.
Next, to Incomm. They needed a solution, and Brock (my boss) gave me ample time to get a lot of this together, and to do it right without interruption. Big thanks there.
Finally, and most importantly, my best bud John Beamon. He encountered and worked with LDAP long before I did. And, while we have considerably different directories in place between our two mutual organizations, his help in putting together how all this works in my head has been invaluable. I daresay I’ve been into his IM client twice (or more) often as he’d like, gave him props nowhere near as much as I should (like at all), and even with all his own work on his plate, he has helped me out immeasurably. Add to that, he’s a long-time friend and fellow-minister, and that’s just good times right there. They don’t come along like John very often, and I’m grateful to have him as both a friend and fellow nerd.
I’m sure there’s still some inaccuracies here, and I’ll be sifting through all this as I continue to build and extend OpenLDAP in this environment. Things will automagically correct before your eyes, most likely. If you find anything that happens to scream “No!” to you, feel free to drop me a line, and I’ll be happy to make that change.
So, let’s see where we are. We have a master server you will be doing all administration work on. This master server replicates to two hosts in the environment that serve LDAP queries to your clients. These servers are replicants and are load-balanced under a VIP that is pointed to by the name you choose. (in our case, ldap.bob.com). You can change passwords at the client level, and have it pushed back up to master and replicated out to the environment immediately.
Finally, we need to talk about security. There’s a number of ways to do security, but RedHat has done a lot of the footwork for you. Unfortunately, it’s very poorly documented, and they really Really REALLY want you to use RedHat Directory Server for everything, so I don’t guess it’s a priority.
Essentially, we want to secure all queries floating around the network with TLS. In a RedHat world, you simply need to make a couple changes at the server, restart LDAP, and then connect from TLS-enabled clients and all works just as it did before, except now it runs over an encrypted channel.
RedHat has tried to ease the pain of generating certificates by placing all you need in a Makefile on-box. navigate to /etc/pki/tls/certs and see that there is a makefile there. Next, run:
to generate the needed files. If it has already been done for you by the system, you will get the answer:
make: `slapd.pem’ is up to date.
If you get this message, you’re halfway there.
Next, edit the /etc/openldap/slapd.conf file. You will need to refer to the appropriate files to allow for secure operation. Insert the following lines into that file:
# TLS Security
Next, edit the file /etc/sysconfig/ldap. Make the following lines:
Then, restart LDAP: /sbin/service ldap restart. This does two things. First, it tells the client where to look for the certificates, and then tells the system to only serve from the secure port 636. (recall that we are on the replicants which are, in turn, servers themselves. We have handled connecting to the master as well as setting the replicant up to receive queries)
Finally, we connect a client.
Connecting the Client
To allow a client to connect, you need the appropriate key on the client (public server key) to be able to exchange identities with the server, and establish the secure session. To do this, you have to distribute this key you just made out to each client you wish to connect back to the server.
The key you will be distributing lives in /etc/pki/tls/certs and is named ca-bundle.crt. Simply move this cert to your client (I use scp for such an operation) and place it into your openldap cacerts directory like so:
scp -rp ca-bundle.crt host.bob.com:/etc/openldap/cacerts
If you don’t have rights to copy straight into the destination, send it to your home directory, then just move the cert there using “sudo”.
Finally, you need to tell the system about the cert. This is done in /etc/openldap/ldap.conf via three lines that tell the system how to connect, and where the cert lives:
In the left column, select “Use LDAP” and in the right column “Use LDAP Authentication”. Tab down to the “Next” button and press “Enter”.
As misleading as “Use TLS” may be, do not select it. Instead, go down to your server line, and modify it like so:
Your base DN should already be filled out (in our case: dc=bob,dc=com). Navigate to the “OK” button, and press “Enter”.
This should conclude your client configuration. Now, you should be able to run a query against LDAP, and the whole path be secure:
uid=123(bob) gid=123(users) groups=123(users),456(bob)
I’m sure I’ve missed or glossed over something highly important. I am in the process of discovery on this particular topic, and this article is serving as my documentation store until I can get the whole thing cleaned up & finalized to push back into my work environment as official documentation. I’ll correct here as I find mistakes and omissions.
Continuing our discussion of LDAP Administration, there’s the matter of Replication.
So far we’ve created an LDAP store, turned up the server, configured a client, and even connected Apache authentication to it. However, if we’re going to use our LDAP server for enterprise authentication, then there’s the small matter of “What happens when my authentication server wets the bed?”.
As with anything in the enterprise, you have backup systems. Sometimes they’re failover systems, sometimes clusters. Sometimes they’re tandem systems, and sometimes they’re load-balanced. No matter the configuration, you have redundancy, resiliency, and scalability. I plan to talk about one of the many scenarios available to LDAP administrators today; the idea of a master server and many replicants.
In my configuration, I have a single administrative parent. This system is where we do all administrative level work. This includes adding users, adding groups, reporting, and the like. It is also the “provider” store to all replicants in our environment. We learned earlier how to turn up a server that is queried directly. Now let’s learn, instead, how to configure this system to replicate itself.
Assume 3 systems total, ldap01.bob.com, ldap02.bob.com, and ldap03.bob.com. ldap01.bob.com is our master server and our replicants are ldap02 & ldap03. To tell the system it will be replicating, you will need to configure it to do so. Shut down LDAP on the primary like so:
/sbin/service ldap stop
This shuts down all daemons and associated processes. Next, we need to edit our /etc/openldap/slapd.conf to include information regarding where our replicants will be. You must add a few lines to the master to make this happen. Like so:
This can be added at the end of the file.
Next, we take our fresh two servers, and turn up a similar system to what ldap01 was before adding the above lines. In these systems, there are only two important lines to tell them they are replicants and not masters. They are as follows:
That is literally the entire configuration.
Populating the Replicants
To have your schema transferred over, and to be working from the same general starting point, I find it important to copy your whole database over to start with. This is easily done utilizing standard LDAP tools.
First, start back up your master server:
/sbin/service ldap start
Once you’ve done this, the database is up and ready for queries. We will essentially dump our database for import on each of the replicants. To do this, we will use the slapcat utility, redirecting the output to a file we can use to move around to the replicants. Run slapcat as follows:
slapcat >> master.ldif
this will output the contents of your LDAP store to a single LDIF-formatted file, suitable for import into other servers. Simply copy this file to a generic location (such as your personal home directory) on each of the other servers, and we are set for import.
Once your file is in the new location, you’re ready to import. First, start LDAP as outlined above. Next, add the LDIF to your store:
slapadd -l master.ldif
Probably unnecessary, but I usually restart my ldap server after the import, and now I’m ready to go. Repeat the process on your third LDAP store, and your full environment is running.
So let’s see where we are.
Master server up and serving.. check.
Two slaves configured as replicants, up and running.. check.
Now that you have your stores up, you have to do some testing. Primarily, that the master replicates to the slaves. The way I usually do this is use the Apache Directory Studio I covered in an earlier article. I simply add a user on the master. Then, I connect to each of the slaves in turn to see that the user has appeared there. If so, then we’re ready for the next steps: High Availability.
You have two query hosts that can equally provide query answers from remote clients. There are several ways you can make these available. Round-robin DNS, HA IP failover, and load-balancing via a hardware load balancer. I prefer the latter. However, to do so, you need a way to tell the load balancer that your LDAP store is up and responding.
I prefer to use a small script on the system that can be served up via HTTP to the load balancer that does a simple operation. First, it does an LDAP search, looks for information, and then prints out to the web page it creates a simple “UP” or “DOWN” message for the load balancer to key on. The script looks like the following:
As you can see, all we do is simply do an ldapsearch against our bob.com domain, look for the home directory for the admin user to look like “/home/admin”. If the answer returns, we say “UP”, if not, we say “DOWN”.
Place this script into your “cgi-bin” directory, make it executable (chmod 0755 <filename>) and simply call it in your browser via the URL: http://yoursite.com/cgi-bin/<filename>. If you have Apache properly configured (outside the scope of this document) to serve CGI Executables, you should get the status of the individual system. Do this for both your replicants.
Finally, ask your network team to configure these two systems in a load-balanced configuration behind a VIP (virtual IP). Have a sensible DNS name pointed at this IP (ldap.bob.com, for instance) and you’re in business. Now, when you configure your clients to authenticate against LDAP (Article #1 in this series), you just point them at the ldap.bob.com name. If either of the systems go out, the load balancer will point you to the machine that is up to serve your requests.
I hope this gives you a basic direction to go in getting high-availability setup for your system through a combination of replication and load balancing. There are other methods for HA in the replicants. Perhaps we will cover that soon.
Next up: Securing your LDAP installation.
In my Internet and sysadmin travels, I find it necessary from time to time to seek out documentation on a particular subject in the Linux world. Sometimes I need something specific to Linux, sometimes a package on Linux, but always technical and many times detailed.
For the longest time (since my Linux infancy in 1995), I have used the Linux documentation project to find important HOWTOs. As a community, we have prided ourselves on a ubiquity of documentation and support, but I found something both interesting and alarming at the same time on my last visit to TLDP.
In 2010, there have only been 12 documents of TLDP modified and in 2009 a mere 5. That means that in nearly two full calendar years, there have only been 17 documents modified in the project. As many of you already know, 24 months is an eternity in Internet time (much less regarding the growth and progress of a major OS like Linux)
While I am confident that many of these docs are solid and still stand on their own, it does concern me that documentation hasn’t changed much and other projects’ documentation stores are starting to see similar atrophy.
Certainly, with each release we find a new set of release notes and subsequent additions to documentation. However, as we go on I find more often than not that a project will have a fundamental change that should really be covered in TLDP or at least documented at the documentation level (rather than the release note level) and simply never has it happen. In these cases, you find a “collective knowledge” of people on the support mailing list that “just know” something to be true, but those new to the project or the list may never know it because it isn’t written down somewhere.
Remember the age-old wisdom: ”If it isn’t written down, it never happened.”
For those of you on a project, please consider your documentation. It may be a time for a rewrite. It happens precious little (if the TLDP is to be believed) and really needs your attention. There are countless volunteers out there who may not be coders, but use your product and would be ecstatic to be counted as one of the team simply to do documentation for you.
System Administration is a funny thing.
Many people think that it’s just adding users, performing requests, and working with project groups to get their “next big thing” out the door.
Certainly, much of your work involves those very things, but what younger admins never quite realize until they’ve built a beast that cannot be fed is that everything is about frameworks that tie everything together. From a system load framework to an authentication framework to an asset management framework, if built correctly a framework can save each and every admin countless hours of administration time. How?
Take distribution. You can push files around considerably more easily when you’ve built a framework to do so rather than every admin having an individual way to do it. In fact, when properly implemented, distribution of a file to “gobs” of similar hosts (that’s a technical term there ) can be as simple as:
distrib <object> <class of host> <drop-point>
Now, many will say that rsync or scp will do much of this for you, and that is correct. However, in the context of your individual site, having symbolic abstractions such as “class of host” goes a long way. Perhaps a certain file needs to be distributed to only your web servers. Or maybe, only web servers running RedHat 5.3. If you correctly build a framework for file shove and management, suddenly heavy lifting becomes a light chore. After all, the more pulleys in the works, the lighter the load becomes.
As has been covered on this site in the past, LDAP is a wonderful authentication framework that can be tied to positively everything in your environment as well. From Apache authentication against the store to UNIX authentication, to various types of applications understanding LDAP as a target for authentication sources, much pain of user administration can be solved by having a centralized authentication mechanism.
Frameworks as Philosophy
Rather than continue with examples on a case-by-case basis, consider the entire concept of frameworks and unifying ties across systems and networks. Many places I’ve worked, organic growth brought about massive numbers of machines that were “siloed” one from another either by project boundaries, function boundaries, or other superimposed logical delineations we as users imposed on them.
Instead of “the DEV servers” or “The PROD servers”, we have logically separated them into “the ECommerce DEV servers” versus “the Web DEV servers” and so forth. Rather than having a framework of systems and their functions in the workplace providing services and features at a certain level to the customer and/or end-user, we cobble together pieces to serve a single purpose rather than modules to expand the greater infrastructure.
Logical Differentiation and Service-Orientation
The clear winner over the “as needed” or “for a purpose” way of doing things is the “tiered” or “services-oriented” model of work. Rather than many groups of things that form a farm of servers, you have the farm of servers that service many different things. This, I know, sounds something like technological double-speak, but let me explain what I mean.
In a normal environment, we would have a somewhat typical scenario with front-end servers, back-end infrastructure, management, and organization. The problem we have with this is that if “New Whiz-Bang project #7″ comes along, new hardware will need to be procured for each piece of the puzzle. New app servers, new web servers, new database components, maybe management and authentication considerations… Each project, every time, capital outlay, budgetary justifications, etc.
If, instead, you think in terms of frameworks, your job becomes one of determining total resources needed across an environment rather than resources needed for an individual project, and thus individual project growth concerns, individual project funding concerns, personnel, etc. ”Siloed” growth and expansion may not bite you today or tomorrow, but as I said early on, will become a beast that cannot be fed.
Consider instead, the following example:
Instead of “theses web servers versus those web servers”, you instead have “the web layer”. Any and all requests to your company come through “the web layer”. It is scaled as an entity and not as individual projects. When scaling happens, all environments benefit.
Instead of “these application servers” versus “those application servers”, you have “The App Layer”. A single applications framework that serves all application server requests back out the front-end web layer by leveraging container and web server features to do the “effing magic ™” on the backend to provide a unified front-facing experience to the user.
Extrapolate these ideas… Instead of the app layer, let’s say the app cluster. Now the power behind this idea becomes clear. Unlimited scalability with unlimited potential. How about “the database cluster”? Regardless of the solution you use, if there is a single database resource (cluster, replication ladder, whatever) that serves back queries you throw at it, how much better is that than “databases for this” and “databases for that”?
Take it a step further.. make an XML services layer that serves out “your data” in a clearly defined API sort of way, and all you do is make XML requests to a services infrastructure rather than directly at your databases. Or, your cluster is comprised of “write databases” versus “read databases” and you’re segmenting the type of traffic you’re serving to reading versus writing, making the read operations light-years faster.
Authentication layers, web layers, XML layers, app layers, database layers… All frameworks that grow as an organism rather than series of unrelated growths.
When going through your next design adjustment or your next expansion or data center rollout, consider thinking differently about growth and planning. I believe that if you think in terms of large organisms with several related parts rather than several growths unrelated except in their location, you’ll build into your infrastructure a power and a scalability that will serve you well for many years (and growths!) to come.
You mean the people we geeks won’t give access to because, if left to themselves, developers will patently destroy anything they come in contact with in the systems world? You mean the people who think root is an account that should be used as a tool to cure ALL their ills and knock down all the “obstacles” they encounter? The people who won’t use “sudo” because it’s too many characters to type and “breaks their flow” when coding?
Oh, I get it, the people who haven’t the slightest clue what it really means to be geek. To give honor and deference to the system. It’s security, design, integrity. They don’t care that there are other people on the box, they just want to meet their date. And they’ll twist every systems admin in every possible contortion to break all the best practices in the world just to meet their date. ”Being Geek” Phah!
These guys like to be called geeks because it is an easy to earn, undeserved moniker for them bestowed by people who have no clue what it means. All the while, they’re breaking every rule and every guideline just to meet a date. Further, when Systems people point out security concerns or elements of systems design these supposed geeks are transgressing, they run to upper management and complain that the systems teams are “blocking their date”, or “They’re blockers”, or “we can’t get anything done.” Geeks.
A real geek would NEVER do that.
A real geek would write beautiful code that followed all the best practices rules for the honor of having written it. A real geek would NEVER even begin to consider using the root account unless it was absolutely necessary. A real geek would take the recommendations of a systems team (the real geeks, by the way) who spend all their time making sure the platform upon which these geek posers perform their witchcraft is ALWAYS up, ALWAYS stable, ALWAYS up to date, and ALWAYS secure.
I’ve been in this business for about 20 years now, and in that time I have met two developers who were tried and true, died-in-the-wool geeks. TWO.
Gimme a break… “Geeks”.
I’ve been asked by the Digg auto-submission system to paste an invisible key into my page so Digg can curate my posts.
Into my next story on my site. Of course, if ANYthing on here catches on, my server is toast, but let’s see what happens, shall we? :)