Tech musings and other things...

LDAP Administration v - Replication

| Comments

Continuing our discussion of LDAP Administration, there’s the matter of Replication.

So far we’ve created an LDAP store, turned up the server, configured a client, and even connected Apache authentication to it.  However, if we’re going to use our LDAP server for enterprise authentication, then there’s the small matter of “What happens when my authentication server wets the bed?”.

As with anything in the enterprise, you have backup systems.  Sometimes they’re failover systems, sometimes clusters.  Sometimes they’re tandem systems, and sometimes they’re load-balanced.  No matter the configuration, you have redundancy, resiliency, and scalability.  I plan to talk about one of the many scenarios available to LDAP administrators today; the idea of a master server and many replicants.


In my configuration, I have a single administrative parent.  This system is where we do all administrative level work.  This includes adding users, adding groups, reporting, and the like.  It is also the “provider” store to all replicants in our environment.  We learned earlier how to turn up a server that is queried directly.  Now let’s learn, instead, how to configure this system to replicate itself.

Assume 3 systems total,,, and is our master server and our replicants are ldap02 & ldap03.  To tell the system it will be replicating, you will need to configure it to do so.  Shut down LDAP on the primary like so:

/sbin/service ldap stop

This shuts down all daemons and associated processes.  Next, we need to edit our /etc/openldap/slapd.conf to include information regarding where our replicants will be.    You must add a few lines to the master to make this happen.  Like so:

replogfile      /var/log/ldap/slapd.replog

replica uri=ldap:// binddn=“cn=admin,dc=bob,dc=com” bindmethod=simple credentials=secret

replica uri=ldap:// binddn=“cn=admin,dc=bob,dc=com” bindmethod=simple credentials=secret

This can be added at the end of the file.

Next, we take our fresh two servers, and turn up a similar system to what ldap01 was before adding the above lines.  In these systems, there are only two important lines to tell them they are replicants and not masters.  They are as follows:

updatedn “cn=admin,dc=bob,dc=com” updateref ldap://

That is literally the entire configuration.

Populating the Replicants

To have your schema transferred over, and to be working from the same general starting point, I find it important to copy your whole database over to start with.  This is easily done utilizing standard LDAP tools.

First, start back up your master server:

/sbin/service ldap start

Once you’ve done this, the database is up and ready for queries.  We will essentially dump our database for import on each of the replicants.  To do this, we will use the slapcat utility, redirecting the output to a file we can use to move around to the replicants.  Run slapcat as follows:

slapcat >> master.ldif

this will output the contents of your LDAP store to a single LDIF-formatted file, suitable for import into other servers.  Simply copy this file to a generic location (such as your personal home directory) on each of the other servers, and we are set for import.

Once your file is in the new location, you’re ready to import.  First, start LDAP as outlined above.  Next, add the LDIF to your store:

slapadd -l master.ldif

Probably unnecessary, but I usually restart my ldap server after the import, and now I’m ready to go.  Repeat the process on your third LDAP store, and your full environment is running.

Next Steps

So let’s see where we are.

Master server up and serving.. check. Two slaves configured as replicants, up and running.. check.

Now that you have your stores up, you have to do some testing.  Primarily, that the master replicates to the slaves.  The way I usually do this is use the Apache Directory Studio I covered in an earlier article.  I simply add a user on the master.  Then, I connect to each of the slaves in turn to see that the user has appeared there.  If so, then we’re ready for the next steps:  High Availability.

You have two query hosts that can equally provide query answers from remote clients.  There are several ways you can make these available.  Round-robin DNS, HA IP failover, and load-balancing via a hardware load balancer.  I prefer the latter.  However, to do so, you need a way to tell the load balancer that your LDAP store is up and responding.

I prefer to use a small script on the system that can be served up via HTTP to the load balancer that does a simple operation.  First, it does an LDAP search, looks for information, and then prints out to the web page it creates a simple “UP” or “DOWN” message for the load balancer to key on.  The script looks like the following:

[caption id=“attachment_265” align=“alignnone” width=“150”] Healthcheck Sample[/caption]

As you can see, all we do is simply do an ldapsearch against our domain, look for the home directory for the admin user to look like “/home/admin”.  If the answer returns, we say “UP”, if not, we say “DOWN”.

Place this script into your “cgi-bin” directory, make it executable (chmod 0755 ) and simply call it in your browser via the URL:  If you have Apache properly configured (outside the scope of this document) to serve CGI Executables, you should get the status of the individual system.  Do this for both your replicants.

Finally, ask your network team to configure these two systems in a load-balanced configuration behind a VIP (virtual IP).  Have a sensible DNS name pointed at this IP (, for instance) and you’re in business.  Now, when you configure your clients to authenticate against LDAP (Article #1 in this series), you just point them at the name.  If either of the systems go out, the load balancer will point you to the machine that is up to serve your requests.


I hope this gives you a basic direction to go in getting high-availability setup for your system through a combination of replication and load balancing.  There are other methods for HA in the replicants.  Perhaps we will cover that soon.

Next up:  Securing your LDAP installation.

Linux Docs Docked?

| Comments

In my Internet and sysadmin travels, I find it necessary from time to time to seek out documentation on a particular subject in the Linux world.  Sometimes I need something specific to Linux, sometimes a package on Linux, but always technical and many times detailed.

For the longest time (since my Linux infancy in 1995), I have used the Linux documentation project to find important HOWTOs.  As a community, we have prided ourselves on a ubiquity of documentation and support, but I found something both interesting and alarming at the same time on my last visit to TLDP.

In 2010, there have only been 12 documents of TLDP modified and in 2009 a mere 5.  That means that in nearly two full calendar years, there have only been 17 documents modified in the project.  As many of you already know, 24 months is an eternity in Internet time (much less regarding the growth and progress of a major OS like Linux)

While I am confident that many of these docs are solid and still stand on their own, it does concern me that documentation hasn’t changed much and other projects' documentation stores are starting to see similar atrophy.

Certainly, with each release we find a new set of release notes and subsequent additions to documentation.  However, as we go on I find more often than not that a project will have a fundamental change that should really be covered in TLDP or at least documented at the documentation level (rather than the release note level) and simply never has it happen. In these cases, you find a “collective knowledge” of people on the support mailing list that “just know” something to be true, but those new to the project or the list may never know it because it isn’t written down somewhere.

Remember the age-old wisdom:  "If it isn’t written down, it never happened."

For those of you on a project, please consider your documentation.  It may be a time for a rewrite.  It happens precious little (if the TLDP is to be believed) and really needs your attention.  There are countless volunteers out there who may not be coders, but use your product and would be ecstatic to be counted as one of the team simply to do documentation for you.

It's All About the Framework

| Comments

System Administration is a funny thing.

Many people think that it’s just adding users, performing requests, and working with project groups to get their “next big thing” out the door.

Certainly, much of your work involves those very things, but what younger admins never quite realize until they’ve built a beast that cannot be fed is that everything is about frameworks that tie everything together.  From a system load framework to an authentication framework to an asset management framework, if built correctly a framework can save each and every admin countless hours of administration time.  How?


Take distribution.  You can push files around considerably more easily when you’ve built a framework to do so rather than every admin having an individual way to do it.  In fact, when properly implemented, distribution of a file to “gobs” of similar hosts (that’s a technical term there :)) can be as simple as:


Now, many will say that rsync or scp will do much of this for you, and that is correct.  However, in the context of your individual site, having symbolic abstractions such as “class of host” goes a long way.  Perhaps a certain file needs to be distributed to only your web servers.  Or maybe, only web servers running RedHat 5.3.  If you correctly build a framework for file shove and management, suddenly heavy lifting becomes a light chore.  After all, the more pulleys in the works, the lighter the load becomes.


As has been covered on this site in the past, LDAP is a wonderful authentication framework that can be tied to positively everything in your environment as well.  From Apache authentication against the store to UNIX authentication, to various types of applications understanding LDAP as a target for authentication sources, much pain of user administration can be solved by having a centralized authentication mechanism.

Frameworks as Philosophy

Rather than continue with examples on a case-by-case basis, consider the entire concept of frameworks and unifying ties across systems and networks.  Many places I’ve worked, organic growth brought about massive numbers of machines that were “siloed” one from another either by project boundaries, function boundaries, or other superimposed logical delineations we as users imposed on them.

Instead of “the DEV servers” or “The PROD servers”, we have logically separated them into “the ECommerce DEV servers” versus “the Web DEV servers” and so forth.  Rather than having a framework of systems and their functions in the workplace providing services and features at a certain level to the customer and/or end-user, we cobble together pieces to serve a single purpose rather than modules to expand the greater infrastructure.

Logical Differentiation and Service-Orientation

The clear winner over the “as needed” or “for a purpose” way of doing things is the “tiered” or “services-oriented” model of work.  Rather than many groups of things that form a farm of servers, you have the farm of servers that service many different things.  This, I know, sounds something like technological double-speak, but let me explain what I mean.

In a normal environment, we would have a somewhat typical scenario with front-end servers, back-end infrastructure, management, and organization.  The problem we have with this is that if “New Whiz-Bang project #7” comes along, new hardware will need to be procured for each piece of the puzzle.  New app servers, new web servers, new database components, maybe management and authentication considerations… Each project, every time, capital outlay, budgetary justifications, etc.

If, instead, you think in terms of frameworks, your job becomes one of determining total resources needed across an environment rather than resources needed for an individual project, and thus individual project growth concerns, individual project funding concerns, personnel, etc.  "Siloed" growth and expansion may not bite you today or tomorrow, but as I said early on, will become a beast that cannot be fed.

Consider instead, the following example:

Instead of “theses web servers versus those web servers”, you instead have “the web layer”.  Any and all requests to your company come through “the web layer”.  It is scaled as an entity and not as individual projects.  When scaling happens, all environments benefit.

Instead of “these application servers” versus “those application servers”, you have “The App Layer”.   A single applications framework that serves all application server requests back out the front-end web layer by leveraging container and web server features to do the “effing magic ™” on the backend to provide a unified front-facing experience to the user.

Extrapolate these ideas… Instead of the app layer, let’s say the app cluster.  Now the power behind this idea becomes clear.  Unlimited scalability with unlimited potential.  How about “the database cluster”?  Regardless of the solution you use, if there is a single database resource (cluster, replication ladder, whatever) that serves back queries you throw at it, how much better is that than “databases for this” and “databases for that”?

Take it a step further.. make an XML services layer that serves out “your data” in a clearly defined API sort of way, and all you do is make XML requests to a services infrastructure rather than directly at your databases.  Or, your cluster is comprised of “write databases” versus “read databases” and you’re segmenting the type of traffic you’re serving to reading versus writing, making the read operations light-years faster.

Authentication layers, web layers, XML layers, app layers, database layers… All frameworks that grow as an organism rather than series of unrelated growths.

When going through your next design adjustment or your next expansion or data center rollout, consider thinking differently about growth and planning.  I believe that if you think in terms of large organisms with several related parts rather than several growths unrelated except in their location, you’ll build into your infrastructure a power and a scalability that will serve you well for many years (and growths!) to come.


| Comments

I saw a p1mp of a new book on Gruber called “Being Geek”.  It bills itself as “The Software Developer’s Career Handbook”.  That sort of set me off, honestly.

You mean the people we geeks won’t give access to because, if left to themselves, developers will patently destroy anything they come in contact with in the systems world?  You mean the people who think root is an account that should be used as a tool to cure ALL their ills and knock down all the “obstacles” they encounter?  The people who won’t use “sudo” because it’s too many characters to type and “breaks their flow” when coding?

Oh, I get it, the people who haven’t the slightest clue what it really means to be geek.  To give honor and deference to the system.  It’s security, design, integrity.  They don’t care that there are other people on the box, they just want to meet their date.  And they’ll twist every systems admin in every possible contortion to break all the best practices in the world just to meet their date.  "Being Geek"  Phah!

These guys like to be called geeks because it is an easy to earn, undeserved moniker for them bestowed by people who have no clue what it means.  All the while, they’re breaking every rule and every guideline just to meet a date.  Further, when Systems people point out security concerns or elements of systems design these supposed geeks are transgressing, they run to upper management and complain that the systems teams are “blocking their date”, or “They’re blockers”, or “we can’t get anything done.”  Geeks.

A real geek would NEVER do that.

A real geek would write beautiful code that followed all the best practices rules for the honor of having written it.  A real geek would NEVER even begin to consider using the root account unless it was absolutely necessary.  A real geek would take the recommendations of a systems team (the real geeks, by the way) who spend all their time making sure the platform upon which these geek posers perform their witchcraft is ALWAYS up, ALWAYS stable, ALWAYS up to date, and ALWAYS secure.

I’ve been in this business for about 20 years now, and in that time I have met two developers who were tried and true, died-in-the-wool geeks.  TWO.

Gimme a break… “Geeks”.

I May Regret This

| Comments

I’ve been asked by the Digg auto-submission system to paste an invisible key into my page so Digg can curate my posts.

Into my next story on my site.  Of course, if ANYthing on here catches on, my server is toast, but let’s see what happens, shall we?  :)

Production and Developers

| Comments

Somewhere in the back recesses of my mind, I was brewing a post about permissioning and the relationship between DEV and Sysadmin teams and when and how elevated access should be granted.

Well, Kyle over at has approached this topic in a post entitled “Should Developers Have Access to Production?”.  I highly recommend this read, as it covers many of the topics I wished to cover.  I do have some additional commentary, but I’ll circle back around to that and share my thoughts at a later time.

Community or One-Upsmanship?

| Comments

Oftentimes I peruse various support forums and IRC channels.  Not for support, and not always to help out, but to just observe the community in action.

I have occasion to see the best of what Linux is all about (as it pertains to community), people literally all over the world at all levels of development asking for and giving help on a wide array of issues with their systems from very basic unix navigation and installation issues to extremely complicated configuration issues that require no less than code modification and custom compilation of new device drivers.

In and among this crucible of collaboration I find an almost pernicious insinuation to the conversation.  Whether it be because of youth, undeveloped social skills,  or some other societal construct, you begin to find those who are there for the sole purpose of detracting from the conversation.  Taking pot-shots and newcomers, ridiculing marginally experienced recent-converts and even getting into heavy argumentation regarding matters of practice in administration and or management of Linux systems.

For whatever reason, these find it a jolly good time to detract from the conversation; to make themselves present for little more than obstruction.  This can (and does) have a chilling effect on the community.

First, the open nature of the community is compromised.  Newcomers experience a chilling effect in not wishing to ask questions and interact for fear of being ridiculed and condescended toward.  Next, it generates a conversation away from these support channels about the character of those in the community, that they are elitist and snobbish in their assistance, and they wish not to help, but to lord over you their level of attainment.  Many times these conversations end with the phrase “I’ll just stick with Windows, I guess.”

Too bad for the community.

Widespread adoption of our beloved OS will not see any significant amount of growth until we move beyond this practice.  In light of this situation, I have some suggestions for community participants to both help them obtain the sense of community they wish and still allowing the newcomer to have a good experience, thus enjoying Linux early on, and then in turn propagating our community to his circle of friends.

First, if you must hang around Linux support forums, be prepared to answer any and all questions no matter how dumb you personally believe them to be. Remember that we all started at ground zero at some point in our lives.  We had those who helped us, and we had to pore through tons of documentation to get up to speed.  Show some understanding and compassion for the newcomer, and help them as much as you can.  If you do this regularly, before you know it, those “newbs” are now helping others only slightly newer than themselves with the very issues they themselves have just dealt with.

This does two things.  First, it brings them much higher in the food chain, making them closer to your level and much easier to relate to.  Next, it places yet another level of support for newcomers where you used to help out.  This is an optimal situation.  It broadens the community, “pays it forward” through newcomers to even newer adherents, and then builds up the ecosystem.

Next, if you don’t have anything to offer, say, or help with, it is perfectly ok to shut up.  If you see a question that you really feel is stupid and you are running over the litany of wonderfully snarky things you can say, maybe it is just best that you not say anything at all.  I firmly believe this benefits everyone.  If you must type something, private message an online friend instead so you can have a private laugh together.

Finally, when you do have things to offer, remember that as users we have preconceived notions about what everyone “should just know”.  The fact of the matter is that everyone does’t “just know” those things, and will never learn them if they’re the butt of finely crafted barbs heading their way.  Be prepared to remediate in a very friendly, non-condescending way and be prepared to even go as far as “Ok, a shell interprets the commands you type on the screen.”  Or even (God forbid) “your keyboard and/or mouse is sometimes referred to as input whereas your screen, network card, or modem is considered output.”

Painful, I know, but the more friendly, welcoming, and helpful you are, you play a part in helping to build up the community in a way that is self-perpetuating.  Help out.  Be friendly.  Be patient.  With a little help and a little patience, you build a group of community members that one day will be doing the very same thing for another generation of users.

LDAP Authentication IV - Apache

| Comments

Once you’ve got your handy LDAP server authenticating users for login throughout your environment, inevitably you find yourself getting around to the question of authentication for all your backend tools/services machines.  Can you authenticate from Apache to that same auth store to reduce administration burden across the environment?

Yes you can.

Recall from part I of this series that we have a server that is serving auth for our mythical “” environment.  Recall also the important info regarding this environment:

Internet domain name:          hostname of the server:          LDAP Search base:                          dc=bob,dc=com

File:  /etc/openldap/ldap.conf


LDAP Defaults


BASE dc=bob,dc=com URI ldap:// TLS_CACERTDIR /etc/openldap/cacerts

Now, we add to the mix an Apache web server that has an area it wants to secure.  Instead of using the old htpasswd method to create a disk-file containing usernames and password hashes, it would make much more sense to simply point Apache’s authentication system at our shiny new LDAP server so we eliminate yet another user/pass system we have to administer.


As many of you are already aware, Apache offers simple authentication to allow/disallow access to specific shared directories over the web.  The Apache Project has detailed documentation to help you get started here. I’ll cover the basics here.

Traditionally, in Apache’s config file(s) you specify a directory you wish to have secured via the “Directory” directive like so:

> > > > > > AllowOverride None > > > > Options ExecCGI Includes > > > > Order allow,deny > > > > Allow from all > > > > AuthUserFile /secure/passwd/file/location > > > > AuthType Basic > > > > AuthName "Cool Security Name" > > > > Require valid-user > > > > > >

You then create your passwd file for Apache with the htpasswd utility, make sure the above path points to it, bounce Apache, and you’re authenticating locally.  Problem with this is that every time you need to add a new user to your environment, every instance like this has to be touched individually and you have to add the user (whether by copy/paste or otherwise is irrelevant) in each.

Enter LDAP.

Fortunately, the configuration for such a change is quite easy; Simple textual changes to the httpd.conf to repoint from the local disk file to the LDAP store is all you need.  First, make sure that your LDAP modules are properly loaded into Apache:

LoadModule ldap_module modules/

LoadModule authnz_ldap_module modules/

This tells the Apache core to load up these at startup so that the directives you supply in the Directory stanza will be understood.  Next, let’s add the appropriate info into the Directory section:

AllowOverride None

Options ExecCGI Includes

Order deny,allow

Deny from all

AuthName “Cool Security Name”

AuthType Basic

AuthBasicProvider ldap

AuthzLDAPAuthoritative Off

AuthLDAPURL ldap://cool.ldap.server/dc=bob,dc=com?uid

AuthLDAPBindDN “cn=admin,dc=bob,dc=com”

AuthLDAPGroupAttribute memberUid

AuthLDAPGroupAttributeIsDN off


Satisfy any

This simple change (after an Apache restart) should give your Apache server security over that directory, authenticating against LDAP.  Since your LDAP store is POSIX compliant for login, you have a few other things you can leverage in there as well.  Among those, are the “Group” field.  That’s right, instead of going out and adding individuals and/or preventing them, you can specify an entire class of people (say LDAP_Logins for instance) that if the user belongs to that group, suddenly they have access to anything that is authenticating against LDAP.

To add that ability, you can add groups to the stanza above like so:

Require ldap-group cn=mygroup,ou=Group,dc=bob,dc=com

Require ldap-attribute gidNumber=500

(you place your group number in place of the “500” above)

Now that you have the magic behind LDAP auth for Apache, you can add this to any Apache web server of the 2.x variety, and do your security centrally.

The ease of addition for new users on your network is now as simple as adding the user to your LDAP store, placing them in the appropriate group.  Once Apache was restarted the very first time to make this work, you never have to restart it for user additions.  It will simply query LDAP, and LDAP will provide the credential response Apache needs to continue forward.

Next time:  Sudoers in LDAP!

Copyright © 2018 - Jerald Sheets - Powered by Octopress