In my Internet and sysadmin travels, I find it necessary from time to time to seek out documentation on a particular subject in the Linux world. Sometimes I need something specific to Linux, sometimes a package on Linux, but always technical and many times detailed.
For the longest time (since my Linux infancy in 1995), I have used the Linux documentation project to find important HOWTOs. As a community, we have prided ourselves on a ubiquity of documentation and support, but I found something both interesting and alarming at the same time on my last visit to TLDP.
In 2010, there have only been 12 documents of TLDP modified and in 2009 a mere 5. That means that in nearly two full calendar years, there have only been 17 documents modified in the project. As many of you already know, 24 months is an eternity in Internet time (much less regarding the growth and progress of a major OS like Linux)
While I am confident that many of these docs are solid and still stand on their own, it does concern me that documentation hasn’t changed much and other projects' documentation stores are starting to see similar atrophy.
Certainly, with each release we find a new set of release notes and subsequent additions to documentation. However, as we go on I find more often than not that a project will have a fundamental change that should really be covered in TLDP or at least documented at the documentation level (rather than the release note level) and simply never has it happen. In these cases, you find a “collective knowledge” of people on the support mailing list that “just know” something to be true, but those new to the project or the list may never know it because it isn’t written down somewhere.
Remember the age-old wisdom: "If it isn’t written down, it never happened."
For those of you on a project, please consider your documentation. It may be a time for a rewrite. It happens precious little (if the TLDP is to be believed) and really needs your attention. There are countless volunteers out there who may not be coders, but use your product and would be ecstatic to be counted as one of the team simply to do documentation for you.
Many people think that it’s just adding users, performing requests, and working with project groups to get their “next big thing” out the door.
Certainly, much of your work involves those very things, but what younger admins never quite realize until they’ve built a beast that cannot be fed is that everything is about frameworks that tie everything together. From a system load framework to an authentication framework to an asset management framework, if built correctly a framework can save each and every admin countless hours of administration time. How?
Take distribution. You can push files around considerably more easily when you’ve built a framework to do so rather than every admin having an individual way to do it. In fact, when properly implemented, distribution of a file to “gobs” of similar hosts (that’s a technical term there :)) can be as simple as:
Now, many will say that rsync or scp will do much of this for you, and that is correct. However, in the context of your individual site, having symbolic abstractions such as “class of host” goes a long way. Perhaps a certain file needs to be distributed to only your web servers. Or maybe, only web servers running RedHat 5.3. If you correctly build a framework for file shove and management, suddenly heavy lifting becomes a light chore. After all, the more pulleys in the works, the lighter the load becomes.
As has been covered on this site in the past, LDAP is a wonderful authentication framework that can be tied to positively everything in your environment as well. From Apache authentication against the store to UNIX authentication, to various types of applications understanding LDAP as a target for authentication sources, much pain of user administration can be solved by having a centralized authentication mechanism.
Frameworks as Philosophy
Rather than continue with examples on a case-by-case basis, consider the entire concept of frameworks and unifying ties across systems and networks. Many places I’ve worked, organic growth brought about massive numbers of machines that were “siloed” one from another either by project boundaries, function boundaries, or other superimposed logical delineations we as users imposed on them.
Instead of “the DEV servers” or “The PROD servers”, we have logically separated them into “the ECommerce DEV servers” versus “the Web DEV servers” and so forth. Rather than having a framework of systems and their functions in the workplace providing services and features at a certain level to the customer and/or end-user, we cobble together pieces to serve a single purpose rather than modules to expand the greater infrastructure.
Logical Differentiation and Service-Orientation
The clear winner over the “as needed” or “for a purpose” way of doing things is the “tiered” or “services-oriented” model of work. Rather than many groups of things that form a farm of servers, you have the farm of servers that service many different things. This, I know, sounds something like technological double-speak, but let me explain what I mean.
In a normal environment, we would have a somewhat typical scenario with front-end servers, back-end infrastructure, management, and organization. The problem we have with this is that if “New Whiz-Bang project #7” comes along, new hardware will need to be procured for each piece of the puzzle. New app servers, new web servers, new database components, maybe management and authentication considerations… Each project, every time, capital outlay, budgetary justifications, etc.
If, instead, you think in terms of frameworks, your job becomes one of determining total resources needed across an environment rather than resources needed for an individual project, and thus individual project growth concerns, individual project funding concerns, personnel, etc. "Siloed" growth and expansion may not bite you today or tomorrow, but as I said early on, will become a beast that cannot be fed.
Consider instead, the following example:
Instead of “theses web servers versus those web servers”, you instead have “the web layer”. Any and all requests to your company come through “the web layer”. It is scaled as an entity and not as individual projects. When scaling happens, all environments benefit.
Instead of “these application servers” versus “those application servers”, you have “The App Layer”. A single applications framework that serves all application server requests back out the front-end web layer by leveraging container and web server features to do the “effing magic ™” on the backend to provide a unified front-facing experience to the user.
Extrapolate these ideas… Instead of the app layer, let’s say the app cluster. Now the power behind this idea becomes clear. Unlimited scalability with unlimited potential. How about “the database cluster”? Regardless of the solution you use, if there is a single database resource (cluster, replication ladder, whatever) that serves back queries you throw at it, how much better is that than “databases for this” and “databases for that”?
Take it a step further.. make an XML services layer that serves out “your data” in a clearly defined API sort of way, and all you do is make XML requests to a services infrastructure rather than directly at your databases. Or, your cluster is comprised of “write databases” versus “read databases” and you’re segmenting the type of traffic you’re serving to reading versus writing, making the read operations light-years faster.
Authentication layers, web layers, XML layers, app layers, database layers… All frameworks that grow as an organism rather than series of unrelated growths.
When going through your next design adjustment or your next expansion or data center rollout, consider thinking differently about growth and planning. I believe that if you think in terms of large organisms with several related parts rather than several growths unrelated except in their location, you’ll build into your infrastructure a power and a scalability that will serve you well for many years (and growths!) to come.
You mean the people we geeks won’t give access to because, if left to themselves, developers will patently destroy anything they come in contact with in the systems world? You mean the people who think root is an account that should be used as a tool to cure ALL their ills and knock down all the “obstacles” they encounter? The people who won’t use “sudo” because it’s too many characters to type and “breaks their flow” when coding?
Oh, I get it, the people who haven’t the slightest clue what it really means to be geek. To give honor and deference to the system. It’s security, design, integrity. They don’t care that there are other people on the box, they just want to meet their date. And they’ll twist every systems admin in every possible contortion to break all the best practices in the world just to meet their date. "Being Geek" Phah!
These guys like to be called geeks because it is an easy to earn, undeserved moniker for them bestowed by people who have no clue what it means. All the while, they’re breaking every rule and every guideline just to meet a date. Further, when Systems people point out security concerns or elements of systems design these supposed geeks are transgressing, they run to upper management and complain that the systems teams are “blocking their date”, or “They’re blockers”, or “we can’t get anything done.” Geeks.
A real geek would NEVER do that.
A real geek would write beautiful code that followed all the best practices rules for the honor of having written it. A real geek would NEVER even begin to consider using the root account unless it was absolutely necessary. A real geek would take the recommendations of a systems team (the real geeks, by the way) who spend all their time making sure the platform upon which these geek posers perform their witchcraft is ALWAYS up, ALWAYS stable, ALWAYS up to date, and ALWAYS secure.
I’ve been in this business for about 20 years now, and in that time I have met two developers who were tried and true, died-in-the-wool geeks. TWO.
Somewhere in the back recesses of my mind, I was brewing a post about permissioning and the relationship between DEV and Sysadmin teams and when and how elevated access should be granted.
Well, Kyle over at serverfault.com has approached this topic in a post entitled “Should Developers Have Access to Production?”. I highly recommend this read, as it covers many of the topics I wished to cover. I do have some additional commentary, but I’ll circle back around to that and share my thoughts at a later time.
Oftentimes I peruse various support forums and IRC channels. Not for support, and not always to help out, but to just observe the community in action.
I have occasion to see the best of what Linux is all about (as it pertains to community), people literally all over the world at all levels of development asking for and giving help on a wide array of issues with their systems from very basic unix navigation and installation issues to extremely complicated configuration issues that require no less than code modification and custom compilation of new device drivers.
In and among this crucible of collaboration I find an almost pernicious insinuation to the conversation. Whether it be because of youth, undeveloped social skills, or some other societal construct, you begin to find those who are there for the sole purpose of detracting from the conversation. Taking pot-shots and newcomers, ridiculing marginally experienced recent-converts and even getting into heavy argumentation regarding matters of practice in administration and or management of Linux systems.
For whatever reason, these find it a jolly good time to detract from the conversation; to make themselves present for little more than obstruction. This can (and does) have a chilling effect on the community.
First, the open nature of the community is compromised. Newcomers experience a chilling effect in not wishing to ask questions and interact for fear of being ridiculed and condescended toward. Next, it generates a conversation away from these support channels about the character of those in the community, that they are elitist and snobbish in their assistance, and they wish not to help, but to lord over you their level of attainment. Many times these conversations end with the phrase “I’ll just stick with Windows, I guess.”
Too bad for the community.
Widespread adoption of our beloved OS will not see any significant amount of growth until we move beyond this practice. In light of this situation, I have some suggestions for community participants to both help them obtain the sense of community they wish and still allowing the newcomer to have a good experience, thus enjoying Linux early on, and then in turn propagating our community to his circle of friends.
First, if you must hang around Linux support forums, be prepared to answer any and all questions no matter how dumb you personally believe them to be. Remember that we all started at ground zero at some point in our lives. We had those who helped us, and we had to pore through tons of documentation to get up to speed. Show some understanding and compassion for the newcomer, and help them as much as you can. If you do this regularly, before you know it, those “newbs” are now helping others only slightly newer than themselves with the very issues they themselves have just dealt with.
This does two things. First, it brings them much higher in the food chain, making them closer to your level and much easier to relate to. Next, it places yet another level of support for newcomers where you used to help out. This is an optimal situation. It broadens the community, “pays it forward” through newcomers to even newer adherents, and then builds up the ecosystem.
Next, if you don’t have anything to offer, say, or help with, it is perfectly ok to shut up. If you see a question that you really feel is stupid and you are running over the litany of wonderfully snarky things you can say, maybe it is just best that you not say anything at all. I firmly believe this benefits everyone. If you must type something, private message an online friend instead so you can have a private laugh together.
Finally, when you do have things to offer, remember that as users we have preconceived notions about what everyone “should just know”. The fact of the matter is that everyone does’t “just know” those things, and will never learn them if they’re the butt of finely crafted barbs heading their way. Be prepared to remediate in a very friendly, non-condescending way and be prepared to even go as far as “Ok, a shell interprets the commands you type on the screen.” Or even (God forbid) “your keyboard and/or mouse is sometimes referred to as input whereas your screen, network card, or modem is considered output.”
Painful, I know, but the more friendly, welcoming, and helpful you are, you play a part in helping to build up the community in a way that is self-perpetuating. Help out. Be friendly. Be patient. With a little help and a little patience, you build a group of community members that one day will be doing the very same thing for another generation of users.
Once you’ve got your handy LDAP server authenticating users for login throughout your environment, inevitably you find yourself getting around to the question of authentication for all your backend tools/services machines. Can you authenticate from Apache to that same auth store to reduce administration burden across the environment?
Yes you can.
Recall from part I of this series that we have a server that is serving auth for our mythical “Bob.com” environment. Recall also the important info regarding this environment:
Internet domain name: bob.com
hostname of the server: ldap.bob.com
LDAP Search base: dc=bob,dc=com
Now, we add to the mix an Apache web server that has an area it wants to secure. Instead of using the old htpasswd method to create a disk-file containing usernames and password hashes, it would make much more sense to simply point Apache’s authentication system at our shiny new LDAP server so we eliminate yet another user/pass system we have to administer.
As many of you are already aware, Apache offers simple authentication to allow/disallow access to specific shared directories over the web. The Apache Project has detailed documentation to help you get started here. I’ll cover the basics here.
Traditionally, in Apache’s config file(s) you specify a directory you wish to have secured via the “Directory” directive like so:
You then create your passwd file for Apache with the htpasswd utility, make sure the above path points to it, bounce Apache, and you’re authenticating locally. Problem with this is that every time you need to add a new user to your environment, every instance like this has to be touched individually and you have to add the user (whether by copy/paste or otherwise is irrelevant) in each.
Fortunately, the configuration for such a change is quite easy; Simple textual changes to the httpd.conf to repoint from the local disk file to the LDAP store is all you need. First, make sure that your LDAP modules are properly loaded into Apache:
This simple change (after an Apache restart) should give your Apache server security over that directory, authenticating against LDAP. Since your LDAP store is POSIX compliant for login, you have a few other things you can leverage in there as well. Among those, are the “Group” field. That’s right, instead of going out and adding individuals and/or preventing them, you can specify an entire class of people (say LDAP_Logins for instance) that if the user belongs to that group, suddenly they have access to anything that is authenticating against LDAP.
To add that ability, you can add groups to the stanza above like so:
(you place your group number in place of the “500” above)
Now that you have the magic behind LDAP auth for Apache, you can add this to any Apache web server of the 2.x variety, and do your security centrally.
The ease of addition for new users on your network is now as simple as adding the user to your LDAP store, placing them in the appropriate group. Once Apache was restarted the very first time to make this work, you never have to restart it for user additions. It will simply query LDAP, and LDAP will provide the credential response Apache needs to continue forward.