WhiteWinterWolf Practical IT security, *nix systems & networking

The five Ws

• What: The Linux Professional Institute Certification (LPIC) is a technical certification on GNU/Linux systems administration. This certification is vendor-neutral and covers the major GNU/Linux distributions (Debian, SUSE, Red Hat) and their derivatives.

  With the Linux Essentials certification aside (it targets end-users, not administrators), the LPIC certification path has three main levels:

  • LPIC-1 “Linux Administrator”: This level studies the GNU/Linux system itself: how it works, how to administrate the local system with some knowledge on troubleshooting and main services.

  • LPIC-2 “Linux Engineer”: This level has two folds: on one side you study advanced administration and troubleshooting techniques, on the other you now envision the GNU/Linux system as part of the corporate ecosystem and study the administration of the most common network services (here again vendor neutral, so you should be comfortable with both Apache and Nginx HTTP servers for instance).

  • LPIC-3 “Linux Enterprise Professional” (previously “Senior level certification”, but I suppose it sounded too elderly ;) ): This is the last level of the LPIC path and allows to specialize on a chosen domain. Three domains are currently available:

    • “Mixed Environment” studies services and functionalities designed to work notably with Windows servers and client, like LDAP, SAMBA, etc.
    • “Security” studies tools and techniques allowing to increase the security at various levels in a enterprise context.
    • “Virtualization and high availability” studies tools and techniques allowing to build virtualized, cloud and clustered environments.

• When: The LPIC-1 has no prerequisites, but it is very technical and demanding so is not adapted for starters in the GNU/Linux world (check the Linux Essentials certification instead).

  Obtaining the LPIC-2 and LPIC-3 certification have both the previous level certification as prerequisite, but the exams themselves can be taken in any order.

• Why: This certification demonstrate how comfortable you are with GNU/Linux environments in corporate environments.

  While vendor certifications cover only a single distribution, often even a single version of their distribution, and limit their scope accordingly, the LPIC certification tests you on various tools and distributions and on new and legacy versions. This shows employers your flexibility and your ease to adapt yourself to any GNU/Linux environment.

• Who: Don’t confuse the LPIC certification with an end-user certification. This one covers a very wide range of technical domains regarding the GNU/Linux internals and is targeting corporate environments. If you are not at least remotely familiar with some of the topics before starting your study, chances are that you will quickly find yourself overwhelmed with new concepts and information (as I said in my general post on certifications, expect the exam to target small details, not generalities).

  “Liking Linux” or regularly using it to for Web browsing for instance is not enough, consider passing the Linux Essentials certification instead.

  That being said, working for this certification is also a great way to learn. There are a lot of people who use LPIC-1 study materials not necessarily with the goal to pass the exam, but simply to complete the gaps and better organize their knowledge of the GNU/Linux environment (a good LPIC-1 study book can effectively be used as one of those “Linux bible” books).

• Where: The exams can be taken in any Pearson VUE test center. The LPI also regularly organize exam sessions at discounted price in Linux-related events, in this case they are announced in the event program.

  The LPIC-1 and LPIC-2 requires to pass two exams each, each exam having its own set of topics. The LPIC-3 requires only single exam.

  Each exam contains 60 multiple-choice and fill-the-blank questions to answer in 90 minutes. It is possible to go back to previously validated questions.

  Each exam costs around $200 USD if you pass it in a Pearson VUE test center. During events, the discounted price may depend but looking at the FOSDEM page for instance it was about half of the normal price.

Training material

Building a lab

Nothing fancy here, all you will need is several virtual machines each running a different distribution of the GNU/Linux operating system. If your system is not very powerful or is lacking RAM, you should be fine with only one virtual machine running at a time so, as long as your system is able to start one virtual machine in good conditions, you should be fine.

You will at least need one copy of each the following Linux distribution:

• Debian, as itself.
• CentOS, as the community maintained version of the commercial Red Hat distribution.
• openSUSE, as a community maintained equivalent of the commercial SUSE distribution.

Of course, if you have access to a genuine Red Hat or SUSE distribution, take advantage of this. But if don’t, there is no major difference as far as the LPIC topics are concerned with their community maintained counterparts.

This is just the minimal requirement, feel free to add any supplementary distribution you would like. Adding an Ubuntu for instance may be interesting as it is a common choice, even in corporate environments, and would allow you to test the differences between an original Debian and this well-known derivative.

You don’t necessarily need the latest version. In fact, given the time to qualify a new system and the upgrade process, most companies do not run the latest version in their production environment. Moreover, in addition to your main images, your will also likely need an intentionally older image of at least one of these systems (like a Debian 6 or 7 for instance) to test older features not present anymore in the most recent versions (like the UNIX System V Init system). If the author of your book or any other training material you use tells the version he is using in his examples, you should use the same to better follow him.

The LPIC is a professional certification which is lead by corporate needs. While interesting to check for your own information, distributions which are more confidential in the corporate world (usually due to the lack of proper business-grade support service from the editor) are not directly covered by the exam (package installation commands, etc.).

Learning resources

First and foremost, I have to tip my hat to the awesome work made by The Urban Penguin. Andrew Mallet’s videos are clear, to-the-point and highly pedagogical. Everything is explained and demonstrated in a calm, clear and concise way. Somehow, his videos really gives the impression to attend a live training, to the point that often when a question pops-up in my mind during his explanation he answers it just right away.

IMHO, The Urban Penguin is a just a mandatory (and nice!) stop for every LPIC student and possibly even more as he regularly publishes videos on other Linux-related topics.

Nevertheless, while videos are a good thing to learn through live demonstration and by hearing, I always like to rely on books for my studies. Sadly for you, dear reader, for some reason for my own studies I happened to use a French book, so I cannot vouch for any English book in particular.

LPI has setup a marketplace where you can buy exam vouchers and all kind of resources including books and other videos trainings. The marketplace can ship books only to the US addresses, but the same books can also be found on any other book-selling websites. The exam vouchers however can be bought internationally and at the same price than Pearson VUE website.

A good LPIC book is a good investment as it can easily serve later as a reference like a “Linux bible” to quickly refresh you memory about how to configure something or how to proceed in some circumstances.

The first levels of the LPIC are usually well covered, including books from the All-In-One and Sybex series. The latter also provides other books covering LPIC-2 and a large set of practice questions covering both LPIC-1 and LPIC-2 which, if done correctly, may be of great help especially if you are not used to pass certification exams).

I mention those two editors simply because they are well-known in the certification industry, as I said I didn’t read them and some alternative may also be as good. For what I know from other certs, Sybex book are usually more wordy, focusing on background explanation, while All-In-One are usually more practical, but may miss some background or secondary information compared to Sybex books. Make your choice!

No matter the book you are choosing, here are a few advises which may help you:

• The LPIC curriculum is very dense, with a lot of knowledge of varying nature covering a large number of different domains. While with some certification focused on a given topic it may possible to learn the whole curriculum from cover to cover at once, don’t try this here or your head will explode.

  • First follow the book without really thinking to the exam. Read the explanation, do the practical exercises, if something looks too complex now simply read the explanations and proceed. Skip all quizzes during this step. This will familiarize you with the content of the book and tell you where are the easier and harder part are located.

  • Still due to the large amount of information and its varied nature, a good LPIC preparation book should be divided in two parts to follow the LPIC division in two exams. On your second read, focus only on the part dedicated to your upcoming exam (if your book mixes all topics, use the official topics list from the LPI website).

    I don’t think it is humanely possible to learn each and every details of the whole curriculum at once (expect the exam to set the bar well higher than what is required for a daily activity, see my general post on certifications for more information). In all cases this is not what is expected by the LPI: that’s why they divided each level in two exams, each one with its own topics list.

• Once you have mastered the content of your training material, check and double-check the official topics list from the LPI website. It seems common for LPI preparation books to miss a various number of topics. In itself, this is not a big deal as usually a simple Internet search is enough the complete your knowledge with the missing information, but you do not want to come to the exam unprepared.

  Note

  As a personal fact, I just completely missed the Unix printing section during my studies for the LPIC-1. I still wonder how I managed to do that.

  As the exam went on, the first question on this topic looked like an alien to me, and the bad news was that this question was only the first from a particularly painful and depressing set.

  While I was feeling down for the rest of the exam, at my own surprise I still managed to pass it with a reasonable score. Nevertheless, don’t tempt the devil, as this is a very uncomfortable situation I wouldn’t wish to anyone.

As a complement, the LPI also manages a list of free training material available from various sources. This material may be more or less complete depending on the source, but in case of doubt it can nicely complement the above mentioned resources.

Some people also published good exam-like questions. While maybe not always up-to-date now, these questions still correctly reflect the kind of questions you will be asked during the exam (note that these questions have been specially written for these exercises, it is forbidden to share actual exam question) and allowed me to find my last weaknesses before taking the exam:

• Penguin Tutor: This one covers both LPIC-1 and LPIC-2, and also hosts an blog with interesting feedbacks from the author’s own experience.
• Gnosis Software: Covers LPIC-1.
• GoCertify: Provides a free access to a few small quizzes covering LPIC-1.

Personal impressions

Curriculum

The LPIC curriculum is very well thought, and regularly updated in a sensible way. It contains real subjects which are of concrete use for Linux system administrators. As I said above, the LPIC curriculum in itself can serve as a basis to write a good “Linux bible” book. Almost every parts of a common corporate GNU/Linux environment and its administration is covered, with here again a sensible weight attributed to each topic.

I regularly read criticism from people claiming that the curriculum or the LPIC training resources often cover old or obsolete technologies and software versions. I guess that the people who have such claim did not work in a lot of large institutions. In large environments, when things are working and each minute of downtime may cost hundreds of thousands of dollars, people are really afraid from touching anything. In such situations, you are more likely to encounter an obsolete, unsupported system than anything even remotely brand-new.

In such environments, “new” is not a quality but a threat. In such environments, “new” doesn’t mean “improved” but it means “unknown” and “untested”. It usually takes months, or even years before upgrading a system, the time required to test and fix the system reaction in every known conditions and events combinations to remove these “unknown” and “untested” threats.

That’s why, as a Linux systems engineer, you need to know both the new and old systems, because it may be very well your duty to maintain the current production, qualify the upgrade, and maybe proceed with the migration when the new system is not new anymore.

Having worked in this area for several years, I can confirm that the LPIC curriculum does a great job in preparing its student to real-life needs.

However as nothing is perfect and, whatever you do, there will still be people to complain, let me still complain a bit too ;) ! Actually, the following points are more some room for improvements, or things that I was a bit surprised to not find covered, but I guess that the LPI has to make choices to keep its curriculum focused on really impacting topics.

In all cases the following points do not remove anything of all the good I think of this certification.

LPIC-2

Linux is a very versatile system with a very heterogeneous community. I feel a bit sorry that the LPI curriculum focuses so exclusively on the three well-known distributions.

There are other distributions which took different technical or architectural approaches, like the way they handle the system installation, updates and software management which may be worth mentioning.

The most notable examples are Arch, Gentoo and Slackware, but for more experimental systems I could also mention GoboLinux (the whole directory tree is reorganized so each application reside in its own tree to provide better isolation) and OpenWall (notably for its per-user /tmp directory to increase overall security).

The goal here is not to be proficient in every Linux distribution out there. Once you get the fundamentals right (and the LPIC curriculum takes care of that) then you can quickly adapt yourself. The goal here is just to be conscious of the richness and flexibility of the Linux system, the very thing which makes it the number one system from low power embedded devices up to the most powerful supercomputers: they all share the very same Linux kernel.

Some LPIC training book attempt to convey this message in their historical and cultural introduction chapters, but I think that the LPI should try to go beyond the simple client/server scheme as, even in corporate environments, Linux can do way more than that.

LPIC-3

You may have noticed how in this article I emphasized that this was GNU/Linux certification as opposed to a Linux certification as the LPI advertises it. This is not some random pedantry as, while this certification does a good job in being distribution agnostic, the Linux kernel is always envisaged in a GNU environment.

I am really missing a section on embedded Linux systems, either as a standalone LPIC-3 specialization or mixed within the already existing LPIC-3 specialties. In particular, Android is not something which can be ignored in the current Linux world, including business environments. To say the least it is Android which solved the historical Ubuntu’s bug #1: “Microsoft has a majority market share”.

An embedded Linux remains a Linux, but due to technical limitations and development choices it presents some specificities compared to a traditional GNU/Linux system.

Knowing, at least from a general perspective, what are these specificities and why there are here should be required for anyone presenting himself as a Linux expert.

For instance, speaking of the LPIC-303 “Security” certification, the topics list mention that “candidates should have a thorough knowledge of SELinux”. Android also relies on SELinux to ensure some of its security, but being an embedded environment while the Android port of SELinux remains a SELinux in its main functionalities it presents some notable differences (more information can be found here).

SELinux being often presented as an obscure topic, I think it should be great to take conscience of its use in major end-users devices and understand how it is used in such contexts.

At last, on a different subject, I take note of the removal of GPG of the LPIC-303 “Security” topics list its latest update (V4). On the other side the HIDS section has been expanded and whole new section on FreeIPA has been added. I understand that LPI needs to keep the number of different subjects to a reasonable amounts, but both SSH and OpenVPN having been already moved to LPIC-202 this should leave enough room for three or four commands.

In these times of mass surveillance and corporate espionage, I think that having at least some basis on GPG (like a bullet point somewhere in the cryptography section, at least mentioning the name) still seems quite important to me.

Exam

The exam questions are clear, non-ambiguous, and closely match the topics list.

Granted, for the LPIC-1 and LPIC-2 exams, there might be a few questions seeming to come from the wrong topic list (for instance one or two questions from the LPIC-102 topics list making their way into the LPIC-101 exam and vice-versa). But these are very broad questions that you should be able to answer without having specifically studied the subject in details.

As a Linux administrator, you are expected to have a minimum of culture on Linux system administration. Sounds fair, doesn’t it?

I was afraid because of a book where the proposed “exam-like” questions were of poor quality, but the actual questions which make their way to the exam are really good quality ones.

While the topics list evolve from one level to another, so does the questions which become more complex as you reach LPIC-2, then LPIC-3 levels:

• In the first level there are very few fill-in-the-blank questions, the exam is composed of mostly multiple-choice questions. But as you progress you find yourself facing these significantly harder fill-in-the-blank questions more and more often.

  Personally these fill-in-the-blank questions where also a cause of worry for me before taking the exam, but they are really done the right way as there is no ambiguity in what you are expected to type.

  They are usually significantly harder as there is no proposed answers anymore to help you refresh your memory as there is with MCQs, but they are fair: as long as you know the correct answer, you will have no problem answering, the real difficulty comes when you need to guess…

• As you progress, the questions will ask you more and more subtle details. For the LPIC-1, the exam questions even learned me new ways to use some commands asking me the expected result, which I found very interesting, dare I say entertaining. But as you progress the questions focus on tinier and tinier technical or configuration details, and entertainment quickly leaves place to headaches!

I see nothing negative in all of this: the LPIC exam is just how I expect it to be.

Conclusion

If you are interested in Linux systems, if you would-like to work with Linux systems, you can’t go wrong with this certification.

While there are other distribution agnostic Linux certifications out there including one from the Linux Foundation itself (about which I’m still curious BTW), the LPI one is the only one I regularly encounter in job offers, demonstrating its reconnaissance in the professional world.

Regarding distribution specific certification, the LPI works hands-in-hands with SUSE at least for their first level LPIC-1 certification. Previously, getting your LPIC-1 automatically awarded you an actual SUSE CLA certification in addition to the LPIC-1 certification. Now SUSE reorganized its certification and, while they now limit their certification to people having really passed a SUSE exam (which I can easily understand, it feels strange to be SUSE-certified without having passed any SUSE exam), they accept the LPIC-1 as a substitute of their Level 1 certification to satisfy their Level 2 certification prerequisites.

I don’t know how are the relation between Red Hat and the LPI foundation, in all case there is no such agreement there. Nevertheless, the LPI certification having acquired a certain reconnaissance in the professional world, you may still have your chances in positions initially requiring a Red Hat certification by showing a LPI certification.

But enough of this blah-blah.

Did I say that when you pass you LPIC-1 exam you even get a ticket for a free issue of the Linux magazine¹? At higher levels you are even invited to join their community as a volunteer to help to design the future of the LPI certification, and no secrets happens there as their Wiki and mailing-list are public.

An open and friendly community to certify your competencies on an open system, what more would you ask?