Categories: Linux
Intro
The Linux From Scratch site provides a tutorial on building a complete linux distribution “by hand”. I recently tried it; these are my notes.
The things you’re probably most curious about are:
- time: how long does it take to do the LFS tutorial?
- quality: how well-written is the tutorial?
- learning: will I learn much about Linux from it?
- long-term: is the end result something I would actually use long-term?
Short answers:
- time: about 20 hours (ie 2.5 full working days - or possible on a single wet weekend for those without spouse and kids..)
- quality: the LFS book is very well written - some of the clearest technical instructions I have ever seen, and 100% error-free
- learning: well, yes I learned some things. But a lot of the time was spent copying/pasting from the instructions and waiting for compilation to finish
- long-term: I am still not sure if I will run the resulting system for a while, or go back to a “normal” distribution. Probably the latter - maintaining a self-built system is a lot of work.
The result is a very basic system - no graphics, sound, wireless networking. All of that requires further work; there is a Beyond LFS (BLFS) book (also well written) with information on how to further extend the system, eg graphics, sound, wireless. I am currently installing some stuff from BLFS, and will write a separate article on that experience later.
Overall, the 2.5 days I spent were not wasted, but neither did I learn anything truly inspirational. Whether it is worth the time for you to work through LFS depends on your goals. I would suggest:
- For those wanting to be “software packagers” (eg Debian maintainers), I think it provides useful insight into how a project is built overall.
- For those who expect to create customised Linux installations for embedded systems or similar, regularly work with cross-compilation, etc. then LFS is definitely worth-while.
- For those wanting to be linux admin gurus, probably not worthwhile - you will be working with redhat/debian/etc - spend the time getting to know those distros better instead
- For software developers, the time could probably be spent in more useful ways
However even if you don’t want to complete the LFS tutorial, I would recommend reading some chapters:
- the foreword
- the Package Management chapter (section 6.3 in edition 7.7 of the book). Note that the BLFS book strongly recommends “Symlink Style Package Management” using
make DESTDIR=... install - the whole of chapter 7 (“Basic System Configuration”) is interesting.
- perhaps chapter 8 (“Making the LFS System Bootable”) which discusses kernel compilation and grub configuration
Ch1-5 took me about 8 hours - this completes construction of all “temporary tools”, ie gets to the point where you can chroot to the new system. This is an important step, as it is then possible to run an entire userspace without using any code from the host (chroot). However these temporary tools have internal dependencies on a temporary “/tools” directory which can be made to work but is not good long-term. The next step is therefore to chroot to the new system and rebuilt everything again with proper paths. I learned a few things here about cross-compilation, chroot, etc.
Ch6 also took me about 8 hours - completes the recompilation of all core software. Rather tedious and not very interesting.
Ch7-9 took just a couple of hours, and was very interesting.
The above durations include some time spent reading interesting documents referenced from the LFS book such as manpages, the Filesystem Hierarchy Standard, and the user-based package manager documentation.
The ALFS (automated linux from scratch) project might be worth trying out; the instructions in the book are marked up in a way that makes it possible for ALFS to extract and execute them. This both (a) ensures that the book is 100% correct, and (b) avoids much tedium. Given that Ch6 takes so much time and offers so little to learn, I would suggest investigating whether ALFS can be applied at least to that part.
Note that I used the systemd-based variant of the book.
Things that could be better explained
The following items are things that confused me at first. Most are actually explained if you read the LFS book carefully, or at least are eventually explained. However it might be useful to know these up-front before starting.
Those who know more about Linux than I might not consider some points below to be “real problems”. Those with less knowledge might not realize there are problems at all…
The problem with GCC and file paths
According to Rob Landley the gnu toolchain (gcc/binutils/etc) is a very poor cross-compiler; it is “full of hardwired assumptions .. such as the C library, header paths”. This is why LFS has such a complicated “pass1”, and includes steps that use sed to manually change paths within generated binaries! Maybe using llvm would improve this?
Many compiled applications have “search paths” hard-wired into their code. In particular, the GNU linker and compiler search for files using default paths encoded into the binaries. The tutorial creates a directory /tools in the host, and “first-pass” (temporary) binaries are compiled for the new LFS system using this directory as the “search path”. It took me a while to realize that the point of this is that after using chroot() to switch to a new environment, the path /tools still works, and contains the same files. Everything compiled under the host-root using ‘/tools’ is eventually recompiled under the lfs-root (ie using chroot) without the temp /tools path, after which the /tools directory can then be removed.
Root or not Root
Commands in the first few chapters of the tutorial need to be run as root. Later, actual compilation of the code is done as a normal user - which is sensible; unfortunately that isn’t mentioned until section 4.3.
I would recommend the “wget” operations (early in the tutorial) be run as a normal user, rather than using root (which the tutorial assumes).
Saving the intermediate state
The steps in chapter-5 (“first pass”) produce a “cross-compilation” environment that can be reused. You can save this state, and then use it many times. It can be used to compile different versions of the software; although the LFS tutorial recompiles exactly the same set of source-packages again, that doesn’t need to be the case. In particular, when “core” software changes versions (eg glibc) then it is recommended to return to this “cross-compilation” environment and recompile everything in LFS (ie repeat chapter 6 but with newer sourcecode).
The –prefix parameter
A “–prefix=/usr” parameter is passed to most makefiles, but it is not explained why. The reason is to override the default prefix of /usr/local. Many packages default to installing in “/usr/local” which is sensible as they are assuming you are running a typical “packaged” distribution (eg redhat/debian) and so should not mix locally-compiled binaries with package-managed binaries. This is not the case for LFS, so overriding prefix is usually required.
The “expected” build and test output
The LFS site provides example output of build and test steps, which is very useful. The link is given in section 4.6 “About The Test Suites”, but I kept forgetting it. For reference, the output for the LFS version I was using could be found at:
Interrupting and Resuming LFS
When resuming work after rebooting do the following (as noted in section 6.4):
# Section 6.2.2: mounting and populating /dev
mount -v --bind /dev $LFS/dev
# Section 6.2.3 Mounting virtual Kernel File Systems
mount -vt devpts devpts $LFS/dev/pts -o gid=5,mode=620
mount -vt proc proc $LFS/proc
mount -vt sysfs sysfs $LFS/sys
mount -vt tmpfs tmpfs $LFS/run
# Section 6.4 Chroot
chroot "$LFS" /tools/bin/env -i \
HOME=/root \
TERM="$TERM" \
PS1='\u:\w\$ ' \
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin \
/tools/bin/bash --login +h
# If you've completed 6.36 (recompile bash) then do this (or use /bin/bash in the step above): see 6.72
exec /bin/bash --login +h
Section 7.2 Networking
This chapter could do with some restructuring and extended explanations.
Section 8.3.1 : building kernel
The first “note” shows an image of a config program, but does not mention that “make menuconfig” is the way to get that menu until after the table.
Section 8.4: configuring GRUB
How about adding instructions to create a bootable USB stick (rather than just a CD)?
And how about instructions to add a grub menu to the grub.cfg of the “host” operating system? Running “update-grub2” on my debian8 host system auto-detected the kernel on the LFS partition and added a menu option - much safer than running grub-install in the LFS chroot.
Links to TDLP (The Linux Documentation Project)
The LFS book has a number of links to documentation on the TLDP website. Sadly, everything on TLDP is completely out-dated, and should be ignored. LFS should really remove such links, as the referenced documents are no longer helpful.
In particular, the “Prerequisites” section recommends reading a TDLP “Software-Building-HOWTO” document. This is really out-of-date (Motif?! “a.out”?! Imake?!), and has many broken links. I’ve created a document that covers similar topics but without the outdated references.
TLDP was a great idea, but sadly has not functioned long-term.
Issues Tracker
The book refers to an LFS “issues tracker”, in particular that suggested changes to the book should be made by opening a ticket. However the issue-tracker is almost impossible to find. The home page has no link to it, nor do the “support”, “contribute” or “site map” subpages. The LFS Book page does have a “Report a Bug” link in the menu, but that leads nowhere relevant.
Only after going to the wiki, then to the LFS component of the wiki, did I find a “View Tickets” option in the top menubar.
Actually, LFS uses “Trac” which is both a wiki and a bugtracker. So in effect, the wiki is the bugtracker - hence the “View Tickets” option in the wiki menubar.
While on the subject of the wiki, it isn’t used in the manner many other projects use their wikis for. There is very little “community content” here; instead it is mainly used by the LFS maintainers. There is nevertheless some useful information here.
Package Managers
Although the LFS tutorial installs everything without using a package-manager, it is possible to install a package-manager on LFS if you wish. There is a section in the LFS book about this, although it doesn’t really recommend any particular option. Nevertheless when reading the BLFS book and the email lists, it appears that many LFSers use the “DESTDIR” approach as documented in LFS. It is probably not worth using this for the basic software in the LFS book itself, but well worth considering before installing anything from BLFS.
The options discussed in the LFS book are all rather unusual for people (like me) used to mainstream distributions such as Debian, Ubuntu or Fedora. Most of the options discussed in LFS could be named “package loggers” rather than “package managers”, ie they just track what was installed and when.
It is theoretically possible to install rpm or dpkg on LFS, but you’d be pretty alone - the LFSers generally like the bare-bones approach.
One person has documented how to install the Debian apt and dpkg tools on LFS, if you are really determined to do so. It might be a good way to get familiar with dpkg/apt..
And of course installing rpm on LFS has also been documented.
Section 5.2: Target Triple
The “target triple” bit has me confused. Running config.guess from binutils gave x86_64-unknown-linux-gnu.
The autoconf manual: Target Triplets and linked page System Type have a reasonable description. From this, I could decode the output of config.guess for my machine as:
- cpu: x86_64
- company: unknown
- system (divided into kernel-os): linux-gnu
I have no idea what a compiler would do with the “company” string. I think that “system” here indicates the convention used for making syscalls and function-calls on this platform (ie which registers are used, who cleans up the stack, etc), but am not 100% certain.
Section 5.3: shell
5.3 states that “/bin/sh” must be a symlink to bash. However on my system (deb8) it is a link to dash. There were no hints on how to elegantly resolve this, other than rewiring sh for my entire system (which I did, but reluctantly).
Note in particular, that commands such as cp -uv $file{,.orig} are used extensively, and this braces-format is a bash extension that will not work with dash.
Section 6.2.1: /dev
I was a bit puzzled why dev/console and dev/null were created when the host /dev was then mounted on top of it anyway. And after rebooting, devtmpfs will be mounted on /dev. So when would these files ever be used?
Section 6.2.3: devpts
I would have appreciated a bit more background on ‘devpts`. This link provides at least some information:
Section 6.3.3 : relative files
Don’t understand. Why would a system have “files that depend on the position of files on a disk system”?
GLIBC and NCSD
An “/etc/nscd.conf” (name service cache daemon) provided by glibc. What does it do?
- related to systemd “nscd.service” unit file => /usr/sbin/ncsd
- daemon appears to depend on passwd,group,hosts,services,netgroup
- it is a daemon that maintains a cross-user cache of “lookup info” of various sorts, including hostnames.
- glibc calls that look up such information query the daemon (if it is available) for performance reasons
Glibc functions which use the daemon include:
- getgrgid, getgrname -> get group info by id or name
- gethostbyname, gethostbyaddr -> get host info
- getservbyname, getservbyport -> get service info by name or port
Actually, glibc provides more than just a library. Interesting things include:
- locale, tzselect, zic – timezone stuff
- nscd – caching daemon
- catchsegv (prints stacktrace for apps that terminate with segfault)
- libc – the “primary” output
- libcrypt – cryptography library
- libmcheck – memory allocation checking
- libpthread – posix threads
Section 6.17.1
Why are links created like “ln -sv ../usr/bin/cpp /lib” rather than “ln -sv /usr/bin/cpp”? They are effectively the same, but isn’t an absolute path easier?
Other Notes
mount-point leakage
Although the compilation in chapter 6 is intended to rid all binaries of traces of the host operating system, it appears that the applications generated by the e2fsprogs package get the mount point of the chroot filesystem embedded into them - ie the path in the host. Don’t worry about this; this path is only present in debug entries and will disappear if /tools/bin/strip is run on the apps (as recommended in section 6.??).
Downloading LFS Files
There are some security issues regarding downloading files - at least LFS doesn’t follow good security practice IMO. Section3.1 recommends checking the downloaded files against an “md5sums” file. However in the downloaded HTML book version, the link is to a file included in the original download. A hacked book could provide a hacked md5sums file. The original directory I downloaded from has the same md5sums file - but that dir is “http” not “https” so content can easily be manipulated via a man-in-the-middle. All unlikely, but perhaps LFS should be a good example of best-practice? - and LFS is mirrored, which means a compromised mirror could do significant damage.
As noted earlier, LFS also recommends running wget as root when downloading the initial set of software. This doesn’t feel like a good idea to me.
Compiling glibc
When running glibc make check the test ntpl/tst-robust8 failed for me, although the “example test output” showed it works. I continued anyway, and haven’t noticed any problems yet.
Section 6.28: e2fsprogs without swapspace
In e2fs “make check”, test “ss” fails: “Regression test for ss library failed!” Presumably this happens when (like me) no swap-space is configured (I have 16GB ram, no need for swap).
Section 6.49 Gettext
Compilation failed on the first pass: makefile variable “dependency_dirs” somehow got set to the mount point of the chroot-filesystem. I recompiled libtool (thinking this might have something to do with it), then recompiled gettext and it worked. Not sure if the libtool rebuild actually did anything, or whether a simple rebuild of gettext would have worked anyway…
Or maybe I screwed up the “–prefix” entry?
Note also that e2fsprogs encodes the mount point of the chroot-filesystem into its binaries, which confused me at first. However these are just debug symbols that get stripped later.
Section 6.61: compiling Make
“make check” failed one test consistently.
misc/fopen-fail .........................................
Test timed out after 5 seconds
Error running /sources/make-4.1/tests/../make (expected 512; got 14): /sources/make-4.1/tests/../make -f work/misc/fopen-fail.mk
Caught signal 14!
FAILED (0/1 passed)
The error-log in “tests/work/misc/fopen-fail.mk” says *** Too many open files. Stop. However the open-file-limit seems adequate:
root:/sources/make-4.1/tests/work# cat /proc/sys/fs/file-max
1605663
root:/sources/make-4.1/tests/work# ulimit -Hn
65536
Section 7.3 Device and Module Handling
In general, great info - including useful background.
In 7.3.2.2, suggested minor changes:
“Device files are created by the kernel by the devtmpfs..” -> “Device files are created automatically by the kernel in the devtmpfs when a driver allocates (major,minor) device numbers. The created file is always owned by root, has mode 0600, and the name of some devices varies depending upon order of discovery” (see 7.3.3.7 and 7.4.1).
A “uevent” is a specially-formatted string sent by the kernel to a netlink socket; the string contains the id of the corresponding sysfs node which provides further information about that device. The udev program is expected to listen on this netlink socket.
udev can maintain a persistent mapping of (device,name) in order to provide “stable names” for devices such as network-cards (which may be discovered in different orders on different boots) or usb-sticks (which should be assigned the same name regardless of which port they are connected to).
Warning: the symlink approach currently has problems with suspend/resume: when an app has opened a device via a symlink, it holds a filehandle that refers to the symlink target. On resume, devices are re-enumerated resulting in devices potentially being allocated different names. udev will then recreate the “stable-named” symlinks pointing to different targets. An app opening the “stable symlink name” would therefore connect to the correct device - but the open filehandle now points to a different device!
Time Required to Build
One SBU (standard build unit) = 2 minutes on my laptop. Compiling GCC first pass took 12 minutes => 6 SBU. The slowest project by far is gcc : 63 SBU according to the docs which worked about approximately correct - 2hrs on my reasonably fast laptop with SSD
When building “from scratch” the problem is that compiling binutils source requires a compiler - but the compiler requires binutils. The solution is to build a temporary binutils with the host compiler, then build a temporary gcc with the custom binutils, then build a final binutils with the temporary compiler and temporary binutils, then build a final gcc with the temporary gcc and final binutils. Phew!
Minor points
I learnt that addr2line (from binutils) can map a program address back to sourcecode file/line (requires debug-info). Useful for crashing apps! (see also catchsegv)