My Linux-focused list of kernel management tasks includes:

  • Following LKML (Linux Kernel Mailing List) for at least the kernel branch of my choice,
  • Following NETDEV list because significant portion of my tasks revolves around network,
  • Using GitWeb to port drivers and updates to the custom tree,
  • Maintaining a custom kernel source tree,
  • Building and packaging custom kernels to run in production.

I start my custom kernel source tree by choosing the most advanced stable, feature-rich branch. After checking out the selected branch, and setting up the build process, I run my custom kernel though a set of functional, compatibility, integration and stress tests adding patches when necessary. When the kenel build passes all tests, the custom build tree becomes stable. Stable build tree is modified only based on the specific need - to add security patches, to add driver updates, to implement specific bug fixes. Limiting modifications to the certified build is essential to controlling production risks. A well-maintained kernel build tree allows to resolve problems quickly and maintain a high level of confidence in production environment.

GitWeb is an outstanding tool for figuring out patch dependancies. For example, my custom kernel tree is based on linux-2.6.32 branch and I want to upgrade bnx2 network driver to the latest version. First, I find the linux-2.6.32 branch on GitWeb. Next, I browse to the bnx2 driver source code - drivers/net. Looking at the History of the specific source file - bnx2.c in this case - I get the list of patches I need to look at to bring me to the desired version of the driver. I study the patches, netpoll crash for example, and apply selected patches in order.

The majority of production systems in my environment are based on RedHat Linux distribution. Thus, my kernel build process generates custom kernel RPM packages.

My kernel build process follows these steps:

  1. Checkout the top of the custom kernel branch with git.
  2. Make adjustments to the .config kernel configuration.
  3. Build and package the kernel with make rpm-pkg.
# Make rpm command
make CONFIG_DEBUG_SECTION_MISMATCH=y rpm-pkg 2> build-error.log \
  | tee build-out.log

Kernel RPM packaging script is incomplete, so I use a modified scripts/package/mkspec.

I add generation of three packages: headers, devel and debug. The headers package is used to build user-space applications, the devel package is needed to build applications with complex kernel dependancies (e.g. drivers), and the debug package provides files to run kernel profiling and systemtap.

The source code for my scripts/package/mkspec can be viewed below.

# Output a simple RPM spec file that uses no fancy features requring
# RPM v4. This is intended to work with any RPM distro.
# The only gothic bit here is redefining install_post to avoid
# stripping the symbols from files in the kernel which we want
# Patched for non-x86 by Opencon (L) 2002

# how we were called determines which rpms we build and how we build them
if [ "$1" = "prebuilt" ]; then

# starting to output the spec
if [ "`grep CONFIG_DRM=y .config | cut -f2 -d\=`" = "y" ]; then

__KERNELRELEASE=`echo $KERNELRELEASE | sed -e "s/-//g"`

echo "Name: kernel"
echo "Summary: The Linux Kernel"
echo "Version: $__KERNELRELEASE"
# we need to determine the NEXT version number so that uname and
# rpm -q will agree
echo "Release: `. $srctree/scripts/mkversion`"
echo "License: GPL"
echo "Group: System Environment/Kernel"
echo "Vendor: The Linux Community"
echo "URL:"

if ! $PREBUILT; then
echo "Source: kernel-$__KERNELRELEASE.tar.gz"

echo "BuildRoot: /var/tmp/%{name}-%{PACKAGE_VERSION}-root"
echo "Provides: $PROVIDES"
echo "%define __spec_install_post /usr/lib/rpm/brp-compress || :"
echo "%define debug_package %{nil}"
echo ""
echo "%description"
echo "The Linux Kernel, the operating system core itself"
echo ""

echo "%package headers"
echo "Summary: Header files for the Linux kernel for use by glibc"
echo "Group: Development/System"
echo ""

echo "%description headers"
echo "Kernel-headers includes the C header files that specify the interface"
echo "between the Linux kernel and userspace libraries and programs.  The"
echo "header files define structures and constants that are needed for"
echo "building most standard programs and are also needed for rebuilding the"
echo "glibc package."
echo ""

echo "%package devel"
echo "Summary: Development package for building kernel modules to match the kernel."
echo "Group: System Environment/Kernel"
echo ""

echo "%description devel"
echo "This package provides kernel headers and makefiles sufficient to build modules"
echo "against the kernel package."
echo ""

echo "%package debug"
echo "Summary: Linux kernel debug files"
echo "Group: Development/System"
echo ""

echo "%description debug"
echo "The debug package provides vmlinux and /usr/lib/debug/lib/modules."
echo ""

if ! $PREBUILT; then
echo "%prep"
echo "%setup -q"
echo ""

echo "%build"

if ! $PREBUILT; then
echo "make clean && make %{?_smp_mflags}"
echo ""

echo "%install"
echo "%ifarch ia64"
echo 'mkdir -p $RPM_BUILD_ROOT/boot/efi $RPM_BUILD_ROOT/lib/modules'
echo 'mkdir -p $RPM_BUILD_ROOT/lib/firmware'
echo "%else"
echo 'mkdir -p $RPM_BUILD_ROOT/boot $RPM_BUILD_ROOT/lib/modules'
echo 'mkdir -p $RPM_BUILD_ROOT/lib/firmware'
echo "%endif"

echo 'INSTALL_MOD_PATH=$RPM_BUILD_ROOT make %{_smp_mflags} KBUILD_SRC= modules_install'
echo "%ifarch ia64"
echo 'cp $KBUILD_IMAGE $RPM_BUILD_ROOT'"/boot/efi/vmlinuz-$KERNELRELEASE"
echo 'ln -s '"efi/vmlinuz-$KERNELRELEASE" '$RPM_BUILD_ROOT'"/boot/"
echo "%else"
echo "%ifarch ppc64"
echo "cp vmlinux arch/powerpc/boot"
echo "cp arch/powerpc/boot/"'$KBUILD_IMAGE $RPM_BUILD_ROOT'"/boot/vmlinuz-$KERNELRELEASE"
echo "%else"
echo "%endif"
echo "%endif"


echo 'cp .config $RPM_BUILD_ROOT'"/boot/config-$KERNELRELEASE"
echo 'cp Module.symvers $RPM_BUILD_ROOT'"/boot/Module.symvers-$KERNELRELEASE" 

echo 'mkdir -p $RPM_BUILD_ROOT'"/usr/src/kernels"
echo 'pushd $RPM_BUILD_ROOT'"/lib/modules/$KERNELRELEASE/build"
echo "find . | cpio -pmadv "'$RPM_BUILD_ROOT/'"/usr/src/kernels/$KERNELRELEASE-x86_64"
echo 'popd'

echo 'pushd $RPM_BUILD_ROOT'"/usr/src/kernels/$KERNELRELEASE-x86_64"
echo "make clean"
echo "rm -f linux-$KERNELRELEASE"
echo "rm -f make-err.log"
echo "rm -f make-out.log"
echo "rm -f .version"
echo "make oldconfig"
echo "make prepare"
echo "popd"

echo 'rm $RPM_BUILD_ROOT'"/lib/modules/$KERNELRELEASE/build"
echo 'rm $RPM_BUILD_ROOT'"/lib/modules/$KERNELRELEASE/source"

echo 'pushd $RPM_BUILD_ROOT'"/lib/modules/$KERNELRELEASE"
echo 'ln -s build source'
echo 'ln -s ../../../usr/src/kernels/$KERNELRELEASE-x86_64 build'
echo 'popd'

#echo "%ifnarch ppc64"
#echo 'cp vmlinux vmlinux.orig'
#echo 'bzip2 -9 vmlinux'
#echo 'mv vmlinux.bz2 $RPM_BUILD_ROOT'"/boot/vmlinux-$KERNELRELEASE.bz2"
#echo 'mv vmlinux.orig vmlinux'
#echo "%endif"

echo 'mv vmlinux $RPM_BUILD_ROOT'"/usr/src/kernels/$KERNELRELEASE-x86_64"
echo 'mkdir -p $RPM_BUILD_ROOT/'"/usr/lib/debug/lib/modules/"
echo 'ln -s '"/usr/src/kernels/$KERNELRELEASE-x86_64/vmlinux "'$RPM_BUILD_ROOT/'"/boot/vmlinux-$KERNELRELEASE"
echo 'ln -s '"/lib/modules/$KERNELRELEASE "'$RPM_BUILD_ROOT/'"/usr/lib/debug/lib/modules/$KERNELRELEASE"

echo ""
echo "# Install kernel headers"
echo 'make INSTALL_HDR_PATH=$RPM_BUILD_ROOT'"/usr headers_install"
echo ""

echo "# glibc provides scsi headers for itself, for now"
echo 'rm -rf $RPM_BUILD_ROOT'"/usr/include/scsi"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/asm*/atomic.h"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/asm*/io.h"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/asm*/irq.h"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/*/.install"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/.install"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/*/..install.cmd"
echo 'rm -f $RPM_BUILD_ROOT'"/usr/include/..install.cmd"
echo ""

echo ""
echo "%clean"
echo 'rm -rf $RPM_BUILD_ROOT'
echo ""
echo "%files"
echo '%defattr (-, root, root)'
echo "%dir /lib/modules"
echo "/lib/modules/$KERNELRELEASE"
echo "/lib/firmware"
echo "/boot/*"
echo ""
echo "%files headers"
echo "%defattr(-,root,root)"
echo "/usr/include/*"
echo ""
echo "%files devel"
echo "%defattr(-,root,root)"
echo "%dir /usr/src/kernels"
echo "/usr/src/kernels/$KERNELRELEASE-x86_64"
echo ""
echo "%files debug"
echo "%defattr(-,root,root)"
echo "/usr/src/kernels/$KERNELRELEASE-x86_64/vmlinux"
echo "/usr/lib/debug/lib/modules/$KERNELRELEASE"
echo "/boot/vmlinux-$KERNELRELEASE"

echo "%post"
echo "if [ \`uname -i\` == \"x86_64\" -o \`uname -i\` == \"i386\" ]; then"
echo "  if [ -f /etc/sysconfig/kernel ]; then"
echo "    /bin/sed -i -e 's/^DEFAULTKERNEL=kernel-smp$/DEFAULTKERNEL=kernel/' /etc/sysconfig/kernel || exit \$?"
echo "  fi"
echo "fi"
echo "/sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install "$KERNELRELEASE" || exit \$?"
echo ""

echo "%preun"
echo "/sbin/new-kernel-pkg --rminitrd --rmmoddep --remove "$KERNELRELEASE" || exit \$?"
echo ""

Leave a Reply

Blogger Templates for WP 2 Blogger sponsored by Cinta.
Content Copyright © 2010 - 2011 Artem Veremey, All Rights Reserved
preload preload preload