DevOps Notes

DevOps Days 2010 had a great pannel discussion on Infrastructure as Code. The responses to one question (asked at 27:30) did surprise me. The question was "Who here goes to kernel.org grabs the sources and compiles them?" No a single panelist raised a hand. I find it shocking that all distinguished DevOps evangelists on the panel are running stock distribution kernels.

I have a hard time picturing a competent production environment running a stock distribution because of the following:

• Stock kernels in stable distributions rarely support new types of hardware. Upgrading or adding support for disk controller and network interface card drivers requires a custom kernel build.
• Development distribution branches (e.g. Fedora 14) provide fairly up-to-date kernels. However, running development distributions in production is a recipe for disaster and/or massive overtime.
• At least 30% of CPU time in production is spent running kernel code. Custom kernel optimizations have direct impact on the bottom line.
• Troubleshooting of operational issues oftentimes requires significant knowledge of kernel internals and kernel development process.

My advice to all developers and operations engineers is to rigorously study kernel internals of the systems used in production.