DevOps Notes

Building AWS AMIs from scratch is the practice many security-focused organizations strictly follow.

I have been building custom AWS AMIs from scratch for my personal AWS environment for many years. Looking at my personal AMI collection, I have custom AMIs dating back to CentOS 5.

A few days ago I decided to do another refresh of my AWS environment and started with a new and improved build of CentOS 7.9. I rehashed my packer build merging in several improvements. OVA creation went flawless and AWS vmimport successfully finished. When I booted my new AMI to upgrade CentOS stock kernel to the latest kernel-lt 5.4, I noticed that DNS resolution is failing and that /etc/resolv.conf did not get updated correctly:

$ cat /etc/resolv.conf
# Generated by NetworkManager

I had to troubleshooting and fix DHCP on my new image.

I have a strange attachment to Nagios web interface. Even though we kicked Nagios itself out of our infrastructure some time ago, we are still keeping a more moden fork of Nagios, Icinga, around as a cornerstone of our event-monitoring infrastructure. With quite a bit of effort spent writing cookbooks and monitors, Icinga is fully automated, and I can enjoy watching and showing my Nagios Web.

Nagios is loved and hated in the industry. Nagios has a great brand recognition, but suffers from less than intuitive setup and difficult to automate configuration. With passive checks and right Chef cookbooks, Nagios/Icinga setup becomes very effective.

Another Nagios weakness is poor implementation. Fortunately, Nagios poorly implements right ideas, so not everything is lost. This war story is about fixing Nagios NSCA implementation. Nagios NSCA is a Nagios passive-check daemon written in C that runs on a Nagios server and processes all passive check sent by Nagios clients.

My Linux-focused list of kernel management tasks includes:

• Following LKML (Linux Kernel Mailing List) for at least the kernel branch of my choice,
• Following NETDEV list because significant portion of my tasks revolves around network,
• Using GitWeb to port drivers and updates to the custom tree,
• Maintaining a custom kernel source tree,
• Building and packaging custom kernels to run in production.