One of my favorite DevOps tools, Chef, is gaining popularity by the day. Github alone has over a thousand Chef cookbooks publicly accessible.

Starting to use Chef is easy. Realizing the full power of Chef is not quite as easy. Many Chef cookbooks that I looked at can be significantly improved. Cookbooks created by developers can be improved by using systems engineering patterns. Cookbooks created by system administrators can be improved by using software engineering patterns.

Opscode does a very good job describing building blocks of Chef cookbooks. Explaining how to write good Chef cooks is something I have not seen done yet. In this post I will go over my cookbook writing process using a syslog-ng cookbook as an example.

Syslog-ng is a server that talks syslog protocol, filters/reformats incoming messages, and forwards/writes results. Syslog-ng has several alternatives: a more open Rsyslog, good-old Syslogd, and upcoming not-exactly-replacements like Logstash. What makes syslog-ng appealing, in my opinion, is the set of features syslog-ng provides, stability, and configuration clarity.

Writing a cookbook starts with generating a cookbook directory structure:

# knife cookbook create cookbook-syslog-ng -o cookbooks/ -r md
The -o option creates the cookbook locally (vs. the cookbook_path specified in ~/.chef/knife.rb) and -r md selects Markdown as a format (Rdoc is the default). Chef, by convention, divides cookbooks into two types: generic cookbooks and site-specific cookbooks. Syslog-ng will be a generic cookbook. Chef's chef-repo directory structure looks like the following:
chef-repo/
   certificates/
   config/
   cookbooks/       # <-- generic cookbooks
   data_bags/
   environments/
   roles/
   site-cookbooks/  # <-- site-specific cookbook
Knife cookbook create cookbook-syslog-ng -o cookbooks/ -r md command creates the following directory structure in the cookbooks directory:
cookbook-syslog-ng/
   README.md
   attributes/
   definitions/
   files/
   libraries/
   metadata.rb
   providers/
   recipes/
   resources/
   templates/
A naive implementation of syslog-ng cookbook is a simple installation of syslog-ng package. All that has to be done in the naive implementation is to create a default recipe, cookbook-syslog-ng/recipes/default.rb:
package "syslog-ng"
The problem with the naive implementation is that it does not do anything useful besides installing a package. Chef is a configuration management system, and one of the goals of the configuration management system is to enforce configuration. A good Chef cookbook enforces file/directory permission, content of configuration files, services running on a system, etc.

A less naive but still naive implementation is cookbook-syslog-ng/recipes/default.rb:

package "syslog-ng"

cookbook_file "/etc/syslog-ng/syslog-ng.conf" do
  owner root
  group root
  mode 00640
end

cookbook_file "/etc/init.d/syslog-ng" do
  owner root
  group root
  mode 00755
end

service "syslog" do
  action [ :disable, :stop ]
end

service "rsyslog" do
  action [ :disable, :stop ]
end

service "syslog-ng" do
  supports :restart => true, :status => true
  action [ :enable, :start ]
end

This version enforces syslog-ng configuration, makes sure that the syslog-ng service is running, and disables syslog/rsyslog services (enabled by default on RedHat/CentOS). However, this version is not good enough because the cookbook is very inflexible (users and filenames are statically defined), and because the cookbook still does not do too much besides installing syslog-ng.

The next step in the cookbook evolution is to parametrize user, group, and file names:

# cookbook-syslog-ng/recipes/default.rb
package "syslog-ng"

cookbook_file "#{node[:syslog_ng][:config_dir]}/syslog-ng.conf" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00640
end

cookbook_file "/etc/init.d/syslog-ng" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00755
end

service "syslog" do
  action [ :disable, :stop ]
end

service "rsyslog" do
  action [ :disable, :stop ]
end

service "syslog-ng" do
  supports :restart => true, :status => true
  action [ :enable, :start ]
end

# cookbook-syslog-ng/attributes/default.rb
default[:syslog_ng][:user]          = "root"
default[:syslog_ng][:group]         = "root"
default[:syslog_ng][:config_dir]    = "/etc/syslog-ng"

Chef cookbooks are parametrized through attributes. Attributes can be defined/changed per cookbook, per environment, per role, and per node.

The cookbook now enforces configuration, provides flexibility through attributes, but still does not do much besides the basic installation and configuration of syslog-ng. As a service, syslog-ng is very much similar to Nginx. Nginx is an HTTP router, and syslog-ng is a syslog router. Nginx configuration can be customized per application. The next feature to add to the syslog-ng cookbook is the ability to include application-specific syslog-ng configurations.

Take an application server node as an example. The system on the node is configured by Chef. The syslog-ng service is configured and installed. An application is added to the server that has specific logging requirements: logging messages generated by the application need to be logged to a separate file on the server and also copied to a centralized logging server via syslog. Syslog-ng configuration on the server needs to be changed based on the application logging requirements.

A traditional syslog configuration approach is to leverage the syslog facility to separate application and system logs. With the limited number of syslog facilities available, this tradition configuration approach does not scale to more than a dozen applications running on a server. Fortunately, syslog-ng features allow to implement an approach similar to HTTP application configuration.

Adam Wiggins, in the Twelve-Factor App gives a very good description of scalable HTTP application configuration: Port Binding. The idea is very simple - configure every distinct HTTP application to use a specific port. Extending this idea to syslog-ng yields a syslog-ng configuration where each distinct application sends logs to a specific syslog-ng port. When an application is added to the server, application-specific syslog-ng configuration is added to the base syslog-ng configuration.

The simplest and the most robust way of adding extra bits to existing service configuration is to make the service source all files out of a configuration directory. Unlike many of the modern unix services, syslog-ng does not support a configuration directory out of the box. A workaround is to keep the syslog-ng main configuration file empty and change syslog-ng control scripts to copy all files from the configuration directory to the syslog-ng main configuration file on start/restart/reload:

start() {
  echo -n "Starting syslog-ng: "
  cat /etc/syslog-ng/conf.d/* > /etc/syslog-ng/syslog-ng.conf
  daemon $binary
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/syslog-ng
}
The cookbook recipe becomes:
package "syslog-ng"

cookbook_file "#{node[:syslog_ng][:config_dir]}/syslog-ng.conf" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00640
end

cookbook_file "/etc/init.d/syslog-ng" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00755
end

directory "#{node[:syslog_ng][:config_dir]}/conf.d" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00750
  action :create
end

cookbook_file "#{node[:syslog_ng][:config_dir]}/conf.d/00base" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00640
end

directory "#{node[:syslog_ng][:log_dir]}" do
  owner node[:syslog_ng][:user]
  group node[:syslog_ng][:group]
  mode 00755
  action :create
end

service "syslog" do
  action [ :disable, :stop ]
end

service "rsyslog" do
  action [ :disable, :stop ]
end

service "syslog-ng" do
  supports :restart => true, :status => true
  action [ :enable, :start ]
end

Dynamic reconfiguration of existing services based on application-specific needs can be done in two ways in Chef: definitions and resources/providers. Looking at the Chef cookbook directory structure listed earlier, definitions, resources and providers have pre-configured directories setup. Definitions and resources/providers (aka LWRP) differ in two ways: loading order and state information. Resources/providers are loaded earlier than definitions and are meant to represent an object with state information. Application-specific syslog-ng configuration bits do not have state information. Thus, adding a cookbook definition is a more appropriate solution in this case.

# cookbook-syslog-ng/definitions/syslog_ng_app.rb
define :syslog_ng_app, :template => "syslog_ng_app.erb" do
  include_recipe "syslog-ng"
  
  application = {
    :name => params[:name],
    :index => params[:index] || "02",
    :cookbook => params[:cookbook] || "syslog-ng",
    :host => params[:host] || "127.0.0.1",
    :port => params[:port] || "514",
    :log_base => params[:log_base] || node[:syslog_ng][:log_dir]
  }
  
  directory "#{application[:log_base]}" do
    owner node[:syslog_ng][:user]
    group node[:syslog_ng][:group]
    mode 00755
    action :create
  end
  
  directory "#{application[:log_base]}/#{application[:name]}" do
    owner node[:syslog_ng][:user]
    group node[:syslog_ng][:group]
    mode 00755
    action :create
  end
  
  template "#{node[:syslog_ng][:config_dir]}/conf.d/
   #{application[:index]}#{application[:name]}" do
    source params[:template]
    owner node[:syslog_ng][:user]
    group node[:syslog_ng][:group]
    mode 00640
    cookbook application[:cookbook]
    
    if params[:cookbook]
      cookbook params[:cookbook]
    end
    
    variables(
      :application => application,
      :params => params
    )
    
    notifies :restart, resources(:service => "syslog-ng"), :immediately
  end
end
The definition above implements application-specific configuration. To configure syslog-ng, the following code will need to be added to the application cookbook:
include_recipe "syslog-ng"

syslog_ng_app "newapplication" do
  index "04"
  port "515"
end

Syslog_ng_app in the example above adds an application configuration from the default template to the syslog-ng configuration, starts a syslog listener on port 515, and creates application log directories.

The last version of the cookbook is

  • Flexible. The cookbook is parametrized through the use of templates and attributes.
  • Extendible. The cookbook defines a uniform interface to extend the configuration.
  • Accessible. The cookbook sets reasonable defaults making new interfaces easy to use.
  • Feature-rich. The cookbook adds new interfaces.
  • Forceful. The cookbook controls and enforces service configuration.

The characteristics listed above is what sets good Chef cookbooks apart.

The full source code of the syslog-ng cookbook can be found at


Leave a Reply

Blogger Templates for WP 2 Blogger sponsored by Cinta.
Content Copyright © 2010 - 2011 Artem Veremey, All Rights Reserved
preload preload preload