Wednesday, December 28, 2011

Machine Names

Fully Qualified Machine Name

Machines on our network obtain their fully qualified domain names from a DHCP server. But where does the DHCP server obtain its name? To configure a machine to return a fully qualified domain name (, you need to edit /etc/hosts horn localhost   horn

Order matters. This will ensure that your fqdn can be queried by Python's socket.getfqdn().

Tuesday, December 27, 2011

Vim Setup

Please install Dropbox before continuing.


Install vim with

$ sudo apt-get install vim

I like to have access to same vim configuration on every machine I use. I created the following structure in my DropBox folder (Note trailing slashes indicate folders):


Next, I created symlinks from my home folder to my Dropbox vim files:

$ ln -s ~/Dropbox/config/vim/.vimrc .
$ ln -s ~/Dropbox/config/vim/.vim

When I make changes to my vim configuration, it immediately synchronizes to all of my other machines.


SnipMate is a plugin that provides sophisticated snippet capabilities in vim. It utilizes vim’s file type detection and wires the appropriate snippets in insert mode.

Download the plugin and follow the installation instructions:

$ unzip -d ~/.vim

Be sure that set paste is not enabled in your .vimrc file.

Study the included snippets in $HOME/.vim/snippets. To add a new file type, open the file e.g. and query vim

:set filetype?

It returns a type (in this case mkd). Create a new file called mkd.snippets in $HOME/.vim/snippets and add your file type specific snippets.

Since you’ve configured this with Dropbox, your snippets will be available everywhere.

Thursday, December 22, 2011

Developer Documentation

We used to store all of our documentation on one wiki but soon realized that we were serving two different audiences: end users and developers. We discovered a nice wiki feature in BitBucket and we now separate end user and developer documentation between Mindtouch and BitBucket respectively.

BitBucket also has convenient REST APIs that make it easy to integrate command line scripts with their source repository, issue tracker and wiki.

BitBucket Wiki

The BitBucket wiki uses the Creole syntax. It is a convenient place to store documentation for source code. I was not aware that it's possible to nest Mercurial repositories. Nesting allows us to conveniently layout our projects as follows (assume our project is called myproj):

$ hg clone
$ cd myproj
$ hg clone

This produces an independent wiki Mercurial repository at the root of myproj. You can now edit these files without switching to your browser.

BitBucket Issue Tracker

Although it's not as sophisticated as Bugzilla, the issue tracker in BitBucket is sufficient for small teams. And the REST APIs make automation relatively easy.

Tuesday, December 13, 2011

Postfix with Amazon SES on Ubuntu 10.04 LTS

We use Postfix as our MTA. However, most ISPs do not allow mail relay so we needed an alternate transport mechanism. Although we use Google Apps, there were restrictions on the from address and hard message limits that we wanted to avoid. This article describes how to integrate Postfix and Amazon's SES.

Create an AWS Account

At the upper right corner of the Amazon SES page, you'll see a link to Create an AWS Account.

Once you've created your account, go to your Security Credentials page and create an Access Key. You'll need your Access Key ID and Secret Access Key. Please make a note of these two values.

Verify Addresses

Next, go the SES Management Console and verify addresses that you intend to use with SES. For example, we use Google Apps to host our domain. We've created groups that we use for various alerts. We added those addresses here.

Install Postfix

Be sure that you've added the fimero repository to your apt list.

Next, install the bbpostfix

$ sudo apt-get install bbpostfix

Just go with the defaults for the Postfix configuration. Once the Postfix configuration completes, you'll be asked for your MX domain. This should be the registered domain that you want to use for your outgoing email.

Package Source

The package sources are available here:

Monday, November 14, 2011

Source Control with Mercurial and BitBucket

The benefits of source control have been covered in numerous places around the Internet. The reference for Mercurial is available here. We'll simply describe how we manage the process.


Create a BitBucket account. This will provide you with unlimited storage for your source code.

Create a base folder for your projects.


The reasoning for the path segments of this layout are:

  • hgdev enables us to segregate source control systems {e.g. svndev, gitdev}
  • org.bitbucket enables us to segregate hosting providers
  • ${username} is your BitBucket username so you can also pull easily from others' repositories

We'll refer to this folder as your BitBucket folder (the BitBucket folder under your username).

New Repositories

To work with a new repository:

  1. Log onto BitBucket and create the project (e.g. myproject) there.
  2. cd into your BitBucket folder.
  3. Clone your newly created project
    hg clone https://${username}${username}/myproject

Bash Development Tools

We introduce our organization of custom bash scripts for development.

Script Types

Bash is essential for automating tasks in Ubuntu. Generally, you'll work with three kinds of bash scripts:

  • Distribution scripts—these come with your Linux distribution
  • Deployed scripts—these come with deb packages that you install
  • Personal scripts—these are fully under your control and live in your home folder

You'll need a folder where you store your personal bash scripts. As with any code, this should be placed under source control.

Please see our short guide on source control and ensure your environment is set up appropriately.

Personal Bash Repository

In the interest of consistency, let's choose mysh as the project name for personal scripts. Create your project on your BitBucket account. Then check out your new project into


Now modify your $HOME/.bashrc to include this folder on your PATH

export PATH=$HOME/hgdev/org.bitbucket/${user}/mysh:$PATH

You'll need to reload your .bashrc for this to take effect. You can just open a new terminal.

Let's create a sample program to test our configuration.

#!/usr/bin/env bash
echo "hello"

This should be in your mysh folder. Make this script executable

chmod +x

Let's change folders to our home.


This should execute your program.

Eclipse Setup for Bash

ShellEd is a good plugin for Eclipse for editing bash scripts. However it can be tricky to install. We assume Eclipse Indigo Service release 1.


ShellEd requires Linux Man Tools.

Add Update Site

Go to Help -> Install New Software... ->

Linux Tools

Install the Linux Man Tools. Then install ShellEd


Using Eclipse for Python on Ubuntu 10.04 LTS

We use Eclipse for Python, Java and C++ development. Eclipse is a fairly large, complicated beast. To stay true to the Unix Philosophy, we use command line tools wherever possible. Also, to reduce potential conflicts between tools and plug-ins, we break up the workspaces so that each development environment has a dedicated workspace.

Download Eclipse

Go to the Eclipse website and download the Eclipse IDE for Java EE Developers. This version comes with a collection of plug-ins that support database access, XML editing, and various other common tasks that you'll need.

Once downloaded, create a folder in your home directory called local/eclipse. Unzip the package you downloaded into this folder.

% cd $HOME/local/eclipse
% tar zvxf $HOME/Downloads/eclipse-jee-indigo-SR1-linux-gtk-x86_64.tar.gz

This unpacks a folder called eclipse. Rename it to eclipse-indigo.

% mv eclipse eclipse-indigo
% ln -s eclipse-indigo current

You will now have a folder layout that looks like the following:

        /eclipse/current -> eclipse-indigo


We've gone to this trouble to make it easier to write scripts that involve Eclipse—even if we upgrade, we only need to point to $HOME/local/eclipse/current. As we mentioned earlier, Eclipse is a complicated application that often requires environmental modifications and parameters tweaks. Rather than typing these out each time, we'll capture our requirements in a bash script.

Eclipse bash Launch Script

Please see the Bash Development Tools article.

Create a script called in mysh.

You can now launch Eclipse with the appropriate parameters and environmental variables &

When you start your Eclipse session, be sure to set the appropriate path to your workspace. In this case, it's $HOME/workspaces/python


PyDev is a useful extension for Eclipse that provides support for code completion and debugging. Once you've launched your Python workspace, install the Pydev extension.

First Project

Let's create our first project. Do not create projects in your workspace! The workspace contains settings unrelated to your project—it's best to keep them separate.

Create a repository for your Python project in BitBucket and check it out into your BitBucket folder. For example, pywork.

  • From Eclipse, create a new PyDev project. Give it the same name as your BitBucket repository (pywork).
  • Uncheck Use default and select the root of pywork.
  • Ensure that Add project directory to PYTHONPATH? is selected.

You'll now have a new PyDev project that you can also add to your shell PYTHONPATH.

Wednesday, November 9, 2011

Bind9 and DHCP3 on Ubuntu 10.04 LTS

We continue documentation of a full Linux server stack. You should consider testing this setup using an experimental network on VirtualBox.

Blackbox Bind9 Package

Make sure you've configured your /etc/apt/sources.list appropriately.

The Blackbox Bind9 package configures a bind9 server for Ubuntu 10.04 LTS. Combined with DHCP3, you'll be able to join machines to your network and have them resolve each other as well as external addresses.

The source code for this package is available on-line.

Begin with

$ sudo apt-get install bbbind9

You'll be asked a series of questions. bbbind9 will attempt to guess reasonable defaults for your settings. You can accept them by pressing [Enter]:

Enter the base domain that your DNS server will service.

base domain name { home.lan }: 

Enter the netmask for the network your server will manage (generally it's a /24)

static interface netmask { }:

Your DNS server should have a static ip address. All your other machines can have DHCP assigned addresses.

static ip assigned to bind9 server { }: 

These are the DNS servers you hit for machines outside your domain.

external dns forwarders {;; }: 

Network interface to which service binds.

interface to which static ip is assigned { eth0 }: 

Fully qualified name of this DNS server.

fully qualified name for dns host { stemcell.home.lan }: 

Email address of administrator. Note that this convention does not allow the @ symbol. hostmaster.home.lan should be read as hostmaster@home.lan This should be a mail alias, or a mailbox, where the person(s) maintaining DNS should read mail frequently. Any mail regarding the domain will be sent to the address listed here. The name need not be hostmaster, it can be your normal e-mail address, but the e-mail address hostmaster is often expected to work as well.

email address for hostmaster (no @ allowed, use .) { hostmaster.home.lan }: 

Gateway for the interface that has a static ip address.

static interface gateway { }: 

Subnet for which this DNS server provides naming services.

ip zone for dns domain { 10.0.2 }: 

To confirm everything is working properly, perform the following:


You should receive the results of a query against your new host:


Non-authoritative answer:

Blackbox DHCP3 Package

The Dynamic Host Configuration Protocol enables machines on your network to be automatically configured with IP addresses, DNS, and other features. The Blackbox package is configured to be installed on the same host as bbbind9.

Subnet for which the DHCP3 server will be handing out leases.

subnet of dhcp zone { }: 

DNS domain to be assigned to client.

base domain name { home.lan }: 

DNS server that clients use.

ip address for nameserver to grant to dhcp clients { }: 

Fully qualified name of DNS server (this host).

fully qualified name for dns host { stemcell.home.lan }: 

IP gateway assigned to clients.

gateway that dhcp clients will use { }: 

Pool of addresses handed out to clients. Enter ranges as a tuple ().

available dhcp pool { ('', '') }: 

Try adding a client to your network and confirm that its properties now point to your new Bind9 server and its address is from your DHCP3 address pool.

Tuesday, November 8, 2011

Experimental Networks with pfSense

Virtual Networks

In addition to providing an easy way to test operating systems, VirtualBox also provides a way to test networks.



Turn off VirtualBox's internal DHCP server (via Preferences).

  1. Create a new guest on VirtualBox as a 64-bit FreeBSD instance. pfSense has very light resource requirements.
  2. Assign 128Mb of RAM and 8Gb of storage.
  3. Create two network interfaces. Both should be PCnet ll (Am79C970A)—FreeBSD has native support for these cards. One should be internal (intnet) and the other should be either NAT or Bridged.

Install pfSense

  1. Download pfSense. I used the Chicago mirror.
  2. Install using Quick Install.
  3. Assign one interface le0 to WAN and the second, le1, to LAN.
  4. Then point your browser (from a machine on your internal network) to https://ipaddress

You may want to change the subnet for your experimental network. I use to avoid colliding with 192.168.x.x and networks that are more common.

pfSense also includes a DHCP server by default. If you are testing DHCP services on your internal network, be sure to turn it off. It's listed under Services.

Finally, if you are running DHCP on your lan (host machine network), you may want to set up a DHCP lease reservation for the WAN interface to ensure you always get the same IP address for your experimental network.

Set Static IP on Ubuntu

If you are going to run DHCP on your experimental network, you'll need to assign a static ip to your DHCP server. Edit /etc/network/interfaces

auto eth0
iface eth0 inet static

Note that your values may differ. Then restart networking

$ sudo /etc/init.d/networking restart

Set Full Hostname on Ubuntu

Edit /etc/hosts to provide your machine with a fully qualified domain name:   stemsrv01.lab.lan stemsrv01 localhost   stemsrv01.localdomain   stemsrv01

Note that order matters. See the following post.

Port Forwarding

You may want to access the machines on your experimental network via SSH. To enable this, you'll need to turn on port forwarding on pfSense. But before you do, make sure you uncheck Block private networks on your WAN interface settings.

Then go to Firewall -> NAT and create a new port forwarding rule (in this case, forward WAN port 2222 to 22 on an internal host).

Wednesday, October 26, 2011

Apt Repository


BitBucket has the ability to host static sites. Although Mercurial is better suited for text files than binary debs, BitBucket does make a reasonable apt-repository host for a small collection of packages.

  1. First, create a BitBucket account for your repository. I recommend a separate account from the one you use for source code development. I created an account with username fimero to host artifacts for the fimero project.
  2. Create a repository named {username} so in this case
  3. Clone this repository to your local machine hg clone{username}/{username}
  4. Add an index.html file to this project, commit and push.

Now you can confirm that things work by pointing your browser to

Apt Repository

Next, install reprepro to set up your apt repository.

$ sudo apt-get install reprepro

Our objective is to have a repository with the following configuration

deb fimero main

To create this, cd into your local copy of your repository and create a folder called ubuntu/conf and create a file called distributions.

You are now ready to deploy a package.

$ reprepro -b ${PACKAGE_FOLDER} -V -P 1 -S main includedeb fimero ${debpackage}

Here ${PACKAGE_FOLDER} is your {username} folder and ${debpackage} is the deb file you want to deploy.

The final step is to push these changes to BitBucket.

$ hg commit
$ hg push

If you've added the deb entry from above in your /etc/apt/sources.list, you'll be able to install your newly deployed package on all of your machines.

Tuesday, October 25, 2011

Portion Package Enhancement

Package Enhancement

I created portion to enhance the standard packaging process. The end goal is to create a packaging and deployment process that is easy to orchestrate with CfEngine. portion allows me to package software in a consistent way while continuing to leverage the standard Ubuntu repositories.

Core Packages

The core packages for portion are bbsh2 and bbpython2. These contain bash and python scripts that are used to bootstrap all other packages. They are special packages--all other packages are built using the facilities included in these two packages.

The apt repository is available at

deb fimero main

We also run with badgerports configured on 10.04 LTS.


The source code for bbsh2 is available here. The primary functions used for portion are in


The source code for bbpython2 is available here.

To create a package, first install bbpython2 with

$ sudo apt-get install bbpython2

This will also install bbsh2 via its dependency. Next, create a folder for your package. Be sure to put it under source control. In this tutorial, we'll package a demo script and call it bbdemo

$ mkdir bbdemo

Creating A Package

A Portion package requires the following

  • manifest file
  • payload.txt file

manifest File

Use the manifest file from bbdemo as a template.

payload File

You must generate a payload.txt file that specifies all of the files that are included with your package.

$ {package} {version} {folder}

This script searches the {folder} and create a payload.txt file. Template Template

Assembling Package

The functions to assemble the package are in

$ source
$ epm_make_package

This creates a .deb package in the target folder.

Tuesday, October 18, 2011

Documentation Bootstrap

I am a fan of good documentation—I feel it's the most undervalued component of many projects but I haven't been able to develop a reliable process to:

  • Collect information
  • Format it
  • Keep it up to date

This is my attempt to come up with that process.


I want to edit my documentation in vim. I like it because it's an extremely lightweight and focused tool for text editing. My .vimrc file is


Markdown is a lightweight markup syntax that avoids the annoying open/close brackets. You can get the syntax extensions for vim from here.


I chose Blogger because it is already integrated with my Google Apps Premier account. I needed something basic and reliable. Documentation is divided into two areas—articles hosted by Blogger and articles hosted on my wiki. We'll begin by setting up documentation on Blogger and then use that to host documentation for setting up the wiki. The wiki will, in turn, host documentation the rest of the server stack.


References to BitBucket code are included via BitBucket's embed function. Note—replace the revision number with default to always pull down the tip version.


Code snippets can also be included via gist on GitHub.


Pastebin is also supported:


We will occasionally need to display equations. There are several options including using the utility available here. Or you can use MathJax by adding the following to your page template:

When \(a \ne 0\), there are two solutions to \(ax^2 + bx + c = 0\) and they are $$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$

I haven't decided which LaTeX method I prefer.


I store all of my articles on Dropbox ~/Dropbox/blogger/content. I'm not convinced that my articles belong in revision control--I don't see myself branching and merging articles. Dropbox provides revision histories.


The original Markdown article must be processed by Pandoc before it can be published to Blogger. I run the following command.

pandoc --no-wrap "${docname}" -o "${tempdoc}"

The output file ${tempdoc} contains the final version that is consumed by Blogger.


My goal is document, start to finish, the steps need to build and deploy our server stack. This includes

  • Building a new machine with Ubuntu LTS
  • Installing well-known services
    • DNS
    • DHCP
    • LDAP
    • Postfix
    • Samba
    • VirtualBox
  • Installation and configuration of servers
    • Wiki
    • Zenoss

The process for deployment is:

  • Build services on a local virtual machine.
  • Test services with other local virtual machines.
  • Deploy to staging environment.
  • Run staging tests.
  • Deploy to production.
  • Monitor.

Saturday, October 15, 2011

Virtualization with VirtualBox

VirtualBox is a free virtualization package available from Oracle. The makers of VirtualBox were originally a company purchased by Sun Microsystems. Some attractive features of this package include:
  • Open source
  • Runs on Macintosh, Linux and Windows
  • Supports OVF

Installation on Host Machine

Download the latest VirtualBox package for your host system (4.1.8 as of this post). Run the installer.
Note that when you're installing VirtualBox on your local machine, you can install it as yourself. However, if you're installing it on shared servers, be sure to use a machine account rather than a personal account.
Be sure to check the Default Machine Folder in Preferences to ensure you're using a path without spaces. On Unix systems, I use $HOME/vm for personal installations and /opt/vm for shared (server) installations.

Installation of Guest Operating System

Once VirtualBox is installed, it's time to create guest operating system. A guest is a virtual machine that runs as an application on your host. It effectively gives you a computer within a computer that you can run side-by-side with your host machine.

Ubuntu 64-bit LTS Desktop

To install Ubuntu, download 64-bit Desktop ISO. I like to create a guest that I call stemcell that I snapshot in well-known states so I can easily test things.
2048 MB RAM
500 GB dynamic hard drive (vdi)
128 MB video ram


It's a good idea to install Dropbox so you can easily copy sample configuration files.
$ sudo apt-get install nautilus-dropbox
$ dropbox start -i # enter your dropbox details
I keep common Ubuntu configuration files in Dropbox/config/ubuntu. For example


Update /etc/apt/sources.list to contain the following:

Next, update your packages and install VirtualBox guest additions.
sudo apt-get update
sudo apt-get upgrade
Now power off your virtual machine and take a snapshot.


Cloning is an extremely useful way to manage machine images that you use over and over. We manage our clones as follows:
  1. Create a base installation of our target OS
  2. Install a package or extension
  3. Take a snapshot
This process is repeated until a full stack machine is created (e.g. WKS server). The full heredity of the machine is now available for cloning.
There is a step that is not well-documented. The clone process does not remove the entries in /etc/udev/rules.d/70-persistent-net.rules. When you first launch your cloned guest, you will get an error with your eth0 interface. Remove all entries in this file and reboot and the interface will reactivate.