Wednesday, October 26, 2011

Apt Repository

BitBucket

BitBucket has the ability to host static sites. Although Mercurial is better suited for text files than binary debs, BitBucket does make a reasonable apt-repository host for a small collection of packages.

  1. First, create a BitBucket account for your repository. I recommend a separate account from the one you use for source code development. I created an account with username fimero to host artifacts for the fimero project.
  2. Create a repository named {username}.bitbucket.org so in this case fimero.bitbucket.org.
  3. Clone this repository to your local machine hg clone https://bitbucket.org/{username}/{username}.bitbucket.org
  4. Add an index.html file to this project, commit and push.

Now you can confirm that things work by pointing your browser to

http://fimero.bitbucket.org

Apt Repository

Next, install reprepro to set up your apt repository.

$ sudo apt-get install reprepro

Our objective is to have a repository with the following configuration

deb http://fimero.bitbucket.org/ubuntu fimero main

To create this, cd into your local copy of your repository and create a folder called ubuntu/conf and create a file called distributions.


You are now ready to deploy a package.

$ reprepro -b ${PACKAGE_FOLDER} -V -P 1 -S main includedeb fimero ${debpackage}

Here ${PACKAGE_FOLDER} is your {username}.bitbucket.org/ubuntu folder and ${debpackage} is the deb file you want to deploy.

The final step is to push these changes to BitBucket.

$ hg commit
$ hg push

If you've added the deb entry from above in your /etc/apt/sources.list, you'll be able to install your newly deployed package on all of your machines.



Tuesday, October 25, 2011

Portion Package Enhancement

Package Enhancement

I created portion to enhance the standard packaging process. The end goal is to create a packaging and deployment process that is easy to orchestrate with CfEngine. portion allows me to package software in a consistent way while continuing to leverage the standard Ubuntu repositories.

Core Packages

The core packages for portion are bbsh2 and bbpython2. These contain bash and python scripts that are used to bootstrap all other packages. They are special packages--all other packages are built using the facilities included in these two packages.

The apt repository is available at

deb http://fimero.bitbucket.org/ubuntu fimero main

We also run with badgerports configured on 10.04 LTS.

bbsh2

The source code for bbsh2 is available here. The primary functions used for portion are in libportion.sh.

bbpython2

The source code for bbpython2 is available here.

To create a package, first install bbpython2 with

$ sudo apt-get install bbpython2

This will also install bbsh2 via its dependency. Next, create a folder for your package. Be sure to put it under source control. In this tutorial, we'll package a demo script and call it bbdemo

$ mkdir bbdemo

Creating A Package

A Portion package requires the following

  • manifest file
  • payload.txt file
  • postinstall.sh
  • preremove.sh

manifest File

Use the manifest file from bbdemo as a template.


payload File

You must generate a payload.txt file that specifies all of the files that are included with your package.

$ bb-mkepmlist.sh {package} {version} {folder}

This script searches the {folder} and create a payload.txt file.

postinstall.sh Template


preremove.sh Template


Assembling Package

The functions to assemble the package are in libportion.sh.

$ source libportion.sh
$ epm_make_package

This creates a .deb package in the target folder.



Tuesday, October 18, 2011

Documentation Bootstrap

I am a fan of good documentation—I feel it's the most undervalued component of many projects but I haven't been able to develop a reliable process to:

  • Collect information
  • Format it
  • Keep it up to date

This is my attempt to come up with that process.

vim

I want to edit my documentation in vim. I like it because it's an extremely lightweight and focused tool for text editing. My .vimrc file is


Markdown

Markdown is a lightweight markup syntax that avoids the annoying open/close brackets. You can get the syntax extensions for vim from here.

Blogger

I chose Blogger because it is already integrated with my Google Apps Premier account. I needed something basic and reliable. Documentation is divided into two areas—articles hosted by Blogger and articles hosted on my wiki. We'll begin by setting up documentation on Blogger and then use that to host documentation for setting up the wiki. The wiki will, in turn, host documentation the rest of the server stack.

BitBucket

References to BitBucket code are included via BitBucket's embed function. Note—replace the revision number with default to always pull down the tip version.


GitHub

Code snippets can also be included via gist on GitHub.

Pastebin

Pastebin is also supported:

LaTeX

We will occasionally need to display equations. There are several options including using the utility available here. Or you can use MathJax by adding the following to your page template:


When \(a \ne 0\), there are two solutions to \(ax^2 + bx + c = 0\) and they are $$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$

I haven't decided which LaTeX method I prefer.

Dropbox

I store all of my articles on Dropbox ~/Dropbox/blogger/content. I'm not convinced that my articles belong in revision control--I don't see myself branching and merging articles. Dropbox provides revision histories.


Pandoc

The original Markdown article must be processed by Pandoc before it can be published to Blogger. I run the following command.

pandoc --no-wrap "${docname}" -o "${tempdoc}"

The output file ${tempdoc} contains the final version that is consumed by Blogger.

Objective

My goal is document, start to finish, the steps need to build and deploy our server stack. This includes

  • Building a new machine with Ubuntu LTS
  • Installing well-known services
    • DNS
    • DHCP
    • LDAP
    • Postfix
    • Samba
    • VirtualBox
  • Installation and configuration of servers
    • Wiki
    • Zenoss

The process for deployment is:

  • Build services on a local virtual machine.
  • Test services with other local virtual machines.
  • Deploy to staging environment.
  • Run staging tests.
  • Deploy to production.
  • Monitor.


Saturday, October 15, 2011

Virtualization with VirtualBox

VirtualBox is a free virtualization package available from Oracle. The makers of VirtualBox were originally a company purchased by Sun Microsystems. Some attractive features of this package include:
  • Open source
  • Runs on Macintosh, Linux and Windows
  • Supports OVF

Installation on Host Machine

Download the latest VirtualBox package for your host system (4.1.8 as of this post). Run the installer.
Note that when you're installing VirtualBox on your local machine, you can install it as yourself. However, if you're installing it on shared servers, be sure to use a machine account rather than a personal account.
Be sure to check the Default Machine Folder in Preferences to ensure you're using a path without spaces. On Unix systems, I use $HOME/vm for personal installations and /opt/vm for shared (server) installations.

Installation of Guest Operating System

Once VirtualBox is installed, it's time to create guest operating system. A guest is a virtual machine that runs as an application on your host. It effectively gives you a computer within a computer that you can run side-by-side with your host machine.

Ubuntu 64-bit LTS Desktop

To install Ubuntu, download 64-bit Desktop ISO. I like to create a guest that I call stemcell that I snapshot in well-known states so I can easily test things.
2048 MB RAM
500 GB dynamic hard drive (vdi)
128 MB video ram

Dropbox

It's a good idea to install Dropbox so you can easily copy sample configuration files.
$ sudo apt-get install nautilus-dropbox
$ dropbox start -i # enter your dropbox details
I keep common Ubuntu configuration files in Dropbox/config/ubuntu. For example
Dropbox/config/ubuntu/etc/apt/sources.list

Packages

Update /etc/apt/sources.list to contain the following:

Next, update your packages and install VirtualBox guest additions.
sudo apt-get update
sudo apt-get upgrade
Now power off your virtual machine and take a snapshot.

Cloning

Cloning is an extremely useful way to manage machine images that you use over and over. We manage our clones as follows:
  1. Create a base installation of our target OS
  2. Install a package or extension
  3. Take a snapshot
This process is repeated until a full stack machine is created (e.g. WKS server). The full heredity of the machine is now available for cloning.
There is a step that is not well-documented. The clone process does not remove the entries in /etc/udev/rules.d/70-persistent-net.rules. When you first launch your cloned guest, you will get an error with your eth0 interface. Remove all entries in this file and reboot and the interface will reactivate.