Shang Wu

After some testing with MAAS, I ran into an issue that I cannot remove a node from MAAS using the “Delete Node” button as its status show “Allocate to Root“.

Assuming you ran the juju bootstrap already and does not need it anymore. The correct way to clear the status from “Allocate to Root” to “Ready” again is by running:

juju destroy-services

Once you done that, you should see all the nodes are back to the ready status.

However, in some occasion, the status cannot be changed that way. Thus, you will need to open a terminal to remove the node from MAAS.

Open a terminal, then see if anything is listed there by using:

cobbler system list

Once you have the name for the system, you can remove it by running:

cobbler system remove –name=xxx

After that is done, you can use the following to remove the node from MAAS properly:

$ sudo maas shell
>> from maasserver.models import Node
>> node = Node.objects.get(hostname=’myhostname’)
>> node.delete()

Check back the MAAS portal, you should have that node removed.

Shang Wu

Thanks to Dustin’s formal introduction, I now have a better understanding about what Ubuntu Orchestra is and how this tool can help system administrators deploying Ubuntu in the data center.

Basically, the Ubuntu orchestra server consist of the following components:

1. Provision Server -> Cobbler
2. Monitoring Server -> Nagios
3. Management Server -> Ensemble
4. Logging Server -> Rsyslog

To build the provision part of the environment, you will need to setup and configure the following servers:

1. DHCP server
2. Ubuntu Orchestra Server
3. PXE boot server

Here is how:
1. For DHCP server (

a. Install the dhcp3-server package:

sudo apt-get install dhcp3-server

b. Configure the dhcpd.conf file under /etc/dhcp directory:
(Leave everything as it is but configure the local network and the static IP address for Ubuntu orchestra server and the PXE boot client)

subnet netmask {
option routers;
option broadcast-address;
default-lease-time 600;
max-lease-time 7200;

host Orchestra {
hardware ethernet 08:00:27:05:5D:15;

host PXE-Client {
hardware ethernet 08:00:27:5E:d4:53;
option subnet-mask;
filename “pxelinux.0”;
option domain-name-servers,;

c. Once that is done, restart the DHCP server:

sudo /etc/init.d/isc-dhcp-server restart

2. For Ubuntu Orchestra server (

a. Install the Ubuntu-orchestra-server package:

sudo apt-get install ubuntu-orchestra-server

b. During the installation, you will be asked to configure the password and if you are going to use the server to manage your DNS/DHCP. For the purpose of this example, I did not use Orchestra as my DNS/DHCP server.

c. Once the setup is completed, you can open the browser and take a look at its interface (Looks great, I have to say!!)

Ubuntu Orchestra Web Interface

(In my case)


d. Import the ubuntu iso file:

sudo cobbler-ubuntu-import natty-i386

(This will create the image profile in the Distro and the Profile entry)


e. After the profile is created, you will need to add the preseed file to the profile entry via the web page, if you want to automate everything. The kickstarts file can be found at /var/lib/cobbler/kickstarts/ directory. In my case, I use the /var/lib/cobbler/kickstarts/ubuntu-server.preseed as my preseed file.


f. Create the system that you want to deploy:
Add a new system from the web interface. You are required to have the name and the MAC address for the setup.


g. Start up the PXE boot server, Et Voilà!



[1] Couple useful cobbler commands:

sudo cobbler check
– Self check on the configuration and provide
sudo cobbler sync – Apply cobbler configuration changes
sudo restart cobbler – restart cobbler services
sudo cobbler-ubuntu-import – import the Ubuntu ISO file for PXE boot installation
sudo cobbler system add –name=example –mac=$mac-address –profile=$profile-name – Specify the system with its MAC address and the cobber profile which is going to be used

[2] References:

Cobbler –


Shang Wu

OpenStack – What is FlatNetworking?

I have set up the openstack couple times now. But, I still have some problem with the networking bits in Openstack. Today, I found the wiki[1] page from which clearly explained what that is.

Basically, the machine that has the nova-network will take care of all the network related stuff in openstack. For example, It will:

1. Give out the IP address to the VMs and keep track of it.
2. The br100 interfaces is used to bridge the private network (–fixed_range=x.x.x.x) that we define in the nova.conf file to the public. This interface can also be set up on the eth0 interface (if you only have one network interface.
3. Nova-Network will use that to route VM’s network traffic to the public .

The second article[2] I found discussed couple options about HA for nova-network to prevent its single point of failure which is very interesting as well.


[1] Understanding FlatNetworking:

[2] Networking in Nova

Shang Wu

After reading the previous post, you should have a basic idea how to deploy services using ensemble.

So I will skip how it works here, but try to explain to you the command that we use before are doing what:

1. The NUMBER ONE Command that you will need to start using ensemble:

juju bootstrap

2. To deploy a service (wordpress, in this example):

juju deploy –repository=. wordpress

So this command will use the wordpress formula and deploy the services. But what is the –repository=. doing here?

Well, actually, the –repository specify where the formula is. If you are not in the correct directory, you will be getting an error saying:
2011-07-28 15:55:10,596 ERROR Formula ‘wordpress’ not found in repository /usr/share/principia-tools

That is why we need to make sure that you have all the formulas in the current directory. Or you can specify the absolute path and tell ensemble where to look for the formula:

juju deploy –repository=/usr/share/principia-tools/formulas wordpress

Sometimes, after the service name, you will see another parameter. What does that mean? It is just an a name that we give to identify the services, myblog in this case:

juju deploy –repository=. wordpress myblog

Same thing goes to the mysql services:

juju deploy –repository=. mysql mydb

3. To bind all the services together, we use the add-relation command:

juju add-relation mydb:db myblog

If you named your database service as mydb, that is what you need to use here and same thing goes to mywebsite. However, I always wondering what the :db means there.

The :db here defines that the mysql providing the db resource which is required by the wordpress service. If you are interested, you can actually look into the metadata.yaml
file under both /usr/share/principia-tools/formulas/wordpress and /usr/share/principia-tools/formulas/mysql directory. In there, it defines what is required and what kind of relation can be provided.

For example, my wordpress metadata.yaml looks like:

ensemble: formula
name: wordpress
revision: 30
summary: “WordPress is a full featured web blogging tool”
description: |
WordPress is a full featured web blogging tool:
– Instant publishing (no rebuilding)
– Comment pingback support with spam protection
– Non-crufty URLs
– Themable
– Plugin support
interface: mysql
interface: http

And the mysql metadata.yaml looks like:

ensemble: formula
name: mysql
revision: 98
summary: MySQL is a fast, stable and true multi-user, multi-threaded SQL database
description: |
MySQL is a fast, stable and true multi-user, multi-threaded SQL database
server. SQL (Structured Query Language) is the most popular database query
language in the world. The main goals of MySQL are speed, robustness and
ease of use.
interface: mysql
interface: mysql
interface: mysql-shared
interface: mysql-oneway-replication
interface: munin-node
interface: mysql-oneway-replication


Shang Wu

Many interesting topic were discussed at the Ubuntu Cloud Days. If you miss it, here are the logs for you to catch up!

Getting started with Ensemble — kim0
Introduction to Cloudinit — koolhead17
Orchestra and Ensemble (part1) — smoser
Orchestra and Ensemble (part2) — RoakSoax
Eucalyptus 3: cloud HA and ID management — nurmi

Getting started with OpenStack Compute — soren
UEC on Ubuntu 10.04 LTS — TeTeT
Node.js/Mongo with Ensemble — m_3
OpenStack: An open cloud infrastructure project — ttx

Shang Wu

Where are all the ensemble formulas?

In case you are wondering this question, the formula is being work on under the project name: principia

To get all the latest ensemble formula, you can:

1. Install the principia-tools package from the ensemble ppa:

sudo add-apt-repository ppa:ensemble/ppa
sudo apt-get update
sudo apt-get install principia-tools

2. Then, create the /usr/share/principia-tools/formulas if it is not exist already and change the permission to yourselves:

sudo mkdir /usr/share/principia-tools/formulas
sudo chown -R `whoami` /usr/share/principia-tools/formulas

3. Now, let’s get all the formulas using:

principia getall

When that is done, run the ls /usr/share/principia-tools/formulas command to see all the formulas can be used by ensemble!

Principia Tools Project:

Ensemble Needs You!:

Shang Wu

For those of you who have not heard or used ensemble before, Ensemble is a quick way to allow administrator to deploy services to the Amazon EC2 cloud. Thanks to the pre-build formulas, you can easily deploy a wordpress blog in less than 5 minutes!!

To start, you will first need to have an Amazon EC2 account. Once the registration is completed, from the Amazon Account Portal, you can find your access key and secret key there. These information will be used later on to start/stop your EC2 instances. With all the said and done, let’s start the fun stuff!

First, add the ensemble ppa to your system:

sudo add-apt-repository ppa:ensemble/ppa
sudo apt-get update && sudo apt-get install ensemble

Then, run the command to setup the environment:


Now, you need to edit the config file put in your EC2 access key and secret key:

sudo nano ~/.ensemble/environments.yaml

After that the file (environments.yaml) should look like:

ensemble: environments

type: ec2
control-bucket: ensemble-e1446ba830f0458fb04a9a86588749bb
admin-secret: 6c14633ea540477691f860e151cde0f7
access-key: _Your_Access_Key_Here_
secret-key: _Your_Secret_Key_Here_

Now, you will be able to use the ensemble command to control/deploy your EC2 cloud~

Let’s take a look at an example:

A. Before you begin, you will need to launch the first instance to manage the rest of the deployment. To do that, you run:

ensemble bootstrap

If the command is working successfully, it should return:

2011-07-19 17:39:18,622 INFO Bootstrapping environment ‘sample’ (type: ec2)…
2011-07-19 17:39:25,079 INFO ‘bootstrap’ command finished successfully

B. You can go check on the Amazon portal to ensure the node is running OR you can run the status command to see if the instance is up and running now:

ensemble status

If the instance is not ready, you will see:

2011-07-19 17:40:55,829 INFO Connecting to environment.
2011-07-19 17:40:59,217 ERROR Connection refused
Unhandled error in Deferred:
Traceback (most recent call last):
Failure: txzookeeper.client.ConnectionTimeoutException: could not connect before timeout
ProviderError: Interaction with machine provider failed: ConnectionTimeoutException(‘could not connect before timeout after 2 retries’,)
2011-07-19 17:41:26,277 ERROR ProviderError: Interaction with machine provider failed: ConnectionTimeoutException(‘could not connect before timeout after 2 retries’,)

Just wait a few minutes and run it again. If everything is working now, you should see:

2011-07-19 17:41:52,668 INFO Connecting to environment.
0: {dns-name:, instance-id: i-37569856}
services: {}
2011-07-19 17:41:58,186 INFO ‘status’ command finished successfully

C. Now you will need to change directory where the ensemble secret formulas are before you deploy the services:

cd /usr/share/doc/ensemble

D. Now, you can start deploying the word press and mysql services on your EC2 cloud by running:

ensemble deploy –repository=examples wordpress
ensemble deploy –repository=examples mysql

When this is done, you should have three instances running on your EC2 cloud.

E. Finally, to let them know the other services exists and to use each other, you run the command:

ensemble add-relation mysql wordpress

If this return:

2011-07-19 17:50:01,038 INFO Connecting to environment.
2011-07-19 17:50:07,706 INFO Added mysql relation to all service units.
2011-07-19 17:50:07,706 INFO ‘add_relation’ command finished successfully

Then, it means you have successfully deploy your own wordpress service to the world!

F. To check the status and the related information, you can use the status command again. It will tell you what/where the systems/services are running right now:

0: {dns-name:, instance-id: i-37569856}
1: {dns-name:, instance-id: i-8d539dec}
2: {dns-name:, instance-id: i-41509e20}
formula: local:mysql-11
relations: {db: wordpress}
machine: 1
relations: {}
state: null
formula: local:wordpress-29
relations: {db: mysql}
machine: 2
relations: {}
state: null
2011-07-19 17:52:49,060 INFO ‘status’ command finished successfully

In my output, the mysql services is running on machine 1, and wordpress is running on machine 2. However, the service is not ready yet, if you notice the line state: null.

Wait a few minutes and try again until you see the output like:

2011-07-19 17:54:53,987 INFO Connecting to environment.
0: {dns-name:, instance-id: i-37569856}
1: {dns-name:, instance-id: i-8d539dec}
2: {dns-name:, instance-id: i-41509e20}
formula: local:mysql-11
relations: {db: wordpress}
machine: 1
db: {state: up}
state: started
formula: local:wordpress-29
relations: {db: mysql}
machine: 2
db: {state: up}
state: started
2011-07-19 17:55:06,462 INFO ‘status’ command finished successfully

Then the system has finish its setup.

G. To really look under the hood and see what is going on, you can use the debug-log command:

ensemble debug-log

Until you see the message:

2011-07-19 17:55:22,591 unit:wordpress/0: hook.executor DEBUG: Hook complete: /var/lib/ensemble/units/wordpress-0/formula/hooks/db-relation-changed

Your wordpress is ready!

H. Now, go to the wordpress machine address from the status ouput, in my case it is machine 1. The address for machine one is “”. Copy and paste that to your browser, you will be asked to configure username and password to complete the setup!

I. When you are done, just use the shutdown command to stop all the services and instances:

ensemble shutdown

Hello WordPress! Hello Ensemble!!

[1] Ensemble Tutorial:
[2] Zero to Ensemble in 5 mins:

Shang Wu

Setting up Apt-Cacher-NG

In order to save some bandwidth for the servers that I have setup, I decided to install the apt-cacher-ng on my natty server box.

To start, using the command:

sudo apt-get install apt-cacher-ng

on the server that you would like to cache all the packages.

Then, configure the server itself to cache all the packages you will need to change the sources.list file from /etc/apt directory using the command below:

sudo sed -i ‘s/http:\/\//http:\/\/localhost:3142\//’ /etc/apt/sources.list

Now, start the update/upgrade/install packages on the machine. All the packages should be cached on the local and ready to be use.

From client side, where you want to take the advantage of the cached packages, run the same command except you will need to change the localhost to the first server ip address ( in my case):

sudo sed -i ‘s/http:\/\//http:\/\/\//’ /etc/apt/sources.list

Once, that is done, you can start update/upgrade/install packages onto the system. All the packages have been cached previously will be picked up by the system.

Also, you can be able to see the statistic report from your browser ( in my case):

Shang Wu

After downloading the images and start the instances, the instances starts up and I saw the status become running for the first time! (Yay!)

BUT! When I try to connect to it, I was not able to. When I look into the console output, I saw the error message:

2011-07-11 09:15:35,306 –[WARNING]: waiting for metadata service at

2011-07-11 09:15:35,308 –[WARNING]: 09:15:35 [ 1/30]: http error [404]

2011-07-11 09:15:36,313 –[WARNING]: 09:15:36 [ 2/30]: http error [404]

2011-07-11 09:15:37,317 –[WARNING]: 09:15:37 [ 3/30]: http error [404]

2011-07-11 09:15:38,323 –[WARNING]: 09:15:38 [ 4/30]: http error [404]

After googling around, I found exactly what I was looking for. Someone posted a message on the Launchpad Answers and having exactly the same issue. Looking through the reply, I realized I made an error in my /etc/nova/nova.conf file about about the –fixed_range= which I set it to the network range the same as my eth0 card and everything is working as expected after.

Shang Wu

Like many others, I follow the CSSOSS instruction to setup my Openstack environment.

After see all my resources appeared to be up-and-running, I want to get the images and test out the final step: start an instances. But, I realised I cannot get to the Internet which was odd because I was able to do that at the beginning to update/upgrade the system. What happened?

Looking around on the Internet, I found myself is not alone, see brendan1495’s post on the forum. As I am sure the Internet connection has no problem, I start to suspect that the issue might be on the machine. However, I tried the host command on the machine. It is able to resolve the DNS name. But does not respond to the ping or apt-get update/install/upgrade command.

Initially, I thought the issue is with iptables. So I tried to save/flush/restore the rules, but in vain. Today after I saw brendan’s post, I realise the issue is probably is related to nova-network because that is when he “lost” his Internet connection.

Then I run the nova-network command, it gives me:

sudo nova-network
2011-07-11 13:57:25,368 AUDIT nova [-] Starting network node (version 2011.2-workspace:tarmac-20110415024701-a9bdb77vaatk99lh)
2011-07-11 13:57:25,726 CRITICAL nova [-] (OperationalError) no such table: networks u’SELECT networks.created_at AS networks_created_at, networks.updated_at AS networks_updated_at, networks.deleted_at AS networks_deleted_at, networks.deleted AS networks_deleted, AS networks_id, networks.label AS networks_label, networks.injected AS networks_injected, networks.cidr AS networks_cidr, networks.cidr_v6 AS networks_cidr_v6, networks.gateway_v6 AS networks_gateway_v6, networks.netmask_v6 AS networks_netmask_v6, networks.netmask AS networks_netmask, networks.bridge AS networks_bridge, networks.gateway AS networks_gateway, networks.broadcast AS networks_broadcast, networks.dns AS networks_dns, networks.vlan AS networks_vlan, networks.vpn_public_address AS networks_vpn_public_address, networks.vpn_public_port AS networks_vpn_public_port, networks.vpn_private_address AS networks_vpn_private_address, networks.dhcp_start AS networks_dhcp_start, networks.project_id AS networks_project_id, AS networks_host \nFROM networks \nWHERE networks.deleted = ? AND = ?’ (False, ‘nova1’)

After this point, my Internet connection is “back.” But I saw the error message at the end, which makes me wonder…..

Create a new blog