Archive for the ‘OpenStack’ Category

Shang Wu

OpenStack – What is FlatNetworking?

I have set up the openstack couple times now. But, I still have some problem with the networking bits in Openstack. Today, I found the wiki[1] page from openstack.org which clearly explained what that is.

Basically, the machine that has the nova-network will take care of all the network related stuff in openstack. For example, It will:

1. Give out the IP address to the VMs and keep track of it.
2. The br100 interfaces is used to bridge the private network (–fixed_range=x.x.x.x) that we define in the nova.conf file to the public. This interface can also be set up on the eth0 interface (if you only have one network interface.
3. Nova-Network will use that to route VM’s network traffic to the public .

The second article[2] I found discussed couple options about HA for nova-network to prevent its single point of failure which is very interesting as well.

Enjoy~

[1] Understanding FlatNetworking:
http://wiki.openstack.org/UnderstandingFlatNetworking

[2] Networking in Nova
http://unchainyourbrain.com/openstack/13-networking-in-nova

Shang Wu

Openstack – Waiting for metadata service

After downloading the images and start the instances, the instances starts up and I saw the status become running for the first time! (Yay!)

BUT! When I try to connect to it, I was not able to. When I look into the console output, I saw the error message:

2011-07-11 09:15:35,306 – DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id

2011-07-11 09:15:35,308 – DataSourceEc2.py[WARNING]: 09:15:35 [ 1/30]: http error [404]

2011-07-11 09:15:36,313 – DataSourceEc2.py[WARNING]: 09:15:36 [ 2/30]: http error [404]

2011-07-11 09:15:37,317 – DataSourceEc2.py[WARNING]: 09:15:37 [ 3/30]: http error [404]

2011-07-11 09:15:38,323 – DataSourceEc2.py[WARNING]: 09:15:38 [ 4/30]: http error [404]

After googling around, I found exactly what I was looking for. Someone posted a message on the Launchpad Answers and having exactly the same issue. Looking through the reply, I realized I made an error in my /etc/nova/nova.conf file about about the –fixed_range= which I set it to the network range the same as my eth0 card and everything is working as expected after.

Shang Wu

Openstack – Lose internet connectivity

Like many others, I follow the CSSOSS instruction to setup my Openstack environment.

After see all my resources appeared to be up-and-running, I want to get the images and test out the final step: start an instances. But, I realised I cannot get to the Internet which was odd because I was able to do that at the beginning to update/upgrade the system. What happened?

Looking around on the Internet, I found myself is not alone, see brendan1495′s post on the forum. As I am sure the Internet connection has no problem, I start to suspect that the issue might be on the machine. However, I tried the host command on the machine. It is able to resolve the DNS name. But does not respond to the ping or apt-get update/install/upgrade command.

Initially, I thought the issue is with iptables. So I tried to save/flush/restore the rules, but in vain. Today after I saw brendan’s post, I realise the issue is probably is related to nova-network because that is when he “lost” his Internet connection.

Then I run the nova-network command, it gives me:

sudo nova-network
2011-07-11 13:57:25,368 AUDIT nova [-] Starting network node (version 2011.2-workspace:tarmac-20110415024701-a9bdb77vaatk99lh)
2011-07-11 13:57:25,726 CRITICAL nova [-] (OperationalError) no such table: networks u’SELECT networks.created_at AS networks_created_at, networks.updated_at AS networks_updated_at, networks.deleted_at AS networks_deleted_at, networks.deleted AS networks_deleted, networks.id AS networks_id, networks.label AS networks_label, networks.injected AS networks_injected, networks.cidr AS networks_cidr, networks.cidr_v6 AS networks_cidr_v6, networks.gateway_v6 AS networks_gateway_v6, networks.netmask_v6 AS networks_netmask_v6, networks.netmask AS networks_netmask, networks.bridge AS networks_bridge, networks.gateway AS networks_gateway, networks.broadcast AS networks_broadcast, networks.dns AS networks_dns, networks.vlan AS networks_vlan, networks.vpn_public_address AS networks_vpn_public_address, networks.vpn_public_port AS networks_vpn_public_port, networks.vpn_private_address AS networks_vpn_private_address, networks.dhcp_start AS networks_dhcp_start, networks.project_id AS networks_project_id, networks.host AS networks_host \nFROM networks \nWHERE networks.deleted = ? AND networks.host = ?’ (False, ‘nova1′)

After this point, my Internet connection is “back.” But I saw the error message at the end, which makes me wonder…..

Shang Wu

Openstack – Failed to spawn instances

Previous, I had some trouble to run Openstack on my test machine. With the help from CSSOSS documentation I was able to see the resources are all up and ready, waiting to be used. But, after I download the tarball images from Ubuntu image store and registered to my openstack. I was not able to get the instances start running.

The error message that I got when I run euca-describe-instances, the isntances status starts from Launching switch to Failed to Spawn almost immediately. Looking through the logs (nova-compute.log), it tells me that the kvm needs to be enabled which I have done so already… So, the issue is somewhere else. But in any case, you are getting the same error, this might be the first thing that you need to check.

To check if your kvm extension is enabled, you can run:

sudo kvm-ok

If it is enabled, the message will look like:

KVM acceleration can be used

Shang Wu

OpenStack – Configure Instance Type

After yesterday’s little trick, now I have my dashboard up and running.

The portal looks like the Eucalyptus admin page without the image store and where to configure the instance type. That is our mission today. Find out where to configure the instance type.

By default, the few instance type looks like the following:

m1.medium – 4096M, 2CPU, 40G Space
m1.large – 8192M, 4CPU, 80G Space
m1.tiny – 512M, 1CPU, 0G Space
m1.xlarge – 16384M, 8CPU, 160G Space
m1.small – 2048M, 1CPU, 20G Space

Which for my testing purpose, I would like to customise it. In Eucalyptus, it used to let user change it directly on the web page. But in OpenStack, I think we will need to use the command line to create/delete the instance types.

First, let’s take a look at what you have already from the command line:

sudo nova-manage instance_type list

which, not surprisingly, you should have the output looks similar to the table above. But, a little surprise that you won’t see in the webpage is the FlavorID, which might gives you a little trouble later.

m1.medium: Memory: 4096MB, VCPUS: 2, Storage: 40GB, FlavorID: 3, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.small: Memory: 2048MB, VCPUS: 1, Storage: 20GB, FlavorID: 2, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.large: Memory: 8192MB, VCPUS: 4, Storage: 80GB, FlavorID: 4, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB
k1.small: Memory: 1024MB, VCPUS: 1, Storage: 8GB, FlavorID: 8, Swap: 2GB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.xlarge: Memory: 16384MB, VCPUS: 8, Storage: 160GB, FlavorID: 5, Swap: 0GB, RXTX Quota: 0GB, RXTX Cap: 0MB

Second, let’s take a look at the help with the nova-manage instance_type create command which we use it to create the instance type:

instance_type create: Creates instance types / flavors
arguments: name memory vcpus local_gb flavorid [swap] [rxtx_quota] [rxtx_cap]

So, to create a “s1.small” (this name could be anything) instance type with 1024M memory, 1 CPU, 5000M storage with flavorID 7, you need:

sudo nova-manage instance_type create s1.small 1024 1 5000 7

with the return of

s1.small created

Then, we want to delete this instance type. Similar command is called but change from create to delete to remove the entry:

sudo nova-manage instance_type delete s1.small

Now, the final twist, if you want to create another entry using the same command which should work, right? Instead, you will get an Unknown error. Doesn’t really tell you much here, does it?

Later I found out the issue is the FlavorID that I mention earlier. Even you delete the entry, the FlavorID cannot be reuse. If you try to use the flavor list command, you still won’t see the deleted entry.

sudo nova-manage flavor list

So, if you are getting the Unknown error, look into the /var/log/nova/nova-manage.log file for clues, but this might be the reason why!

Shang Wu

OpenStack – Changing the database to MySQL from Sqllite

Again, following the documentationon OpenStack Dashboard with Mysql Database

I ran into issues when I directly copy and paste the:

DATABASES = {
‘default’: {
‘ENGINE’: ‘django.db.backends.mysql’,
‘NAME’: ‘dashboarddb’,
‘USER’: ‘nova’,
‘PASSWORD’: ‘mygreatsecret’,
‘HOST’: ‘localhost’,
‘default-character-set’: ‘utf8′,
}
}

into my /opt/osdb/openstack-dashboard/local/local_settings.py

I am getting the error message like:

(.dashboard-venv)root@nova-stack:/opt/osdb/openstack-dashboard# sudo tools/with_venv.sh dashboard/manage.py syncdb
Creating table auth_permission
Creating table auth_group_permissions
Traceback (most recent call last):
File “dashboard/manage.py”, line 11, in
execute_manager(settings)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/core/management/__init__.py”, line 438, in execute_manager
utility.execute()
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/core/management/__init__.py”, line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/core/management/base.py”, line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/core/management/base.py”, line 220, in execute
output = self.handle(*args, **options)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/core/management/base.py”, line 351, in handle
return self.handle_noargs(**options)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/core/management/commands/syncdb.py”, line 95, in handle_noargs
cursor.execute(statement)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/db/backends/util.py”, line 15, in execute
return self.cursor.execute(sql, params)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py”, line 86, in execute
return self.cursor.execute(query, args)
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/cursors.py”, line 174, in execute
File “/opt/osdb/openstack-dashboard/.dashboard-venv/lib/python2.7/site-packages/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py”, line 36, in defaulterrorhandler
_mysql_exceptions.OperationalError: (1142, “ALTER command denied to user ‘nova’@’localhost’ for table ‘auth_group_permissions’”)

At the last line, it looks like the Mysql issue. So, I comment out the USER line from the config and ran the command:

sudo tools/with_venv.sh dashboard/manage.py syncdb

And everything is working as expected…

Shang Wu

After struggling with the Openstack dashboard yesterday, I was able to bring up the dashboard interface. But just when I thought this is the end……. Boom! I can’t login to the system!

According to the CSS documentation, as soon as I use the command:

sudo tools/with_venv.sh dashboard/manage.py syncdb

I should be asked to create the password. Looking at the wiki page from OpenStack DashBoard, it should look like this:

You just installed Django’s auth system, which means you don’t have any superusers defined.
Would you like to create one now? (yes/no): *YES*
Username (Leave blank to use ‘root’): *ENTER YOUR NOVA_USERNAME (The name of the Nova admin user you created) FROM NOVARC*
E-mail address: *ENTER YOUR EMAIL ADDRESS*
Password: *MAKE UP A PASSWORD*
Password (again): *REPEAT YOUR PASSWORD*

But instead, I got (whether using the code from Github or Launchpad, I got the same result):

shang@shang:~/openstack-dashboard/openstack-dashboard$ sudo tools/with_venv.sh dashboard/manage.py syncdb
^[3DEBUG:django.db.backends:(0.001)
SELECT name FROM sqlite_master
WHERE type=’table’ AND NOT name=’sqlite_sequence’
ORDER BY name; args=()
Creating tables …
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = contenttypes ORDER BY “django_content_type”.”name” ASC; args=(‘contenttypes’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = contenttype AND “django_content_type”.”app_label” = contenttypes ); args=(‘contenttype’, ‘contenttypes’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = sessions ORDER BY “django_content_type”.”name” ASC; args=(‘sessions’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = session AND “django_content_type”.”app_label” = sessions ); args=(‘session’, ‘sessions’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = messages ORDER BY “django_content_type”.”name” ASC; args=(‘messages’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = comments ORDER BY “django_content_type”.”name” ASC; args=(‘comments’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = comment AND “django_content_type”.”app_label” = comments ); args=(‘comment’, ‘comments’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = commentflag AND “django_content_type”.”app_label” = comments ); args=(‘commentflag’, ‘comments’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = sites ORDER BY “django_content_type”.”name” ASC; args=(‘sites’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = site AND “django_content_type”.”app_label” = sites ); args=(‘site’, ‘sites’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = markup ORDER BY “django_content_type”.”name” ASC; args=(‘markup’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = django_openstack ORDER BY “django_content_type”.”name” ASC; args=(‘django_openstack’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE “django_content_type”.”app_label” = mailer ORDER BY “django_content_type”.”name” ASC; args=(‘mailer’,)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = message AND “django_content_type”.”app_label” = mailer ); args=(‘message’, ‘mailer’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = dontsendentry AND “django_content_type”.”app_label” = mailer ); args=(‘dontsendentry’, ‘mailer’)
DEBUG:django.db.backends:(0.000) SELECT “django_content_type”.”id”, “django_content_type”.”name”, “django_content_type”.”app_label”, “django_content_type”.”model” FROM “django_content_type” WHERE (“django_content_type”.”model” = messagelog AND “django_content_type”.”app_label” = mailer ); args=(‘messagelog’, ‘mailer’)
Installing custom SQL …
Installing indexes …
No fixtures found.

Apparently, I am not the only one has this issue. From the Openstack-operators mailing list, the answer to that question is to use bzr command below to reverse to the previous version, and everything will work fine:

bzr revert -r 46

and it did! I am able to login to the Openstack Dashboard now.

So to conclude, you do need to use the bzr command to get the code, but need to use the command above to fix the code and get it going after that.

Enjoy!

Shang Wu

Update notes from Jun 28th, 2011: I encounter some issues after using the code from GitHub. Thus, I switch back to use bzr and get it working after that. Please refer to part 2, for more detail information.

I have been trying to install OpenStack these days just to see how different it is than Eucalyptus. Like many others, back when I was still using Eucalyptus, the book/documentation from http://cssoss.wordpress.com is fantastic. At least, that was how I use to get started at the beginning.

This time around, checking on the CSS website, I found they also have the documentation for OpenStack (They are great, aren’t they?!) That is where I will base my article about OpenStack today.

The article is still great and helpful like before. However, there were some changes made on the OpenStack Dashboard side as they have moved to Github after the article has been published. That is why if you follow the session about Nova Dashboard, you will run into problems.

To be more specific, if you use the bzr command:

sudo bzr branch lp:openstack-dashboard /opt/osdb

to download the openstack-dashboard to /opt/osdb, you will soon find out that the size for your run_tests.sh is zero.

To workaround this problem, you will need to use the following command at your preferred directory:

git clone https://github.com/4P/openstack-dashboard.git

After this, you will have “almost” everything you need and continue on with the article now.



Create a new blog