Canonical Voices

Posts tagged with 'linaro'

Michael Hudson-Doyle

I wanted to use PostgreSQL‘s array_agg function from Django to help bound the number of queries a complex page makes.

I managed to make it work, but it was a bit painful. I’d love to hear from anyone else that’s managed to make it work, especially if you didn’t have to hack as hard as me :-)

First, let’s define some models for us to play with:

from django.db import models

class Location(models.Model):
    name = models.CharField(max_length=1024)

class Hurricane(models.Model):
    year = models.IntegerField()
    name = models.CharField(max_length=1024)
    location = models.ForeignKey(Location)
 

The problem I want to solve is: for a given Location, show the hurricanes that occurred for the last ten years in which there were hurricanes.

In this simplified case, this is probably quite easy to solve without being as fancy as I am about to be, but the thing that makes it non-trivial is the bit at the end: displaying from the last ten years is easy (“WHERE YEAR > 2002“) but there may be years where there were not hurricanes. The trick I want to use is to GROUP ON year, use array_agg to collect the ids of the hurricanes for each year, and use LIMIT to ensure I only get 10 years worth. Then I can gather the Hurricane ids up in Python code and issue another query to fetch them all in a single query. I’m aiming for something like this:

hurr=> SELECT year, array_agg(id) FROM hurr_hurricane
hurr-> WHERE hurr_hurricane.location_id = 1
hurr-> GROUP BY year ORDER BY year LIMIT 10;
 year | array_agg
------+-----------
 2004 | {2,1}
 2006 | {3}
 2007 | {4}
(3 rows)
 

I’ve never quite gotten my head around Django’s support for aggregates, but it’s not hard to get Django to emit a query of the right shape:

>>> qs = Hurricane.objects.filter(
...     location=loc).values('year').annotate(Sum('id'))[:10]
>>> print qs.query
SELECT "hurr_hurricane"."year", SUM("hurr_hurricane"."id") AS "id__sum"
FROM "hurr_hurricane" WHERE "hurr_hurricane"."location_id" = 1
GROUP BY "hurr_hurricane"."year", "hurr_hurricane"."year"
ORDER BY "hurr_hurricane"."year" LIMIT 10
>>> pprint(list(qs))
[{'id__sum': 3, 'year': 2004},
 {'id__sum': 3, 'year': 2006},
 {'id__sum': 4, 'year': 2007}]
 

We don’t want to SUM though, we want to array_agg. I don’t really know to what extent it’s actually supported, but googling can find enough of a clue on how to use custom aggregates with Django. Culting the approprate cargo leads to:

from django.db.models.sql.aggregates import Aggregate as SQLAggregate
from django.db.models import Aggregate

class SQLArrayAgg(SQLAggregate):
    sql_function = 'array_agg'

class ArrayAgg(Aggregate):
    name = 'ArrayAgg'
    def add_to_query(self, query, alias, col, source, is_summary):
        klass = SQLArrayAgg
        aggregate = klass(
            col, source=source, is_summary=is_summary, **self.extra)
        query.aggregates[alias] = aggregate
 

and then:

>>> qs = Hurricane.objects.filter(
...     location=loc).values('year').annotate(ArrayAgg('id')).order_by('year')[:10]
>>> print qs.query
SELECT "hurr_hurricane"."year", array_agg("hurr_hurricane"."id") AS "id__arrayagg"
FROM "hurr_hurricane" WHERE "hurr_hurricane"."location_id" = 1
GROUP BY "hurr_hurricane"."year", "hurr_hurricane"."year"
ORDER BY "hurr_hurricane"."year" ASC LIMIT 10
 

Yay! Except:

>>> list(qs)
[]
 

Huh? This sent me around the houses for ages and ages, but it turns out that there is an easy way of seeing the problem:

>>> for hurricane in qs:
...    print hurricane
Traceback (most recent call last):
    ...
    ...
File "/srv/lava/.cache/eggs/Django-1.4.1-py2.7.egg/django/db/models/sql/query.py", line 316, in convert_values
    return connection.ops.convert_values(value, field)
File "/srv/lava/.cache/eggs/Django-1.4.1-py2.7.egg/django/db/backends/__init__.py", line 843, in convert_values
    return int(value)
TypeError: int() argument must be a string or a number, not 'list'
 

(Something in executing list(qs) is swallowing the exception – I presume this is a bug, I’ll file it soon if it’s unreported.)

So let’s look at where the exception comes from (django/db/backends/__init__.py, Django version 1.4.1 below):

def convert_values(self, value, field):
    """Coerce the value returned by the database backend into a consistent type that
    is compatible with the field type.
    """
    internal_type = field.get_internal_type()
    if internal_type == 'DecimalField':
        return value
    elif internal_type and internal_type.endswith('IntegerField') or internal_type == 'AutoField':
        return int(value)
    elif internal_type in ('DateField', 'DateTimeField', 'TimeField'):
        return value
    # No field, or the field isn't known to be a decimal or integer
    # Default to a float
    return float(value)
 

Django appears to assume that all aggregates return numeric types. This is a bit annoying (another bug to file?), but there is a slimy hack we can pull: DecimalFields are assumed to be the correct type already and do not get converted. So here’s the final, working, array_agg support:

from django.db.models import DecimalField

class SQLArrayAgg(SQLAggregate):
    sql_function = 'array_agg'

class ArrayAgg(Aggregate):
    name = 'ArrayAgg'
    def add_to_query(self, query, alias, col, source, is_summary):
        klass = SQLArrayAgg
        aggregate = klass(
            col, source=source, is_summary=is_summary, **self.extra)
        aggregate.field = DecimalField() # vomit
        query.aggregates[alias] = aggregate
 

And to prove it works:

>>> pprint(list(qs))
[{'id__arrayagg': [2, 1], 'year': 2004},
 {'id__arrayagg': [3], 'year': 2006},
 {'id__arrayagg': [4], 'year': 2007}]
 

Yay! Now: does it really have to be this hard?

Read more
rsalveti

This week I’m proudly participating at the Ubuntu Developer Summit to help planning and defining what will the Quantal Quetzal (12.10) release be in the next following months.

As usual I’m wearing not only the Linaro hat, but also my Ubuntu and Canonical ones, interested and participating actively at most topics that are related with ARM in general.

And what can I say after the first 3 days at UDS-Q? Well, busy as never before and with great opportunities to help getting Ubuntu to rock even more at ARM, with current devices/platforms and with the exciting new ones that will be coming in the next few months.

Here are a few highlights from the first days:

Monday – May 7th

  • Introduction and Keynote
    • Great start as usual by Mark, showing the great opportunities for both Canonical and Ubuntu, describing the new target and use cases, and also showing how important Cloud is now for Ubuntu. After that we had, finally, the announcement of a real hardware availability from Calxeda, proving that ARM server are indeed real! (which is a quite important accomplishment)
  • Schedule displays all working with our member’s boards
    • This was the first time that all the schedule displays available at UDS were all covered by the ARM boards provided by Linaro. This time we got Pandaboard, Origen and also Snowball constantly showing the schedule through all the day. Low power and powerful devices all around :-)
  • Plans for a minimum filesystem for embedded devices
    • Discussion to cover all the possible embedded related use cases for Ubuntu, and trying to understand the real requirements for a minimum filesystem (rootfs) for those devices. While we didn’t decide to generate the smallest-still-apt/dpkg-compatible rootfs for our users (as ubuntu-core is already covering most of the cases), we’ll provide enough tools and documentation on how to easily generate them. At Linaro side the Ubuntu Nano image should probably reflect such suggestions.
  • Identify impact of the switch to pure live images for ARM platforms
    • Here the focus was basically to review and understand if we would really continue providing pre-installed based images instead of just supporting live based ones. Having the images provided only at the SD cards are very useful to make the bootstrap and install quite easy, but it hurts badly the performance. As we’re now getting ARM boards that are very powerful in many ways, the I/O bound shouldn’t limit what the users would be able to get from them. The decision for Quantal is to drop support for the pre-installed images, and provide live based ones at the SD cards (think like the live-sd image as we have with CD on other archs), where the user would install Ubuntu the same way as done with x86, and using USB/Sata based devices as rootfs by default.
  • OpenStack Deployment on ARM Server
    • The focus of this session was basically to better understand what might be the missing pieces for a proper OpenStack support at ARM. Quite a few open questions still, but the missing pkgs enablement, LXC testing and support and KVM for a few platforms will help making sure the support is at least correctly in place. After initial support, continuous test and validation should happen to make sure the ARM platforms keeps well supported over the time (which will be better stressed and tested once MAAS/Juju is also supported properly at ARM).

Tuesday – May 8th

  • Detail and begin the arm64/aarch64 port in Ubuntu
    • Clearly the most important session of the day for ARM. Great discussion on how to prepare and start the ARMv8 port at Ubuntu and Debian, by starting with cross-build support with multiarch and later support with Fast Models and Qemu. A lot is still to be covered once ARM is able to publish the ARMv8 support for Toolchain and Kernel, and session will be reviewed again at Linaro Connect at the end of this month.
  • Ubuntu Kernel Delta Review
    • Usual review of the patches the Ubuntu Kernel team is maintaining at the Ubuntu Kernel tree. At Linaro this is important as we also enable the Ubuntu specific patch-set at the packages provided by the LEB, for proper kernel and user-space support. Luckily this time it seems the delta is really minimum, which should probably also start to be part of Linux Linaro in the following month.
  • Integrate Linaro hwpacks for ARM with the Ubuntu image build infrastructure
    • Usual discussion about trying to avoid replicated work that is strictly related with each ARM board we support at both Ubuntu and Linaro. Decision is to finally sync with the latest flash-kernel available at Debian and try to get the common project/package with the hardware specific bits in place, so it can be used by linaro-image-tools, flash-kernel and debian-cd.

Wednesday – May 9th

  • MAAS Next Steps
    • Session to review and plan what are the next steps for the MAAS project, which is also missing proper ARM support for now. Great discussions on understanding all the requirements, as they will not necessarily match entirely with the usual ARM devices we have at the moment. Here the goal for ARM is to continue improving the PXE support at U-Boot (even with UEFI chainload later), and understanding what might be missing to also have IPMI support (even if not entirely provided by the hardware).
  • System Compositor
    • Great session covering what might be the improvements and development on the graphics side for next release. Goal is to use a system compositor that would be started right at the beginning at the boot, which will then be controlled and used properly once lightdm is up (with X11). This will improve a lot the user experience on normal x86 based desktops, and luckily on ARM we’re also in a quite nice situation with the work done by Linaro helping getting the proper DRM/KMS support for the boards we support, so I hope ARM will be in a great shape here :-)
  • ARM Server general enhancements (for ARMv7 and perhaps v8)
    • At this session we could cover what seems to be the most recurrent and problematically thing at supporting ARM servers, which is the lack of a single and supported boot method and boot loader. UEFI should be able to help on this front soon, but until then the focus will be to keep checking and making sure the current PXE implementation at u-boot works as expected (chainloading UEFI on u-boot is also another possibility Linaro is investigating). There is also the request for IPMI support, which is still unclear in general how it’ll be done generically speaking.
  • Integration testing for the bootloader
    • As Ubuntu is also moving to the direction of continuous validating and testing all important components available, there’s the need for a proper validation of the bootloader, and the effect at the user experience while booting the system. For ARM it’s also a special case, as U-Boot is still the main bootloader used across the boards. Test case descriptions in place, and discussion will probably continue at Linaro Connect as this is also an area where we also want to help validating/testing.
  • ARM Server Benchmarking and Performance
    • Here the Ubuntu Server Team presented how they are benchmarking and checking performance at the server level at x86, and covering what might still be needed to run and validate the ARM boards the same way. For ARM the plan is to run the same test cases on the available scenarios, and also try to get Linaro involved by making sure this is also part of the continuous validation and testing done with LAVA. Another important topic that will probably be extended at Linaro Connect is finding a way to get the power consumption data when running the test cases/benchmarks, so it can be further optimised later on.
  • Compiz GLES2 Handover
    • Last session of the day, trying to find the missing gaps to finally get the OpenGL ES2.0 support merged at the Compiz and Unity upstream branches used by the entire Ubuntu desktop (across all archs). Following work and actions will basically be to fix the remaining and important plugins after merging the changes, and also getting a few test cases to properly validate the support at Ubuntu. Once all done, it should be merged ASAP.

These are just a few topics which I was able to participate. There are a lot of more exciting work coming on, which can all be found at http://summit.ubuntu.com/uds-q/. Remember that you’re still able to participate in a few of them tomorrow and friday, as remote access is provided for all the sessions we have.

I’m sure a lot of more exciting stuff will be discussed for ARM support until the end of this week, and at Linaro Connect, at the end of the month, we’ll be able to review and get our hands dirty as well :-)

Exciting times for ARM!


Read more
Chris Johnston

Over the past few months a lot of work has been done to the Summit Scheduler. This all culminated this past week when I sent in an RT to update the production instance of Summit. This included somewhere in the neighborhood of 8,000 lines of code change between Summit and the two themes that are run.

There are two major changes to Summit from the changes this past week. The first is that Summit has been rethemed to meet the new design guidelines for Ubuntu. This work was completed with major assistance by Alexander Fougner, Stephen Williams, and Brandon Holtsclaw who each put in numerous hours to make this happen.

The other major change, which affects the way that you will work with Summit is that you now have the ability to propose meetings in Summit removing the requirement that you create a blueprint. Also, as long as you’re using the new propose meeting feature in Summit, you no longer have to have a special name for your meeting, this is now done automatically. I also created a video which will walk you through the process of proposing a meeting in Summit.

As always, please file bugs if you find any issues!

Read more
Chris Johnston

The Summit Hackers have been hard at work again creating new features to make Summit easier to use. However, we need your help testing these features before the public release. This is where you, the user, come in! I have created a Test Summit on my server and created a test sprint in Launchpad and here is where I need your help with testing:

First, visit the test sprint in Launchpad, next mark yourself as attending. Please note, it could take up to 20 minutes before you are able to use summit after marking yourself as attending. You can either attend physically or remotely, it doesn’t matter. While you are there, take the time to create a couple of test meetings. Please be sure when you create these test meetings to follow the naming guidelines below.

Hey Chris, what are the new features? I am so glad you asked! The new features are:

For Attendees:

Summit now supports the ability for attendees to propose meetings right inside of Summit! On certain pages in Summit, you will now see an “Actions” area. Depending on what part of the cycle we are in, you may see different links. During the sponsorship application process, you will see links to apply for sponsorship. If you have the ability to review sponsorship applications, this is where you will find that link during that time period. During the scheduling phase of the cycle, you will be able to “Propose a Meeting.” What this means is that you will create a meeting in Summit, which will then go into a holding pattern waiting for a Track Lead or Summit Organizer to approve or deny your meeting. You may be familiar with this behavior, as it isThis is a very similar behavior as it has been with what you may have seen with Launchpad.

For Track Leads, Schedulers, and Organizers:

If you are listed as a Track Lead, Scheduler, or Organizer in Summit, you will have the ability to “Create a Meeting” and “Review Proposed Meetings.” When creating a meeting, you have the ability to mark a meeting as Private. If a meeting is marked as private, you will have to get it scheduled through a scheduler (Michelle and Marianna for UDS, Arwen for Linaro Connect), it will not be automatically scheduled. Non-private meetings that are created by a Track Lead, Scheduler, or Organizer will be automatically scheduled when slots are available.

The other option that this group has is reviewing proposed meetings. There are three different options for the status, Approved, Pending, and Declined, which you get to by clicking the current status. Pending and Declined meetings show up on this page, while Approved meetings are auto-scheduled and no not show up here.

Known Issues:
* All tracks are shown in the track choice list, not just the ones for the current summit
* Edit meeting links don’t work
* Issues with the form theme

Creating Meetings through Launchpad

The good news is–the old way still works! You will still able to create meetings through Launchpad Blueprints just like before. There are a few things you will need to be aware of as you create meetings this way, these are as follows:

Tracks:

I have created a couple of tracks for this Test Summit. They are as follows:

Name                         Slug
Community        community
Linaro                     linaro
QA                               qa
Ubuntu                  ubuntu

Blueprint naming standards

When creating a blueprint in Launchpad, please follow the required naming standards whereis a custom name that you create for your meeting:

For the Community track:

community-

For the Linaro track:

linaro-

For the QA track:

qa-

For the Ubuntu track:

ubuntu-

Managers, Organizers, Track Leads, oh my!

If you are a member of ~summit-users on Launchpad, please take the time to approve/deny blueprints in launchpad.. I would like to have a couple denied, just to make sure everything still works properly. The same thing with the new meeting review feature in Summit. Please do not approve all meetings! Some of them need to be declined to ensure that everything is working properly!

IT’S VERY IMPORTANT that people who are in the role of Track Lead, Scheduler, or Organizer for UDS and Linaro Connect test things and give your feedback, please don’t wait until the week before your event to test this, please be considerate to those volunteers who are working to make this better for everyone. If you feel that you should have this access but don’t, please contact me and I will get you setup.

If you run into any problems, please file a bug and talk to me on IRC!

I look forward to your feedback and continuing to make summit the best possible scheduler it can be, and to do so requires your help. Thank in advance.

Read more
rsalveti

For those following the development of the next Ubuntu release (12.04 – Precise Pangolin), you all know that we’re quite close to the release date already, and to make sure Precise rocks since day 0, we all need to work hard to get most of the bugs sorted out during the next few weeks.

At Linaro, the Linaro Developer Platform team will be organizing an ARM porting Jam this Friday, with the goal of getting all developers interested in fixing and working on bugs and portability issues related with the Ubuntu ARM port (mostly issues with ARMHF at the moment).

The idea of having the Porting Jam at Friday is to have it as a joint effort with Ubuntu’s Fix Friday and Ubuntu Global Jam, so expect quite a few other developers helping improving Ubuntu as well!

It’s quite easy to participate:

Remember that for ARM this release will be a quite huge milestone, as it’ll be the first LTS release supporting ARM, besides delivering support for ARM servers and ARMHF as default, so let’s make sure it rocks!

Looking forward for a great porting Jam!

Happy bug fixing!


Read more
Chris Johnston

My buddy Dustin Kirkland pointed me to a neat little utility that he wrote with Scott Moser called ssh-import-id. Since he showed it to me a few months ago, I have used it many times and it has made my life quite a bit easier.

ssh-import-id fetches a the defined user(s) public keys from Launchpad, validates them, and then adds them to the ~/.ssh/authorized_keys file. That’s it, but if you need to add multiple people, or don’t know which key they are going to want used, this will save you time.

Dustin has tried to get it added to OpenSSH, but he hasn’t been able to succeed at this yet.

To use ssh-import-id, you first need to install it if it already isn’t:

sudo apt-get install ssh-import-id

Then to run it you would run:

ssh-import-id chrisjohnston

This would import my public keys. You are also able to import multiple users at the same time:

ssh-import-id chrisjohnston kirkland

If you are looking for the latest version of the code it is available in a ppa:

ppa:ssh-import-id/ppa

If you have problems or want to check out the code, check out the package on Launchpad.

Read more
Chris Johnston

This week I will be attending Linaro Connect Q1.12 in Redwood City, California. Infact, I’m in an American Airlines plane at 34,000 feet heading there now. In-flight WiFi is awesome!

Over the past two months Michael Hall and myself have been doing a large amount of work on The Summit Scheduler to get it ready for Connect this week including modifying more than 2,400 lines of Summit code. You can find out more about that in my previous post.

I have a few things that I want to get out of Connect. The first is that I want to get feedback on the changes to Summit, as well as figure out what other things we may need to change. The second thing that I want to do is to learn more about the Beagleboard-xM that I have and how to use it for the many different things it can be used for. The third thing that I want to do is to learn about Linaro’s LAVA.

LAVA is an automated validation suite used to run all sorts of different tests on the products that Linaro produces. The things that I would like to get out of Connect in relation to LAVA are how to setup and run LAVA, how to set it up to run tests, and how to produce results and display those results the way that I want them.

If you are at Linaro Connect, and would be willing to talk with me about Summit and the way you use it and your thoughts on the changes, please contact me and we will set aside a time to meet.

Read more
rsalveti

Yesterday Canonical announced the first UI concept for the Ubuntu TV. Together with the announcement, the first code drop was released, so we could read and understand better the technologies used, and how this will behave on an ARM environment, mostly at a Pandaboard (that we already have OpenGL ES 2 and video decode working).

Getting Ubuntu TV to work

If are still using Oneiric, you can just follow the guide presented at https://wiki.ubuntu.com/UbuntuTV/Contributing, where you’ll find all needed steps to try Ubuntu TV at your machine.

As it’s quite close with Unity 2D (similar code base), and also based on Qt, I decided to follow the steps described at wiki page and see if it should work correctly.

First issue we found with Qt, was that it wasn’t rendering at full screen when using with latest PowerVR SGX drivers, so any application you wanted to use with Qt Opengl would just show itself on a small part of the screen. Luckily TI (Nicolas Dechesne and Xavier Boudet) quickly provided me a new release of the driver, fixing this issue (version that should be around later today at the Linaro Overlay), so I could continue my journey :-)

Next problem was that Qt was enabling brokenTexSubImage and brokenFBOReadBack for the SGX drivers based on the old versions available for Beagle, and seems this is not needed anymore with the current version available at Pandaboard (still to be reviewed with TI, so a proper solution can be forwarded to Qt).

Code removed, patch applied and package built (after many hours), and I was finally able to successfully open the Ubuntu TV interface at my Panda :-)

UI Navigation on a Pandaboard, with Qt and OpenGL ES2.0

Running Ubuntu TV is quite simple if you’re already running the Unity 2D interface. All you need to do is to make sure you kill all unity-2d components and that you’re running metacity without composite enabled. Other than that you just run ”unity-2d-shell -opengl” and voilà ;-)

Here’s a video of the current interface running on my Panda:

As you can see from the video, I didn’t actually play any video, and that’s because currently we’re lacking a generic texture handler for OpenGL ES with Gstreamer at Qtmobility (there’s only one available, but specifically for Meego). Once that’s fixed, the video playback should behave similarly as with XBMC (but with less hacks, as it’s a native GST backend).

Next steps, enabling proper video decode

Looking at what would be needed to finally be able to play the videos, and to make it something useful at your Pandaboard, the first thing is that we need to improve Qtmobility to have a more generic (but unfortunately still specific to Omap) way handle texture streaming with Gstreamer and OpenGL ES. Rob Clark added a similar functionality at XBMC, creating support for ”eglImage”, so we just need to port the work and make sure it works properly with Qtmobility.

Once that’s ported, the video should be streamed as a texture at the video surface, making it also work transparently with QML (the way it’s done with Ubuntu TV).

If you know Qt and Gstreamer, and also want to help getting it to work properly on your panda, here follows a few resources:

As soon video decoding is working properly, a new blog post should be around explaining the details and how to reproduce it at your own Panda with Ubuntu LEB :-)

Cheers!


Read more
rsalveti

As described on my previous post about Ubuntu TV support on a Pandaboard, we were still missing proper support for texture streaming on a Pandaboard, to have the video playback also working and fully accelerated.

This weekend Rob Clark managed to create the first version of the TI’s specific eglImage support at Qtmobility, posting the code at his gitorious account, and for the first time we’re fully able to use Ubuntu TV on a ARM device, using a Pandaboard.

Demo video with the Ubuntu TV UI (accelerated with Qt and OpenGL ES 2.0) and with video decode support of 720p and 1080p:

The code support for TI’s eglImage still needs a few clean-ups, but we hope to be able to push the support at Ubuntu in the following weeks (make it good enough to try at least a package patch).

For people wanting to try it out, a few packages are already available at Linaro’s Overlay PPA, and the remaining ones should be available later today (Qt and Qtmobility), so people can easily run it with our images.

Hope you enjoy, and we’ll make sure we’re always working on keeping and improving the current support, so Ubuntu TV also rocks with ARM :-)

Cheers!


Read more
rsalveti

During the end of October and beginning of November we had the last Linaro Connect for the year. This time we also had it together with the Ubuntu Developer Summit, giving us the opportunity to better discuss the roadmap with both Linaro and the Ubuntu team.

From the Developer Platform team perspective, we had a quite nice week, with demos happening at Monday and Friday (showing people what we’ve been working on), and also sharing some great news with the Ubuntu team, now that Mark Shuttleworth announced that Ubuntu will go to Tablets, TVs and Phones (and ARM for sure will be a huge part of that).

Some nice links and videos of what happened during that week (related with our team):
* Sessions related with the Developer Platform Team (Ubuntu)
* Linaro Demo: Ubuntu Unity with OpenGL ES on Pandaboard
* Linaro Developer Platform Tech Lead Ricardo Salveti Interview at Linaro Connect
* Linaro Connect Q4.11 – Ubuntu LEB tutorial
* Linaro Connect Q4.11 – Interview with Marcin Juszkiewicz

Linaro 11.11 Release

Another quite good achievement for us during November was the 11.11 release.

During this release we had a quite a few great highlights, including some that we were planning for quite a while already:
* Ability to cross build Firefox using Multiarch
* OMAP4 SPL USB Booting, enabling USB boot at Pandaboard
* ARM DS-5 support for the 5.8 release
* CI Builds for Linaro GCC both for cross and native
* And a lot of bug fixes

Now it’s time to get ready to develop the blueprints we’re planning for 11.12, to also make December another great and solid month :-) (will do another post about the 11.12 planning later this one).


Read more
Michael Hudson-Doyle

The Ubuntu developers and, because Linaro started out with a fork of Ubuntu’s processes, the Linaro developers track much of their work in blueprints on Launchpad and work items in those blueprints.  We’ve even built a fairly sophisticated web site that tracks work item completion in a manager friendly way.

Editing these work items is a fairly tedious job however.  The syntax is fairly simple but still easy to get wrong, and fiddling around in a textarea just to change one work item to DONE is not very friendly.  So, after a challenge laid down by James, I’ve built a Greasemonkey script that adds a button to blueprint pages which opens an editor that lets you easily change the status or and add work items to the blueprint.

In action, it looks like this (it uses the LP JavaScript widgets, so it almost feels like a native part of Launchpad):

There are probably a few bugs, and certainly things it doesn’t do (off the top of my head: track the milestone a work item is allocated to, let you delete or reorder work items or allow you to change the assignee of a workitem with a nice person picker) but I think it will save us all a bit of time every day even in its current state.

The script is now part of the launchpad-gm-scripts project on Launchpad.  I don’t really know the best way of installing greasemonkey scripts yet.  Grabbing the branch, opening it in Nautilus and dragging it into a firefox window worked for me.

Chrome/chromium has a similar extension, but it hasn’t been tested there.  If you do, I’d love to know if it works :)  Please submit bug reports or merge proposals on LP if you find problems!

Read more
Michael Hudson-Doyle

It seems like it’s taken a long time to get all the pieces hooked up, but I’m very happy to report that finally you can see the results of testing an android build in LAVA directly in the build page!  If you go to a build page for a build that is automatically tested such as https://android-build.linaro.org/builds/~linaro-android/panda/#build=250 (after a little while for some ajax to happen), you should see a table listing the test runs we ran and the summary of the pass/fail counts:

It may not seem all that earth shattering, but there have been many bits and pieces that needed to be put together to get this to work:

  • We needed to build the scheduler, so that we could submit this job to run on the first Panda board to become available.
  • We needed to build the infrastructure to allow the job to be submitted with some measure of security from a remote host.
  • We needed to work on the build system scripts to submit the job without disclosing the authorization token.
  • We needed to add views to the scheduler and dashboard that present data in a ajax-friendly way.
  • We needed to work on the build system frontend to make uses of these views and format the data.
  • And finally, we needed to test and deploy all of these changes.

So I think I’m justified in being happy to have this finally working in production :-)  Of course, it’s just a start: we want to build similar facilities for the other working groups to use, if nothing else.

Read more
Michael Hudson-Doyle

A few weeks ago now, most of the Linaro engineers met at “Linaro Connect”, the new name for our get-together.  Linaro bootstrapped its processes by borrow heavily from Ubuntu, including the “two planning meetings, two hacking meetings” pattern. Over the last year though it’s become clear that this isn’t totally appropriate for Linaro and while we’re sticking to the same number of meetings, 4 a year, each meeting now has the same status and will be a mix of planning and hacking.  Based on a sample size of 1, this seems to be a good idea – last week’s meeting was excellent.  Very intense, which is why I never got around to blogging during the event, but also very productive.

The validation team had a dedicated hacking room, and on Monday we set up a “mini-Lab” that we could run tests on.  This took a surprisingly (and pleasingly) short amount of time, although we weren’t as neat about the cabling as we are in the real lab:

The main awkwardness in this kind of setup where you are connecting to the serial ports via USB rather than a console server is that the device names of the usb serial dongles is not predictable, and so naming boards becomes a challenge.  Dave worked out a set of hacks to mostly make this work, although I know nothing about the details.

Now that a few weeks have passed I can’t really remember what we did next :)  There was a lot of hacking and a lot talking.  These are some things I remember:

  • I spent some time talking to the Android developers about getting the results of the tests to display on the build page. Luckily there were no new surprises and I managed to come up with a plan for getting this to work (have the process that runs the tests and uploads the bundle to the dashboard print out the URL to the result bundle and have the lava scheduler read this and record the link).
  • We all talked to the kernel team about how to test their work on an automated basis.
  • I talked to Michael Hope about the toolchain builds that are currently done in his basement, although we mostly deferred that conversation until after the event itself.
  • There was a lot of talk about making the front page of the validation server show something more useful.
  • I implemented a prototype for replacing QATracker with something that could guide a user through manual tests and upload the results directly to the dashboard.
  • We talked to ARM about possibly using some of the LAVA components we have built for their internal testing,
  • There was talk about the practicalities of using the LAVA lab to measure the effect of power management changes.

I’m sure there was lots of other stuff, but this should give some impression of how much went on!

Read more

At Linaro we want to get some metrics for the patches we submit upstream, so we've built a system based on Patchwork that parses email messages to extract patches and categorize them by project/author. This works fine for projects that use mailing lists to keep track of patches (e.g. the kernel, qemu, etc), but it doesn't work for one project to which Linaro has already contributed and expect to contribute a lot more: the Android Open Source Project, which uses Gerrit to track changes.

For that reason we decided to pull that data directly from AOSP's Gerrit instance into our system. Unfortunately, the only public API provided by Gerrit is this one over SSH, which doesn't give a piece of data that is very important to us: the date a change was proposed.

Luckily, James pointed me to this discussion where a neat trick is suggested: watch the requests your browser sends when rendering http://review.source.android.com to figure out how to use Gerrit's internal JSON-RPC API. Yes, it is not a public API (so we should not expect it to be stable), and having to watch your browser requests to learn how to use it is not the kind of documentation I'd like, but that was all we had (ok, I could check Gerrit's source, but since that's Java I'd rather watch the requests sent by the browser) so it had to do.

After experimenting a bit I was able to get a list of changes proposed by a given person as well as all the details of a given change. Here are the methods one can use for that:

Retrieving a list of changes


  endpoint: /gerrit/rpc/ChangeListService
    method: allQueryNext(query, pos, page_size)

  Return up to page_size items that match the given query whose changeSortKey is lower than pos.

  # This will get the list of changes authored by jserv@0xlab.org
  curl -i -X POST -H "Accept: application/json,application/jsonrequest" \
       -H "Content-Type: application/json; charset=UTF-8" \
       --data '{"jsonrpc":"2.0","method":"allQueryNext","params":["owner: jserv@0xlab.org","z",10],"id":1}'\
       https://review.source.android.com/gerrit/rpc/ChangeListService


Getting the details of a change


  endpoint: gerrit/rpc/ChangeDetailService
    method: changeDetail(id)

  Return the details of the change with the given ID.

  # This will get all details of change 16615
  curl -i -X POST -H "Accept: application/json,application/jsonrequest" \
       -H "Content-Type: application/json; charset=UTF-8" \
       --data '{"jsonrpc":"2.0","method":"changeDetail","params":[{"id":16615}],"id":1}'\
       https://review.source.android.com/gerrit/rpc/ChangeDetailService


Pretty simple, right? Just note that you need to specify your charset in the Content-Type header or else you'll get a server error from Gerrit, and the
JSON-RPC requires the 'id' param to correlate a response with its request, but you don't need to worry about that if you're doing things synchronously.

That was all I needed to know in order to write the code that keeps the changes authored by Linaro engineers in sync between our system and AOSP's Gerrit instance, and it should be enough for you to get started if you ever need to get data out of a Gerrit instance (assuming the API hasn't changed in the meantime ;).

Read more
Michael Hudson

This is the first of a hopefully weekly series of posts describing the work my team is doing.  This means that this post is probably mostly background about the team’s goals, but in the coming weeks I intend to outline what we’ve done in the past week and plans for the next week.

We’re all about validation obviously – telling whether the code the other Linaro engineers are producing “works” in whatever sense that means.  It could be a simple compile or boot test for the kernel, testing whether the code produced by gcc is smaller or faster, whether a kernel scheduler change reduces power consumption for a certain workload, or many other things.

Beyond simple validation though, what we’re really about is automated validation.  We want to build and test the kernel on all supported boards every day.  We want to build and test proposed android changes in gerrit before they are landed, and the same for the gcc work.

We have built up a validation lab in Cambridge – the boards from the Linaro members we want to test on, but also Cyclades serial console servers, routers, and a few servers.  It looks a bit like this:

The thing that makes our task more complicated than “just install jenkins” is the hardware we run on, of course, and the fact that for many of our tests we need to boot a custom kernel on said hardware.  We’ve written a program (called “lava-dispatcher” or just “the dispatcher”) that knows how to install a custom hwpack and root filesystem by manipulating a board over a serial link, another (“lava-scheduler”) that handles incoming requests to run tests and runs the dispatcher as appropriate and yet another (“lava-dashboard”, aka “launch-control”) that displays the results from the tests.  We’ve also built up a number of infrastructure projects that help us run these main three, and command line clients for most things.  You can see all the code at https://code.launchpad.net/lava and the running instance is at http://validation.linaro.org/lava-server.

So, what are we working on this week?  The main areas are to improve the UI of the scheduler – currently it runs jobs, but is very opaque about what they are doing, improving the front page to make it clearer to the uninitiated what validation is happening and improving the reliability of the dispatcher.  We’re also hard at work “joining the dots” so that the daily builds of Android that are already being produced can be tested daily, and have the build output and test results all visible from the same page.

Next week I’ll be at Linaro Connect in the UK, but I’ll try to update here on what we get done this week and what our plans are for the Connect.

Read more
rsalveti

As at Linaro we usually work with many PPAs over the releases, there was a need to generate a proper changelog for a PPA, in a way we could know what packages got changed before doing the release.

At first I thought I could just parse the repository metadata (as a PPA is nothing more than a debian repository), but then I realized I could just use the awesome (yes, *awesome*) launchpadlib, if it had a way to get the data I needed.

So I called the launchpadlib master I know (Ursinha), and in 15 minutes we saw that we could use it to parse the “.changes” file, and from there get the data I needed. As Launchpad stores the PPA packages publishing history, it’s quite easy to get all the changes over period of time.

A few minutes later (after also noticing that there’s a python-debian module to parse the changes file), I created the first version of the generate-ppa-changelog.py script, that does exactly what I needed, and with just a few python lines :-)

Here’s the link: http://bazaar.launchpad.net/~rsalveti/+junk/launchpad/view/head:/generate-ppa-changelog.py.

Usage:

rsalveti@evatp:~/projects/launchpad/scripts$ python generate-ppa-changelog.py -h
usage: generate-ppa-changelog.py [-h] [-d YYYYMMDD] [-s SERIES] -t TEAM
[-p PPA] [--version]

Print Changelog for a PPA

optional arguments:
-h, –help show this help message and exit
-d YYYYMMDD, –date YYYYMMDD
start date to probe for changes
-s SERIES, –series SERIES
ubuntu series to look for changes (default: natty)
-t TEAM, –team TEAM launchpad team that owns the PPA
-p PPA, –ppa PPA ppa name to probe the changelog (default: first PPA)
–version show program’s version number and exit

If no argument is given, it will probe all the changes for the default series.

Output Example:

rsalveti@evatp:~/projects/launchpad/scripts$ python generate-ppa-changelog.py -t linaro-maintainers -p overlay -s natty -d 20110701
Changelog for linaro-maintainers’s overlay PPA (series natty) since 2011-07-01 00:00:00

base-files (5.0.0ubuntu28linaro3) natty; urgency=low

* Updating Linaro LEB version to 11.07 (development branch)

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 04:19:40 -0300

libjpeg-turbo (1.1.1-1inaro2) natty; urgency=low

* release
* add timestamp code to cjpeg.c
* default cjpeg.c and djpeg.c timestamp code to off

— Tom Gall Mon, 11 Jul 2011 20:32:23 +0000

linaro-meta (017) natty; urgency=low

* Refreshed dependencies
* Added libjpeg-turbo62 to linaro-alip, linaro-graphical-engineering,
linaro-multimedia-engineering, linaro-ubuntu-desktop

— Tom Gall Wed, 13 Jul 2011 15:41:29 +0000

u-boot-linaro (2011.07.1-0ubuntu1~natty1) natty; urgency=low

* New upstream 2011.07.1 which includes
– PXE FDT fix from previous ubuntu release so patch has been removed
– Added missing PXE env vars (LP: #808815)
– Generated unique usbethaddr (LP: #809015)
– Modify pxe command to look for usbethaddr if ethaddr is NULL

— John Rigby Wed, 13 Jul 2011 23:31:39 -0600

x-loader (1.5.1+git20110715+fca7cd2-1ubuntu1~natty1) natty; urgency=low

* New upstream release
– several fixes for panda and igep
– OMAP3 code cleanup
– Beagle Rev C5 support
– Support for IGEPv3 board

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 05:55:09 -0300

x-loader (1.5.0+git20110714+cdc887b-1ubuntu1~natty1) natty; urgency=low

* New upstream release
– Adding support for IGEPv3 board
* debian/patches/01-Beagle-Rev-C5-support.patch:
– Adding support for the new Beagle C5

— Ricardo Salveti de Araujo Fri, 15 Jul 2011 02:36:58 -0300

And now we can just generate the proper changelog anytime we want to do a call for testing or a Linaro Ubuntu release :-)


Read more
rsalveti

Over the past month I’ve being working with John Rigby to integrate the SMSC95XX and OMAP4 EHCI patches into Linaro U-Boot, so we could deliver the network booting feature for people using Pandaboards.

Those patches are published at the U-Boot mailing list, but still as a working in progress. While we work helping the original developers to get the patches accepted upstream, we also want to deliver the functionality for our users, so all those patches are now integrated at the Linaro U-Boot tree.

You can check the patches by going at http://git.linaro.org/gitweb?p=boot/u-boot-linaro-stable.git;a=shortlog.

Testing with Pandaboard

To make it work properly, besides using Linaro U-Boot you’ll also need to use the upstream X-Loader tree, with one additional patch that’s not yet merged. You can clone the upstream tree from http://gitorious.org/x-loader/x-loader, then just apply the patch http://people.canonical.com/~rsalveti/pxe/0001-omap4-pandaboard-ehci-fref_clkout-per-board-revision.patch and build for the Pandaboard target.

If you just want to test without building your own X-Loader and U-Boot, you can just grab both files from  http://people.canonical.com/~rsalveti:

Building your TFTP + DHCP server for PXE

To build your TFTP + DCHP server just follow the instructions described at https://help.ubuntu.com/community/Desktop/PXE. Don’t worry about the ‘filename “pxelinux.0″;’ line at the dhcpd.conf file, you can remove it.

Then just create your PXE config file at the right place:

$ cat /tftpboot/pxelinux.cfg/0A2A2B0A
default panda-natty
prompt 0
timeout 3

label panda-natty
kernel panda/uImage
append console=ttyO2,115200n8 root=/dev/mmcblk0p2 ro fixrtc vram=48M omapfb.vram=0:24M mem=1G@0x80000000 text earlyprintk=ttyO2
initrd panda/uInitrd

PXE Booting

With the proper X-Loader and U-Boot files in place (at your first SD card partition), and with the TFTP + DHCP server also properly installed, you can just jump and try TFTP/PXE boot.

Stop the U-Boot autoload and call the following commands:

  • setenv pxecfg_ram 0x88000000: location in RAM to load the pxecfg file
  • setenv kernel_ram 0x80000000: location in RAM to load the kernel
  • setenv initrd_ram 0x81600000: location in RAM to load the initrd
  • setenv autoload no: disable autoload while calling bootp (so you can just set up your network without autoboot)
  • usb start: start USB and enables the SMSC95xx ethernet interface
  • bootp: initialize the network, probing the ip address settings from your DHCP server
  • pxecfg get: probe the pxecfg config file
  • pxecfg boot: boot :-)

You should get a similar output as:

Texas Instruments X-Loader 1.5.0 (Jul 11 2011 – 07:52:49)
Reading boot sector
Loading u-boot.bin from mmc

U-Boot 2011.06 (Jul 11 2011 – 02:49:51)

CPU : OMAP4430
Board: OMAP4 Panda
I2C: ready
DRAM: 1 GiB
MMC: OMAP SD/MMC: 0
Using default environment

In: serial
Out: serial
Err: serial
Net: No ethernet found.
Hit any key to stop autoboot: 0
Panda # setenv pxecfg_ram 0x88000000
Panda # setenv kernel_ram 0x80000000
Panda # setenv initrd_ram 0x81600000
Panda # setenv autoload no
Panda # usb start
(Re)start USB…
USB: Register 1313 NbrPorts 3
USB EHCI 1.00
scanning bus for devices… The request port(2) is not configured
EHCI timed out on TD – token=0x80008c80
The request port(2) is not configured
4 USB Device(s) found
scanning bus for storage devices… 0 Storage Device(s) found
scanning bus for ethernet devices… 1 Ethernet Device(s) found
Panda # bootp
Waiting for Ethernet connection… done.
BOOTP broadcast 1
DHCP client bound to address 10.42.43.10
Panda # pxecfg get
missing environment variable: pxeuuid
missing environment variable: ethaddr
Retreiving file: pxelinux.cfg/0A2A2B0A
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘pxelinux.cfg/0A2A2B0A’.
Load address: 0x88000000
Loading: #
done
Bytes transferred = 239 (ef hex)
Config file found
Panda # pxecfg boot
Hit any key to stop autoboot: 0
Label: panda-natty
kernel: panda/uImage
append: console=ttyO2,115200n8 root=/dev/mmcblk0p2 ro fixrtc vram=48M omapfb.vram=0:24M mem=1G@0x80000000 text earlyprintk=ttyO2
initrd: panda/uInitrd
Retreiving file: panda/uInitrd
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘panda/uInitrd’.
Load address: 0x81600000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
############
done
Bytes transferred = 3982715 (3cc57b hex)
Retreiving file: panda/uImage
Waiting for Ethernet connection… done.
Using sms0 device
TFTP from server 10.42.43.1; our IP address is 10.42.43.10
Filename ‘panda/uImage’.
Load address: 0x80000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#########################
done
Bytes transferred = 4174480 (3fb290 hex)
## Booting kernel from Legacy Image at 80000000 …
Image Name: Ubuntu Kernel
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4174416 Bytes = 4 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum … OK
## Loading init Ramdisk from Legacy Image at 81600000 …
Image Name: Ubuntu Initrd
Image Type: ARM Linux RAMDisk Image (uncompressed)
Data Size: 3982651 Bytes = 3.8 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum … OK
Loading Kernel Image … OK
OK

Starting kernel …

Uncompressing Linux… done, booting the kernel.
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu

This should be enough for you to get your Pandaboard booting with PXE. You can also script these commands at your boot.scr file that U-Boot loads automatically from your SD card, so you don’t have to call them by hand every time you reboot your board.

In case it doesn’t work for you, just ping me (rsalveti) at #linaro on freenode :-)


Read more
Michael Hudson-Doyle

Installing Linaro for a Beagle xM

After I’d unpacked and booted my xM, I wanted to install a Linaro daily build on it.  This was actually fairly complicated for me because of a few bugs I ran into on the way, but as they’re all fixed now I’ll describe the way it’s meant to work :-)

Basically, the instructions on the wiki are what you want.  You can download the latest snapshot from http://snapshots.linaro.org/10.11-daily/linaro-headless/ (which is what I’d recommend at this point; I can state that the 20100915-1 build works for me) or you can navigate your way to a more official release from https://wiki.linaro.org/Releases (but don’t use the Linaro 10.11 Beta — it has a not very serious kernel bug that makes upgrades harder than needed on xMs).

Once you’ve downloaded the file (using dlimage or just boringly) and run linaro-media-create with a command line like

sudo linaro-media-create --dev beagle --rootfs ext3 --mmc /dev/sdb \
    --binary ~/Downloads/linaro-m-headless-tar-20100915-1.tar.gz

(make sure you get the –mmc bit right!), pop the card into your board, power it up and with the serial console connected run “screen /dev/ttyUSB0 115200″ again.  The Linaro image is slightly different to the one that comes with the board in that you get a root prompt directly on the serial console, no need to log in.

As an aside, when I want to boot on a different card, I usually type ‘poweroff’ on the serial console, pull the card out, pop the new one in and press the reset button.  I don’t know if this is the best process :-)  There is a kernel bug that prevents clean shutdown after the card has been on for a while, but it happens late enough in the shutdown process that I ignore it.

Next up, I’ll talk about how I set up my cards for networking and general user-level hackery.

Read more
Michael Hudson

As I said in my previous post, I recently received a Beagle Board xM to test some of my work stuff on and also just to get a bit more familiar with the world of ARM development.

When I got the board, I had no idea what to do with it.  None.  There didn’t seem to be a guide that I could find with google for people with my level of utter inexperience, so I thought I’d try to write one up.

First things first, you need some gadgets.  Well, to get started the only thing you really need is a USB->Serial adapter.  I don’t know much about these gadgets but was recommended this one and it works fine for me.  While you’re in the shop, you’ll probably want to buy a handful of microSD cards.  The faster and bigger the better probably, but 2 gigs or bigger is fine for playing around.  You’ll need some way of writing to these cards as well, of course.

The xM comes with a microSD card that contains “The Ångström Distribution”.  I don’t know what this is really, but it’s good to test that the board works, so pop the card in the slot in the board and plug the board in (I agonized for ages about this — I didn’t want to just plug it in in case a brief power contact would damage the board or something, but it seems to be the thing to do).  Then connect up the USB/serial adapter and run “screen /dev/ttyUSB0 115200″.  If the board has booted fully, you should get to a tty login.  Type ‘root’ as the user name and you’re in!  This default install is fairly plain — it’s a fairly minimal linux using busybox.  But if it boots, the board likely at least somewhat works.

Next up is installing a Linaro daily build on the board and getting networking working so that you can install all the fun software that makes up Ubuntu!

If anything in this post strikes you as wrong or unclear, please let me know in the comments.

Read more
Michael Hudson

My new(-ish) job

After three years of working on Launchpad, back in May I transferred over to working for Linaro.  On the face of it, this was a large move, moving from hacking on a programmer-supporting tool to making Linux work better on ARM processors, but I’m not working on the kernel or anything, in fact I’m still working on programmer-supporting tools, even hacking on Launchpad a bit from time to time.

Last week I received an ARM board of my very own to play with, a Beagle Board xM.  My first steps with this will be the subject of my next blog post…

Read more