Canonical Voices

Michael Hall

As part of the continued development of the Ubuntu platform, the Content Hub has gained the ability to share links (and soon text) as a content type, just as it has been able to share images and other file-based content in the past. This allows applications to more easily, and more consistently, share things to a user’s social media accounts.

Consolidating APIs

facebook-sharing
Thanks to the collaborative work going on between the Content Hub and the Ubuntu Webapps developers, it is now possible for remote websites to be packaged with local user scripts that provide deep integration with our platform services. One of the first to take advantage of this is the Facebook webapp, which while displaying remote content via a web browser wrapper, is also a Content Hub importer. This means that when you go to share an image from the Gallery app, the Facebook webapp is displayed as an optional sharing target for that image. If you select it, it will use the Facebook web interface to upload that image to your timeline, without having to go through the separate Friends API.

This work not only brings the social sharing user experience inline with the rest of the system’s content sharing experience, it also provide a much simpler API for application developers to use for accomplishing the same thing. As a result, the Friends API is being deprecated in favor of the new Content Hub functionality.

What it means for App Devs

Because this is an API change, there are things that you as an app developer need to be aware of. First, though the API is being deprecated immediately, it is not being removed from the device images until after the release of 14.10, which will continue to support the ubuntu-sdk-14.04 framework which included the Friends API. The API will not be included in the final ubuntu-sdk-14.10 framework, or any new 14.10-dev frameworks after -dev2.

After the 14.10 release in October, when device images start to build for utopic+1, the ubuntu-sdk-14.04 framework will no longer be on the images. So if you haven’t updated your Click package by then to use the ubuntu-sdk-14.10 framework, it won’t be available to install on devices with the new image. If you are not using the Friends API, this would simply be a matter of changing your package metadata to the new framework version.  For new apps, it will default to the newer version to begin with, so you shouldn’t have to do anything.

Read more
Jane Silber

The social and business value of having a diverse workforce is well documented.  Equally well documented is the relative lack of women in technology, and in open source.

At Canonical we are working hard to build a globally diverse workforce. We are well positioned to do so, particularly building on our open source roots, and in areas such as supporting geographic diversity we are quite successful.   However, in terms of gender diversity, women make up only 13% of Canonical and, slightly more encouragingly, 18% of our managers.   It is disappointing to me that despite having one of the most welcoming, collaborative, flexible and meritocratic environments I have known, we still have such a large gender disparity.

As a woman in technology and a CEO, I am aware of the power of positive examples.  While we need to learn from and eliminate the discouragement, barriers and illegal behaviour which continues to haunt women in technology, we should also celebrate the possibilities, highlight the opportunities and help illuminate a path for others to follow.  In that vein, I’d like to introduce you to a few of the amazing women in technical leadership roles in Canonical.

 

Alexis Bruemmer is the Engineering Manager for Canonical’s Juju team – a team of brilliant engineers working to make cloud orchestration easy, portable and flawless.  Alexis has been working in Linux since her graduation in 2005 and is passionate about open source.  Prior to Canonical, Alexis was at IBM’s Linux Technology Center.  Beyond her work as a professional, she is active in the community promoting STEM outreach as Vice Chair for Saturday Academy and long time member of Society of Women Engineers.

 

Ara Pulido is the Hardware Certification Manager at Canonical, leading the team that defines and ensures the quality bar for desktops and laptops pre-installed with Ubuntu. She discovered Free Software at college, where she was a founding member of the local LUG back in 2002. She joined Canonical 6 years ago in the Ubuntu Engineering QA team. You can follow her at https://twitter.com/arapulido.

 

Leann Ogasawara is the Engineering Manager for our Kernel Team, following a series of promotions at Canonical from Kernel QA to Kernel Engineer to overall team manager.  She has been involved in Linux and Open Source for over a decade.  Before coming to Canonical in 2007, Leann was at the Open Source Development Labs.

 

Pat Gaughen is the Engineering Manager for the fabulous Ubuntu Server and Openstack Development team.  She’s worked in Linux since 1999, and has been in love with Operating System internals for even longer. Prior to Canonical, Pat was at the IBM Linux Technology Center.

 

Roxanne Fan is the Quality Assurance Manager in our Devices Commercial Engineering team. She has been working in data mining for software quality improvement and automation tool development for the past 12 years. She wrote her Masters thesis on the performance of innovative routing for wireless sensor networks in the Ubuntu system. Before Canonical, she was at Pegatron Corp.

 

There are of course many reasons why women join and succeed at Canonical – great technology, inspirational colleagues, the opportunity to innovate, and to fundamentally have an impact on people’s mobile and cloud computing experiences.  Some of the less visible yet fundamental characteristics of Canonical which allow women to succeed in leadership positions include:

  • A commitment to a respectful, collaborative, meritocratic environment sets the stage. One of the earliest manifestations of this commitment was encoded in the Ubuntu Code of Conduct.  This clear statement of expectations has helped make the Ubuntu community a welcoming place for women, and applies in equal measure to Canonical.
  • Our recruitment philosophy of ‘hire only the best people’,  largely unrestricted by geographical boundaries, provides us with the opportunity to grow and support a diverse workforce.   It enables us to consider candidates of varying locations,  economic circumstances, gender, and physical ability.   Like all organisations we want the best person for the role, and leveraging our expertise in distributed, multi-cultural environments allows us to widen our recruiting net significantly.  Across all Canonical companies, our staff is 30% UK, 32% US, and 38% rest of world.  Those percentages are approximately the same when looking at all staff or management/leadership roles, thus providing excellent leadership opportunities in sometimes underserved markets.
  • We operate on a largely distributed environment and strive to support both home-based and office-based workers in equal measure.    With 75% of our employees working remotely we have an extremely high trust environment, thereby empowering employees to integrate working life with home life.  This approach has enabled us to retain men and women who otherwise may have left due to family demands.

I find the women above inspiring and am proud to work with them and many others of the same calibre. But we still have a long road to travel for our diversity figures to be where they should be.    As with the root causes of the problem, the solution is multi-faceted and complex.  We know that there is much more we can do to attract and retain greater diversity at Canonical, and are redoubling our efforts to do so.  As a first step, come join us!

Read more
Colin Ian King

a final few more features in stress-ng

While hoping to get a feature complete stress-ng sooner than later, I found a few more ways to fiendishly stress a system.

Stress-ng 0.01.22 will be landing soon in Ubuntu 14.10 with three more stress mechanisms:

  • CPU affinity stressing; this rapidly changes CPU affinity of the stress processes just to keep the scheduling busy wasting effort.
  • Timer stressing using the real-time clock; this allows one to generate a large amount of timer interrupts, so it is a useful interrupt saturation test.
  • Directory entry thrashing; this creates and deletes a selectable number of zero length files and hence populates and destroys directory entries.
I have also removed the need to use rand() for random number generation for some of the stress tests and re-used a the faster MWC "random" number generator to add in some well known and very simple math operations for CPU stressing.

Stress-ng now has 15 different simple stress mechanisms that exercise CPU, cache, memory, file system, I/O and CPU schedulers.  I could add more tests, but I think this is a large enough set to allow one to thrash a machine and see how well it performs under pressure.

Read more
facundo


Algunas tardes de sábado libre, más un par de viajes, hicieron que no me quede atrás con las películas...

  • A good old fashioned orgy: +0. De esas comedias livianas sobre la amistad y el amor. Divertida, no es gran cosa, pero está bien.
  • Apollo 18: -0. La idea general es interesante, pero ese querer simular una película con pedazos de filmaciones "reales" hace que todo sea muy forzado, especialmente cuando se nota lo "no real".
  • Catch .44: -0. Desordenada, aburrida, sin una historia que valga la pena.
  • Jack Ryan: Shadow recruit: +0. Una de acción, bien hecha, pero no más que eso.
  • Le noms des gens: +0. Una comedia simpática que plantea puntos interesantes sobre los franceses... y sobre el amor.
  • Like crazy: -0. Una historia de amor, que muestra las dificultades de las distancias. Aunque tiene sus momentos, en general la peli es muy lenta, y no entusiasma.
  • Prometheus: +0. Es una más de "alien", pero bien hecha, me gustó bastante, aunque al final no dejar de ser eso... una más de "alien" :)
  • Sherlock Holmes: A game of shadows: +0. Está perdiendo un poco la gracia (y es la segunda!), pero las buenas actuaciones y una historia interesante la salvan.
  • The darkest hour: -0. Un par de conceptos interesantes... pero no deja de ser una peli yanqui de adolescentes.
  • The girl with the dragon tattoo: +1. Una gran película, muy densa en contenidos (deja entrever que el libro es mucho más sabroso) y que te tiene atrapado hasta el final. Eso sí, muy dura (es un warning, no le resta).
  • The grey: +0. Fuerte, dura, pero bastante conceptual, con planteos que me gustaron. Estaría mejor si no estuviese bastante llena de golpes bajos al pedo (como ruidos que te quieren sorprender cuando lo único que sucedes es un cambio de encuadre que no tiene nada que ver).
  • The monuments men: +0. Un punto de vista sobre la guerra como destructor de culturas que no había pensado antes. Bien armada, con actuaciones decentes.
  • The rum diary: -0. Tiene un trasfondo interesante, pero la peli en general es aburrida, y no vale la pena.
  • Underworld: Awakening: -0. Ni siquiera más de lo mismo, antes las "underworld" tenían conceptos o historias interesantes... está es una mezcla mala de resident evil y blade; no más underworld para mí, gracias.


Pocas nuevas, eso sí, incluso metiendo una peli vieja en la lista...


Finalmente, el conteo de pendientes por fecha:

(Ene-2009)   12   1   1
(May-2009)   10   5
(Oct-2009)   15  14
(Mar-2010)   18  18  16   4
(Sep-2010)   18  18  18  18   9   2   1
(Dic-2010)   13  13  12  12  12   5   1
(Abr-2011)   23  23  23  23  23  22  17   4
(Ago-2011)   12  12  11  11  11  11  11  11   4
(Ene-2012)       21  21  18  17  17  17  17  11   3
(Jul-2012)           15  15  15  15  15  15  14  11
(Nov-2012)               12  12  11  11  11  11  11
(Feb-2013)                   19  19  16  15  14  14
(Jun-2013)                       19  18  16  15  15
(Sep-2013)                           18  18  18  18
(Dic-2013)                               14  14  12
(Abr-2014)                                    9   9
(Jul-2014)                                       10
Total:      127 125 117 113 118 121 125 121 110 103

Read more
Dustin Kirkland

Transcoding video is a very resource intensive process.

It can take many minutes to process a small, 30-second clip, or even hours to process a full movie.  There are numerous, excellent, open source video transcoding and processing tools freely available in Ubuntu, including libav-toolsffmpegmencoder, and handbrake.  Surprisingly, however, none of those support parallel computing easily or out of the box.  And disappointingly, I couldn't find any MPI support readily available either.

I happened to have an Orange Box for a few days recently, so I decided to tackle the problem myself, and develop a scalable, parallel video transcoding solution myself.  I'm delighted to share the result with you today!

When it comes to commercial video production, it can take thousands of machines, hundreds of compute hours to render a full movie.  I had the distinct privilege some time ago to visit WETA Digital in Wellington, New Zealand and tour the render farm that processed The Lord of the Rings triology, Avatar, and The Hobbit, etc.  And just a few weeks ago, I visited another quite visionary, cloud savvy digital film processing firm in Hollywood, called Digital Film Tree.

Windows and Mac OS may be the first platforms that come to mind, when you think about front end video production, Linux is far more widely used for batch video processing, and with Ubuntu, in particular, being extensively at both WETA Digital and Digital Film Tree, among others.

While I could have worked with any of a number of tools, I settled on avconv (the successor(?) of ffmpeg), as it was the first one that I got working well on my laptop, before scaling it out to the cluster.

I designed an approach on my whiteboard, in fact quite similar to some work I did parallelizing and scaling the john-the-ripper password quality checker.

At a high level, the algorithm looks like this:
  1. Create a shared network filesystem, simultaneously readable and writable by all nodes
  2. Have the master node split the work into even sized chunks for each worker
  3. Have each worker process their segment of the video, and raise a flag when done
  4. Have the master node wait for each of the all-done flags, and then concatenate the result
And that's exactly what I implemented that in a new transcode charm and transcode-cluster bundle.  It provides linear scalability and performance improvements, as you add additional units to the cluster.  A transcode job that takes 24 minutes on a single node, is down to 3 minutes on 8 worker nodes in the Orange Box, using Juju and MAAS against physical hardware nodes.


For the curious, the real magic is in the config-changed hook, which has decent inline documentation.



The trick, for anyone who might make their way into this by way of various StackExchange questions and (incorrect) answers, is in the command that splits up the original video (around line 54):

avconv -ss $start_time -i $filename -t $length -s $size -vcodec libx264 -acodec aac -bsf:v h264_mp4toannexb -f mpegts -strict experimental -y ${filename}.part${current_node}.ts

And the one that puts it back together (around line 72):

avconv -i concat:"$concat" -c copy -bsf:a aac_adtstoasc -y ${filename}_${size}_x264_aac.${format}

I found this post and this documentation particularly helpful in understanding and solving the problem.

In any case, once deployed, my cluster bundle looks like this.  8 units of transcoders, all connected to a shared filesystem, and performance monitoring too.


I was able to leverage the shared-fs relation provided by the nfs charm, as well as the ganglia charm to monitor the utilization of the cluster.  You can see the spikes in the cpu, disk, and network in the graphs below, during the course of a transcode job.




For my testing, I downloaded the movie Code Rushfreely available under the CC-BY-NC-SA 3.0 license.  If you haven't seen it, it's an excellent documentary about the open source software around Netscape/Mozilla/Firefox and the dotcom bubble of the late 1990s.

Oddly enough, the stock, 746MB high quality MP4 video doesn't play in Firefox, since it's an mpeg4 stream, rather than H264.  Fail.  (Yes, of course I could have used mplayer, vlc, etc., that's not the point ;-)


Perhaps one of the most useful, intriguing features of HTML5 is it's support for embedding multimedia, video, and sound into webpages.  HTML5 even supports multiple video formats.  Sounds nice, right?  If it only were that simple...  As it turns out, different browsers have, and lack support for the different formats.  While there is no one format to rule them all, MP4 is supported by the majority of browsers, including the two that I use (Chromium and Firefox).  This matrix from w3schools.com illustrates the mess.

http://www.w3schools.com/html/html5_video.asp

The file format, however, is only half of the story.  The audio and video contents within the file also have to be encoded and compressed with very specific codecs, in order to work properly within the browsers.  For MP4, the video has to be encoded with H264, and the audio with AAC.

Among the various brands of phones, webcams, digital cameras, etc., the output format and codecs are seriously all over the map.  If you've ever wondered what's happening, when you upload a video to YouTube or Facebook, and it's a while before it's ready to be viewed, it's being transcoded and scaled in the background. 

In any case, I find it quite useful to transcode my videos to MP4/H264/AAC format.  And for that, a scalable, parallel computing approach to video processing would be quite helpful.

During the course of the 3 minute run, I liked watching the avconv log files of all of the nodes, using Byobu and Tmux in a tiled split screen format, like this:


Also, the transcode charm installs an Apache2 webserver on each node, so you can expose the service and point a browser to any of the nodes, where you can find the input, output, and intermediary data files, as well as the logs and DONE flags.



Once the job completes, I can simply click on the output file, Code_Rush.mp4_1280x720_x264_aac.mp4, and see that it's now perfectly viewable in the browser!


In case you're curious, I have verified the same charm with a couple of other OGG, AVI, MPEG, and MOV input files, too.


Beyond transcoding the format and codecs, I have also added configuration support within the charm itself to scale the video frame size, too.  This is useful to take a larger video, and scale it down to a more appropriate size, perhaps for a phone or tablet.  Again, this resource intensive procedure perfectly benefits from additional compute units.


File format, audio/video codec, and frame size changes are hardly the extent of video transcoding workloads.  There are hundreds of options and thousands of combinations, as the manpages of avconv and mencoder attest.  All of my scripts and configurations are free software, open source.  Your contributions and extensions are certainly welcome!

In the mean time, I hope you'll take a look at this charm and consider using it, if you have the need to scale up your own video transcoding ;-)

Cheers,
Dustin

Read more
facundo

¿Dónde estaba Dios?


Hay un micro de la segunda temporada del genial programa "Filosofía aquí y ahora", de José Pablo Feinmann, que se llama ¿Dónde estaba Dios?. Lo pueden ver acá, pero les transcribo el texto, que es cortito...

    En el tema de Auschwitz está el tema de Dios.

    Es un tema muy muy complejo, y más de un teólogo se pone de muy mal humor cuando se menciona el tema de Dios con respecto a Auschwitz, pero más de un filósofo ha preguntado ¿dónde estaba Dios en Auschwitz?

    E incluso Primo Levi, que es el gran escritor judío, que escribe "Si esto es un hombre", "Los hundidos y los salvados"... dice "Existe Auschwitz, no existe Dios".

    Y Karl Löwith, que también es un gran pensador judío, discípulo de Heidegger, dice "Luego de Auschwitz es imposible pensar un Dios totalmente bueno".

    Es muy contundente, todo esto. Hay un tango, incluso, genial, que dice "¿dónde estaba Dios cuando te fuiste?". O sea, nuestra tanguística es tan excepcional que puede ser comparada con los grandes filósofos del siglo XX.

    Es lo mismo, ¿dónde estaba Dios cuando te fuiste?.

Y como cierre, tenemos al Chino Laborde, invitado de Demoliendo Tangos, interpretando, justamente, "Canción desesperada", de Enrique Santos Discépolo.

Read more
Nicholas Skaggs

The first testing day of the utopic cycle is coming this week on Thursday, July 10th. You can catch us live in a hangout on ubuntuonair.com starting at 1900 UTC. We'll be demonstrating running and testing the development release of ubuntu, reporting test results, reporting bugs, and doing triage work. We'll also be availible to answer your questions and help you get started testing as well.

Please join us in testing utopic and helping the next release of ubuntu become the best it can be. Hope to see everyone there!

P.S. We have a team calendar that can help you keep track of the release schedule along with this and other events. Check it out!

Read more
Colin Ian King

more stress with stress-ng

Since my last article about stress-ng I have been adding a few more stress mechanisms to stress-ng:

  • file locking - exercise file locking with one or more processes (the more processes the better).
  • fallocate - this allocates a 4MB file, sync's, truncates to zero size and syncs repeatedly
  • yield - this loops on sched_yield() to repeatedly relinquish the CPU forcing a high context switch rate when run with multiple yielding processes.
Also, I have added some new features to tweak scheduling, I/O characteristics and memory allocations of the running stress processes:
  • --sched and --sched-prio options to specify the scheduler type and priority
  • --ionice-class and --ionice-level options to tweak I/O niceness
  • --vm-populate option to populate (pre-fault) page tables for a mapping for the --vm stress test.
If I think of other mechanisms to stress the kernel I will add them, but for now, stress-ng is becoming almost feature complete.

Read more
Greg Lutostanski

2014-07-01 Meeting Minutes

Agenda

  • Review ACTION points from previous meeting
  • U Development
  • Server & Cloud Bugs (caribou)
  • Weekly Updates & Questions for the QA Team (psivaa)
  • Weekly Updates & Questions for the Kernel Team (smb, sforshee)
  • Ubuntu Server Team Events
  • Open Discussion
  • Announce next meeting date, time and chair

Minutes

  • bug 1317587 is in progress

Next Meeting

Next meeting will be on Tuesday, July 8th at 16:00 UTC in #ubuntu-meeting.

Additional logs @ https://wiki.ubuntu.com/MeetingLogs/Server/20140701

Read more
Tom Macfarlane

Mobile Asia Expo 2014

Following the success of our new stand design at MWC earlier this
year, we applied the same design principles to the Ubuntu stand at
last month’s Mobile Asia Expo in Shanghai.

MAE_2014_stand

With increased floor space, compared to last year, and a new stand
location that was approachable from three key directions, we were
faced with a few new design challenges:

  • How to effectively incorporate existing 7m wide banners into
    the new 8m wide stand?
  • How to make the stand open and approachable from three sides
    with optimum use of floor space and maintaining the maximum
    amount storage space possible?
  • How to maintain our strong brand presence after any necessary
    structural changes?

Proposed layout ideas

MAE_drawing_1

MAE_drawing_2

MAE_drawing_3 Final layout
The final design utilised maximum floor space and incorporated the
positioning of our bespoke demo pods, that proved successful at MWC.
Along with strong branding featuring our folded paper background
with large graphics showcasing app and scope designs and a new aisle
banner. The main stand banners were then positioned in an alternating
arrangement aligned to the left and to the right above the stand.

MAE_drawing_4

Aisle banner

Ubuntu_hanging_banner_Preview

MAE_stand_front

MAE_stand_back

Read more
David Planella

Ubuntu loves HTML5Here’s a reminder about next Monday’s 7th of July Ubuntu HTML5 apps session in Barcelona.

At this free event, I’ll be presenting Ubuntu’s HTML5 development story, together with a live coding session and a Q&A round at the end. You’ll learn how to use the Ubuntu SDK and the UI toolkit to easily reuse your web skills to create stunning Ubuntu apps.

HTML5 is the other side of the coin of the Ubuntu app developer offering, where both web and native are first class citizens, offering a very flexible yet focused approach for application development. Teaming up with BeMyApp meetups, the session will start at 7 p.m. at Barcelona’s Mobile World Centre.

I look forward to seeing you there!

Register here for the HTML5 session >

Read more
Michael Hall

It was less than a month that we announced crossing the 10,000 users milestone for Ubuntu phones and tablets, and we’ve already reached another: 100,000 app downloads!

Downloads

10k_downloads_by_countryThe new Ubuntu store used by phones, tablets, and soon the desktop as well, provides app developers with some useful statistics about how many times their app was downloaded, which version was downloaded, and what country the download originated from. This is very useful as it it lets the developer gauge how many users they currently have for their app, and how quickly they are updating to new versions.  One side-effect of these statistics is that we can see how many total downloads there have been across all of the apps in the store, and this week we reached (and quickly passed) the 100,000th download.

Users

app_storeWe’re getting close to having Ubuntu phones go on sale from our partners at Bq and Meizu, but there are still no devices on the market that came with Ubuntu.  This means that we’ve reached this milestone solely from developers and enthusiasts who have installed Ubuntu on one of their own devices (probably a Nexus device) or the device emulator.  

The continued growth in the download number validates the earlier milestone of 10,000 users, a large number of them are clearly still using Ubuntu on their device (or emulator) and keeping their apps up to date (the number represents new app installs and updates). This means that not only are people trying Ubuntu already, many of them are sticking with it too.  Yet another datapoint in support of this is the 600 new unique users who have been using the store since the last milestone announcement.

Pioneers

pioneers_shirtTo supply all of these users with the apps they want, we’re continuing to build our community of app developers around Ubuntu. The first of these have already received their limited edition t-shirts, and are listed on the Ubuntu Pioneers page of the developer portal.

There is still time to get your app published, and claim your place on that page and your t-shirt, but they’re filling up fast so don’t delay. Go to our Developer Portal and get started today, you could be only a few hours away from publishing your first app in the store!

Read more
Joseph Salisbury

Meeting Minutes

IRC Log of the meeting.

Meeting minutes.

Agenda

20140701 Meeting Agenda


Release Metrics and Incoming Bugs

Release metrics and incoming bug data can be reviewed at the following link:

  • http://people.canonical.com/~kernel/reports/kt-meeting.txt


Status: Utopic Development Kernel

We have rebased our Utopic kernel “unstable” branch to v3.16-rc3 and
uploaded to our ckt ppa (ppa:canonical-kernel-team/ppa). Please test.
I don’t anticipate an official v3.16 based upload to the archive until
further testing and baking has taken place.
—–
Important upcoming dates:
Thurs Jul 24 – 14.04.1 (~3 weeks away)
Thurs Jul 31 – 14.10 Alpha 2 (~4 weeks away)
Thurs Aug 07 – 12.04.5 (~5 weeks away)


Status: CVE’s

The current CVE status can be reviewed at the following link:

http://people.canonical.com/~kernel/cve/pkg/ALL-linux.html


Status: Stable, Security, and Bugfix Kernel Updates – Trusty/Saucy/Precise/Lucid

Status for the main kernels, until today (Jul. 1):

  • Lucid – Kernel Prep
  • Precise – Kernel Prep
  • Saucy – Kernel Prep
  • Trusty – Kernel Prep

    Current opened tracking bugs details:

  • http://people.canonical.com/~kernel/reports/kernel-sru-workflow.html

    For SRUs, SRU report is a good source of information:

  • http://people.canonical.com/~kernel/reports/sru-report.html

    Schedule:

    14.04.1 cycle: 29-Jun through 07-Aug
    ====================================================================
    27-Jun Last day for kernel commits for this cycle
    29-Jun – 05-Jul Kernel prep week.
    06-Jul – 12-Jul Bug verification & Regression testing.
    13-Jul – 19-Jul Regression testing & Release to -updates.
    20-Jul – 24-Jul Release prep
    24-Jul 14.04.1 Release [1]
    07-Aug 12.04.5 Release [2]

    [1] This will be the very last kernels for lts-backport-quantal, lts-backport-raring,
    and lts-backport-saucy.

    [2] This will be the lts-backport-trusty kernel as the default in the precise point
    release iso.


Open Discussion or Questions? Raise your hand to be recognized

No open discussion.

Read more
pitti

We currently use completely different methods and tools of building test beds and running tests for Debian vs. Click packages, for normal uploads vs. CI airline landings vs. upstream project merge proposal testing, and keep lots of knowledge about Click package test metadata external and not easily accessible/discoverable.

Today I released autopkgtest 3.0 (and 3.0.1 with a few minor updates) which is a major milestone in unifying how we run package tests both locally and in production CI. The goals of this are:

  • Keep all test metadata, such as test dependencies, commands to run the test etc., in the project/package source itself instead of external. We have had that for a long time for Debian packages with DEP-8 and debian/tests/control, but not yet for Ubuntu’s Click packages.
  • Use the same tools for Debian and Click packages to simplify what developers have to know about and to reduce the amount of test infrastructure code to maintain.
  • Use the exact same testbeds and test runners in production CI than what developers use locally, so that you can reproduce and investigate failures.
  • Re-use the existing autopkgtest capabilities for using various kinds of testbeds, and conversely, making all new testbed types immediately available to all package formats.
  • Stop putting tests into the Ubuntu archive as packages (such as mediaplayer-app-autopilot). This just adds packaging and archive space overhead and also makes updating tests a lot harder and taking longer than it should.

So, let’s dive into the new features!

New runner: adt-virt-ssh

We want to run tests on real hardware such as a laptop of a particular brand with a particular graphics card, or an Ubuntu phone. We also want to restructure our current CI machinery to run tests on a real OpenStack cloud and gradually get rid of our hand-maintained QA lab with its test machines. While these use cases seem rather different, they both have in common that there is an already existing machine which is pretty much only accessible with ssh. Once you have an ssh connection, they look pretty much the same, you just need different initial setup (like fiddling with adb, calling nova boot, etc.) to prepare them.

So the new adt-virt-ssh runner factorizes all the common bits such as communicating with adt-run, auto-detecting sudo availability, doing SSH connection sharing etc., and delegates the target specific bits to a “setup script”. E. g. we could specify --setup-script ssh-setup-nova or --setup-script ssh-setup-adb which would then get called with open at the appropriate time by adt-run; it calls the nova commands to create a VM, or run a few adb commands to install/start ssh and install the public key. Then autopkgtest does its thing, and eventually calls the script with cleanup again. The actual protocol is a bit more involved (see manpage), but that’s the general idea.

autopkgtest now ships readymade scripts for these two use cases. So you could e. g. run the libpng tests in a temporary cloud VM:

# if you don't have one, create it with "nova keypair-create"
$ nova keypair-list
[...]
| pitti | 9f:31:cf:78:50:4f:42:04:7a:87:d7:2a:75:5e:46:56 |

# find a suitable image
$ nova image-list 
[...]
| ca2e362c-62c9-4c0d-82a6-5d6a37fcb251 | Ubuntu Server 14.04 LTS (amd64 20140607.1) - Partner Image                         | ACTIVE |  

$ nova flavor-list 
[...]
| 100 | standard.xsmall  | 1024      | 10   | 10        |      | 1     | 1.0         | N/A       |

# now run the tests: please be patient, this takes a few mins!
$ adt-run libpng --setup-commands="apt-get update" --- ssh -s /usr/share/autopkgtest/ssh-setup/nova -- \
   -f standard.xsmall -i ca2e362c-62c9-4c0d-82a6-5d6a37fcb251 -k pitti
[...]
adt-run [16:23:16]: test build:  - - - - - - - - - - results - - - - - - - - - -
build                PASS
adt-run: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ tests done.

Please see man adt-virt-ssh for details how to use it and how to write setup scripts. There is also a commented /usr/share/autopkgtest/ssh-setup/SKELETON template for writing your own for your use cases. You can also not use any setup script and just specify user and host name as options, but please remember that the ssh runner cannot clean up after itself, so never use this on important machines which you can’t reset/reinstall!

Test dependency installation without apt/root

Ubuntu phones with system images have a read-only file system where you can’t install test dependencies with apt. A similar case is using the “null” runner without root. When apt-get install is not available, autopkgtest now has a reduced fallback mode: it downloads the required test dependencies, unpacks them into a temporary directory, and runs the tests with $PATH, $PYTHONPATH, $GI_TYPELIB_PATH, etc. pointing to the unpacked temp dir. Of course this only works for packages which are relocatable in that way, i. e. libraries, Python modules, or command line tools; it will totally fail for things which look for config files, plugins etc. in hardcoded directory paths. But it’s good enough for the purposes of Click package testing such as installing autopilot, libautopilot-qt etc.

Click package support

autopkgtest now recognizes click source directories and *.click package arguments, and introduces a new test metadata specification syntax in a click package manifest. This is similar in spirit and capabilities to DEP-8 debian/tests/control, except that it’s using JSON:

    "x-test": {
        "unit": "tests/unittests",
        "smoke": {
            "path": "tests/smoketest",
            "depends": ["shunit2", "moreutils"],
            "restrictions": ["allow-stderr"]
        },
        "another": {
            "command": "echo hello > /tmp/world.txt"
        }
    }

For convenience, there is also some magic to make running autopilot tests particularly simple. E. g. our existing click packages usually specify something like

    "x-test": {
        "autopilot": "ubuntu_calculator_app"
    }

which is enough to “do what I mean”, i. e. implicitly add the autopilot test depends and run autopilot with the specified test module name. You can specify your own dependencies and/or commands, and restrictions etc., of course.

So with this, and the previous support for non-apt test dependencies and the ssh runner, we can put all this together to run the tests for e. g. the Ubuntu calculator app on the phone:

$ bzr branch lp:ubuntu-calculator-app
# built straight from that branch; TODO: where is the official" download URL?
$ wget http://people.canonical.com/~pitti/tmp/com.ubuntu.calculator_1.3.283_all.click
$ adt-run ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- \
      ssh -s /usr/share/autopkgtest/ssh-setup/adb
[..]
Traceback (most recent call last):
  File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 93, in test_divide_with_infinity_length_result_number
    self._assert_result("0.33333333")
  File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 63, in _assert_result
    self.main_view.get_result, Eventually(Equals(expected_result)))
  File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: After 10.0 seconds test failed: '0.33333333' != '0.3'

Ran 33 tests in 295.586s
FAILED (failures=1)

Note that the current adb ssh setup script deals with some things like applying the autopilot click AppArmor hooks and disabling screen dimming, but it does not do the first-time setup (connecting to network, doing the gesture intro) and unlocking the screen. These are still on the TODO list, but I need to find out how to do these properly. Help appreciated!

Click app tests in schroot/containers

But, that’s not the only thing you can do! autopkgtest has all these other runners, so why not try and run them in a schroot or container? To emulate the environment of an Ubuntu Touch session I wrote a --setup-commands script:

adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
    ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- schroot utopic

This will actually work in the sense of running (and succeeding) the autopilot tests, but it will fail due to a lot of libust[11345/11358]: Error: Error opening shm /lttng-ust-wait... warnings on stderr. I don’t know what these mean, just that I also see them on the phone itself occasionally.

I also wrote another setup-commands script which emulates “read-only apt”, so that you can test the “unpack only” fallback. So you could prepare a container with click and the App framework preinstalled (so that it doesn’t always take ages to install them), starting from a standard adt-build-lxc container:

$ sudo lxc-clone -o adt-utopic -n click
$ sudo lxc-start -n click
  # run "sudo apt-get install click ubuntu-sdk-libs ubuntu-app-launch-tools" there
  # then "sudo powerdown"

# current apparmor profile doesn't allow remounting something read-only
$ echo "lxc.aa_profile = unconfined" | sudo tee -a /var/lib/lxc/click/config

Now that container has enough stuff preinstalled to be reasonably fast to set up, and the remaining test dependencies (mostly autopilot) work fine with the unpack/$*_PATH fallback:

$ adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
          --setup-commands /usr/share/autopkgtest/setup-commands/ro-apt \
          ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click \
          --- lxc -es click

This will successfully run all the tests, and provided you have apt-cacher-ng installed, it only takes a few seconds to set up. This might be a nice thing to do on merge proposals, if you don’t have an actual phone at hand, or don’t want to clutter it up.

autopkgtest 3.0.1 will be available in Utopic tomorrow (through autosyncs). If you can’t wait to try it out, download it from my people.c.c page ?.

Feedback appreciated!

Read more
Alan Pope

As previously blogged we’re inviting the community to hack on Core Apps and Community Apps this week.

All the details are in the post above, but here’s the executive summary:-

  • Hack days run from 30th June till 4th July
  • We’re hacking on the Core Apps Music, Calendar, Calendar, Clock, Weather & Calculator
  • In addition we’re also hacking on community apps including Beru Ebook Reader, OSM Touch mapping software, and Trojita email client
  • Join us in the #ubuntu-app-devel IRC channel on freenode, and on the ubuntu-phone mailing list to get started
  • Get all the details from the hack days wiki page

As always we welcome new contributions during the Hack Days, but also beyond that.

Read more
Colin Ian King

Linaro's idlestat is another useful tool in the arsenal of CPU monitoring utilities.  Idlestat monitors and captures CPU C-state and P-state transitions using the kernel Ftrace tracer and outputs statistics based on entering/exiting each state for each CPU.  Idlestat  also captures IRQ activity as well which ones caused a CPU to exit an idle state -  knowing why a processor came out of a deep C state is always very useful way to help diagnose power consumption issues.

Using idlestat is easy, to capture 20 seconds of activity into a log file called example.log, run:

 sudo idlestat --trace -f example.log -t 20    
..and this will display the per CPU C-state and P-state and IRQ statistics for that run.

One can also take the saved log file and parse it again to calculate the statistics again using:
 idlestat --import -f example.log  

One can get the source from here and I've packaged version 0.3 (plus a bunch of minor fixes that will land in 0.4) for Ubuntu 14.10 Utopic Unicorn.

Read more
Daniel Holbach

The idea

At the last Ubuntu Online Summit, we had a session in which we discussed (among other things) the idea to make it easier for LoCo teams to share news, stories, pictures, events and everything else. A lot of great work is happening around the world already, but a lot of us don’t get to hear about it.

I took an action item to schedule a meeting to discuss the idea some more. The rough plan being that we add functionality to the LoCo Team Portal which allows teams to share their stories, which then end up on Planet Ubuntu.

We held the meeting last week, find the IRC logs here.

The mini-spec

Find the spec here and please either comments on the loco-contacts mailing list or below in the comments. If you are a designer, a web developer, know Django or just generally want to get involved, please let us know! :)

We will discuss the spec some more, turn it into bug reports and schedule a series of hack days to work on this together.

Read more
Prakash Advani

Top cities with Tech Skills are: Bangalore, Pune, Hyderabad and Chennai.

These 4 Indian cities are ahead of San Francisco Bay Area.

Read the complete report: http://blog.linkedin.com/2014/06/24/indias-got-tech-talent-cities-in-india-top-list-of-cities-attracting-technology-talent/

Read more
facundo

Satélites argentinos


Estos días fue lanzado exitosamente el tercer nanosatélite argentino, "Tita" (llamado así en honor a Tita Merello).

Se llaman "nanosatélites" porque, justamente, son mucho más chicos (y baratos) que los satélites "tradicionales". En particular, Tita pesa unos 25 kilos, está equipado con tres antenas y lleva una cámara para tomar fotos y videos en alta definición.

El satélite Tita, siendo instalado en el lanzador

Lo desarrolló la empresa argentina Satellogic, pero no lo lanzamos nosotros al espacio (todavía no tenemos esa capacidad), sino que fue lanzado desde la ciudad rusa de Yasny. Su objetivo es tomar imágenes durante tres años, en colaboración con otros nanosatélites, los ya lanzados Capitán Beto (llamado así obviamente en referencia a la canción de Spinetta) y Manolito (por el personaje de Mafalda), y a otros 15 satélites que Satellogic planea lanzar durante el próximo año.

Pero Tita es diferente a los dos anteriores, que pesaban alrededor de dos kilos. También es un prototipo, y usa las mismas estrategias de diseño y fabricación con componentes de uso comercial (resortes de ferretería, electrónica de teléfonos celulares y computadoras personales), pero este permite tomar imágenes y videos de dos metros de resolución. Esencialmente, la gente de Satellogic está haciendo lo mismo que hace un satélite convencional, pero a un precio entre cien y mil veces menor.

En este video bastante interesante podemos ver a Lino Barañao (Ministro de Ciencia y Tecnología) y Emiliano Kargieman (CEO de Satellogic), contándonos un poco todo esto (y de paso se ven pasos de la construcción, y las oficinas, ¡donde se ve bastante gente de PyAr trabajando!).



Como detalle final, les dejo este audio de Adrián Paenza hablando sobre los satélites (en general) en el programa La Mañana de Victor Hugo Morales.

Read more
Alan Pope

Ready for RTM*: Ubuntu Touch Core App Hack Days!

* Release to Manufacturing

device-2014-06-25-121330

We’re running another set of Core Apps Hack Days next week. Starting Monday 30th June through to Friday 4th July we’ll be hacking on Core Apps, getting them polished for our upcoming RTM (Release To Manufacture) images. The goal of our hack days is as always to implement missing features, fix bugs, get new developers involved in coding on Ubuntu using the SDK and to have some fun hacking on Free Software.

For those who’ve not seen the hack days before, it’s really simple. We get together from 09:00 UTC till 21:00 UTC on #ubuntu-app-devel on freenode IRC and hack on the Core Apps. We will be testing the apps to destruction, filing and triaging bugs, creating patches, discussing and testing proposals and generally do whatever we can to get these apps ready for RTM. It’s good fun, relaxed and a great way to get started in Ubuntu app development with the SDK

We’ll have developers hanging around to answer questions, and can call on platform and SDK experts for assistance when required. We focus on specific apps each day, but as always we welcome contributions to all the core apps both during the designated days, and beyond.

Not just Core Apps

This time around we’re also doing things a little differently. Typically we only focus attention on the main community maintained Core Apps we ship on our device images. For this set of Hack Days we’d like to invite 3rd party community app developers to bring their apps along as well and hack with us. We’re looking for developers who have already developed their Ubuntu app using the SDK but maybe need help with the “last mile”. Perhaps you have design questions, bugs or feature enhancements which you’d like to get people involved in.

device-2014-06-25-122105 device-2014-06-25-122334

We won’t be writing your code for you, but we can certainly help to find experienced people to answer your questions and advise of platform and SDK details. We’d expect you to make your code available somewhere, to allow contributions and perhaps enable some kind of bug tracker or task manager. It’s up to you to manage your own community app, we’re here to help though!

Get involved

If you’re interested in bringing your app to hack days, then get in touch with popey (Alan Pope) on IRC or via email [popey@ubuntu.com] and we’ll schedule it in for next week and get the word out.

You can find out more about the Core Apps Hack Days on the wiki, and can discuss this with us on IRC in #ubuntu-app-devel.

Read more