We are working on the Juju GUI. This weekVery good interaction with design team Improved team velocity after deceleration last week Real users are starting to trickle in We have had some very positive interactions with the design team. We have dug up some good data, requirements, and decisions together, and we've increased our dialog. They have been very responsive to what we need in the short

UPDATE: A command porting walk-through has beed added to the documentation.

Back around UDS time, I began work on a reboot of Quickly, Ubuntu’s application development tool.  After two months and just short of 4000 lines of code written, I’m pleased to announce that the inner-workings of the new code is very nearly complete!  Now I’ve reached the point where I need your help.

The Recap

First, let me go back to what I said needed to be done last time.  Port from Python 2 to Python 3: Done. Add built-in argument handling: Done. Add meta-data output: Well, not quite.  I’m working on that though, and now I can add it without requiring anything from template authors.

But here are some other things I did get done. Add Bash shell completion: Done.  Added Help command (that works with all other commands): Done.  Created command class decorators: Done.  Support templates installed in any XDG_DATA_DIRS: Done.  Allow template overriding on the command-line: Done.  Started documentation for template authors: Done.

Now it’s your turn

With the core of the Quickly reboot nearly done, focus can now turn to the templates.  At this point I’m reasonably confident that the API used by the templates and commands won’t change (at least not much).  The ‘create’ and ‘run’ commands from the ubuntu-application template have already been ported, I used those to help develop the API.  But that leaves all the rest of the commands that need to be updated (see list at the bottom of this post).  If you want to help make application development in Ubuntu better, this is a great way to contribute.

For now, I want to focus on finishing the port of the ubuntu-application template.  This will reveal any changes that might still need to be made to the new API and code, without disrupting multiple templates.

How to port a Command

The first thing you need to do is understand how the new Quickly handles templates and commands.  I’ve started on some documentation for template developers, with a Getting Started guide that covers the basics.  You can also find me in #quickly in Freenode IRC for help.

Next you’ll need to find the code for the command you want to port.  If you already have the current Quickly installed, you can find them in /usr/share/quickly/templates/ubuntu-application/, or you can bzr branch lp:quickly to get the source.

The commands are already in Python, but they are stand-alone scripts.  You will need to convert them into Python classes, with the code to be executed being called in the run() method.  You can add your class to the ./data/templates/ubuntu-application/commands.py file in the new Quickly branch (lp:quickly/reboot).  Then submit it as a merge proposal against lp:quickly/reboot.

Grab one and go!

So here’s the full list of ubuntu-application template commands.  I’ll update this list with progress as it happens.  If you want to help, grab one of the TODO commands, and start porting.  Email me or ping me on IRC if you need help.

add: TODO
configure: TODO
create: DONE!
debug: TODO
design: TODO
edit: DONE!
license: TODO
package: DONE!
release: TODO
run: DONE!
save: DONE!
share: TODO
submitubuntu: TODO
test: TODO
tutorial: TODO
upgrade: TODO

Welcome back! We've been away working in secret for awhile. Our work is open now, so our retrospective notes are too. Our current project is a web-based GUI for Juju. Juju lets you deploy connected services to the cloud in a convenient, vendor-neutral, and powerful way. The GUI lets you visualize and manage your work (see also another blog; a demo of our trunk, which is reset every 15 minutes

In our last blog on Landscape and Puppet, we talked about using Landscape to automatically deploy Puppet on new computers. In this article we’ll dive deeper, and look at how to use Landscape as an External Node Classifier for Puppet.

In a typical Puppet configuration, all your nodes are defined in site.pp under the “node” definitions section. Each node is assigned Puppet classes by manually editing the file. When your number of nodes grows, manually managing the site.pp file becomes tedious and error prone.

An External Node Classifier (ENC) is Puppet’s way of offloading the tedious node maintenance to an external program. The interface is dead simple – puppet executes the external node classifier program with a single full node name as the only argument. The classifier just has to write a YAML blob out to stdout before exiting.

To start, let’s create a simple python ENC in /etc/puppet/landscape_enc – Don’t forget to make the file executable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/usr/bin/env python
import sys
import yaml

# The node name to be classified is supplied as an argument to the script
node_name = sys.argv[1]

classes = ["basenode"]

# Output must be a YAML document
print(yaml.dump({
    "classes": classes,
    }))

It ignores the node name and just puts everything into the “basenode” class. Not very interesting but it’s enough to get started with Puppet.

NOTE: These examples are all using puppet 2.7 which ships on Ubuntu 12.04 Precise LTS. The ENC functionality behaves a bit differently in versions of puppet earlier than 2.65 – See http://docs.puppetlabs.com/guides/external_nodes.html for details.

To test the ENC I put together a minimal puppet configuration with two simple classes and put everything into my site.pp in /etc/puppet/manifests/site.pp

1
2
3
4
5
6
7

class basenode {
  notify {"I am a basenode!":}
}

class specialnode {
  notify {"I am a specialnode!":}
}

Notice that no nodes are actually defined. That is the ENC’s job. To enable the ENC you need to add two lines to your puppetmaster’s config file /etc/puppet/puppet.conf

Add these lines at the end of the “[master]” section:

1
2

node_terminus = exec
external_nodes = /etc/puppet/landscape_enc

You can now test that the puppetmaster is using your new, rather basic, ENC.

1
2
3
4
5
6
7

ubuntu@ubuntu:~$ sudo puppet agent --test

info: Caching catalog for ubuntu
info: Applying configuration version '1354824718'
notice: I am a basenode!
notice: /Stage[main]/Basenode/Notify[I am a basenode!]/message: defined 'message' as 'I am a basenode!'
notice: Finished catalog run in 0.06 seconds

As you can see, with our trivial ENC everyone is a basenode.

Now we’re going to enhance our ENC to ask Landscape to classify nodes for us.

The Landscape API
To use the Landscape API you need three pieces of information: the Landscape API endpoint URI, the user key, and the secret for your user.

To find your API credentials, log in to the Landscape web interface and click your username on the top right. Your API credentials are in the “API Access” section.

For this example, we’ll use the python API client provided with Landscape (NOTE, you must install the landscape-api package first). Here’s how to query for a computer registered with Landscape using it’s host name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from landscape_api.base import API

landscape_uri = "https://landscape.canonical.com/api/"
landscape_key = "43NW6OV71L32CSOPCJGX"
landscape_secret = "agBf3v267DqO8vtVRnzjseWfYdV4ueklj5a81iIT"
api = API(landscape_uri, landscape_key, landscape_secret)

api.get_computers(query="my_hostname_here")
[{u'access_group': u'server',
u'comment': u'',
u'hostname': u'appserv1',
u'id': 1,
u'last_exchange_time': None,
u'last_ping_time': u'2012-09-07T15:19:22Z',
u'reboot_required_flag': False,
u'tags': [u'lucid', u'server', u'puppet-webfarm'],
u'title': u'Application Server 1',
u'total_memory': 1024,
u'total_swap': 1024}]

Now if we combine that with our ENC we get the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

#!/usr/bin/env python
import sys
import yaml
from landscape_api.base import API

# Create our connection to Landscape
landscape_uri = "https://landscape.canonical.com/api/"
landscape_key = "43NW6OV71L32CSOPCJGX"
landscape_secret = "agBf3v267DqO8vtVRnzjseWfYdV4ueklj5a81iIT"
api = API(landscape_uri, landscape_key, landscape_secret)

# The node name to be classified is supplied as an argument to the script
node_name = sys.argv[1]

# Ask Landscape about the computer
computers = api.get_computers(query=node_name)

# If we don't get back any computers or if we get more than one, error out.
# You could also handle this case by simply giving the node a default class.
if len(computers) != 1:
    sys.stderr.write("Only expecting one computer, instead got this: %s" % computers)
    sys.exit(1)

# Extract the tags from our computer
tags = computer[0]["tags"]

# Now here you can use whatever logic you want to convert
# tags into classes. I'm going to use any tag that starts with "puppet-"
# as a class name. I'm also going to make sure every node gets the
# "basenode" class
classes = ["basenode"]
for tag in tags:
    if tag.startswith("puppet-"):
        class_name = tag.split("-",1)[1]
        classes.append(class_name)

# Output must be a YAML document
print(yaml.dump({
    "classes": classes,
    }))

That’s all there is to it. Now if you tag a computer “puppet-database” in Landscape, it will automatically get the “database” class in Puppet.

You can see in the script comments that it’s very easy to customize the behaviour to match your environment. Now tag away and let Landscape and Puppet take over your world!

During this latest round of arguing over the inclusion of Amazon search results in the Unity Dash, Alan Bell pointed out the fact that while the default scopes shipped in Ubuntu were made to check the new privacy settings, we didn’t do a very good job of telling third-party developers how to do it.

(Update: I was told a better way of doing this, be sure to read the bottom of the post before implementing it in your own code)

Since I am also a third-party lens developer, I decided to add it to my come of my own code and share how to do it with other lens/scope developers.  It turns out, it’s remarkably easy to do.

Since the privacy setting is stored in DConf, which we can access via the Gio library, we need to include that in our GObject Introspection imports:

from gi.repository import GLib, Unity, Gio

Then, before performing a search, we need to fetch the Unity Lens settings:

lens_settings = Gio.Settings(‘com.canonical.Unity.Lenses’)

The key we are interested in is ’remote-content-search’, and it can have one of two value, ‘all’ or ‘none’.  Since my locoteams-scope performs only remote searches, by calling the API on http://loco.ubuntu.com, if the user has asked that no remote searches be made, this scope will return without doing anything.

And that’s it!  That’s all you need to do in order to make your lens or scope follow the user’s privacy settings.

Now, before we get to the comments, I’d like to kindly point out that this post is about how to check the privacy setting in your lens or scope.  It is not about whether or not we should be doing remote searches in the dash, or how you would rather the feature be implemented.  If you want to pile on to those argument some more, there are dozens of open threads all over the internet where you can do that.  Please don’t do it here.
&nbps;

Update

I wasn’t aware, but there is a PreferencesManager class in Unity 6 (Ubuntu 12.10) that lets you access the same settings:

You should use this API instead of going directly to GSettings/DConf.

For my birthday in October, I received a Fitbit One. The reason that I wanted it is that I thought with better data tracking I could push myself to be more active during the day. The Fitbit One is a “fitness tracker”, essentially a technologically enhanced pedometer, that can also measure elevation gain (steps climbed they call it), and even track your sleep patterns. The device, which is slightly larger than a large paperclip, syncs wirelessly  to iPhones or to a computer. It uploads all your statistics to Fitbit.com, which provides a cool dashboard which you can use to track your steps, floors climbed, calories burned, etc. Here’s my dashboard from yesterday:

My Fitbit dashboard from yesterday

Like most geeks, I love data, and nice charts and graphs too, so I’ve really enjoyed the dashboard. I’ve also found that the maxim, “What gets measured gets done” really applies here. Two nights ago at 11:30PM, I noticed I was 300 steps short of 10000 steps, so I made sure to walk around while brushing my teeth, took the trash out, and generally wandered until I got past 10000 steps. That was only 300 steps, but I’ve also found myself walking the dog more, walking to the library more, etc.

So what does this have to do with Ubuntu? Well you can see at the bottom of that dashboard that Fitbit gives “badges”, which Chris Wayne thought would be a perfect fit for the Ubuntu Accomplishments system.  So Chris hacked all weekend and created an online account plugin for Fitbit. On Monday we hooked the oauth account created by Chris’s plugin into Fitbit’s web API and now we had Fitbit accomplishments!

Badges I can earn

My Trophies

You need a Fitbit to use it, and if you buy one, use this link so that Chris and I can support our daily beer and daily steps habits. The same link is also in the collection itself.

Installing

Note: That this requires Quantal or Raring because it uses Online Accounts. The raring build broke for some reason earlier but it should be ready an hour from the time this posts.

Installing is easy, although if you don’t already have Ubuntu Accomplishments installed it’s a two step process.

First, install Ubuntu Accomplishments if you’ve not already done so:

sudo add-apt-repository ppa:ubuntu-accomplishments/releases

sudo apt-get update

sudo apt-get install accomplishments-daemon accomplishments-viewer ubuntu-community-accomplishments ubuntu-desktop-accomplishments accomplishments-lens

Then install the Fitbit Accomplishments collection:

sudo add-apt-repository ppa:fitbit-accomplishment-maintainers/daily

sudo apt-get update

sudo apt-get install account-plugin-fitbit ubuntu-fitbit-accomplishments

If you’re already running Ubuntu Accomplishments, you’ll need to close the viewer and restart the Accomplishments Daemon to get the new collection to show up.  You can restart the daemon by doing accomplishments-daemon –restart.  A simple logout/login will also work.

The first accomplishment you need to get is connecting to your Fitbit account. Chris also wrote a post with some screenshots if you get stuck.

You need to setup your Fitbit Online Account before you can get any Fitbit badges, follow the steps in the Accomplishment to do so.

Follow the directions in the accomplishment to set this up. Once you do that, the other fitbit accomplishments will unlock in a logical progression as you achieve things (for example, the 10000 steps in a day accomplishment requires you to complete the 5000 steps in a day accomplishment first).

Note that Fitbit admits that the Badge API is still new and there are some quirks, for example, Fitbit provides badges for 50 and 250 lifetime kilometers, but for lifetime miles, they offer 50, 250, 1000, and 5000. Also some badges are transparent, some are not, which I know we could fix, but I haven’t had time yet. As this API improved and is expanded, we’ll add more accomplishments, or better yet, you can add more by sending us a merge proposal (the code is here).

Why?

Fitbit accomplishments, like walking 10000 steps in a day, obviously have nothing to do with Ubuntu, but this collection highlights the flexibility of the Ubuntu Accomplishments system. Anything that can be tested via script can be an accomplishment. I’m sure there are lots of other websites that people use that could be added as collections like this one. If you’re interested and you need help setting one up, you can find me (mfisch) in #ubuntu-accomplishments on Freenode.

About the Accomplishments Code

The code for checking these accomplishments in the accomplishments scripts is very very simple:

    badgeid = "10000 DAILY_STEPS"
    me = FitBit.fetch(None)
    if badgeid in me.badges:
        sys.exit(0)
    else:
        sys.exit(1)

This is because all the hard logic is in helpers.py, which provides the FitBit class and handles caching for us. Since the way accomplishments work is that each accomplishment has a script associated with it, we want to cache the info so that we don’t hammer the Fitbit web API once per script every 15 minutes (all unlocked accomplishments are checked every 15 minutes). The caching solution in helpers.py, was copied from the model used by AskUbuntu and Launchpad in the Ubuntu Community Accomplishments package. helpers.py also is how we interact with the Online Accounts plugin and Fitbit’s web API, so if you want to see the “interesting code”, look there.

Note: Expect a follow-up blog post from Chris Wayne on how to write an online accounts plugin in the next couple of weeks.

Help Needed

If you live outside of the US and you have a Fitbit and are willing to help, I need some assistance to see what happens if the Fitbit API returns localized Badge info. I also need to see if what it looks like when you get a badge marked in Kilometers. I don’t think I get these because of where I live (the US). Drop me an email to matt@<this_domain>.com if you can assist or find me in #ubuntu-accomplishments on freenode, I’m mfisch. I think I’ll only need a few minutes of your time.

The Ubuntu 12.10 release saw the Dash take another important step towards fulfilling its intended purpose of being an online, global search tool that helps users find anything, instantly, right from their home environment. There are a number of exciting improvements planned for 13.04 that will make the Dash more comprehensive, more online and as a result – slicker and even more useful. Here’s a snapshot of what is likely to land in the next release:

Smart Scopes – Scopes are the daemons capable of presenting local or remote information right in the Dash. In 13.04, we will increase the number of scopes installed by default in Ubuntu (including many existing community developed scopes) and introduce the ability to automatically light up the right ones based on their relevancy to a user’s search query. For example, a search for “The Beatles” is likely to trigger the Music and Video scopes, showing results that will contain local and online sources – with the online sources querying your personal cloud as well as other free and commercial sources like YouTube, Last.fm, Amazon, etc. To achieve this, the Dash will call a new smart scope service which will return ranked online search results, which the Dash will then balance against local results to return the most relevant information to the user. Scopes are becoming a really interesting contribution area for our developer community – and we can’t wait to see what people will submit to make the Dash an even richer experience.

Instant Purchasing – being able to purchase music or apps directly from the Dash, without opening a browser or a separate client. In 13.04, we expect to enable instant payments, powered by Ubuntu One, for both applications from the Software Center and music from the Music Store – to deliver the fastest possible purchasing experience directly from the Dash.

More Suggestions and User Controls – the More Suggestions scope, which currently returns relevant commercial content available from the Ubuntu One Music Store and Amazon, will expand to include more retailers. We are also testing a few additional user controls like filters for local and global searching – more to come on this front as we learn from those sessions. In the meantime, users can already focus a search to local files only with a simple super-f keystroke.

There are several principles around the Dash that are also worth reiterating:

Its raison d’etre is to provide Ubuntu users the fastest, slickest way to find things right from their home environment – independent of whether those “things” are on your machine, available online, free or commercial.  The music and video lenses in the Dash have queried online sources since their introduction, and we will continue to expand our online sources over the next releases. Our testing has overwhelmingly shown that this integrated and unified search feature is the best experience for the vast majority of users – and the best user experience will always be included as a default on Ubuntu.

 
Privacy is extremely important to Canonical. The data we collect is not user-identifiable (we automatically anonymize user logs and that information is never available to the teams delivering services to end users), we make users aware of what data will be collected and which third party services will be queried through a notice right in the Dash, and we only collect data that allows us to deliver a great search experience to Ubuntu users.  We also recognize that there is always a minority of users who prefer complete data protection, often choosing to avoid services like Google, Facebook or Twitter for those reasons – and for those users, we have made it dead easy to switch the online search tools off with a simple toggle in settings.

Onwards and upwards.

The Ubuntu Skunkworks program is now officially underway, we have some projects already staffed and running, and others where I am gathering final details about the work that needs to be done.  Today I decided to look at who has applied, and what they listed as their areas of interest, programming language experience and toolkit knowledge.  I can’t share details about the program, but I can give a general look at the people who have applied so far.

Most applicants listed more than one area of interest, including ones not listed in Mark’s original post about Skunkworks.  I’m not surprised that UI/UX and Web were two of the most popular areas.  I was expecting more people interested in the artistic/design side of things though.

Not as many people listed multiple languages as multiple areas of interest.  As a programmer myself, I’d encourage other programmers to expand their knowledge to other languages.  Python and web languages being the most popular isn’t at all surprising.  I would like to see more C/C++ applicants, given the number of important projects that are written in them.  Strangely absent was any mention of Vala or Go, surely we have some community members who have some experience with those.

The technology section had the most unexpected results.  Gtk has the largest single slice, sure, but it’s still much, much smaller than I would have expected.  Qt/QML even more so, where are all you KDE folks?  The Django slice makes sense, given the number of Python and Web applicants.

So in total, we’ve had a pretty wide variety of skills and interests from Skunkworks applicants, but we can still use more, especially in areas that are under-represented compared to the wider Ubuntu community.  If you are interested, the application process is simple: just create a wiki page using the ParticipationTemplate, and email me the link (mhall119@ubuntu.com).

Now that Google+ has added a Communities feature, and seeing as how Jorge Castro has already created one for the wider Ubuntu community, I went ahead and created one specifically for our application developers.  If you are an existing app developer, or someone who is interested in getting started with app development, or thinking about porting an existing app to Ubuntu, be sure to join.

• Ubuntu App Developer Community

Google+ communities are brand new, so we’ll be figuring out how best to use them in the coming days and weeks, but it seems like a great addition.

Based on the questions I’ve seen on AskUbuntu, lightdm.conf is one of the most misunderstood files on your system. So, I decided I’d write a post on how you can easily modify this file and what the modification are useful for. I hope to show how to modify the most asked about settings.

Safely Modifying lightdm.conf

Before you do anything to your lightdm.conf file, you should make a backup, simply run:

sudo cp /etc/lightdm/lightdm.conf /etc/lightdm/lightdm.conf.old

Once you’ve made a backup, the simplest and safest way to modify lightdm.conf is to use lightdm-set-defaults. lightdm-set-defaults was written so that lightdm.conf could be modified via script, but you can also use it to easily make changes. I’ve made several changes to this tool to add new features that I needed, and best of all, I even wrote a manpage for it, which should show up in raring at some point. If you’re not using raring, then just run /usr/lib/lightdm/lightdm-set-defaults with no arguments and you’ll get a clear picture on what it can do.

Usage:
lightdm-set-defaults [OPTION...] - set lightdm default values

Help Options:
-h, --help Show help options

Application Options:
-d, --debug Enable debugging
-k, --keep-old Only update if no default already set
-r, --remove Remove default value if it's the current one
-s, --session Set default session
-g, --greeter Set default greeter
-a, --autologin Set autologin user
-i, --hide-users Set greeter-hide-users to true or false
-m, --show-manual-login Set show-manual-login to true or false
-R, --show-remote-login Set show-remote-login to true or false
-l, --allow-guest Set allow-guest to true or false

You can also edit the file manually, but in either case, manual edit or set-defaults, you’ll need to use sudo. And now that you know how to modify the file, let’s cover what the most frequently asked about items are.

Disabling Guest Login

Some people really get annoyed by guest login, so if you want to disable it, simply use:

sudo /usr/lib/lightdm/lightdm-set-defaults --allow-guest false

Or, you can manually add the following line in the [SeatDefaults] section:

allow-guest=false

The default for this option is true, so if unset, the guest account will be enabled.  Note: See how great the command option for lightdm-set-defaults was named? Whoever added that was a genius.

Hiding the User List

If you don’t want a user list to be displayed by the greeter, you can enable this option. This should also be used with the enabling manual login (below) or logging in may be a challenge (actually I’ve never tried one without the other, I’m not sure what will happen).

sudo /usr/lib/lightdm/lightdm-set-defaults --hide-users true

Or, you can manually add the following line in the [SeatDefaults] section:

greeter-hide-users=true

The default for this option is false, so if unset, you will get a user list in the greeter.

Show Manual Login Box

If you previously hid your user list and would like a box where you can manually type in a user name then this option is for you.

sudo /usr/lib/lightdm/lightdm-set-defaults --show-manual-login true

Or, you can manually add the following line in the [SeatDefaults] section:

greeter-show-manual-login=true

The default for this option is false, so if unset, you won’t get a manual login box.

Autologin

You can enable autologin by specifying the autologin user.

sudo /usr/lib/lightdm/lightdm-set-defaults --autologin username

Or, you can manually add the following line in the [SeatDefaults] section:

autologin-user=username

There are other autologin related options which you may want to set, but none of these can be set using lightdm-set-defaults:

To change how long the greeter will delay before starting autologin.  If not set, the delay is 0, so if you want this to be 0, you don’t need to change it.  Note: the default for all unset integers in the [SeatDefaults] section is 0.

autologin-user-timeout=delay

To enable autologin of the guest account:

autologin-guest=true

Run a Command When X Starts, When The Greeter Starts, When the User Session Starts

When lightdm starts X you can run a command or script, like xset perhaps.

display-setup-script=[script|command]

You can do something similar when the greeter starts:

greeter-setup-script=[script|command]

or when the user session starts:

session-setup-script=[script|command]

Change the Default Session

If you want a different session for the default, you can modify this option. I think that the greeter will default to give you the last session you chose, so this option will only change the default session. Note: The session switcher will only show up if you have more than one VALID session; a valid session is one that points to a valid executable. By default in 12.10 you will have a session file for gnome-shell, but gnome-shell won’t be installed, so the session is invalid, leaving you with a single valid session (Ubuntu), and hence no session selector!

/usr/lib/lightdm/lightdm-set-defaults --session [session name]

Or, you can manually add the following line in the [SeatDefaults] section:

user-session=true

The list of user sessions is in /usr/share/xsessions, although even that location is configurable (see Advanced Options).

You can change the default greeter in the same manner, using –greeter for lightdm-set-defaults or greeter-session for the config file. The list of installed greeters is in /usr/share/xgreeters.

Advanced Options and All Other Options

There is no manpage for lighdm.conf, but there is an example that lists all the options and a bit about what they do, just look in /usr/share/doc/lightdm/lightdm.conf.gz.  If you use vim, you can just edit the file and it will be automagically ungzipped for you, users of other editors are on their own.

Several months ago, I wrote a stock quote lens using Michael Hall‘s Singlet. After installing Quantal in September, I started playing with the preview feature in lenses and I really liked it, so I got motivated to add it to my lens. Turns out, it’s super easy. During the process, I also added real charts for the preview icons and fixed a bug when displaying news stories (they were missing the publication date).

First a quick look at the new graphs and previews, then I’ll go over the code.

Real charts with the quotes

The preview mode offers two buttons for stock quotes, the first one “More Info” takes you to the standard stock quote page. The second one, Interactive Chart, takes you to a large interactive stock chart, both at Yahoo Finance.

If you want to install the new version with quotes, you need version 0.6 or later. You can get them from the Scopes Packagers PPA, I’ve uploaded versions for Precise, Quantal, and Raring.

Now, let’s look at the code, you can find it here, or just branch it with: bzr branch lp:~mfisch/onehundredscopes/unity-stock-ticker-lens.  The relevant changes are in revision 11, look at the preview function in the scope code (yahoostock-scope). I’ve simplified what’s in my code some to make it easier to follow:

def preview(self, result_item, result_model):
   preview = Unity.GenericPreview.new(result_item['title'], result_item['description'], None)
   preview.props.image_source_uri = 'http://chart.finance.yahoo.com/t?s=%s&lang=en-US&region=US&width=380&height=380' % result_item['title']
   open_chart = Unity.PreviewAction.new("open_chart", "Interactive Chart", None)
   open_chart.connect('activated', self.open_chart)
   preview.add_action(open_chart)
   return preview

The first two lines setup the preview, the first one sets the large font title and the smaller font text. The second line sets the large preview image. The open_chart lines define an action, which becomes a button in the UI. The button text is “Interactive Chart” and the button action is a function called open_chart. The open_chart function (not listed, but available in the bzr branch munges the inbound URL some and then opens a webbrowser to the Yahoo interactive chart page.

And that’s it!  Very simple! I had this up and running in about 30 minutes, while watching The Pacific on TV,

You can read more about previews in Singlet 0.3 in Michael Hall’s blog post about it. Special thanks to Chris Wayne for his git hub lens which was inspiration for these changes. You can look at Chris’s code for some other examples, his may be easier to follow than mine.

At the UDS in Copenhagen I finally had time to attend a session on Juju Charms. I knew the theory of Juju, which is that allows you to easily deploy and link services on public clouds, locally, or even on bare metal, but I never had time to try it out. The Charm School (registration required) session in Copenhagen really showed me the power of what Juju can give you. For example, when I first setup my blog, I had to find a webhost, get an an ssh account, download WordPress, install it, and dependencies, setup mysql, configure WordPress, debug why they weren’t communicating, etc. It was super annoying and took way too long. Now, imagine you want to setup ten blogs, or ten instances of couchdb, or one hundred, or one thousand, and it quickly becomes untenable.  With juju, setting up a blog is as simple as:

• juju deploy wordpress
• juju deploy mysql
• juju add-relation wordpress mysql
• juju expose wordpress

A few minutes later, and I have a functioning WordPress install. For more complex setups and installs Juju helps to manage the relationships between charms and sends events that the charms react to. This makes it easy to add and remove services like haproxy and memcached to an existing webapp. This interaction between charms implies that the more charms that are available the more useful they all become; the network effect applies to charms!

So after I got home, Charm School had left me energized and ready to write a charm, but I didn’t have any great ideas, until I remembered an app that I’ve used before called Tracks. Tracks is a GTD app, in other words, a fancy todo list. I’d used it hosted before, but my free host went offline and I lost all my to do items. Hosting my own would be much safer. So I started working on a Tracks charm.

If you need an idea for a charm, think about what tools you use that you have to setup, what software have you installed and configured recently? If you need an idea and nothing stands out, you can check out the list of “Charm Needed” bugs. Actually you should check that list regardless to make sure nobody else is already writing the same one.

With an idea in hand, I sat down to write my Charm. Step one is the documentation, most of which was contained on this page “Writing a Charm“. I fully expected to spend three weeks learning a new programming language with arcane black magic commands, but I was pleasantly surprised to learn that you can write a charm in any language you want. Most charms seem to be shell scripts or Python and my charm was simple enough that I wrote it in bash.

During the process of charm writing you may have some questions, and there’s plenty of help to be had. First, the examples that are contained in the juju trunk are OLD and I wouldn’t recommend you follow them. They are missing things like README files and don’t expose http interfaces, which was requested for my charm. Instead I’d recommend you pull the wordpress, mysql, and drupal charms from the charm store. If the examples aren’t enough, you can always ask in #juju on freenode or use askubuntu.com. Once your charm works, you can submit it for review. You’ll probably learn a lot during the review, every person I’ve talked to has.

Finally after a bit of work off and on, my charm was done! I submitted it for review, made a few fixes and it made it into the store.

I can now have a Tracks instance up and running in just a few minutes

In our last exciting episode, we learned how to capture a valgrind log. Today we’re going to take the next step and learn how to actually use it to debug memory leaks.

There are a few prerequisites:

1. know C. If you don’t know it, go read the C programming language which is often referred to as K&R C. Be sure to understand the sections on pointers, and after you do, come back to my blog. See you in 2 weeks!
2. a nice supply of your favorite beverages and snacks. I prefer coffee and bacon, myself. Get ready because you’re about to read an epic 2276 word blog entry.

That’s it. Ok, ready? Let’s go!

navigate the valgrind log
Open the valgrind log that you collected. If you don’t have one, you can grab one that I’ve already collected. Take a deep breath. It looks scary but it’s not so bad. I like to skip straight to the good part near the bottom. Search the file for “LEAK SUMMARY”. You’ll see something like:

==13124== LEAK SUMMARY:
==13124==    definitely lost: 916,130 bytes in 37,528 blocks
==13124==    indirectly lost: 531,034 bytes in 12,735 blocks
==13124==      possibly lost: 82,297 bytes in 891 blocks
==13124==    still reachable: 2,578,733 bytes in 42,856 blocks
==13124==         suppressed: 0 bytes in 0 blocks
==13124== Reachable blocks (those to which a pointer was found) are not shown.
==13124== To see them, rerun with: --leak-check=full --show-reachable=yes

You can see that valgrind thinks we’ve definitely leaked some memory. So let’s go figure out what leaked.

Valgrind lists all the leaks, in order from smallest to largest. The leaks are also categorized as “possibly” or “definitely”. We’ll want to focus on “definitely” for now. Right above the summary, you’ll see the worst, definite leak:

==13124== 317,347 (77,312 direct, 240,035 indirect) bytes in 4,832 blocks are definitely lost in loss record 10,353 of 10,353
==13124==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13124==    by 0x74E3A78: g_malloc (gmem.c:159)
==13124==    by 0x74F6CA2: g_slice_alloc (gslice.c:1003)
==13124==    by 0x74F7ABD: g_slist_prepend (gslist.c:265)
==13124==    by 0x4275A4: get_menu_item_for_ap (applet-device-wifi.c:879)
==13124==    by 0x427ACE: wireless_add_menu_item (applet-device-wifi.c:1138)
==13124==    by 0x41815B: nma_menu_show_cb (applet.c:1643)
==13124==    by 0x4189EC: applet_update_indicator_menu (applet.c:2218)
==13124==    by 0x74DDD52: g_main_context_dispatch (gmain.c:2539)
==13124==    by 0x74DE09F: g_main_context_iterate.isra.23 (gmain.c:3146)
==13124==    by 0x74DE499: g_main_loop_run (gmain.c:3340)
==13124==    by 0x414266: main (main.c:106)

Wow, we lost 300K of memory in just a few hours. Now imagine if you don’t reboot your laptop for a week. Yeah, that’s not so good. Time for a coffee and bacon break, the next part is about to get fun.

read the stack trace
What you saw above is a stack trace, and it’s printed chronologically “backwards”. In this example, malloc() was called by g_malloc(), which was called by g_slice_alloc(), which in turn was called by g_slist_prepend(), which itself was called by get_menu_item_for_ap() and so forth. The first function ever called was main(), which should hopefully make sense.

At this point, we need to use a little bit of extra knowledge to understand what is happening. The first function, main() is in our program, nm-applet. That’s fairly easy to understand. However, the next few functions that begin with g_main_ don’t actually live inside nm-applet. They are part of glib, which is a library that nm-applet depends on. I happened to have just known this off the top of my head, but if you’re ever unsure, you can just google for the function name. After searching, we can see that those functions are in glib, and while there is some magic that is happening, we can blissfully ignore it because we see that we soon jump back into nm-applet code, starting with applet_update_indicator_menu().

a quick side note
Many Linux programs will have a stack trace similar to the above. The program starts off in its own main(), but will call various other libraries on your system, such as glib, and then jump back to itself. What’s going on? Well, glib provides a feature known as a “main loop” which is used by the program to look for inputs and events, and then react to them. It’s a common programming paradigm, and rather than have every application in the world write their own main loop, it’s easier if everyone just uses the one provided by glib.

The other observation is to note how the function names appear prominently in the stack trace. Pundits wryly say that naming things is one of the hardest things in computer science, and I completely agree. So take care when naming your functions, because people other than you will definitely see them and need to understand them!

Alright, let’s get back to the stack trace. We can see a few functions that look like they belong to nm-applet, based on their names and their associated filenames. For example, the function wireless_add_menu_item() is in the file applet-device-wifi.c on line 1138. Now you see why we wanted symbols from the last episode. Without the debug symbols, all we would have seen would have been a bunch of useless ??? and we’d be gnashing our teeth and wishing for more bacon right now.

Finally, we see a few more g_* functions, which means we’re back in the memory allocation functions provided by glib. It’s important to understand at this point that g_malloc() is not the memory leak. g_malloc() is simply doing whatever nm-applet asks it to do, which is to allocate memory. The leak is highly likely to be in nm-applet losing a reference to the pointer returned by g_malloc().

What does it mean?
Now we’re ready to start the real debugging. We know approximately where we are leaking memory inside nm-applet: get_menu_item_for_ap() which is the last function before calling the g_* memory functions. Time to top off on coffee because we’re about to get our hands dirty.

reading the source
The whole point of open source is being able to read the source. Are you as excited as I am? I know you are!

First, let’s get the source to nm-applet. Assuming you are using Ubuntu and you are using 12.04, you’d simply say:

$ cd Projects
$ mkdir network-manager-gnome
$ cd network-manager-gnome
$ apt-get source network-manager-gnome
$ cd network-manager-applet-0.9.4.1

Woo hoo! That wasn’t hard, right?

side note #2
Contrary to popular belief, reading code is harder than writing code. When you write code, you are transmitting the thoughts of your messy brain into an editor, and as long as it kinda works, you’re happy. When you read code, now you’re faced with the problem of trying to understand exactly what the previous messy brain wrote down and making sense of it. Depending on how messy that previous brain was, you may have real trouble understanding the code. This is where pencil and paper and plenty of coffee come into play, where you literally trace through what the program is doing to try and understand it.

Luckily there are at least a few tools to help you do this. My favorite tools are cscope and ctags, which help me to rapidly understand the skeleton of a program and navigate around its complex structure.

Assuming you are in the network-manager-applet-0.9.4.1 source tree:

$ apt-get install cscope ctags
$ cscope -bqR
$ ctags -R
$ cscope -dp4

You are now presented with a menu. Use control-n and control-p to navigate input fields at the bottom. Try navigating to “Find this C symbol:” and then type in get_menu_item_for_ap, and press enter. The search results are returned, and you can press ’0′ or ’1′ to jump to either of the locations where the function is referenced. You can also press the space bar to see the final search result. Play around with some of the other search types and see what happens. I’ll talk about ctags in a bit.

Alrighty, let’s go looking for our suspicious nm-applet function. Start up cscope as described above. Navigate to “Find this global definition:” and search for get_menu_item_for_ap. cscope should just directly put you in the right spot.

Based on our stack trace, it looks like we’re doing something suspicious on line 879, so let’s go see what it means.

 869         if (dup_data.found) {
 870 #ifndef ENABLE_INDICATOR
 871                 nm_network_menu_item_best_strength (dup_data.found, nm_acce
 872                 nm_network_menu_item_add_dupe (dup_data.found, ap);
 873 #else
 874                 GSList *dupes = NULL;
 875                 const char *path;
 876 
 877                 dupes = g_object_get_data (G_OBJECT (dup_data.found), "dupe
 878                 path = nm_object_get_path (NM_OBJECT (ap));
 879                 dupes = g_slist_prepend (dupes, g_strdup (path));
 880 #endif
 881                 return NULL;
 882         }

Cool, we can now see where the source code is matching up with the valgrind log.

Let’s start doing some analysis. The first thing to note are the #ifdef blocks on lines 870, 873, and 880. You should know that ENABLE_INDICATOR is defined, meaning we do not execute the code in lines 871 and 872. Instead, we do lines 874 to 879, and then we do 881. Why do we do 881 if it is after the #endif? That’s because we fell off the end of the #ifdef block, and then we do whatever is next, after we fall off, namely returning NULL.

Don’t worry, I don’t know what’s going on yet, either. Time for a refill!

Back? Great. Alright, valgrind says that we’re doing something funky with g_slist_prepend().

==13124==    by 0x74F7ABD: g_slist_prepend (gslist.c:265)

And our relevant code is:

 874                 GSList *dupes = NULL;
 875                 const char *path;
 876 
 877                 dupes = g_object_get_data (G_OBJECT (dup_data.found), "dupe
 878                 path = nm_object_get_path (NM_OBJECT (ap));
 879                 dupes = g_slist_prepend (dupes, g_strdup (path));
 880 #endif
 881                 return NULL;

We can see that we declare the pointer *dupes on line 874, but we don’t do anything with it. Then, we assign something to it on line 877. Then, we assign something to it again on line 879. Finally, we end up not doing anything with *dupes at all, and just return NULL on line 881.

This definitely seems weird and worth a second glance. At this point, I’m asking myself the following questions:

• did g_object_get_data() allocate memory?
• did g_slist_prepend() allocate memory?
• are we overwriting *dupes on line 879? that might be a leak.
• is it safe to just return NULL without doing anything to dupes? maybe that’s our leak?

Let’s take them in order.

did g_object_get_data() allocate memory?
g_object_get_data has online documentation, so that’s our first stop. The documentation says:

Returns :
the data if found, or NULL if no such data exists. [transfer none]

Since I am not 100% familiar with glib terminology, I guess [transfer none] means that g_object_get_data() doesn’t actually allocate memory on its own. But let’s be 100% sure. Time to grab the glib source and find out for ourselves.

$ apt-get source libglib2.0-0
$ cd glib2.0-2.32.1
$ cscope -bqR
$ ctags -R
$ cscope -dp4
search for global definition of g_object_get_data

Pretty simple function.

3208 gpointer
3209 g_object_get_data (GObject     *object,
3210                    const gchar *key)
3211 {
3212   g_return_val_if_fail (G_IS_OBJECT (object), NULL);
3213   g_return_val_if_fail (key != NULL, NULL);
3214 
3215   return g_datalist_get_data (&object->qdata, key);
3216 }

Except I have no idea what g_datalist_get_data() does. Maybe that guy is allocating memory. Now I’ll use ctags to make my life easier. In vim, put your cursor over the “g” in “g_datalist_get_data” and then press control-]. This will “step into” the function. Magic!

 844 gpointer
 845 g_datalist_get_data (GData       **datalist,
 846                      const gchar *key)
 847 {
 848   gpointer res = NULL; 
 ... 
 856   d = G_DATALIST_GET_POINTER (datalist);
 ...
 859       data = d->data;
 860       data_end = data + d->len;
 861       while (data < data_end)
 862         {
 863           if (strcmp (g_quark_to_string (data->key), key) == 0)
 864             {
 865               res = data->data;
 866               break;
 867             }
 868           data++;
 869         }
 ... 
 874   return res;
 875 }

This is a pretty simple loop, walking through an existing list of pointers which have already been allocated somewhere else, starting on line 861. We do our comparison on line 863, and if we get a match, we assign whatever we found to res on line 865. Note that all we are doing here is a simple assignment. We are not allocating any memory!

Finally, we return our pointer on line 874. Press control-t in vim to pop back to your last location.

Now we know for sure that g_object_get_data() and g_datalist_get_data() do not allocate any memory at all, so there can be no possibility of a leak here. Let’s try the next function.

did g_slist_prepend() allocate memory?
First, read the documentation, which says:

The return value is the new start of the list, which may have changed, so make sure you store the new value.

This probably means it allocates memory for us, but let’s double-check just to be sure. Back to cscope!

 259 GSList*
 260 g_slist_prepend (GSList   *list,
 261                  gpointer  data)
 262 {
 263   GSList *new_list;
 264 
 265   new_list = _g_slist_alloc ();
 266   new_list->data = data;
 267   new_list->next = list;
 268 
 269   return new_list;
 270 }

Ah ha! Look at line 265. We are 100% definitely allocating memory, and returning it on line 269. Things are looking up! Let’s keep going with our questions.

are we overwriting *dupes on line 879? that might be a leak.
Remember:

 877                 dupes = g_object_get_data (G_OBJECT (dup_data.found), "dupe
 878                 path = nm_object_get_path (NM_OBJECT (ap));
 879                 dupes = g_slist_prepend (dupes, g_strdup (path));

We’ve already proven to ourselves that line 877 doesn’t allocate any memory. It just sets dupes to some value. However, on line 879, we do allocate memory. It is equivalent to this code:

  int *dupes;
  dupes = 0x12345678;
  dupes = malloc(128);

So simply setting dupes to the return value of g_object_get_data() and later overwriting it with the return value of malloc() does not inherently cause a leak.

By way of counter-example, the below code is a memory leak:

  int *dupes;
  dupes = malloc(64);
  dupes = malloc(128);    /* leak! */

The above essentially illustrates the scenario I was worried about. I was worried that g_object_get_data() allocated memory, and then g_slist_prepend() also allocated memory which would have been a leak because the first value of dupes got scribbled over by the second value. My worry turned out to be incorrect, but that is the type of detective work you have to think about.

As a clearer example of why the above is a leak, consider the next snippet:

  int *dupes1, *dupes2;
  dupes1 = malloc(64);     /* ok */
  dupes2 = malloc(128);    /* ok */
  dupes1 = dupes2;         /* leak! */

First we allocate dupes1. Then allocate dupes2. Finally, we set dupes1 = dupes2, and now we have a leak. No one knows what the old value of dupes1 was, because we scribbled over it, and it is gone forever.

is it safe to just return NULL without doing anything to dupes? maybe that’s our leak?
We can definitively say that it is not safe to return NULL without doing anything to dupes. We definitely allocated memory, stuck it into dupes, and then threw dupes away. This is our smoking gun.

Next time, we’ll see how to actually fix the problem.

This is a a brief post to alert everyone that the 32Gb Nexus7 with 3g cannot currently install Ubuntu.  The problems have been fixed, but we’re not going to re-roll a Quantal based image for this. You’ll get a fix when the Raring builds are ready, which should be soon, hopefully this week. The issue, if you’re curious, is that the new radio changed the device ID where we write the rootfs into. The new code will not make assumptions about the layout and will determine it dynamically.

leaky plumbing?

An important piece of optimizing the Ubuntu core on the Nexus 7 is slimming down Ubuntu’s memory requirements. It turns out this focus area has plenty of opportunity to help contribute, and today, I’ll talk about how to find memory leaks in an individual application using valgrind.

The best part? You don’t even have to be a developer to help. The second best part? You don’t even need a Nexus 7! What I describe below works on any Ubuntu machine. Let’s get started!

The first step is to find an application to profile. This is the easiest step. Maybe you have an app you use all the time and really care about making it perform as well as possible. Or maybe you’re experiencing a strange behavior problem in an app that takes a little while to show up. Or maybe you just pick a random application from the dash because you’re in a great mood. They’re all good.

In my case, I’ll use nm-applet as my example, since I’ve been struggling with LP: #780602 for a while, where the list of wifi access points would stop displaying after a day or two. Trés annoying!

Next, install valgrind if it is not already installed.

sudo apt-get install valgrind

Pay attention to the next bit because it is important. In order for your valgrind report to be as helpful as possible for developers, you will also need to install debug packages related to your app. The debug packages contain information to help developers narrow in on exactly where problems might be. “Great” you say, “what do I need to install?”

UPDATE: 29 January 2013

After a bit more thinking and discussing with smart folks like infinity, xnox, and pitti, we realized that I was essentially reinventing a lot of code that already exists in apport-retrace, as that tool already knows how to go from a binary to a package and then solve dependencies.

I tossed the idea (and a really rough crappy version of a prototype) to Kyle Nitzsche who took the idea, ran with it, and fixed all my crap! Woo hoo! With a little bit of effort, we ended up with apport-valgrind which has already landed in raring (along with the required valgrind support patch). Even better, Kyle wrote a great apport-valgrind introduction explaining how it works.

So ignore the script below and use apport-valgrind instead (unfortunately only available in raring).

Today is your lucky day because I’ve written a small script to help you figure out which debug packages you’ll need. Go ahead and grab the python version of valgrind-ubuntu-dbg-packages. (Ignore the go version for now, that’s just something I’m playing with in my other spare time!)

Ok, now comes the tricky part. We have to do a quick valgrind run to see what libraries your app uses. Then we’ll use the helper script to see if there are debug packages for those libraries. Ready?

To run valgrind, use this command:

G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log --track-origins=yes 

Replace with the name of your app.

Let this run long enough for your app to launch (which may take a while under valgrind) and then play with your app just a bit where you would reproduce your bug but without actually reproducing the bug. In the case of nm-applet, I did the following sequence:

killall nm-applet	# stop earlier instances of nm-applet

G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log --track-origins=yes nm-applet

Then I clicked the “More networks” menu item in the applet just to get it to display the other wifi access points, since this is the thing that was breaking for me. After doing that just once, I stopped my valgrind run completely by pressing control-c in the terminal where I launched it.

A valgrind log file should now exist, and you can run the helper script on the log:

./valgrind-ubuntu-dbg-packages.py valgrind.log

You will see quite a bit of output, but at the end, you will get a list of recommended extra packages to install.

It is recommended to install the following packages:
libnss3-dbg libdbus-glib-1-2-dbg libdconf-dbg gvfs-dbg libcanberra-gtk3-module-dbg libatk1.0-dbg librsvg2-dbg libfontconfig1-dbg

Go ahead and install the packages.

Now we are finally ready to collect our real logs.

Update: 29 January 2013

Instead of doing all that janky stuff above, just:

1. apt-get install apport-valgrind
2. run: apport-valgrind <executable>
3. Do step 2 for as long as it takes to reproduce the bug. There is no step 3!

Re-run valgrind exactly as above, but this time, let the app run as long as it needs to reproduce the bug. In the case of nm-applet, I had to let it just sit there and run normally for 24 hours before I saw the bug again. Hopefully your bug reproduces faster! Patience is key. I recommend eating a delicious sandwich if you can’t think of anything better to do.

After your bug has reproduced itself, kill the valgrind run. File a bug — you can use the Ubuntu Nexus7 project — and be sure to attach the valgrind log. It would also be great if you could describe how you reproduced the bug. Be sure to read the bug filing guidelines for more detail.

Huzzah, you’ve contributed something extremely valuable to making Ubuntu leaner and meaner — a great log file. With any luck, a developer will be able to pick up your bug and fix the problem.

And… if we’re even luckier, maybe that developer will be you! Next time I’ll show you how to actually analyze the valgrind log. Stay tuned.

I was asked last Friday about what type of bugs we see the most on Nexus7.  Right now we have about 75 unfixed bugs, and from having walked through that bugs list about twenty times, I know that we can break the bugs down into some categories. These are arbitrary categories and subject to debate, but they show the patterns I see in the bug list:

Kernel/Kernel Config/Drivers: 9 bugs

The initial kernel we used was the Android 3.1 kernel, which included binary drivers. This kernel has configuration and code differences from the standard Ubuntu kernel. The kernel team at Canonical is working on trying to get the kernel we’re using as close to Ubuntu as possible. This includes things as simple as enabling more modules and as complex as merging in patches so we can support things like overlayfs.

Onboard Related Issues: 9 bugs

For mobility impaired users, Onboard is a part of daily life. For most of us, we only have to use it when we’re playing with a tablet device, so I think it’s great that these bugs are getting some attention. One bug that was recently fixed by marmuta that should lead to Onboard launching 7x faster with certain themes, including the default theme. Some of the bugs in this category are not issues with Onboard itself, but impact Onboard specifically.

Unity/Nux: 6 bugs

Unity and nux have some bugs that impair the usability of the device and a couple bugs that lead to crashes or lock-ups. This is the area I know the least about. Many of the bugs here are sitting in the upstream project as “New”, if you can help confirm them upstream or even find an older dupe, please do.

Tegra3/nVidia: 6 bugs

We found several bugs that seem to be Tegra3 related, Tegra drivers related, or bugs that we need some input from nVidia on, for example, the sound only works after suspend/resume issue.  Not all of these may really be tegra related issues in the end, many require more investigation.

—–

So these are my top four categories, but that still leaves over half the bugs out. There are some smaller categories, which I’d list as Touch, Performance, and Bluetooth, and then there is Misc aka Everything Else.

Categories aside, another interesting thing I’ve noticed that aside from bugs that are specific to the kernel, drivers, or chipset, almost all the bugs we’ve found were confirmed on other platforms; usually they’re confirmed on a someone’s amd64/i386 laptop. Finding, bringing attention to, and fixing these bugs means shows that we’re achieving one of our goals, which is to fix issues in Ubuntu Core. These fixes will benefit all platforms. There are only a small number of bugs that we have not been able to confirm on other platforms yet, can any of my readers do so?  Here are the ones that stand-out in my mind:

• 1070374 unity cannot be cleanly restarted from the command-line on Nexus7
• 1077099 Color management on the Nexus7 does not work

Also I’d like to thank all the new contributors we’ve had in the past couple of weeks, we’re glad for all your help on bugs in any form you can provide it.

Below are the full bug lists for my generated categories:

Kernel:

• 1068672 webcam support
• 1072320 please consider adding OTG charging support to kernel
• 1075549 please include fw_bcmdhd.bin and bcm4330.hcd in linux-firmware for support of the nexus7
• 1076317 overlayfs support
• 1070770 bluetoothd dies with glibc malloc memory corruption when used with brcm_patchram
• 1071259 Setting brightness all the way down actually switches off the display completely
• 1073499 please consider turning on all possible modules for external USB devices
• 1073840 Sync kernel configuration with the one from the Ubuntu kernel
• 1074673 JACK server fails to start

OnBoard:

• 960537 Dash search box doesn’t unhide Onboard on-screen keyboard
• 1078554 Onboard doesn’t respect launcher icon size
• 1081227 onboard should optionally stay hidden if a keyboard is present
• 1075326 On screen keyboard doesn’t re-position in order to see input
• 421660 gksu’s and gksudo’s modal password prompt prevents OnBoard’s virtual keyboard input, causing accessibility issues
• 1079591 onboard can be made thin to the point of unusable
• 1071508 Onboard onscreen keyboard isn’t always shown when text input selected
• 1077260 When using software center search, onboard goes away until text box is reslected after entering 2 chars
• 1077277 Keyboard can’t type into Firefox bookmark dialog

Unity/Nux:

• 1065638 Unity panels don’t display visuals
• 1072249 Using desktop switcher via touchscreen causes Unity launcher to stop working
• 1045256 Dash – It should be possible to vertically scroll the Dash left clicking and dragging
• 1055949 Unity panel shadow appears as solid black bar on GLES/ARM (Pandaboard, Nexus 7)
• 1075417 Unity panel/launcher width don’t scale with system DPI/font settings
• 1070374 unity cannot be cleanly restarted from the command-line on Nexus7

Tegra3/nVidia:

• 1065644 plymouth causes a hard reset of the nexus
• 1068804 sound only works after suspend/resume cycle
• 1070283 after reboot, framebuffer of previous boot appears on screen
• 1073096 Screen is corrupted between rotations
• 1067954 control-alt-f1 to bring up a VT shows a blank (black) screen
• 1070755 screen rotates to portrait sometimes

PS – Thanks to Chris Wayne for vetting my bug category list.

Week ending 16 November 2012

Accomplishments

• New kernel has been uploaded to raring archive. Now we’re just waiting on a fix for nux to land before everyone can dist-upgrade to raring. Look for an announcement soon. Thanks to Jani Monoses for doing the heavy lifting here and the Ubuntu kernel team for taking care of the last mile.
• New benchmarking packages — ubuntu-benchmarking-tools and ubuntu-remote-debug-host-tools — uploaded to raring. Once your Nexus 7 is on raring, you’ll be able to install these convenient metapackages and help us start Ubuntu Pilates! Well done, Chris Wayne!
• A juju pbuilder charm has been submitted to the charm store. Once this is accepted, developers will be able to easily build ARM packages in the cloud. Thanks Scott Sweeny.
• Performance optimizations are already landing. Our onscreen keyboard, onboard, recently reduced its startup time from 45 seconds down to 6 seconds. Check out all the gory details in the bug, and big thanks to marmuta and Francesco Fumanti!
• We had our first weekly status meeting in #ubuntu-meeting. Come back every week for more.

Worst 5 Bugs

• #1065638 – Unity panels don’t display visuals – INPROGRESS. Thanks to Jay Taoko, we have a merge proposal for this bug, and once it is resolved, people will be able to dist-upgrade their tablets to raring.
• #1068994 – button1 gets stuck after a while – INPROGRESS. Daniel d’Andrada is fighting his way past grues in the dungeons of the X input stack and could use your help in debugging!
• #1068804 – Sounds only works after s3/resume cycle – BLOCKED.
  Alex Chiang will escalate this bug to nVidia. Thanks to David Henningsson for collecting up some great notes and logs are in the bug.

• #1072920 – Rotating screen using gnome-settings does not rotate touch input – CONFIRMED. Needs an owner. The script that Bryce Harrington wrote (/usr/bin/xrotate) might be a good starting point for investigating this. Possibly related: Sometimes the screen seems to rotate to portrait mode by itself when coming out of suspend (http://pad.lv/1070755)
• #1072249 – Using desktop switcher causes unity to break – CONFIRMED. Needs an owner. Who wants to help?

Upcoming Plans

• We are working with Platform QA on creating a set of guidelines and tools for the community to help us start benchmarking memory consumption and usage. Expect an announcement around 23 November.
• Converting our FAQ to a more friendly and community maintainable AskUbuntu format.

Grab Bag
Brave souls can try upgrading to raring today with this apt-pinning recipe. Create the following file /etc/apt/preferences.d/ubuntu-nexus7-ppa and add the contents below. Thanks to Colin Watson for the tidbit.

Package: *
Pin: release o=LP-PPA-ubuntu-nexus7
Pin-Priority: 600

Shortly after the Ubuntu App Showdown earlier this year, Didier Roche and Michael Terry kicked off a series of discussions about a ground-up re-write of Quickly.  Not only would this fix many of the complications app developers experienced during the Showdown competition, but it would also make it easier to write tools around Quickly itself.

Unfortunately, neither Didier nor Michael were going to have much time this cycle to work on the reboot.  We had a UDS session to discuss the initiative, but we were going to need community contributions in order to get it done.

JFDI

I was very excited about the prospects of a Quickly reboot, but knowing that the current maintainers weren’t going to have time to work on it was a bit of a concern.  So much so, that during my 9+ hour flight from Orlando to Copenhagen, I decided to have a go at it myself. Between the flight, a layover in Frankfurt without wifi, and a few late nights in the Bella Sky hotel, I had the start of something promising enough to present during the UDS session.  I was pleased that both Didier and Michael liked my approach, and gave me some very good feedback on where to take it next.  Add another 9+ hour flight home, and I had a foundation on which a reboot can begin.

Where is stands now

My code branch is now a part of the Quickly project on Launchpad, you can grab a copy of it by running bzr branch lp:quickly/reboot.  The code currently provides some basic command-line functionality (including shell completion), as well as base classes for Templates, Projects and Commands.  I’ve begun porting the ubuntu-application template, reusing the current project_root files, but built on the new foundation.  Currently only the ‘create’ and ‘run’ commands have been converted to the new object-oriented command class.

I also have examples showing how this new approach will allow template authors to easily sub-class Templates and Commands, by starting both a port of the ubuntu-cli template, and also creating an ubuntu-git-application template that uses git instead of bzr.

What comes next

This is only the very beginning of the reboot process, and there is still a massive amount of work to be done.  For starters, the whole thing needs to be converted from Python 2 to Python 3, which should be relatively easy except for one area that does some import trickery (to keep Templates as python modules, without having to install them to PYTHON_PATH).  The Command class also needs to gain argument parameters, so they can be easily introspected to see what arguments they can take on the command line.  And the whole thing needs to gain a structured meta-data output mechanism so that non-Python application can still query it for information about available templates, a project’s commands and their arguments.

Where you come in

As I said at the beginning of the post, this reboot can only succeed if it has community contributions.  The groundwork has been laid, but there’s a lot more work to be done than I can do myself.  Our 13.04 goal is to have all of the existing functionality and templates (with the exception of the Flash template) ported to the reboot.  I can use help with the inner-working of Quickly core, but I absolutely need help porting the existing templates.

The new Template and Command classes make this much easier (in my opinion, anyway), so it will mostly be a matter of copy/paste/tweak from the old commands to the new ones. In many cases, it will make sense to sub-class and re-use parts of one Template or Command in another, further reducing the amount of work.

Getting started

If you are interested in helping with this effort, or if you simply want to take the current work for a spin, the first thing you should do is grab the code (bzr branch lp:quickly/reboot).  You can call the quickly binary by running ./bin/quickly from within the project’s root.

Some things you can try are:

./bin/quickly create ubuntu-application /tmp/foo

This will create a new python-gtk project called ‘foo’ in /tmp/foo.  You can then call:

./bin/quickly -p /tmp/foo run

This will run the applicaiton.  Note that you can use -p /path/to/project to make the command run against a specific project, without having to actually be in that directory.  If you are in that directory, you won’t need to use -p (but you will need to give the full path to the new quickly binary).

If you are interested in the templates, they are in ./data/templates/, each folder name corresponds to a template name.  The code will look for a class called Template in the base python namespace for the template (in ubuntu-application/__init__.py for example), which must be a subclass of the BaseTemplate class.  You don’t have to define the class there, but you do need to import it there.  Commands are added to the Template class definition, they can take arguments at the time you define them (see code for examples), and their .run() method will be called when invoked from the command line.  Unlike Templates, Commands can be defined anywhere, with any name, as long as they subclass BaseCommand and are attached to a template.

You have critical decisions ahead as you take your first steps into cloud computing.

One of them will be whether to build a private cloud infrastructure in your own data centre, make use of one of the public cloud services offered by vendors like Amazon, Rackspace and HP, or combine the two in a ‘hybrid cloud’ approach.

You can get closer to the right decision by considering the right questions now:

• Budget - How much do you have (or how much don’t you have) to support your cloud strategy?
• Speed - When do you need this done? Tomorrow, next year, yesterday…
• Demand - How many users will you need to support? And will they call come at once?
• Resources - What kind of resources do you have in-house? And how many can you realistically get your hands on?
• Privacy -How sensitive is your data? Where are you doing business?

This short, sharp checklist takes you through the process that points you in the right direction and ensures your investments pay off from the start. Download it today.

There are dozens of ways that the rest of the community can participate in the Nexus7 project, but I’d like to call out one facet that I’ve been working hard on since I got back from Copenhagen. Chris Wayne and I spend a large portion of our day going through the Nexus7 bug list doing bug triage. In brief, what we do is:

1. If the bug is unclear, ask the submitter for more info in a comment
2. Checking for duplicate bug reports inside the Nexus7 project
3. Confirm that the bug really happens on the Nexus7 device
4. See if the bug occurs on our laptops/dev boxes (x86 running Quantal)
5. Search the upstream launchpad projects to see if the bug is known already, and mark the Nexus7 one as a duplicate if so
6. Search gnome bug reports to see if we can link to one
7. File upstream LP and gnome bug reports if none exist
8. Testing fixes from upstream
9. Set bug priority

So today, I’m officially asking the community to help, if you’re interested in helping with the Nexus7 work and don’t know where to start, bug triage is a great place and we could use your help.

So how do you start?  A good place to start is by reviewing the New bug queue and then following the Triaging guide and try to get enough information so that a developer can get started on it. You should also join the #ubuntu-arm channel on freenode, where we can discuss bug status and priority. Feel free to ping me (mfisch) or Chris (cwayne) if you have questions about a bug or how to help.

You may also consider joining the Ubuntu Bug Squad, and later Ubuntu Bug Control if you want to keep doing this work generally in Ubuntu.

On a personal note: when I first started working on Ubuntu, I joined the Ubuntu Bug Squad because there’s really no better way to learn about the components of Ubuntu than to dive into a bunch of bug reports. After a while doing Bug triage work, I was very comfortable with launchpad and bug triage procedures. I also had a better feel for how the components of the system worked together, it’s amazing how much you can learn from digging into bug reports!