Canonical Voices

Posts tagged with 'ubuntu one'

beuno

Now that we've open sourced the code for Ubuntu One filesync, I thoughts I'd highlight some of the interesting challenges we had while building and scaling the service to several million users.

The teams that built the service were roughly split into two: the foundations team, who was responsible for the lowest levels of the service (storage and retrieval of files, data model, client and server protocol for syncing) and the web team, focused on user-visible services (website to manage files, photos, music streaming, contacts and Android/iOS equivalent clients).
I joined the web team early on and stayed with it until we shut it down, so that's where a lot of my stories will be focused on.

Today I'm going to focus on the challenge we faced when launching the Photos and Music streaming services. Given that by the time we launched them we had a few years of experience serving files at scale, our challenge turned out to be in presenting and manipulating the metadata quickly to each user, and be able to show the data in appealing ways to users (showing music by artist, genre and searching, for example). Photos was a similar story, people tended to have many thousands of photos and songs and we needed to extract metadata, parse it, store it and then be able to present it back to users quickly in different ways. Easy, right? It is, until a certain scale  :)
Our architecture for storing metadata at the time was about 8 PostgreSQL master databases where we sharded metadata across (essentially your metadata lived on a different DB server depending on your user id) plus at least one read-only slave per shard. These were really beefy servers with a truck load of CPUs, more than 128GB of RAM and very fast disks (when reading this, remember this was 2009-2013, hardware specs seem tiny as time goes by!).  However, no matter how big these DB servers got, given how busy they were and how much metadata was stored (for years, we didn't delete any metadata, so for every change to every file we duplicated the metadata) after a certain time we couldn't get a simple listing of a user's photos or songs (essentially, some of their files filtered by mimetype) in a reasonable time-frame (less than 5 seconds). As it grew we added caches, indexes, optimized queries and code paths but we quickly hit a performance wall that left us no choice but a much feared major architectural change. I say much feared, because major architectural changes come with a lot of risk to running services that have low tolerance for outages or data loss, whenever you change something that's already running in a significant way you're basically throwing out most of your previous optimizations. On top of that as users we expect things to be fast, we take it for granted. A 5 person team spending 6 months to make things as you expect them isn't really something you can brag about in the middle of a race with many other companies to capture a growing market.
In the time since we had started the project, NoSQL had taken off and matured enough for it to be a viable alternative to SQL and seemed to fit many of our use cases much better (webscale!). After some research and prototyping, we decided to generate pre-computed views of each user's data in a NoSQL DB (Cassandra), and we decided to do that by extending our existing architecture instead of revamping it completely. Given our code was pretty well built into proper layers of responsibility we hooked up to the lowest layer of our code,-database transactions- an async process that would send messages to a queue whenever new data was written or modified. This meant essentially duplicating the metadata we stored for each user, but trading storage for computing is usually a good trade-off to make, both in cost and performance. So now we had a firehose queue of every change that went on in the system, and we could build a separate piece of infrastructure who's focus would only be to provide per-user metadata *fast* for any type of file so we could build interesting and flexible user interfaces for people to consume back their own content. The stated internal goals were: 1) Fast responses (under 1 second), 2) Less than 10 seconds between user action and UI update and 3) Complete isolation from existing infrastructure.
Here's a rough diagram of how the information flowed throw the system:

U1 Diagram

It's a little bit scary when looking at it like that, but in essence it was pretty simple: write each relevant change that happened in the system to a temporary table in PG in the same transaction that it's written to the permanent table. That way you get transactional guarantees that you won't loose any data on that layer for free and use PG's built in cache that keeps recently added records cheaply accessible.
Then we built a bunch of workers that looked through those rows, parsed them, sent them to a persistent queue in RabbitMQ and once it got confirmation it was queued it would delete it from the temporary PG table.
Following that we took advantage of Rabbit's queue exchange features to build different types of workers that processes the data differently depending on what it was (music was stored differently than photos, for example).
Once we completed all of this, accessing someone's photos was a quick and predictable read operation that would give us all their data back in an easy-to-parse format that would fit in memory. Eventually we moved all the metadata accessed from the website and REST APIs to these new pre-computed views and the result was a significant reduction in load on the main DB servers, while now getting predictable sub-second request times for all types of metadata in a horizontally scalable system (just add more workers and cassandra nodes).

All in all, it took about 6 months end-to-end, which included a prototype phase that used memcache as a key/value store.

You can see the code that wrote and read from the temporary PG table if you branch the code and look under: src/backends/txlog/
The worker code, as well as the web ui is still not available but will be in the future once we finish cleaning it up to make it available. I decided to write this up and publish it now because I believe the value is more in the architecture rather than the code itself   :)

Read more
beuno

This week has been bitter-sweet. On the one hand, we announced that a project many of us had poured our hearts and minds into was going to be shut down. It’s made many of us sad and some of us haven’t even figured out what to do with their files yet    :)

On the other hand, we’ve been laser-focused on making Ubuntu on phones and tablets a success, our attention has moved to making sure we have a rock-solid, scalable, secure and pleasant to use for developers and users alike. We just didn’t have the time to continue racing against other companies whose only focus is on file syncing, which was very frustrating as we saw a project we were proud of be left behind. It was hard to keep feeling proud of the service, so shutting it down felt like the right thing to do.

I am, however, very excited about open sourcing the server-side of the file syncing infrastructure. It’s a huge beast that contains many services and has scaled well into the millions of users.

We are proud of the code that is being released and in many ways we feel that the code itself was successful despite the business side of things not turning out the way we hoped for.

This will be a great opportunity to those of you who’ve been itching to have an open source service for personal cloud syncing at scale, the code comes battle-tested and with a wide array of features.

As usual, some people have taken this generous gesture “as an attempt to gain interest in a failing codebase”, which couldn’t be more wrong. The agenda here is to make Ubuntu for phones a runaway success, and in order to do that we need to double down on our efforts and focus on what matters right now.

Instead of storing away those tens of thousands of expensive man-hours of work in an internal repository somewhere, we’ve decided to share that work with the world, allow others to build on top of that work, benefit from it.

It’s hard sometimes to see some people trying to make a career out of trying to make everything that Canonical does as inherently evil, although at the end of the day what matters is making open source available to the masses. That’s what we’ve been doing for a long time and that’s the only thing that will count in the end.

 

So in the coming months we’re going to be cleaning things up a bit, trying to release the code in the best shape possible and work out the details on how to best release it to make it useful for others.

All of us who worked on this project for so many years are looking forward to sharing it and look forward to seeing many open source personal cloud syncing services blossoming from it.

Read more
Jane Silber

Today we are announcing plans to shut down the Ubuntu One file services.  This is a tough decision, particularly when our users rely so heavily on the functionality that Ubuntu One provides.  However, like any company, we want to focus our efforts on our most important strategic initiatives and ensure we are not spread too thin.

Our strategic priority for Ubuntu is making the best converged operating system for phones, tablets, desktops and more. In fact, our user experience, developer tools for apps and scopes, and commercial relationships have been constructed specifically to highlight third party content and services (as opposed to our own); this is one of our many differentiators from our competitors.  Additionally, the free storage wars aren’t a sustainable place for us to be, particularly with other services now regularly offering 25GB-50GB free storage.  If we offer a service, we want it to compete on a global scale, and for Ubuntu One to continue to do that would require more investment than we are willing to make. We choose instead to invest in making the absolute best, open platform  and to highlight the best of our partners’ services and content.

As of today, it will no longer be possible to purchase storage or music from the Ubuntu One store. The Ubuntu One file services will not be included in the upcoming Ubuntu 14.04 LTS release, and the Ubuntu One apps in older versions of Ubuntu and in the Ubuntu, Google, and Apple stores will be updated appropriately. The current services will be unavailable from 1 June 2014; user content will remain available for download until 31 July, at which time it will be deleted.

We will work to ensure that customers have an easy path to download all their content from Ubuntu One to migrate to other personal cloud services.  Additionally, we continue to believe in the Ubuntu One file services, the quality of the code, and the user experience, so will release the code as open source software to give others an opportunity to build on this code to create an open source file syncing platform.

Customers who have an active annual subscription will have their unused fees refunded. We will calculate the refund amount from today’s announcement, even though the service will remain available until 1 June and data available for a further two months.

We will contact customers separately with additional information about what to expect.  We will also publish further blog posts with advice on how to download content and with details on the open sourcing of the code.

The shutdown will not affect the Ubuntu One single sign on service, the Ubuntu One payment service, or the backend U1DB database service.

We’ve always been inspired by the support, feedback and enthusiasm of our users and want to thank you for the support you’ve shown for Ubuntu One. We hope that you’ll continue to support us as together we bring a revolutionary experience to new devices.

UPDATE:  See this post for updated information on downloading all your content from Ubuntu One.  We are aware that in some rare cases (large amount of content or very large number of files), the bulk download to a single archive is failing. Don’t worry – your content is not lost and we’ll post an updated bulk download tool which generates multiple archives rather than a single large one. We know of no issues with the other options discussed in that post.

 

Read more
Cristian Parrino

Today we are announcing plans to shut down the Ubuntu One file services.  This is a tough decision, particularly when our users rely so heavily on the functionality that Ubuntu One provides.  However, like any company, we want to focus our efforts on our most important strategic initiatives and ensure we are not spread too thin.

Our strategic priority for Ubuntu is making the best converged operating system for phones, tablets, desktops and more. In fact, our user experience, developer tools for apps and scopes, and commercial relationships have been constructed specifically to highlight third party content and services (as opposed to our own); this is one of our many differentiators from our competitors.  Additionally, the free storage wars aren’t a sustainable place for us to be, particularly with other services now regularly offering 25GB-50GB free storage.  If we offer a service, we want it to compete on a global scale, and for Ubuntu One to continue to do that would require more investment than we are willing to make. We choose instead to invest in making the absolute best, open platform  and to highlight the best of our partners’ services and content.

As of today, it will no longer be possible to purchase storage or music from the Ubuntu One store. The Ubuntu One file services will not be included in the upcoming Ubuntu 14.04 LTS release, and the Ubuntu One apps in older versions of Ubuntu and in the Ubuntu, Google, and Apple stores will be updated appropriately. The current services will be unavailable from 1 June 2014; user content will remain available for download until 31 July, at which time it will be deleted.

We will work to ensure that customers have an easy path to download all their content from Ubuntu One to migrate to other personal cloud services.  Additionally, we continue to believe in the Ubuntu One file services, the quality of the code, and the user experience, so will release the code as open source software to give others an opportunity to build on this code to create an open source file syncing platform.

Customers who have an active annual subscription will have their unused fees refunded. We will calculate the refund amount from today’s announcement, even though the service will remain available until 1 June and data available for a further two months.

We will contact customers separately with additional information about what to expect.  We will also publish further blog posts with advice on how to download content and with details on the open sourcing of the code.

The shutdown will not affect the Ubuntu One single sign on service, the Ubuntu One payment service, or the backend U1DB database service.

We’ve always been inspired by the support, feedback and enthusiasm of our users and want to thank you for the support you’ve shown for Ubuntu One. We hope that you’ll continue to support us as together we bring a revolutionary experience to new devices.

 

Read more
login.ubuntu.com-id-hY4GFhr

Ubuntu Phone OS integrates with Orange and Deutsche Telekom in GSMA OneAPI initiative

Mobile World Congress kicks off today and we’re gearing up to show off Ubuntu running on multiple devices. We’ll be demonstrating phones, tablets and desktops at the stand, have Ubuntu developers flashing spare hardware, as well as be showing integration and interoperability with Orange and Deutsche Telekom through the GSMA’s One API initiative.

GSMA’s OneAPI initiative aims to provide application programming interfaces (APIs) that enable applications to exploit mobile network capabilities, such as messaging, authentication, payments and location-finding with a cross-operator reach. For example, a payment network API could be used to add an in-app purchase directly to the user’s mobile phone bill.

Ubuntu is the first smartphone operating system to be able to demonstrate integration and interoperability with a carrier’s authentication and billing systems. Working with Deutsche Telekom and Orange, we’ll show how a single API can be used to instantly log users in with their operator identity and seamlessly link that with Ubuntu One, Ubuntu’s identity and payments services, and provide carrier billing options upon purchase of music and eventually, apps.

This is a massive step forward for the industry as the GSMA and partners such as Canonical, are spearheading an initiative to standardise access to operator facilities via network APIs across all operators. The initiative will benefit operators, developers and consumers:

  • It puts operators in a position to forge stronger relationships with their customers.
  • For developers, OneAPI reduces the time and effort needed to create applications for and content that is portable across mobile operators, increasing reach and ultimately enhancing the consumer experience.
  • For consumers, it makes it really quick and easy to make application purchases directly from their phone. It’s also more secure because it’s not necessary to input credit card details for each purchase.

Also at Mobile World Congress:

  • Mark Shuttleworth, founder of Ubuntu, will participate in a keynote panel discussion alongside Mozilla and Tizen on Tuesday 26th Feb at 18.00 at the MWC Conference Auditorium and broadcast live on Mobile World Live
  • We’ll be taking part in the App Developer Day on Tuesday 26th Feb. Stuart Langridge, technical architect at Canonical will be presenting the Ubuntu phone, SDK, HTML5 and native apps as well as discussing app development for Ubuntu on phones and tablets. We’ll also have engineers available at the event to flash spare handsets with Touch Developer Preview of Ubuntu. This will take place from 9.00-9.30 and 11.40-11.55, and 13.30-14.00 in Hall 8.0, Theatre A.
  • The GSMA Seminar on “Unlocking Value with Network APIs” will run on Thursday 28th from 9am to 10.30 am in Room CC1.1. Canonical’s Stuart Langridge will present and demo the Ubuntu Phone during the session. We’ll also be demonstrating Ubuntu’s OneAPI solution at the GSMA stand daily.
  • Look out for Ubuntu engineers who will flash spare hardware with developer images for phone and tablet throughout the show close to the Ubuntu stand.

Read more
James Henstridge

One of the projects I’ve been working on has been to improve aspects of the Ubuntu One Developer Documentation web site.  While there are still some layout problems we are working on, it is now in a state where it is a lot easier for us to update.

I have been working on updating our authentication/authorisation documentation and revising some of the file storage documentation (the API used by the mobile Ubuntu One clients).  To help verify that the documentation was useful, I wrote a small program to exercise those APIs.  The result is u1ftp: a program that exposes a user’s files via an FTP daemon running on localhost.  In conjunction with the OS file manager or a dedicated FTP client, this can be used to conveniently access your files on a system without the full Ubuntu One client installed.

You can download the program from:

https://launchpad.net/u1ftp/trunk/0.1/+download/u1ftp-0.1.zip

To make it easy to run on as many systems as possible, I packaged it up as a runnable zip file so can be run directly by the Python interpreter.  As well as a Python interpreter, you will need the following installed to run it:

  • On Linux systems, either the gnomekeyring extension (if you are using a GNOME derived desktop), or PyKDE4 (if you have a KDE derived desktop).
  • On Windows, you will need pywin32.
  • On MacOS X, you shouldn’t need any additional modules.

These could not be included in the zip file because they are extension modules rather than pure Python.

Once you’ve downloaded the program, you can run it with the following command:

python u1ftp-0.1.zip

This will start the FTP server listening at ftp://localhost:2121/.  Pointing a file manager at that URL should prompt you to log in, where you can use your standard Ubuntu One credentials and start browsing your files.  It will verify the credentials against the Ubuntu SSO service and issue an OAuth token that it stores in the keyring.  The OAuth token is then used to authenticate requests to the file storage REST API.

While I expect this program to be useful on its own, it was also intended to act as an example of how the Ubuntu One API can be used.  One way to browse the source is to simply unzip the package and poke around.  Alternatively, you can check out the source directly from Launchpad:

bzr branch lp:u1ftp

If you come up with an interesting extension to u1ftp, feel free to upload your changes as a branch on Launchpad.

Read more
mandel

Recently a very interesting bug has been reported agains Ubuntu One on Windows. Apparently we try to sync a number of system folders that are present on Windows 7 to be backward compatible.

The problem

The actual problem in the code is that we are using os.listdir. While lisdir on python does return system folders (at the end of the day, they are there) os.walk does not, for example, lets imaging hat we have the following scenario:

Documents
    My Pictures (System folder)
    My Videos (System folder)
    Random dir
    Random Text.txt

If we run os.listdir we would have the following:

import os
>> os.listdir('Documents')
['My Pictures', 'My Videos', 'Random dir', 'Random Text.txt']

While if we use os.walk we have:

import os
path, dirs, files = os.walk('Documents')
print dirs
>> ['Random dir']
print files
>> ['Random Text.txt']

The fix is very simple, simply filter the result from os.listdir using the following function:

import win32file
 
INVALID_FILE_ATTRIBUTES = -1
 
 
def is_system_path(path):
    """Return if the function is a system path."""
    attrs = win32file.GetFileAttributesW(path)
    if attrs == INVALID_FILE_ATTRIBUTES:
        return False
    return win32file.FILE_ATTRIBUTE_SYSTEM & attrs ==\
        win32file.FILE_ATTRIBUTE_SYSTEM

File system events

An interesting question to ask after the above is, how does ReadDirectoryChangesW work with systen directories? Well, thankfully it works correctly. What does that mean? Well, it means the following:

  • Changes in the system folders do not get notified.
  • Moves from a watch directory to a system folder is not a MOVE_TO, MOVE_FROM couple but a FILE_DELETED

The above means that if you have a system folder in a watch path you do not need to worry since the events will work correctly, which are very very good news.

Read more
mandel

On Ubuntu One we use BtiRock to create the packages for Windows. One of the new features I’m working on is to check if there are updates every so often so that the user gets notified. This code on Ubuntu is not needed because the Update Manger takes care of that, but when you work in an inferior OS…

Generate the auto-update.exe

In order to check for updates we use the generated auto-update.exe wizard from BitRock. Generating the wizard is very straight forward first, as with most of the BitRock stuff, we generate the XML to configure the generated .exe.

<autoUpdateProject>
    <fullName>Ubuntu One</fullName>
    <shortName>ubuntuone</shortName>
    <vendor>Canonical</vendor>
    <version>201</version>
    <singleInstanceCheck>1</singleInstanceCheck>
    <requireInstallationByRootUser>0</requireInstallationByRootUser>
    <requestedExecutionLevel>asInvoker</requestedExecutionLevel>
</autoUpdateProject>

There is just a single thing that is worth mentioning about the above XML. The requireInstallationByRootUser is true because we use the generated .exe to check if there are updates present and we do not what the user to have to be root for that, it does not make sense. Once you have the above or similar XML you can execute:

{$bitrock_installation$}\autoupdate\bin\customize.exe" build ubuntuone_autoupdate.xml windows

Which generates the .exe (the output is in ~\Documents\AutoUpdate\output).

Putting it together with Twisted

The following code provides an example of a couple of functions that can be used by the application, first to check for an update, and to perform the actual update.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import os
import sys
 
# Avoid pylint error on Linux
# pylint: disable=F0401
import win32api
# pylint: enable=F0401
 
from twisted.internet import defer
from twisted.internet.utils import getProcessValue
 
AUTOUPDATE_EXE_NAME = 'autoupdate-windows.exe'
 
def _get_update_path():
    """Return the path in which the autoupdate command is found."""
    if hasattr(sys, "frozen"):
        exec_path = os.path.abspath(sys.executable)
    else:
        exec_path = os.path.dirname(__file__)
    folder = os.path.dirname(exec_path)
    update_path = os.path.join(folder, AUTOUPDATE_EXE_NAME)
    if os.path.exists(update_path):
        return update_path
    return None
 
 
@defer.inlineCallbacks
def are_updates_present(logger):
    """Return if there are updates for Ubuntu One."""
    update_path = _get_update_path()
    logger.debug('Update path %s', update_path)
    if update_path is not None:
        # If there is an update present we will get 0 and other number
        # otherwise
        retcode = yield getProcessValue(update_path, args=('--mode',
            'unattended'), path=os.path.dirname(update_path))
        logger.debug('Return code %s', retcode)
        if retcode == 0:
            logger.debug('Returning True')
            defer.returnValue(True)
    logger.debug('Returning False')
    defer.returnValue(False)
 
 
def perform_update():
    """Spawn the autoupdate process and call the stop function."""
    update_path = _get_update_path()
    if update_path is not None:
        # lets call the updater with the commands that are required,
        win32api.ShellExecute(None, 'runas',
            update_path,
            '--unattendedmodeui none', '', 0)

With the above you should be able to easily update the installation of your frozen python app on Windows when using BitRock.

Read more
mandel

Following my last post regarding how to list all installed applications using python here is the code that one will require to remove an installed msi from a Windows machine using python.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class MsiException(Exception):
    """Raised when there are issues with the msi actions."""
 
 
def uninstall_product(uid):
    """Will remove the old beta from the users machine."""
    # we use the msi lib to be able to uninstall apps
    property_name = u'LocalPackage'
    uninstall_path = get_property_for_product(uid, property_name)
    if uninstall_path is not None:
        # lets remove the package.
        command_line = u'REMOVE=ALL'
        result = windll.msi.MsiInstallProductW(uninstall_path, command_line)
        if result != ERROR_SUCCESS:
            raise MsiException('Could not remove product %s' % uid)

The missing functions can be found in the last post about the topic.

Read more
David

Another edition of the Ubuntu App Developer Week and another amazing knowledge sharing fest around everything related to application development in Ubuntu. Brought to you by a range of the best experts in the field, here’s just a sample of the topics they talked about: App Developer Strategy, Bazaar, Bazaar Explorer, Launchpad, Python, Internationalization, Launchpad Translations, Unity, Unity 2D, Gedit Developer Plugins, the MyApps Portal, the App Review Board, the UbuntuSoftware Centre, Unity Mail, Launchpad Daily Builds, Ubuntu One APIs, Rapid App Development, Quickly, GooCanvas, PyGame, Unity Launcher, Vala, the App Developer Site, Indicators, Python Desktop Integration, Libgrip, Multitouch, Unity Lenses, Ubuntu One Files Integration, The Business Side of Apps, Go, Qt Quick… and more. Oh my!

And a pick of what they had to say:

We believe that to get Ubuntu from 20 million to 200 million users, we need more and better apps on Ubuntu
Jonathan Lange on making Ubuntu a target for app developers

Bazaar is the world’s finest revision control system
Jonathan Riddell on Bazaar

So you’ve got your stuff, wherever you are, whichever device you’re on
Stuart Langridge on Ubuntu One

Oneiric’s EOG and Evince will be gesture-enabled out of the box
Jussi Pakkanen on multitouch in Ubuntu 11.10

I control the upper right corner of your screen ;-)
Ted Gould on Indicators

If you happened to miss any of the sessions, you’ll find the logs for all of them on the Ubuntu App Developer Week page, and the summaries for each day on the links below:

Ubuntu App Developer Week – Day 5 Summary

The last day came with a surprise: an extra session for all of those who wanted to know more about Qt Quick and QML. Here are the summaries:

Getting A Grip on Your Apps: Multitouch on GTK apps using Libgrip

By Jussi Pakkanen

In his session, Jussi talked about one of the most interesting technologies where Ubuntu is leading the way in the open source world: multitouch. Walking the audience through the Grip Tutorial, he described how to add gesture support to existing applications based on GTK+ 3. He chose to focus on the higher layer of the uTouch stack, where he explained the concepts on which libgrip, the gesture library, is built upon, such as device types and subscriptions. After having explored in detail the code examples, he then revealed that in Oneiric Eye Of GNOME and Evince, Ubuntu’s default image viewer and default PDF reader, will be gesture-enabled.

Check out the session log.

Creating a Google Docs Lens

By Neil Patel

Neil introduced his session explaining the background behind Lenses: a re-architecture effort of the now superseded Places concept to make them more powerful, provide more features and make it easier to add features through a re-engineered API. Lenses create its own instance, add categories, filters and leave the searching to Scopes. The Lenses/Scopes pairs are purely requests for data, independent of the type of UI, and being provided by the libunity library, they can be written in any of the programming languages supported by GObject Introspection (Python, Javascript, C/C++, Vala, etc.). To illustrate all of this concepts, Neil devoted the rest of the session to a real example of creating a Lens for Google Docs.

Check out the session log.

Practical Ubuntu One Files Integration

By Michael Terry

Another hands-on session from Michael, with a real world example on how to supercharge apps with cloud support. Using his experience in integrating the Ubuntu One Files API to Deja Dup, the default backup application in Ubuntu, he went in detail through the code of a simple program to talk to a user’s personal Ubuntu One file storage area. We liked Michael’s session so much that it will very soon be featured as a tutorial on developer.ubuntu.com!

Check out the session log and Michael’s awesome notes.

Publishing Your Apps in the Software Center: The Business Side

By John Pugh

Ubuntu directly benefits from Canonical becoming a sustainable business to support its development, and that’s exactly what John talked about. Being responsible for business development in the Ubuntu Software Centre, he’s got a privileged  insight on how to make it happen. He started off explaining that the main goal is to present Ubuntu users with a large catalog of apps available for purchase, and then continued concentrating on how to submit paid applications to be published in the Software Centre. A simple 5-step process, the behind-the-scenes work can be summarized in: Canonical helps packaging the app, it hosts the app and provides the payment via pay.ubuntu.com, in a 80%/20% split. Other highlights include the facts that only non-DRM, non-licensed apps cannot be submitted right now, but there is ongoing work to implement license key support, and that MyApps, the online app submission portal, can take any nearly any content: apps with adverts, “free” online game clients and HTML5 apps.

Check out the session log.

Writing an App with Go

By Gustavo Niemeyer

Gustavo’s enthusiasm for Go, the new programming language created by Google shows every time you start a conversation with him on that topic. And it showed as well on this session, in which he created yet another “Hello world” application in a new language -you guessed-: Go. Along the way, he had time to describe all of the features of this new addition of the extensive family of programming languages: statically compiled with good reflection capabilities, structural typing, interfaces and more.

Check out the session log.

Qt Quick At A Pace

By Donald Carr

Closing the week on the last -and surprise- session, we had the luxury of having Donald, from the Nokia Qt team, the makers of Qt itself, to talk about Qt Quick. Using a clear and concise definition, Qt Quick is an umbrella term used to refer to QML and its associated tooling; QML being a declarative markup language with tight bindings to Javascript. A technology equally suited to mobile or to the desktop, QML enables developers to rapidly create animation-rich, pixmap-oriented UIs. Through the qtmediahub and Qt tutorial examples, he explored QML’s capabilities and offered good practices for succesfully developing QML-based projects.

Check out the session log.

Wrapping Up

Finally, if you’ve got any feedback on UADW, on how to make it better, things you enjoyed or things you believe should be improved, your comments will be very appreciated and useful to tailor this event to your needs.

Thanks a lot for participating. I hope you enjoyed it  as much as I did, and see you again in 6 months time for another week full with app development goodness!


Read more
David

Ubuntu App Developer Week – Day 3 Summary

Time flies and we’re already halfway through UADW, but there is still much to come! Here’s yesterday report for your reading pleasure:

Unity Mail: Webmail Notification on Your Desktop

By Dmitry Shachnev

Starting off with a description of the features of Unity Mail, such as displaying webmail unread message count, notifications and mail subjects, we then learned more about how it was developed and the technologies that were used to create it. It’s written in Python, using GObject introspection (PyGI) and integrates with Ubuntu through the Unity, Notify and Indicate modules. After describing each one in more detail, Dmitry continued talking about how the app can be translated using Launchpad, and how he uses the Bazaar  source revision control system to work with code history. Wrapping up, he went through the plans for the future: more configuration options, marking all messages as read and the need for a new icon. Any takers? ;)

Check out the session log here.

Launchpad Daily Builds and Rapid Feedback: Writing Recipe Builds

By Jelmer Vernooij

Assuming some previous knowledge on Debian packaging, in his session Jelmer walked the audience through a practical example of a basic recipe build for a small project: pydoctor. Drawing the cooking recipe analogy, package recipes are a description of the ingredients (source code branches) and how to put them together, ending up with a delicious Debian package for users to enjoy. Launchpad can build packages from recipes once or automatically on a daily basis provided the code has changed, conveniently placing the result in a PPA. In the last part of the session, he described in detail the contents of an existing recipe and added some notes on best practices when building from a recipe.

Check out the session log here.

Using the Ubuntu One APIs for Your Apps: An Overview

By Stuart Langridge

The idea bahind the Ubuntu One developer programme is to make it easy to add the cloud to your apps and make new apps for the cloud. With this opening line, Stuart delivered a talk about a high-level overview on the cool things you can do as an app developer adding Ubuntu One support. One aspect it data: for example building applications that work on the desktop, on mobile phones and on the web, securely sharing data among users. Another is music: streaming, streaming music and sharing playlists on the desktop, on mobile and from the web, all through a simple REST HTTP API. He also mentioned some examples of cloud enabled applications: Shutter and Deja-Dup, and many other interesting ways to use Ubuntu One to do exciting thigs with data. And you can get started already using the available documentation.

Check out the session log here.

Supercharging Your Apps with Unity Launcher Integration

By Jason Smith

In his talk, Jason first went through the terminology that covers the elements related to the Unity Launcher, and the bachground behind the Launcher API, implemented in the libunity library. Libunity can be used in many programming languages: Python, C, Vala and others supported by GObject Introspection. Going through what you can do with the Launcher (marking/unmarking apps as urgent, setting object counts, setting progress on objects and adding quicklist menu items to the object), he used Vala snippets to illustrate each feature with code.

Check out the session log here.

Hello Vala: An Introduction to the Vala Language

By Luca Bruno

Vala, a new programming language with C#-like syntax that compiles to C and targets the GObject type system: with a clear statement of what Vala is and what it can do, Luca, a contributor to the project introduced one by one the mostkey features of the language through his “Hello world” example: namespaces, types, classes, properties, keywords and more. As a highlight he mentioned Vala’s automatic memory management using reference counting, andits interoperability with other languages, most notably C, but it can also work with many others supported by GObject Introspection. Other cool featuresto note were also error handling on top of GError, support for async operations, closures and DBus client/server, on each of which he elaborated before finishing the session.

Check out the session log here.

The Day Ahead: Upcoming Sessions for Day 3

Another day, another awesome set of sessions coming up:

16.00 UTCCreating an App Developer Website: developer.ubuntu.com

Ubuntu 11.10 will not only bring new features to the OS itself. In time for the release we’ll be launching the new Ubuntu App Developer site, a place for developers to find all the infromation and the resources they need to get started creating, submitting and publishing their apps in Ubuntu. John Oxton, David Planella and many other people have worked to make the next developer.ubuntu.com possible and will tell you all about it.

17:00 UTCRapid App Development with Quickly

Quickly is a wrapper that pulls together all the recommended tools and technologies to bring apps from creation and through their whole life cycle in Ubuntu. With an easy set of commands that hide all the complexity for your, it effectively enables developers to follow rapid development principles and worry only about writing code. Michael Terry, from the Quickly development team will be looking forward to guide you through the first steps with this awesome tool.

18:00 UTCDeveloping with Freeform Design Surfaces: GooCanvas and PyGame

Have you ever wondered what freeform design surfaces, or canvases are? You probably have now. Well, lucky you then, because Rick Spencer will be here to tell you what they’re good for and how to get started with them ;)

19:00 UTCMaking your app appear in the Indicators

In another session on how to integrate with the platform, Ted Gould, the man who knows most about them, will describe how to add indicator features  to your apps, both in terms of panel indicators and messaging menu support.

20:00 UTC – Will it Blend? Python Libraries for Desktop Integration

You certainly will want your app to have that familiar look and feel at home in the OS it’s running on, but you’ll also want it to use all the backend technologies to integrate even deeper and provide a great user experience. Well, fear not, for Marcelo Hashimoto is here to tell you exactly how to do that!

Looking forward to seeing you all there in a few hours!


Read more
mandel

In a few days (well, if I find some kind person to take care of Iron) I will be attending the Ubuntu One Developer evening in which we should be able to hear Stuart will be talking about the bunch of crazy ideas he has for developers to use or infrastructure to do cool stuff. I’ll be there for two reason:

  • Hear what Stuart has been planning. I’ve got to recognized I should know a lot more of the Ubuntu One developer program but unfortunately I have been in the working in the Windows port too much and I have ignored the rests of the teams… mea culpa :( .
  • Learn a few more things of the APIs so that I can find my little Chrome extension that uses Ubuntu One (no, it is not bookmark sync, I cannot be less interested in that!).
  • See a bunch of developers and learn about there ideas and what they are doing.
  • Drinks, drinks, drinks! I’m a University of Manchester alumni and a bloody miss Manchester, I don’t care what people say, it is a great city.

If you are going to be in Manchester you should join us, the event is FREE and trust me Stuart is a great guy to go out for drinks with (I’m not bad either, but I always get in trouble :P ).

Read more
beuno

After a long and interesting journey, today we've released Ubuntu One Files for Android.

The app started being developed by Micha? Karnicki as a Google Summer of Code project, and he did such a fantastic job at it that we hired him on full time and teamed him up Chad Miller to end up releasing a fantastically polished app. It got immediately featured in the press!
It was built on top of our public APIs, documented here: https://one.ubuntu.com/developer/

Besides it letting you access all your files stored in Ubuntu One, it has a very cool feature to auto-sync all the pictures on your phone, having an instant backup of them, and a convenient place to share them!

I'm super proud of the work we put out.

Also, as with all the rest of our clients, it's open source and you can get it in Launchpad

Read more
mandel

At Ubuntu One we required to be able to use named pipes on windows for IPC. This is a ver normal process in multi-process applications like the one we are going to provide, but in our case we had a twist, we are using twisted. As some of you may know there is not default reactor that would allow you to write a protocol in twisted and allows to use named pipes as the transport of the protocol. Well this was until very recently.

Txnamedpipes (lp:txnamedpipes) is a project that provides a ICOP based reactor that allows to use namedpipes for the transport of your protocol. At the moment we are confident that the implementation would allow you to use spred.pb or a custom protocol on twisted 10 and later on Windows 7 (we have been able to find a number of issues on Windows XP). The following is a small example of a spread.pb service and client that uses a named pipe for communication.

from txnamedpipes.reactor import install
install()
from twisted.spread import pb
from twisted.internet import reactor
 
class Echoer(pb.Root):
    def remote_echo(self, st):
        print 'echoing:', st
        return st
 
if __name__ == '__main__':
    reactor.listenPipe('\\\\.\\pipe\\test_pipe',
                               pb.PBServerFactory(Echoer()))
    reactor.run()
from txnamedpipes.reactor import install
install()
from twisted.spread import pb
from twisted.internet import reactor
from twisted.python import util
 
factory = pb.PBClientFactory()
reactor.connectPipe('\\\\.\\pipe\\test_pipe', factory)
d = factory.getRootObject()
d.addCallback(lambda object: object.callRemote("echo", 
                      "hello network"))
d.addCallback(lambda echo: 'server echoed: '+echo)
d.addErrback(lambda reason: 'error: '+str(reason.value))
d.addCallback(util.println)
d.addCallback(lambda _: reactor.stop())
reactor.run()

The code has the MIT license and we hope that other people find it useful.

Read more
beuno

A very healthy and civilised session about switching to Thunderbird by default just ended here in the Ubuntu Developer Summit, and the outcome was that if the Thunderbird developers manage to do some needed work (to be defined) by a certain time in our cycle (to be defined), we will ship Oneiric and more importantly, the 12.04 LTS with Thunderbird by default.

The bits I can remember that need to be done are:
- Evolution data server integration
- Tighter integration with Unity
- Shrink the size of the overall application so it fits in the CD
- A good upgrade story
- Migration plan for Evolution users

We will also make sure it ships with integration with contacts in Ubuntu One, thanks to James Tait's head start with the Hedera project.

I'm a big fan of Thunderbird, so I'll be doing my best to help them achieve their goals  :)

Read more
mandel

At the moment some of the tests (and I cannot point out which ones) of ubuntuone-client fail when they are ran on Windows. The reason for this is due to the way in which we get the notifications out of the file system and the way the tests are written. Before I blame the OS or the tests, let me explain a number of facts about the Windows filesystem and the possible ways to interact with it.

To be able to get file system changes from the OS the Win32 API provides the following:

SHChangedNotifyRegister

This function was broken up to Vista when it was fixed, Unfortunately AFAIK we also support Windows XP which means that we cannot trust this function. On top of that taking this path means that we can have a performance issue. Because the function is build on top of Windows messages, if too many changes occur the sync daemon would start receiving roll up messages that just state that something changed and it would be up to the sync daemon to decide what really happened. Therefore we can all agree that this is a no no, right?

FindFirstChangeNotification

This is a really easy function to use which is based on ReadDirectoryChangesW (I think is a simple wrapper around it) that lets you know that something changed but gives no information about what changed. Because if is based on ReadDirectoryChangesW it suffers from the same issues.

ReadDirectoryChangesW

This is by far the most common way to get the notification changes from the system. Now, in theory there are two possible cases which can go wrong that would affect the events raised by this function:

  1. There are too many events and the buffer gets overloaded and we start loosing events. A simple way to solve this issues is to process the events in a diff thread asap so that we can keep reading the changes.
  2. We use the sync version of the function which means that we could have the following issues:
    • Blue screen of death because we used too much memory from the kernel space.
    • We cannot close the handles used to watch the changes in the directories. This makes the threads to end up blocked.

As I mentioned this is the theory and therefore makes perfect sense to choose this option as the way to get notified by the changes until… you hit a great little feature of Windows called write-behind caching. The idea of write-behind caching is the following one:

When you attempt to write a new file on your HD Windows does not directly modify the HD. Instead it makes a not of the fact that your intention is to write on disk and saves your changes in memory. Ins’t that smart?

Well, that lovely feature does come set as default AFAIK from XP onwards. Any smart person would wonder how does that interact with FindFirstChangeNotification/ReadDirectoryChangesW, well after some work here is what I have managed to find out:

The IO Manager (internal to the kernel) is queueing up disk-write requests in an internal buffer, and the actual changes are not physically committed until some condition is met which I believe is for the “write-behind caching” feature. The problem appears to be that the user-space callback via FileSystemWatcher/ReadDirectoryChanges does not occur when disk-write requests are inserted into the queue, but rather occurs when they are leaving the queue and being physically committed to disk. For what I have been able to manage through observation, the life time of a queue is based on:

  1. Whether more writes are being inserted in the q.
  2. Is another app request a read from an item in the q.

This means that when using FileSystemWatcher/ReadDirectoryChanges the events are fired only when the changes are actually committed and as for a user-space program this follows a non-deterministic process (insert spanish swearing here). a way to work around this issue is to use the FluxhFileBuffers function on the volume, which does need admin rights, yeah!

Change Journal records

Well, this allows to track the changes that have been committed in an NTFS system (that means that we do not have support to FAT). This technique allows to keep track of the changes using an update sequence number that keeps track of changes in an interesting manner. At first look, although parsing the data is hard, this solution seems to be very similar to the one used by pyinotify and therefore someone will say, hey, let just ell twisted to do a select on that file and read the changes. Well, no, is not that easy, files do not provide the functionality used for select, just sockets (http://msdn.microsoft.com/en-us/library/aa363803%28VS.85%29.aspx) /me jumps of happiness

File system filterr

Well, this is an easy one to summarize, you have to write a driver like piece of code. Means C, COM and being able to crash the entire system with a nice blue screen (although I can change the color to aubergine before we crash)

Conclusion

At this point I hope I have convinced a few to believe that ReadDirectoryChangesW is the best option to take but might be wondering why I mentioned the write-behind caching feature, well here comes my complain towards the tests. We do use the real file system notifications for testing and the trial test cases do have a timeout! Those two facts plus the lovely write-behind caching feature mean that the tests on Windows fail just because the bloody evens are not raise until the leave the q from the IO manager.

Read more
beuno

In the last few months, I've been lucky enough to be able to hire some exceptional people that were contributing to Ubuntu One in their free time. Every time someone comes in from the community, filled with excitement about being able to work on their pet project full time my job gets that much better.
So, everyone say hello to James Tait and Micha? Karnicki!

Now we're looking for a new team member to help us make the Ubuntu One website awesome. Someone who knows CSS and HTML inside out, cares deeply about doing things the best way possible and is passionate about their work.

If you're interested or know anyone who may, the job posting is up on Canonical's website.

Read more
David

Ubuntu App Developer Week – Day 3 Summary

Right into the middle of the week and still delivering the most diverse set of sessions from the most interesting technologies. QML, Cloud, D-Bus, Multitouch, Unity, Bazaar… Wednesday had a bit of everything. Most importantly, this sessions are for you all, so I was really glad to hear feedback on how people liked the content of App Developer Week! So here’s a new summary for all of those who couldn’t attend.

Qt Quick: QML the Language

By Jürgen Bocklage-Ryannel

In his first session, Jürgen gave a short intro to Qt Quick’s QML language and how to use it. The first steps were to install Qt and Qt Creator, followed by a description of what Qt Quick is and how developers came up with a declarative way, similar to CSS or JSON to write in the language. All that clear, he then started with the Qt Quick tutorial and code examples that could be run with qmlviewer, the qml interpreter. Onto the second part, he focused on the QML languate, and going into the detail on how to create custom QML components. There were also lots of pointers to the excellent Qt documentation.

Check out the session log here.

Make your applications work in the cloud with Ubuntu One

By Stuart Langridge

Stuart gave a great overview on how to add the cloud to existing apps and how to make new apps for the cloud, letting Ubuntu One do all the hard work for you: from managing identities, password renewal to sharing data between applications. And all that on the web, the desktop, mobile… all your stuff everywhere! He then showed us some simple code to sync playlists on the cloud, ready for streaming. File sync is also an important Ubuntu One feature apps can make use of for sharing, and he also went through a couple of the many cool ways you can use it. The last mention was on API documentation, something Stuart is working on in this cycle.

Check out the session log here.

Take control of your desktop easily with DBus

By Alejandro J. Cura

In this session Alejandro showed us in a hands-on and easy to follow way different bits and pieces of D-Bus, and how applications in the desktop can communicate through it. He went through real life examples to show how to do simple tasks and explained how they can be achieved with D-Bus.

Check out the session log here.

Touchégg: Bringing Multitouch Gestures to your Desktop

In the second multitouch session of the week, app developer José Expósito started showcasing Touchégg, how it works and its features: recognizing multitouch gestures and getting the most of multitouch devices. He then went on describing which gestures it supports, such as tap, drag, pinch or tap & hold, and the different actions that can be associated to gestures, showing us a really cool video of Touchégg in action. The second part of the talk focused on describing the technologies used to develop Touchégg: uTouch-GEIS, through its simplified interface, and Qt.

By José Expósito

Check out the session log here.

Unity: Integrating with Launcher and Places

By Mikkel Kamstrup Erlandsen

Mikkel used the intro of the talk to set a couple of things straight: “Places” are going to be called “Lenses” in the next cycle, and libunity does not yet guarantee API or ABI stability. He then followed with the Unity Launcher integration, and how applications can use static quicklists, and more advanced features such as count, progress bar, window flashing and dynamic quicklists. The second part were Places: remote databases that provide data for Unity to render. Through a Python code example he showed us in detail all the aspects of creating a Unity Place.

Check out the session log here.

Tracking Source Code History with Bazaar

By Jelmer Vernooij

Jelmer, in his experience of seasoned Bazaar hacker started off introducing what bzr is: a modern distributed version control system. He then went on with the basics with a hands-on example, going through the creation of a branch, the first commit, and describing several of the most handy bzr commands. As a wrap-up, he showcased more advanced features such as source recipes: scripts that combine branches and build daily Debian packages from them.

Check out the session log here.

The Day Ahead: Upcoming Sessions for Day 4

We’re featuring a Qt Quick Marathon today: 2 sessions in a row. Following that, how to do RAD with yet another framework: Quickly, how to get your applications in Ubuntu, and how to get them translated in Launchpad. Enjoy!

16:00 UTC
Qt Quick: Elements/Animations/States – Jürgen Bocklage-Ryannel
Another day and more featured Qt content: this time Jürgen will take us through different elements/animations and states Qt Quick provides, and will show us through examples how to make use of them.

17:00 UTC
Qt Quick: Rapid Prototyping – Jürgen Bocklage-Ryannel
If one session weren’t enough, here’s the continuation: more Qt goodness, this time a hands-on session to develop a small application from start to finish and experience the whole process from the front row.

18:00 UTC
Rapid App Development with QuicklyMichael Terry
Mike will show you how to write applications in no time with the power of Python and Quickly: bringing back the fun in programming.

19:00 UTC
Getting Your App in the Distro: the Application Review ProcessAllison Randal
A while back we created an easy process defining how to get applications into Ubuntu, so in order to be able to add them in a matter of weeks, rather than waiting for the next release. Allison, in her Ubuntu Technical Architect and Application Review Board member hat, will walk you through the Application Review Process

20:00 UTC
Adding Indicator Support to your AppsTed Gould
Join the man who knows most about indicators in a session that will teach you how to integrate your application even more into Ubuntu. They’re slick, robust and consistent: bringing indicator support to your apps.

21:00 UTC
Using Launchpad to get your application translatedHenning Eggers
One of the coolest features of Launchpad is that it helps growing a translation community around your project. You can make your application translatable in Launchpad and be able to deliver it into almost any language. Henning will teach you how to do this, picking up where the previous session on translations left.

Looking forward to seeing you all there!


Read more
beuno

I have to confess, after I heard I found out we where shipping Unity in Ubuntu by default I was nervous. I got asked many times what my feelings were, and I think I generally dodged the question. This was a pretty risky move, which we are still a few months away from finding out how well the risk pays off.
Given that a lot of the design behind Unity wasn't done in the open and hadn't had a long time to mature, I've been sceptical of whether we  (as in, the Ubuntu project) could pull of such a massive change in a such a short period of time, and still have happy users.

I've been using Unity on and off on my netbook (which is my secondary computer), but while enjoying a long weekend I've spent the last few days using it a lot, and my feeling towards the it have changed quite a bit.

I think it was the right decision. Overall, it feels like an overall improved experience, even with its current rough edges. Exactly what I think we need to win over a wider audience and have them fall in love with Ubuntu head over heels. Everything is starting to feel much more tightly integrated and with a purpose, as well as some eye-candy sprinkled in a lot of the right places.
I'm really glad Canonical decided to invest to heavily in such a risky and insanely complicated task, Natty is probably one of the most exciting releases I can remember.

There are still a few key challenges ahead, most notably to me is making the design process more open and inclusive, but still being able to deliver something that feels polished and not a pile of consensus between people who have gotten good at arguing. The Ayatana community does seem to be slowly growing, though, so the future looks pretty bright. Getting the right balance between Canonical and a community around design feels like one of the hardest problems to solve, luckily, Canonical continues to hire the brightest and most enthusiastic minds around, so I'm sure it will eventually feel like a solved problem.

I think it's been almost 6 years since I landed in the Ubuntu world, I've done all kinds of things in the community ranging from starting and building the Argentine LoCo to editing the Ubuntu Weekly Newsletter, to evaluating new Ubuntu members in the Americas region. With its ups and down, great press and wild controversies, it still feels like the best place to be.

Read more
mandel

For those that do not what is the keyring module here is the official description:

The Python keyring lib provides a easy way to access the system keyring service from python. It can be used in any application that needs safe password storage.

The module is a very nice idea and has been rather useful during the Ubuntu One port. I just have a problem with it which is the lack of a method to delete a password.

I have forked the project in bitbucket and added the missing methods. Of course I have requested a pull from the original project, so unless there are problems the new code should be ‘landable’ (is landable even a word?) in trunk and usable.

For those that cannot wait for that, you can grab the code by doing:

hg clone https://bitbucket.org/mandel/pykeyring-delete-password

Read more