Canonical Voices

Posts tagged with 'canonical'


During the past few days I have been trying to track down an issue in the Ubuntu One client tests when ran on Windows that would use all the threads that the python process could have. As you can imaging finding out why there are deadlocks is quite hard, specially when I though that the code was thread safe, guess what? it wasn’t

The bug I had in the code was related to the way in which ReadDirectoryChangesW works. This functions has two different ways to be executed:


The ReadDirectoryChangesW can be executed in a sync mode by NOT providing a OVERLAPPED structure to perform the IO operations, for example:

def _watcherThread(self, dn, dh, changes):
        flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME
        while 1:
                print "waiting", dh
                changes = win32file.ReadDirectoryChangesW(dh,
                print "got", changes

The above example has the following two problems:

  • ReadDirectoryChangesW without an OVERLAPPED blocks infinitely.
  • If another thread attempts to close the handle while ReadDirectoryChangesW is waiting on it, the CloseHandle() method blocks (which has nothing to do with the GIL – it is correctly managed)

I got bitten in the ass by the second item which broke my tests in two different ways since it let thread block and a Handle used so that the rest of the tests could not remove the tmp directories that were under used by the block threads.


In other to be able to use the async version of the function we just have to use an OVERLAPPED structure, this way the IO operations will no block and we will also be able to close the handle from a diff thread.

def _watcherThreadOverlapped(self, dn, dh, changes):
        flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME
        buf = win32file.AllocateReadBuffer(8192)
        overlapped = pywintypes.OVERLAPPED()
        overlapped.hEvent = win32event.CreateEvent(None, 0, 0, None)
        while 1:
                                            False, #sub-tree
            # Wait for our event, or for 5 seconds.
            rc = win32event.WaitForSingleObject(overlapped.hEvent, 5000)
            if rc == win32event.WAIT_OBJECT_0:
                # got some data!  Must use GetOverlappedResult to find out
                # how much is valid!  0 generally means the handle has
                # been closed.  Blocking is OK here, as the event has
                # already been set.
                nbytes = win32file.GetOverlappedResult(dh, overlapped, True)
                if nbytes:
                    bits = win32file.FILE_NOTIFY_INFORMATION(buf, nbytes)
                    # This is "normal" exit - our 'tearDown' closes the
                    # handle.
                    # print "looks like dir handle was closed!"
                print "ERROR: Watcher thread timed-out!"
                return # kill the thread!

Using the ReadDirectoryW function in this way does solve all the other issues that are found on the sync version and the only extra overhead added is that you need to understand how to deal with COM events which is not that hard after you have worked with it for a little.

I leave this here for people that might find the same issue and for me to remember how much my ass hurt.


Read more

So far using py2exe has not been walk in the park with several issues so far and this time it could not be different…. The interesting issue brought to me by py2exe this time was during the runtime of a pyqt application in which the following was being print to stderr:

QObject::moveToThread: Current thread (0x21a3410) is not the object's thread (0x19af0d0).

Funny enough that error would not happen if the application was not froze (WTF!). After some help from ralsina and some googling we found the following:

In my case I was pretty scared that the actual issue was related to the that the operation of the QObject was taking place in a twisted deferred callback and that qtreactor might be doing something naughty. At the end it turned out that the issue was related to the use of a jpg image which requires to use an image plugin in Qt… I fixed the issue as per this. Nevertheless I have made the required changes so that such a hack is hidden in the Windows code and that the Qt UI can be used in Kubuntu without dirty code so if everything goes as planned, SSO should have a buggy t UI that runs on Kubuntu.

Read more

A while back I posted an example of how to create a rounded rectangle actor with clutter using the Python bindings. Last night I started to write a small UI in which I’m using Clutter and wanted to get the same effect but this time I used gobject introspection. The differences are not huge but nevertheless here you have the example code of the same exercise using gi.

from gi.repository import Cogl, Clutter, GtkClutter
class RoundedRectangle(GtkClutter.Actor):
    """Base actor for a rounded retangle."""
    def __init__(self, width, height, arc, step, 
                 color=None, border_color=None, border_width=0):
        """Create a new instance."""
        super(RoundedRectangle, self).__init__()
        self._width = width
        self._height = height
        self._arc = arc
        self._step = step
        if color:
            self._color = color
            self._color = Cogl.Color()
            self._color.init_from_4f(0, 0, 0, 1)
        if border_color:
            self._border_color = border_color
            self._border_color = Cogl.Color()
            self._border_color.init_from_4f(0, 0, 0, 1)
        self._border_width = border_width
    def do_paint(self):
        # Draw a rectangle for the clipping
        Cogl.path_round_rectangle(0, 0, self._width, self._height, self._arc, self._step)
        # Start the clip
        # set color to border color
        # draw the rectangle for the border which is the same size and the
        # object
        Cogl.path_round_rectangle(0, 0, self._width, self._height, self._arc, self._step)
        # color the path usign the border color
        # draw the content with is the same size minus the wirth of the border
        # finish the clip
        Cogl.path_round_rectangle(self._border_width, self._border_width, 
        self._width - self._border_width, 
        self._height - self._border_width, self._arc, self._step)
    def do_pick(self, color):
        if not self.should_pick_paint():
        # do pick gets a Clutter color but no a Cogl one, we need to convert it
        color = Cogl.Color()
        color.init_from_4f( , , , color.alpha)
        Cogl.path_round_rectangle(0, 0, self._width, self._height, self._arc, self._step)
        # Start the clip
        # set color to border color
        # draw the rectangle for the border which is the same size and the
        # object
        Cogl.path_round_rectangle(0, 0, self._width, self._height, self._arc, self._step)

I’ve got to say, that although introspection does bring a great opportunity for Python developers to get to the gobject libraries easiyly, it does make the Python code look very un-pynthonic…

Read more

The following bug I have been faced with has made me loose more time that I would have expected and therefore I think is a good idea to describe it so that the rest of the internet can take advantage of my wasted time and also to keep a record of my stupidity.

After building the .exe file of the ubuntu-sso port to windows I was getting the following error at runtime:

Cannot mix incompatible Qt library (version 0x40701) with this library (version 0x40702)

Usually this means that you have two different version of Qt installed are you are mixing the libraries (.dll in this case because I was dealing with Windows). My initial reaction was to look at my Qt setup in the machine and compared the version installed in the system and that used by PyQt. Let me tell you that is a waste of time. The real reason behind this runtime error was the fact that I was using the qtreactor and I had PyQt and PySide installed in my system.

When not freezing the application, the fact that you have both packages installed is not a problem what so ever, but with py2exe it is. Py2exe bundles all the dependencies you app has and due to the fact hat qtreactor does the following:

    from PyQt4.QtCore import QSocketNotifier, QObject, SIGNAL, QTimer, QCoreApplication
    from PyQt4.QtCore import QEventLoop
except ImportError:
    from PySide.QtCore import QSocketNotifier, QObject, SIGNAL, QTimer, QCoreApplication
    from PySide.QtCore import QEventLoop

both, PySide and PyQt were included in the frozen app. The problem arises due to this fact. When py2exe adds both libs, it copies the Qt dlls you depend on, and if PySide and PYQt depend on different versions (which is what was happening in my system) you might run into the issue of getting dlls from different versions because py2exe will override the already copied dlls without telling you.

In summary if you get the above runtime error, take a look to see if PySide and PyQt have ben included in your frozen app and if they depend in different versions of Qt.

Read more

One of the things we wanted to achieve for the Windows port of Ubuntu One was to deploy to the users systems .exe files rather than requiring them to have python and all the different dependencies installed in their machine. There are different reasons we wanted to do this, but this post is not related to that. The goal of this post is to explain what to do when you are using py2exe and you depend on a package such as lazr.restfulclient.

Why lazr.restfulclient?

There are different reasons why I’m using lazr.restfulclient as an example:

  • It is a dependency we do have on Ubuntu One, and therefore I already have done the work with it.
  • It uses two features of setuptools that do not play well with py2exe:
    • It uses namespaced packages.
    • I uses pkg_resources to load resources used for the client.

Working around the use of namedspaced packages

This is actually a fairly easy thing to solve and it is well documented in the py2exe wiki, nevertheless I’d like to show it in this post so that the inclusion of the lazr.restfulclient is complete.

The main issue with namedspaced packages is that you have to tell the module finder from py2exe where to find those packages, which in our example are lazr.authentication, lazr.restfulclient and lazr.uri. A way to do that would be the following:

import lazr
    import as modulefinder
except ImportError:
    import modulefinder
for p in lazr.__path__:
        modulefinder.AddPackagePath(__name__, p)

Adding the lazr resources

This is a more problematic issue to solve since we have to work around a limitation found in py2exe. The lazr.restfulcient tries to load a resource from the py2exe but as the zipfile is reserved for compiled files, and therefore the module fails. In py2exe there is no way to state that those resource files have to be copied to the which would mean that an error is raised at runtime when trying to use the lib but not at build time.

The best way (if not the only one) to solve this is to extend the py2exe command to copy the resource files to the folders that are zipped before they are embedded, that way pkg_resource will be able to load the file with no problems.

import os
import glob
import lazr.restfulclient
from py2exe.build_exe import py2exe as build_exe
class LazrMediaCollector(build_exe):
    """Extension that copies lazr missing data."""
    def copy_extensions(self, extensions):
        """Copy the missing extensions."""
        build_exe.copy_extensions(self, extensions)
        # Create the media subdir where the
        # Python files are collected.
        media = os.path.join('lazr', 'restfulclient')
        full = os.path.join(self.collect_dir, media)
        if not os.path.exists(full):
        # Copy the media files to the collection dir.
        # Also add the copied file to the list of compiled
        # files so it will be included in zipfile.
        for f in glob.glob(lazr.restfulclient.__path__[0] + '/*.txt'):
            name = os.path.basename(f)
            self.copy_file(f, os.path.join(full, name))
            self.compiled_files.append(os.path.join(media, name))

In order to use the above command class to perform the compilation you simply have to tell setup tools which command class to use.

cmdclass = {'py2exe' : LazrMediaCollector}

With the above done, you can use the usual ‘python install py2exe’. Now, the question for the Internet, can this be done with Pyinstaller?

Read more

Sometimes the Moirae (lovely three women, aren’t they?) decide that your project is going to have a complicated live, and this is what I have been facing so far with the port of Ubuntu One to Windows. This means that things that I do not anticipate to go wrong will go wrong. As an example of this is what has currently broken the nightlies of Ubuntu One on any platform (at least we have the same features in all platforms now ;) ). The issue has happened due to some changed that added in Ubuntu SSO Client that would allow to use pykeyring on windows and the COM to detect network changes.

In Ubuntu SSO Client there was an error in the that would have the following trace

ERROR: Python module pythoncom not found
Traceback (most recent call last):
  File "", line 105, in <module>
    'clean' : SSOClean})
  File "/usr/lib/python2.6/dist-packages/DistUtilsExtra/", line 95, in setup
    __requires(attrs, src_all)
  File "/usr/lib/python2.6/dist-packages/DistUtilsExtra/", line 392, in __requires
    __add_imports(imports, s, attrs)
  File "/usr/lib/python2.6/dist-packages/DistUtilsExtra/", line 341, in __add_imports
    if __external_mod(node.module, attrs):
  File "/usr/lib/python2.6/dist-packages/DistUtilsExtra/", line 317, in __external_mod
    path = __import__(module).__file__
  File "/usr/lib/python2.6/ctypes/", line 23, in <module>
    class VARIANT_BOOL(_SimpleCData):
ValueError: _type_ 'v' not supported

Well that is little odd, isn’t it? Why would Distutils-extra have an issue with wintypes, shouldn’t it just return an error to the stderr and leave it like that?. Well interestingly enough, the following returns a ValueError on Linux:

import ctypes.wintypes

He, interesting (I can assure you I was not this polite when I saw the error). So why is distutils extra raising this? Well the main reason resides in the __add_imports method in distutils extra that uses the ast module to find all the modules that you import and tries to import them to see if they are in the system. All of this is wrap by a try statement, but unfortunately the except clause looks for the common exceptions for error hen importing, and ValueError is not one of them. I have sent a patch to disutils-extra to work around this and sent a mail to python-dev asking where is the best place to submit a patch for ctypes…. Who said this project would not help open-source?

Read more

In some cases you might find yourself in the situation of wanting to use gettext in a PyQt project in which you have .ui files generated using QtDesigner.

For those kind of situations is a good idea to extend the uic compiler form PyQt. he following example shows how to do so in a distutils command.

class QtBuild(build_extra.build_extra):
    """Build PyQt (.ui) files and resources."""
    description = "build PyQt GUIs (.ui)."
    def compile_ui(self, ui_file, py_file=None):
        """Compile the .ui files to python modules."""
        # Search for pyuic4 in python bin dir, then in the $Path.
        if py_file is None:
            # go from the ui_file in the data folder to the
            # python file in the qt moodule
            py_file = os.path.split(ui_file)[1]
            py_file = os.path.splitext(py_file)[0] + ''
            py_file = os.path.join('package', 'qt', py_file)
        # we indeed want to catch Exception, is ugle but w need it
        # pylint: disable=W0703
            # import the uic compiler from pyqt and generate the 
            # .py files something similar could be done with pyside
            # but that is left as an exercise for the reader.
            from PyQt4 import uic
            fp = open(py_file, 'w')
            uic.compileUi(ui_file, fp)
  'Compiled %s into %s', ui_file, py_file)
        except Exception, e:
            self.warn('Unable to compile user interface %s: %s',
                           py_file, e)
            if not os.path.exists(py_file) or\
                                            not file(py_file).read():
                raise SystemExit(1)
        # pylint: enable=W0703
    def run(self):
        """Execute the command."""
        basepath = os.path.join('data',  'qt')
        for dirpath, _, filenames in os.walk(basepath):
            for filename in filenames:
                if filename.endswith('.ui'):
                    self.compile_ui(os.path.join(dirpath, filename))
    # pylint: disable=E1002
    _wrappeduic = False
    def _wrapuic(cls):
        """Wrap uic to use gettext's _() in place of tr()"""
        if cls._wrappeduic:
        from PyQt4.uic.Compiler import compiler, qtproxies, indenter
        # pylint: disable=C0103
        class _UICompiler(compiler.UICompiler):
            """Speciallized compiler for qt .ui files."""
            def createToplevelWidget(self, classname, widgetname):
                o = indenter.getIndenter()
                o.level = 0
                o.write('from module.with.gettext.setup import _')
                return super(_UICompiler, self).createToplevelWidget(
                                   classname, widgetname)
        compiler.UICompiler = _UICompiler
        class _i18n_string(qtproxies.i18n_string):
            """Provide a translated text."""
            def __str__(self):
                return "_('%s')" % self.string.encode(
        qtproxies.i18n_string = _i18n_string
        cls._wrappeduic = True
        # pylint: enable=C0103
    # pylint: enable=E1002

The above should be doable with PySide, but that is left as an exercise for the reader.

Read more

For those that do not what is the keyring module here is the official description:

The Python keyring lib provides a easy way to access the system keyring service from python. It can be used in any application that needs safe password storage.

The module is a very nice idea and has been rather useful during the Ubuntu One port. I just have a problem with it which is the lack of a method to delete a password.

I have forked the project in bitbucket and added the missing methods. Of course I have requested a pull from the original project, so unless there are problems the new code should be ‘landable’ (is landable even a word?) in trunk and usable.

For those that cannot wait for that, you can grab the code by doing:

hg clone

Read more

Some of you might know that I’m a rugby nutter. I love watching it, playing it, training… this has had the following consequence:


Unfortunately this will slow me down the following 4 weeks which will affect those people that wanted to see the new code coming to Ubuntu One on Windows. Sorry for that, I hope you can understand :)

Read more

Before I introduce the code, let me say that this is not a 100% exact implementation of the interfaces that can be found in pyinotify but the implementation of a subset that matches my needs. The main idea of creating this post is to give an example of the implementation of such a library for Windows trying to reuse the code that can be found in pyinotify.

Once I have excused my self, let get into the code. First of all, there are a number of classes from pyinotify that we can use in our code. That subset of classes is the below code which I grabbed from pyinotify git:

#!/usr/bin/env python
# - python interface to inotify
# Copyright (c) 2010 Sebastien Martini <>
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
"""Platform agnostic code grabed from pyinotify."""
import logging
import os
class RawOutputFormat:
    Format string representations.
    def __init__(self, format=None):
        self.format = format or {}
    def simple(self, s, attribute):
        if not isinstance(s, str):
            s = str(s)
        return (self.format.get(attribute, '') + s +
                self.format.get('normal', ''))
    def punctuation(self, s):
        """Punctuation color."""
        return self.simple(s, 'normal')
    def field_value(self, s):
        """Field value color."""
        return self.simple(s, 'purple')
    def field_name(self, s):
        """Field name color."""
        return self.simple(s, 'blue')
    def class_name(self, s):
        """Class name color."""
        return self.format.get('red', '') + self.simple(s, 'bold')
output_format = RawOutputFormat()
class EventsCodes:
    Set of codes corresponding to each kind of events.
    Some of these flags are used to communicate with inotify, whereas
    the others are sent to userspace by inotify notifying some events.
    @cvar IN_ACCESS: File was accessed.
    @type IN_ACCESS: int
    @cvar IN_MODIFY: File was modified.
    @type IN_MODIFY: int
    @cvar IN_ATTRIB: Metadata changed.
    @type IN_ATTRIB: int
    @cvar IN_CLOSE_WRITE: Writtable file was closed.
    @type IN_CLOSE_WRITE: int
    @cvar IN_CLOSE_NOWRITE: Unwrittable file closed.
    @type IN_CLOSE_NOWRITE: int
    @cvar IN_OPEN: File was opened.
    @type IN_OPEN: int
    @cvar IN_MOVED_FROM: File was moved from X.
    @type IN_MOVED_FROM: int
    @cvar IN_MOVED_TO: File was moved to Y.
    @type IN_MOVED_TO: int
    @cvar IN_CREATE: Subfile was created.
    @type IN_CREATE: int
    @cvar IN_DELETE: Subfile was deleted.
    @type IN_DELETE: int
    @cvar IN_DELETE_SELF: Self (watched item itself) was deleted.
    @type IN_DELETE_SELF: int
    @cvar IN_MOVE_SELF: Self (watched item itself) was moved.
    @type IN_MOVE_SELF: int
    @cvar IN_UNMOUNT: Backing fs was unmounted.
    @type IN_UNMOUNT: int
    @cvar IN_Q_OVERFLOW: Event queued overflowed.
    @type IN_Q_OVERFLOW: int
    @cvar IN_IGNORED: File was ignored.
    @type IN_IGNORED: int
    @cvar IN_ONLYDIR: only watch the path if it is a directory (new
                      in kernel 2.6.15).
    @type IN_ONLYDIR: int
    @cvar IN_DONT_FOLLOW: don't follow a symlink (new in kernel 2.6.15).
                          IN_ONLYDIR we can make sure that we don't watch
                          the target of symlinks.
    @type IN_DONT_FOLLOW: int
    @cvar IN_MASK_ADD: add to the mask of an already existing watch (new
                       in kernel 2.6.14).
    @type IN_MASK_ADD: int
    @cvar IN_ISDIR: Event occurred against dir.
    @type IN_ISDIR: int
    @cvar IN_ONESHOT: Only send event once.
    @type IN_ONESHOT: int
    @cvar ALL_EVENTS: Alias for considering all of the events.
    @type ALL_EVENTS: int
    # The idea here is 'configuration-as-code' - this way, we get
    # our nice class constants, but we also get nice human-friendly text
    # mappings to do lookups against as well, for free:
        'IN_ACCESS'        : 0x00000001,  # File was accessed
        'IN_MODIFY'        : 0x00000002,  # File was modified
        'IN_ATTRIB'        : 0x00000004,  # Metadata changed
        'IN_CLOSE_WRITE'   : 0x00000008,  # Writable file was closed
        'IN_CLOSE_NOWRITE' : 0x00000010,  # Unwritable file closed
        'IN_OPEN'          : 0x00000020,  # File was opened
        'IN_MOVED_FROM'    : 0x00000040,  # File was moved from X
        'IN_MOVED_TO'      : 0x00000080,  # File was moved to Y
        'IN_CREATE'        : 0x00000100,  # Subfile was created
        'IN_DELETE'        : 0x00000200,  # Subfile was deleted
        'IN_DELETE_SELF'   : 0x00000400,  # Self (watched item itself)
                                          # was deleted
        'IN_MOVE_SELF'     : 0x00000800,  # Self(watched item itself) was moved
                        'EVENT_FLAGS': {
        'IN_UNMOUNT'       : 0x00002000,  # Backing fs was unmounted
        'IN_Q_OVERFLOW'    : 0x00004000,  # Event queued overflowed
        'IN_IGNORED'       : 0x00008000,  # File was ignored
                        'SPECIAL_FLAGS': {
        'IN_ONLYDIR'       : 0x01000000,  # only watch the path if it is a
                                          # directory
        'IN_DONT_FOLLOW'   : 0x02000000,  # don't follow a symlink
        'IN_MASK_ADD'      : 0x20000000,  # add to the mask of an already
                                          # existing watch
        'IN_ISDIR'         : 0x40000000,  # event occurred against dir
        'IN_ONESHOT'       : 0x80000000,  # only send event once
    def maskname(mask):
        Returns the event name associated to mask. IN_ISDIR is appended to
        the result when appropriate. Note: only one event is returned, because
        only one event can be raised at a given time.
        @param mask: mask.
        @type mask: int
        @return: event name.
        @rtype: str
        ms = mask
        name = '%s'
        if mask & IN_ISDIR:
            ms = mask - IN_ISDIR
            name = '%s|IN_ISDIR'
        return name % EventsCodes.ALL_VALUES[ms]
    maskname = staticmethod(maskname)
# So let's now turn the configuration into code
EventsCodes.ALL_FLAGS = {}
EventsCodes.ALL_VALUES = {}
for flagc, valc in EventsCodes.FLAG_COLLECTIONS.items():
    # Make the collections' members directly accessible through the
    # class dictionary
    setattr(EventsCodes, flagc, valc)
    # Collect all the flags under a common umbrella
    # Make the individual masks accessible as 'constants' at globals() scope
    # and masknames accessible by values.
    for name, val in valc.items():
        globals()[name] = val
        EventsCodes.ALL_VALUES[val] = name
# all 'normal' events
ALL_EVENTS = reduce(lambda x, y: x | y, EventsCodes.OP_FLAGS.values())
class _Event:
    Event structure, represent events raised by the system. This
    is the base class and should be subclassed.
    def __init__(self, dict_):
        Attach attributes (contained in dict_) to self.
        @param dict_: Set of attributes.
        @type dict_: dictionary
        for tpl in dict_.items():
            setattr(self, *tpl)
    def __repr__(self):
        @return: Generic event string representation.
        @rtype: str
        s = ''
        for attr, value in sorted(self.__dict__.items(), key=lambda x: x[0]):
            if attr.startswith('_'):
            if attr == 'mask':
                value = hex(getattr(self, attr))
            elif isinstance(value, basestring) and not value:
                value = "''"
            s += ' %s%s%s' % (output_format.field_name(attr),
        s = '%s%s%s %s' % (output_format.punctuation('<'),
        return s
    def __str__(self):
        return repr(self)
class _RawEvent(_Event):
    Raw event, it contains only the informations provided by the system.
    It doesn't infer anything.
    def __init__(self, wd, mask, cookie, name):
        @param wd: Watch Descriptor.
        @type wd: int
        @param mask: Bitmask of events.
        @type mask: int
        @param cookie: Cookie.
        @type cookie: int
        @param name: Basename of the file or directory against which the
                     event was raised in case where the watched directory
                     is the parent directory. None if the event was raised
                     on the watched item itself.
        @type name: string or None
        # Use this variable to cache the result of str(self), this object
        # is immutable.
        self._str = None
        # name: remove trailing '\0'
        d = {'wd': wd,
             'mask': mask,
             'cookie': cookie,
             'name': name.rstrip('\0')}
        _Event.__init__(self, d)
    def __str__(self):
        if self._str is None:
            self._str = _Event.__str__(self)
        return self._str
class Event(_Event):
    This class contains all the useful informations about the observed
    event. However, the presence of each field is not guaranteed and
    depends on the type of event. In effect, some fields are irrelevant
    for some kind of event (for example 'cookie' is meaningless for
    IN_CREATE whereas it is mandatory for IN_MOVE_TO).
    The possible fields are:
      - wd (int): Watch Descriptor.
      - mask (int): Mask.
      - maskname (str): Readable event name.
      - path (str): path of the file or directory being watched.
      - name (str): Basename of the file or directory against which the
              event was raised in case where the watched directory
              is the parent directory. None if the event was raised
              on the watched item itself. This field is always provided
              even if the string is ''.
      - pathname (str): Concatenation of 'path' and 'name'.
      - src_pathname (str): Only present for IN_MOVED_TO events and only in
              the case where IN_MOVED_FROM events are watched too. Holds the
              source pathname from where pathname was moved from.
      - cookie (int): Cookie.
      - dir (bool): True if the event was raised against a directory.
    def __init__(self, raw):
        Concretely, this is the raw event plus inferred infos.
        _Event.__init__(self, raw)
        self.maskname = EventsCodes.maskname(self.mask)
            self.event_name = self.maskname
                self.pathname = os.path.abspath(os.path.join(self.path,
                self.pathname = os.path.abspath(self.path)
        except AttributeError, err:
            # Usually it is not an error some events are perfectly valids
            # despite the lack of these attributes.
class _ProcessEvent:
    Abstract processing event class.
    def __call__(self, event):
        To behave like a functor the object must be callable.
        This method is a dispatch method. Its lookup order is:
          1. process_MASKNAME method
          2. process_FAMILY_NAME method
          3. otherwise calls process_default
        @param event: Event to be processed.
        @type event: Event object
        @return: By convention when used from the ProcessEvent class:
                 - Returning False or None (default value) means keep on
                 executing next chained functors (see example).
                 - Returning True instead means do not execute next
                   processing functions.
        @rtype: bool
        @raise ProcessEventError: Event object undispatchable,
                                  unknown event.
        stripped_mask = event.mask - (event.mask & IN_ISDIR)
        maskname = EventsCodes.ALL_VALUES.get(stripped_mask)
        if maskname is None:
            raise ProcessEventError("Unknown mask 0x%08x" % stripped_mask)
        # 1- look for process_MASKNAME
        meth = getattr(self, 'process_' + maskname, None)
        if meth is not None:
            return meth(event)
        # 2- look for process_FAMILY_NAME
        meth = getattr(self, 'process_IN_' + maskname.split('_')[1], None)
        if meth is not None:
            return meth(event)
        # 3- default call method process_default
        return self.process_default(event)
    def __repr__(self):
        return '<%s>' % self.__class__.__name__
class ProcessEvent(_ProcessEvent):
    Process events objects, can be specialized via subclassing, thus its
    behavior can be overriden:
    Note: you should not override __init__ in your subclass instead define
    a my_init() method, this method will be called automatically from the
    constructor of this class with its optionals parameters.
      1. Provide specialized individual methods, e.g. process_IN_DELETE for
         processing a precise type of event (e.g. IN_DELETE in this case).
      2. Or/and provide methods for processing events by 'family', e.g.
         process_IN_CLOSE method will process both IN_CLOSE_WRITE and
         IN_CLOSE_NOWRITE events (if process_IN_CLOSE_WRITE and
         process_IN_CLOSE_NOWRITE aren't defined though).
      3. Or/and override process_default for catching and processing all
         the remaining types of events.
    pevent = None
    def __init__(self, pevent=None, **kargs):
        Enable chaining of ProcessEvent instances.
        @param pevent: Optional callable object, will be called on event
                       processing (before self).
        @type pevent: callable
        @param kargs: This constructor is implemented as a template method
                      delegating its optionals keyworded arguments to the
                      method my_init().
        @type kargs: dict
        self.pevent = pevent
    def my_init(self, **kargs):
        This method is called from ProcessEvent.__init__(). This method is
        empty here and must be redefined to be useful. In effect, if you
        need to specifically initialize your subclass' instance then you
        just have to override this method in your subclass. Then all the
        keyworded arguments passed to ProcessEvent.__init__() will be
        transmitted as parameters to this method. Beware you MUST pass
        keyword arguments though.
        @param kargs: optional delegated arguments from __init__().
        @type kargs: dict
    def __call__(self, event):
        stop_chaining = False
        if self.pevent is not None:
            # By default methods return None so we set as guideline
            # that methods asking for stop chaining must explicitely
            # return non None or non False values, otherwise the default
            # behavior will be to accept chain call to the corresponding
            # local method.
            stop_chaining = self.pevent(event)
        if not stop_chaining:
            return _ProcessEvent.__call__(self, event)
    def nested_pevent(self):
        return self.pevent
    def process_IN_Q_OVERFLOW(self, event):
        By default this method only reports warning messages, you can
        overredide it by subclassing ProcessEvent and implement your own
        process_IN_Q_OVERFLOW method. The actions you can take on receiving
        this event is either to update the variable max_queued_events in order
        to handle more simultaneous events or to modify your code in order to
        accomplish a better filtering diminishing the number of raised events.
        Because this method is defined, IN_Q_OVERFLOW will never get
        transmitted as arguments to process_default calls.
        @param event: IN_Q_OVERFLOW event.
        @type event: dict
        log.warning('Event queue overflowed.')
    def process_default(self, event):
        Default processing event method. By default does nothing. Subclass
        ProcessEvent and redefine this method in order to modify its behavior.
        @param event: Event to be processed. Can be of any type of events but
                      IN_Q_OVERFLOW events (see method process_IN_Q_OVERFLOW).
        @type event: Event instance
class PrintAllEvents(ProcessEvent):
    Dummy class used to print events strings representations. For instance this
    class is used from command line to print all received events to stdout.
    def my_init(self, out=None):
        @param out: Where events will be written.
        @type out: Object providing a valid file object interface.
        if out is None:
            out = sys.stdout
        self._out = out
    def process_default(self, event):
        Writes event string representation to file object provided to
        @param event: Event to be processed. Can be of any type of events but
                      IN_Q_OVERFLOW events (see method process_IN_Q_OVERFLOW).
        @type event: Event instance
class WatchManagerError(Exception):
    WatchManager Exception. Raised on error encountered on watches
    def __init__(self, msg, wmd):
        @param msg: Exception string's description.
        @type msg: string
        @param wmd: This dictionary contains the wd assigned to paths of the
                    same call for which watches were successfully added.
        @type wmd: dict
        self.wmd = wmd
        Exception.__init__(self, msg)

Unfortunatly we need to implement the code that talks with the Win32 API to be able to retrieve the events in the file system. In my design this is done by the Watch class that looks like this:

# Author: Manuel de la Pena <>
# Copyright 2011 Canonical Ltd.
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License version 3, as published
# by the Free Software Foundation.
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranties of
# PURPOSE.  See the GNU General Public License for more details.
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <>.
"""File notifications on windows."""
import logging
import os
import re
import winerror
from Queue import Queue, Empty
from threading import Thread
from uuid import uuid4
from twisted.internet import task, reactor
from win32con import (
from win32file import CreateFile, ReadDirectoryChangesW
from import (
from ubuntuone.syncdaemon.filesystem_notifications import (
from import (
# constant found in the msdn documentation:
# a map between the few events that we have on windows and those
# found in pyinotify
# translates quickly the event and it's is_dir state to our standard events
# the default mask to be used in the watches added by the FilesystemMonitor
# class
# The implementation of the code that is provided as the pyinotify
# substitute
class Watch(object):
    """Implement the same functions as pyinotify.Watch."""
    def __init__(self, watch_descriptor, path, mask, auto_add,
        events_queue=None, exclude_filter=None, proc_fun=None):
        super(Watch, self).__init__()
        self.log = logging.getLogger('' +
        self._watching = False
        self._descriptor = watch_descriptor
        self._auto_add = auto_add
        self.exclude_filter = None
        self._proc_fun = proc_fun
        self._cookie = None
        self._source_pathname = None
        # remember the subdirs we have so that when we have a delete we can
        # check if it was a remove
        self._subdirs = []
        # ensure that we work with an abspath and that we can deal with
        # long paths over 260 chars.
        self._path = os.path.abspath(path)
        if not self._path.startswith(LONG_PATH_PREFIX):
            self._path = LONG_PATH_PREFIX + self._path
        self._mask = mask
        # lets make the q as big as possible
        self._raw_events_queue = Queue()
        if not events_queue:
            events_queue = Queue()
        self.events_queue = events_queue
    def _path_is_dir(self, path):
        """"Check if the path is a dir and update the local subdir list."""
        self.log.debug('Testing if path "%s" is a dir', path)
        is_dir = False
        if os.path.exists(path):
            is_dir = os.path.isdir(path)
            self.log.debug('Path "%s" was deleted subdirs are %s.',
                path, self._subdirs)
            # we removed the path, we look in the internal list
            if path in self._subdirs:
                is_dir = True
        if is_dir:
            self.log.debug('Adding %s to subdirs %s', path, self._subdirs)
        return is_dir
    def _process_events(self):
        """Process the events form the queue."""
        # we transform the events to be the same as the one in pyinotify
        # and then use the proc_fun
        while self._watching or not self._raw_events_queue.empty():
            file_name, action = self._raw_events_queue.get()
            # map the windows events to the pyinotify ones, tis is dirty but
            # makes the multiplatform better, linux was first :P
            is_dir = self._path_is_dir(file_name)
            if os.path.exists(file_name):
                is_dir = os.path.isdir(file_name)
                # we removed the path, we look in the internal list
                if file_name in self._subdirs:
                    is_dir = True
            if is_dir:
            mask = WINDOWS_ACTIONS[action]
            head, tail = os.path.split(file_name)
            if is_dir:
                mask |= IN_ISDIR
            event_raw_data = {
                'wd': self._descriptor,
                'dir': is_dir,
                'mask': mask,
                'name': tail,
                'path': head.replace(self.path, '.')
            # by the way in which the win api fires the events we know for
            # sure that no move events will be added in the wrong order, this
            # is kind of hacky, I dont like it too much
            if WINDOWS_ACTIONS[action] == IN_MOVED_FROM:
                self._cookie = str(uuid4())
                self._source_pathname = tail
                event_raw_data['cookie'] = self._cookie
            if WINDOWS_ACTIONS[action] == IN_MOVED_TO:
                event_raw_data['src_pathname'] = self._source_pathname
                event_raw_data['cookie'] = self._cookie
            event = Event(event_raw_data)
            # FIXME: event deduces the pathname wrong and we need manually
            # set it
            event.pathname = file_name
            # add the event only if we do not have an exclude filter or
            # the exclude filter returns False, that is, the event will not
            # be excluded
            if not self.exclude_filter or not self.exclude_filter(event):
                self.log.debug('Addding event %s to queue.', event)
    def _watch(self):
        """Watch a path that is a directory."""
        # we are going to be using the ReadDirectoryChangesW whihc requires
        # a direcotry handle and the mask to be used.
        handle = CreateFile(
        self.log.debug('Watchng path %s.', self._path)
        while self._watching:
            # important information to know about the parameters:
            # param 1: the handle to the dir
            # param 2: the size to be used in the kernel to store events
            # that might be lost whilw the call is being performed. This
            # is complicates to fine tune since if you make lots of watcher
            # you migh used to much memory and make your OS to BSOD
            results = ReadDirectoryChangesW(
            # add the diff events to the q so that the can be processed no
            # matter the speed.
            for action, file in results:
                full_filename = os.path.join(self._path, file)
                self._raw_events_queue.put((full_filename, action))
                self.log.debug('Added %s to raw events queue.',
                    (full_filename, action))
    def start_watching(self):
        """Tell the watch to start processing events."""
        # get the diff dirs in the path
        for current_child in listdir(self._path):
            full_child_path = os.path.join(self._path, current_child)
            if os.path.isdir(full_child_path):
        # start to diff threads, one to watch the path, the other to
        # process the events.
        self.log.debug('Sart watching path.')
        self._watching = True
        watch_thread = Thread(target=self._watch,
            name='Watch(%s)' % self._path)
        process_thread = Thread(target=self._process_events,
            name='Process(%s)' % self._path)
    def stop_watching(self):
        """Tell the watch to stop processing events."""
        self._watching = False
        self._subdirs = []
    def update(self, mask, proc_fun=None, auto_add=False):
        """Update the info used by the watcher."""
        self.log.debug('update(%s, %s, %s)', mask, proc_fun, auto_add)
        self._mask = mask
        self._proc_fun = proc_fun
        self._auto_add = auto_add
    def path(self):
        """Return the patch watched."""
        return self._path
    def auto_add(self):
        return self._auto_add
    def proc_fun(self):
        return self._proc_fun
class WatchManager(object):
    """Implement the same functions as pyinotify.WatchManager."""
    def __init__(self, exclude_filter=lambda path: False):
        """Init the manager to keep trak of the different watches."""
        super(WatchManager, self).__init__()
        self.log = logging.getLogger(''
            + 'filesystem_notifications.WatchManager')
        self._wdm = {}
        self._wd_count = 0
        self._exclude_filter = exclude_filter
        self._events_queue = Queue()
        self._ignored_paths = []
    def stop(self):
        """Close the manager and stop all watches."""
        self.log.debug('Stopping watches.')
        for current_wd in self._wdm:
            self.log.debug('Watch for %s stopped.', self._wdm[current_wd].path)
    def get_watch(self, wd):
        """Return the watch with the given descriptor."""
        return self._wdm[wd]
    def del_watch(self, wd):
        """Delete the watch with the given descriptor."""
            watch = self._wdm[wd]
            del self._wdm[wd]
            self.log.debug('Watch %s removed.', wd)
        except KeyError, e:
    def _add_single_watch(self, path, mask, proc_fun=None, auto_add=False,
        quiet=True, exclude_filter=None):
        self.log.debug('add_single_watch(%s, %s, %s, %s, %s, %s)', path, mask,
            proc_fun, auto_add, quiet, exclude_filter)
        self._wdm[self._wd_count] = Watch(self._wd_count, path, mask,
            auto_add, events_queue=self._events_queue,
            exclude_filter=exclude_filter, proc_fun=proc_fun)
        self._wd_count += 1
        self.log.debug('Watch count increased to %s', self._wd_count)
    def add_watch(self, path, mask, proc_fun=None, auto_add=False,
        quiet=True, exclude_filter=None):
        if hasattr(path, '__iter__'):
            self.log.debug('Added collection of watches.')
            # we are dealing with a collection of paths
            for current_path in path:
                if not self.get_wd(current_path):
                    self._add_single_watch(current_path, mask, proc_fun,
                        auto_add, quiet, exclude_filter)
        elif not self.get_wd(path):
            self.log.debug('Adding single watch.')
            self._add_single_watch(path, mask, proc_fun, auto_add,
                quiet, exclude_filter)
    def update_watch(self, wd, mask=None, proc_fun=None, rec=False,
                     auto_add=False, quiet=True):
            watch = self._wdm[wd]
            self.log.debug('Stopped watch on %s for update.', watch.path)
            # update the data and restart watching
            auto_add = auto_add or rec
            watch.update(mask, proc_fun=proc_fun, auto_add=auto_add)
            # only start the watcher again if the mask was given, otherwhise
            # we are not watchng and therefore do not care
            if mask:
        except KeyError, e:
            if not quiet:
                raise WatchManagerError('Watch %s was not found' % wd, {})
    def get_wd(self, path):
        """Return the watcher that is used to watch the given path."""
        for current_wd in self._wdm:
            if self._wdm[current_wd].path in path and \
                return current_wd
    def get_path(self, wd):
        """Return the path watched by the wath with the given wd."""
        watch_ = self._wmd.get(wd)
        if watch:
            return watch.path
    def rm_watch(self, wd, rec=False, quiet=True):
        """Remove the the watch with the given wd."""
            watch = self._wdm[wd]
            del self._wdm[wd]
        except KeyrError, err:
            if not quiet:
                raise WatchManagerError('Watch %s was not found' % wd, {})
    def rm_path(self, path):
        """Remove a watch to the given path."""
        # it would be very tricky to remove a subpath from a watcher that is
        # looking at changes in ther kids. To make it simpler and less error
        # prone (and even better performant since we use less threads) we will
        # add a filter to the events in the watcher so that the events from
        # that child are not received :)
        def ignore_path(event):
            """Ignore an event if it has a given path."""
            is_ignored = False
            for ignored_path in self._ignored_paths:
                if ignore_path in event.pathname:
                    return True
            return False
        wd = self.get_wd(path)
        if wd:
            if self._wdm[wd].path == path:
                self.log.debug('Removing watch for path "%s"', path)
                self.log.debug('Adding exclude filter for "%s"', path)
                # we have a watch that cotains the path as a child path
                if not path in self._ignored_paths:
                # FIXME: This assumes that we do not have other function
                # which in our usecase is correct, but what is we move this
                # to other projects evet?!? Maybe using the manager
                # exclude_filter is better
                if not self._wdm[wd].exclude_filter:
                    self._wdm[wd].exclude_filter = ignore_path
    def watches(self):
        """Return a reference to the dictionary that contains the watches."""
        return self._wdm
    def events_queue(self):
        """Return the queue with the events that the manager contains."""
        return self._events_queue
class Notifier(object):
    Read notifications, process events. Inspired by the pyinotify.Notifier
    def __init__(self, watch_manager, default_proc_fun=None, read_freq=0,
                 threshold=10, timeout=-1):
        """Init to process event according to the given timeout & threshold."""
        super(Notifier, self).__init__()
        self.log = logging.getLogger(''
            + 'filesystem_notifications.Notifier')
        # Watch Manager instance
        self._watch_manager = watch_manager
        # Default processing method
        self._default_proc_fun = default_proc_fun
        if default_proc_fun is None:
            self._default_proc_fun = PrintAllEvents()
        # Loop parameters
        self._read_freq = read_freq
        self._threshold = threshold
        self._timeout = timeout
    def proc_fun(self):
        return self._default_proc_fun
    def process_events(self):
        Process the event given the threshold and the timeout.
        self.log.debug('Processing events with threashold: %s and timeout: %s',
            self._threshold, self._timeout)
        # we will process an amount of events equal to the threshold of
        # the notifier and will block for the amount given by the timeout
        processed_events = 0
        while processed_events < self._threshold:
                raw_event = None
                if not self._timeout or self._timeout < 0:
                    raw_event = self._watch_manager.events_queue.get(
                    raw_event = self._watch_manager.events_queue.get(
                watch = self._watch_manager.get_watch(raw_event.wd)
                if watch is None:
                    # Not really sure how we ended up here, nor how we should
                    # handle these types of events and if it is appropriate to
                    # completly skip them (like we are doing here).
                    self.log.warning('Unable to retrieve Watch object '
                        + 'associated to %s', raw_event)
                    processed_events += 1
                if watch and watch.proc_fun:
                    self.log.debug('Executing proc_fun from watch.')
                    watch.proc_fun(raw_event)  # user processings
                    self.log.debug('Executing default_proc_fun')
                processed_events += 1
            except Empty:
                # increase the number of processed events, and continue
                processed_events += 1
    def stop(self):
        """Stop processing events and the watch manager."""

While one of the threads is retrieving the events from the file system, the second one process them so that the will be exposed as pyinotify events. I have done so because I did not want to deal with OVERLAP structures for asyn operations in Win32 and because I wanted to use pyinotify events so that if someone with experience in pyinotify looks at the output, he can easily understand it. I really like this approach because it allowed me to reuse a fair amount of logic hat we had in the Ubuntu client and to approach the port in a very TDD way since the tests I’ve used are the same ones as the ones found on Ubuntu :)

Read more

Yet again Windows has presented me a challenge when trying to work with its file system, this time in the form of lock files. The Ubuntu One client on linux uses pyinotify to be able to listen to the file system events this, for example, allows the daemon to be updating your files when a new version has been created without the direct intervention of the user.

Although Windows does not have pyinotify (for obvious reasons) a developer that wants to perform such a directory monitoring can rely on the ReadDirectoryChangesW function. This function provides a similar behavior but unfortunately the information it provides is limited when compared with the one from pyinotify. On one hand, there are less events you can listen on Windows (IN_OPEN and IN_CLOSE for example are not present) but it also provides very little information by just giving 5 actions back, that is while on Windows you can listen to:


You will only get back 5 values which are integers that represent the action that was performed. YesterdayI decide to see if it was possible to query the Windows Object Manager to see the currently used FILE HANDLES which would returned the open files. My idea was to write such a function and the pool (ouch!) to find when a file was opened or close. The result of such an attempt is the following:

import os
import struct
import winerror
import win32file
import win32con
from ctypes import *
from ctypes.wintypes import *
from Queue import Queue
from threading import Thread
from win32api import GetCurrentProcess, OpenProcess, DuplicateHandle
from win32api import error as ApiError
from win32con import (
from win32event import WaitForSingleObject, WAIT_TIMEOUT, WAIT_ABANDONED
from win32event import error as EventError
from win32file import CreateFile, ReadDirectoryChangesW, CloseHandle
from win32file import error as FileError
# from import LONG_PATH_PREFIX, abspath
LONG_PATH_PREFIX = '\\\\?\\'
# constant found in the msdn documentation:
# XXX: the following code is some kind of hack that allows to get the opened
# files in a system. The techinique uses an no documented API from windows nt
# that is internal to MS and might change in the future braking our code :(
UCHAR = c_ubyte
PVOID = c_void_p
ntdll = windll.ntdll
SystemHandleInformation = 16
CURRENT_PROCESS = GetCurrentProcess ()
for d in "abcdefghijklmnopqrstuvwxyz":
        DEVICE_DRIVES[win32file.QueryDosDevice (d + ":").strip ("\x00").lower ()] = d + ":"
    except FileError, (errno, errctx, errmsg):
        if errno == 2:
class x_file_handles(Exception):
def signed_to_unsigned(signed):
    unsigned, = struct.unpack ("L", struct.pack ("l", signed))
    return unsigned
    """Represent the SYSTEM_HANDLE_TABLE_ENTRY_INFO on ntdll."""
    _fields_ = [
        ("UniqueProcessId", USHORT),
        ("CreatorBackTraceIndex", USHORT),
        ("ObjectTypeIndex", UCHAR),
        ("HandleAttributes", UCHAR),
        ("HandleValue", USHORT),
        ("Object", PVOID),
        ("GrantedAccess", ULONG),
    """Represent the SYSTEM_HANDLE_INFORMATION on ntdll."""
    _fields_ = [
        ("NumberOfHandles", ULONG),
        ("Handles", SYSTEM_HANDLE_TABLE_ENTRY_INFO * 1),
class LSA_UNICODE_STRING(Structure):
    """Represent the LSA_UNICODE_STRING on ntdll."""
    _fields_ = [
        ("Length", USHORT),
        ("MaximumLength", USHORT),
        ("Buffer", LPWSTR),
    """Represent the PUBLIC_OBJECT_TYPE_INFORMATION on ntdll."""
    _fields_ = [
        ("Name", LSA_UNICODE_STRING),
        ("Reserved", ULONG * 22),
    """Represent the OBJECT_NAME_INFORMATION on ntdll."""
    _fields_ = [
        ("Name", LSA_UNICODE_STRING),
    """Represent the IO_STATUS_BLOCK_UNION on ntdll."""
    _fields_ = [
        ("Status", LONG),
        ("Pointer", PVOID),
class IO_STATUS_BLOCK (Structure):
    """Represent the IO_STATUS_BLOCK on ntdll."""
    _anonymous_ = ("u",)
    _fields_ = [
        ("u", IO_STATUS_BLOCK_UNION),
        ("Information", POINTER (ULONG)),
class FILE_NAME_INFORMATION (Structure):
    """Represent the on FILE_NAME_INFORMATION ntdll."""
    filename_size = 4096
    _fields_ = [
        ("FilenameLength", ULONG),
        ("FileName", WCHAR * filename_size),
def get_handles():
    """Return all the processes handles in the system atm."""
    system_handle_information = SYSTEM_HANDLE_INFORMATION()
    size = DWORD (sizeof (system_handle_information))
    while True:
        result = ntdll.NtQuerySystemInformation(
        result = signed_to_unsigned(result)
        if result == STATUS_SUCCESS:
        elif result == STATUS_INFO_LENGTH_MISMATCH:
            size = DWORD(size.value * 4)
            resize(system_handle_information, size.value)
            raise x_file_handles("NtQuerySystemInformation", hex(result))
    pHandles = cast(
    for handle in pHandles.contents:
        yield handle.UniqueProcessId, handle.HandleValue
def get_process_handle (pid, handle):
    """Get a handle for the process with the given pid."""
        hProcess = OpenProcess(win32con.PROCESS_DUP_HANDLE, 0, pid)
        return DuplicateHandle(hProcess, handle, CURRENT_PROCESS,
            0, 0, DUPLICATE_SAME_ACCESS)
    except ApiError,(errno, errctx, errmsg):
        if errno in (
            return None
def get_type_info (handle):
    """Get the handle type information."""
    public_object_type_information = PUBLIC_OBJECT_TYPE_INFORMATION()
    size = DWORD(sizeof(public_object_type_information))
    while True:
        result = signed_to_unsigned(
                handle, 2, byref(public_object_type_information), size, None))
        if result == STATUS_SUCCESS:
            return public_object_type_information.Name.Buffer
        elif result == STATUS_INFO_LENGTH_MISMATCH:
            size = DWORD(size.value * 4)
            resize(public_object_type_information, size.value)
        elif result == STATUS_INVALID_HANDLE:
            return None
            raise x_file_handles("NtQueryObject.2", hex (result))
def get_name_info (handle):
    """Get the handle name information."""
    object_name_information = OBJECT_NAME_INFORMATION()
    size = DWORD(sizeof(object_name_information))
    while True:
        result = signed_to_unsigned(
            ntdll.NtQueryObject(handle, 1, byref (object_name_information),
            size, None))
        if result == STATUS_SUCCESS:
            return object_name_information.Name.Buffer
            size = DWORD(size.value * 4)
            resize (object_name_information, size.value)
            return None
def filepath_from_devicepath (devicepath):
    """Return a file path from a device path."""
    if devicepath is None:
        return None
    devicepath = devicepath.lower()
    for device, drive in DEVICE_DRIVES.items():
        if devicepath.startswith(device):
            return drive + devicepath[len(device):]
        return devicepath
def get_real_path(path):
    """Return the real path avoiding issues with the Library a in Windows 7"""
    assert os.path.isdir(path)
    handle = CreateFile(
    name = get_name_info(int(handle))
    return filepath_from_devicepath(name)
def get_open_file_handles():
    """Return all the open file handles."""
    print 'get_open_file_handles'
    result = set()
    this_pid = os.getpid()
    for pid, handle in get_handles():
        if pid == this_pid:
        duplicate = get_process_handle(pid, handle)
        if duplicate is None:
            # get the type info and name info of the handle
            type = get_type_info(handle)
            name = get_name_info(handle)
            # add the handle to the result only if it is a file
            if type and type == 'File':
                # the name info represents the path to the object,
                # we need to convert it to a file path and then
                # test that it does exist
                if name:
                    file_path = filepath_from_devicepath(name)
                    if os.path.exists(file_path):
    return result
def get_open_file_handles_under_directory(directory):
    """get the open files under a directory."""
    result = set()
    all_handles = get_open_file_handles()
    # to avoid issues with Libraries on Windows 7 and later, we will
    # have to get the real path
    directory = get_real_path(os.path.abspath(directory))
    print 'Dir ' + directory
    if not directory.endswith(os.path.sep):
        directory += os.path.sep
    for file in all_handles:
        print 'Current file ' + file
        if directory in file:
    return result

The above code uses undocumented functions from the ntdll which I supposed Microsoft does not want me to use. An while it works, the solution does no scale since the process of querying the Object Manager is vey expensive and can rocket your CPU if performed several times. Nevertheless the above code works correctly and could be used to write a tools similar to those written by sysinternals.

I hope someone will find a use for the code, in my case it is code that I’ll have to throw away :(

Read more

In the last post I explained how to set the security attributes of a file on Windows. What naturally follows such a post is explaining how to implement the os.access method that takes into account such settings because the default implementation of python will ignore them. Lets first define when does a user have read access in our use case:

I user has read access if the user sid has read access our the sid of the ‘Everyone’ group has read access.

The above also includes any type of configuration like rw or rx. In order to be able to do this we have to understand how does Windows NT set the security of a file. On Windows NT the security of a file is set by using a bitmask of type DWORD which can be compared to a 32 bit unsigned long in ANSI C, and this is as far as the normal things go, let continue with the bizarre Windows implementation. For some reason I cannot understand the Windows developers rather than going with the more intuitive solution of using a bit per right, they instead, have decided to use a combination of bits per right. For example, to set the read flag 5 bits have to be set, for the write flag they use 6 bits and for the execute 4 bits are used. To make matters more simple the used bitmask overlap, that is if we remove the read flag we will be removing bit for the execute mask, and there is no documentation to be found about the different masks that are used…

Thankfully for use the cfengine project has had to go through this process already and by trial an error discovered the exact bits that provide the read rights. Such a magic number is:


Therefore we can easily and this flag to an existing right to remove the read flag. The number also means that the only import bit that we are interested in are bits 0 and 3 which when set mean that the read flag was added. To make matters more complicated the ‘Full Access’ rights does not use such flag. In order to know if a user has the Full Access rights we have to look at bit 28 which if set does represent the ‘Full Access’ flag.

So to summarize, to know if a user has the read flag we have to look at bit 28 to test for the ‘Full Access’ flag, if the ‘Full Access’ was not granted we have to look at bits 0 and 3 and when both of them are set the usre has the read flag, easy right ;) . Now to the practical example, the bellow code does exactly what I just explained using python and the win32api and win32security modules.

from win32api import GetUserName
from win32security import (
from ntsecuritycon import (
platform = 'win32'
def _int_to_bin(n):
    """Convert an int to a bin string of 32 bits."""
    return "".join([str((n >> y) & 1) for y in range(32-1, -1, -1)])
def _has_read_mask(number):
    """Return if the read flag is present."""
    # get the bin representation of the mask
    binary = _int_to_bin(number)
    # there is actual no documentation of this in MSDN but if bt 28 is set,
    # the mask has full access, more info can be found here:
    if binary[28] == '1':
        return True
    # there is no documentation in MSDN about this, but if bit 0 and 3 are true
    # we have the read flag, more info can be found here:
    return binary[0] == '1' and binary[3] == '1'
def access(path):
    """Return if the path is at least readable."""
    # for a file to be readable it has to be readable either by the user or
    # by the everyone group
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = security_descriptor.GetSecurityDescriptorDacl()
    sids = []
    for index in range(0, dacl.GetAceCount()):
        # add the sid of the ace if it can read to test that we remove
        # the r bitmask and test if the bitmask is the same, if not, it means
        # we could read and removed it.
        ace = dacl.GetAce(index)
        if _has_read_mask(ace[1]):
    accounts = [LookupAccountSid('',x)[0] for x in sids]
    return GetUserName() in accounts or EVERYONE_GROUP in accounts

When I wrote this my brain was in a WTF state so I’m sure that the horrible _int_to_bin function can be exchanged by the bin build in function from python. If you fancy doing it I would greatly appreciate it I cannot take this any longer ;)

Read more

While working on making the Ubuntu One code more multiplatform I founded myself having to write some code that would set the attributes of a file on Windows. Ideally os.chmod would do the trick, but of course this is windows, and it is not fully supported. According to the python documentation:

Note: Although Windows supports chmod(), you can only set the file’s read-only flag with it (via the stat.S_IWRITE and stat.S_IREAD constants or a corresponding integer value). All other bits are ignored.

Grrrreat… To solve this issue I have written a small function that will allow to set the attributes of a file by using the win32api and win32security modules. This solves partially the issues since 0444 and others cannot be perfectly map to the Windows world. In my code I have made the assumption that using the groups ‘Everyone’, ‘Administrators’ and the user name would be close enough for our use cases.

Here is the code in case anyone has to go through this:

from win32api import MoveFileEx, GetUserName
from win32file import (
from win32security import (
from ntsecuritycon import (
def _get_group_sid(group_name):
    """Return the SID for a group with the given name."""
    return LookupAccountName('', group_name)[0]
def _set_file_attributes(path, groups):
    """Set file attributes using the wind32api."""
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = ACL()
    for group_name in groups:
        # set the attributes of the group only if not null
        if groups[group_name]:
            group_sid = _get_group_sid(group_name)
            dacl.AddAccessAllowedAce(ACL_REVISION, groups[group_name],
    # the dacl has all the info of the dff groups passed in the parameters
    security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
    SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)
def set_file_readonly(path):
    """Change path permissions to readonly in a file."""
    # we use the win32 api because chmod just sets the readonly flag and
    # we want to have imore control over the permissions
    groups = {}
    groups[GetUserName()] = FILE_GENERIC_READ
    # the above equals more or less to 0444
    _set_file_attributes(path, groups)

For those who might want to remove the read access from a group, you just have to not pass the group in the groups parameter which would remove the group from the security descriptor.

Read more

I’m currently working on a “small” pet project (that I want to take advantage of at some point) in which I’m json serializing (Json.Net) a Tuple in which the Item type can be of any kind. Of course when deserializing the object, I’d like to have my object type as close as possible to its original type, that is, if I serialized a Tuple<string> I want to get the same object type and not a Tuple<object>.

To solve this I have written the following small extension method to do the job using Reflection:

/// <summary>
/// Downcast the more generic tuple with an object type 
/// to a more specific one.
/// </summary>
/// <param name="tuple">The tuple to down cast.</param>
/// <returns>The downcasted instance of the object.</returns>
public static object DownCast(this Tuple<object> tuple)
    var tupleType = typeof (Tuple<>);
    var downcast = tupleType.MakeGenericType(tuple.Item1.GetType());
    return Activator.CreateInstance(downcast, tuple.Item1);

The same idea can be used for any type of tuple (Tuple<,>, Tuple<,,> etc…). I hope this small code helps someone else :)

Read more

At the moment I am sprinting in Argentina trying to make the Ubuntu One port to Windows better by adding support to the sync daemon used on Linux. While the rest of the guys are focused in accomodating the current code to my “multiplatform” requirements, I’m working on getting a number of missing parts to work on windows. One of this parts is the lack of network manager on Windows.

One of the things we need to know to coninusly sync you files on windows is to get an event when your network is present, or dies. As usual this is far easier on Linux than on Windows. To get this event you have to implement the ISesNetwork interface from COM that will allow your object to register to network status changes. Due to the absolute lack of examples on the net (or how bad google is getting ;) ) I’ve decided to share the code I managed to get working:

"""Implementation of ISesNework in Python."""
import logging
import logging.handlers
import pythoncom
from win32com.server.policy import DesignatedWrapPolicy
from win32com.client import Dispatch
# set te logging to store the data in the ubuntuone folder
handler = logging.handlers.RotatingFileHandler('network_manager.log', 
                    maxBytes=400, backupCount=5)
service_logger = logging.getLogger('NetworkManager')
## from EventSys.h
PROGID_EventSystem = "EventSystem.EventSystem"
PROGID_EventSubscription = "EventSystem.EventSubscription"
# sens values for the events, this events contain the uuid of the
# event, the name of the event to be used as well as the method name 
# of the method in the ISesNetwork interface that will be executed for
# the event.
SUBSCRIPTION_NETALIVE = ('{cd1dcbd6-a14d-4823-a0d2-8473afde360f}',
                         'UbuntuOne Network Alive',
SUBSCRIPTION_NETALIVE_NOQOC = ('{a82f0e80-1305-400c-ba56-375ae04264a1}',
                               'UbuntuOne Net Alive No Info',
SUBSCRIPTION_NETLOST = ('{45233130-b6c3-44fb-a6af-487c47cee611}',
                        'UbuntuOne Network Lost',
SUBSCRIPTION_REACH = ('{4c6b2afa-3235-4185-8558-57a7a922ac7b}',
                       'UbuntuOne Network Reach',
SUBSCRIPTION_REACH_NOQOC = ('{db62fa23-4c3e-47a3-aef2-b843016177cf}',
                            'UbuntuOne Network Reach No Info',
SUBSCRIPTION_REACH_NOQOC2 = ('{d4d8097a-60c6-440d-a6da-918b619ae4b7}',
                             'UbuntuOne Network Reach No Info 2',
                 SUBSCRIPTION_REACH_NOQOC2 ]
SENSGUID_EVENTCLASS_NETWORK = '{d5978620-5b9f-11d1-8dd2-00aa004abd5e}'
SENSGUID_PUBLISHER = "{5fee1bd6-5b9b-11d1-8dd2-00aa004abd5e}"
# uuid of the implemented com interface
IID_ISesNetwork = '{d597bab1-5b9f-11d1-8dd2-00aa004abd5e}'
class NetworkManager(DesignatedWrapPolicy):
    """Implement ISesNetwork to know about the network status."""
    _com_interfaces_ = [IID_ISesNetwork]
    _public_methods_ = ['ConnectionMade',
    _reg_clsid_ = '{41B032DA-86B5-4907-A7F7-958E59333010}' 
    _reg_progid_ = "UbuntuOne.NetworkManager"
    def __init__(self, connected_cb, disconnected_cb):
        self.connected_cb = connected_cb 
        self.disconnected_cb = disconnected_cb
    def ConnectionMade(self, *args):
        """Tell that the connection is up again."""'Connection was made.')
    def ConnectionMadeNoQOCInfo(self, *args):
        """Tell that the connection is up again."""'Connection was made no info.')
    def ConnectionLost(self, *args):
        """Tell the connection was lost."""'Connection was lost.')
    def register(self):
        """Register to listen to network events."""
        # call the CoInitialize to allow the registration to run in an other
        # thread
        # interface to be used by com
        manager_interface = pythoncom.WrapObject(self)
        event_system = Dispatch(PROGID_EventSystem)
        # register to listent to each of the events to make sure that
        # the code will work on all platforms.
        for current_event in SUBSCRIPTIONS:
            # create an event subscription and add it to the event
            # service
            event_subscription = Dispatch(PROGID_EventSubscription)
            event_subscription.EventClassId = SENSGUID_EVENTCLASS_NETWORK
            event_subscription.PublisherID = SENSGUID_PUBLISHER
            event_subscription.SubscriptionID = current_event[0]
            event_subscription.SubscriptionName = current_event[1]
            event_subscription.MethodName = current_event[2]
            event_subscription.SubscriberInterface = manager_interface
            event_subscription.PerUser = True
            # store the event
            except pythoncom.com_error as e:
                    'Error registering to event %s', current_event[1])
if __name__ == '__main__':
    from threading import Thread
    def connected():
        print 'Connected'
    def disconnected():
        print 'Disconnected'
    manager = NetworkManager(connected, disconnected)
    p = Thread(target=manager.register)

The above code represents a NetworkManager class that will execute a callback according to the event that was raised by the sens subsystem. It is important to note that in the above code the ‘Connected’ event will be fired 3 times since we registered to three different connect events while it will fire a single ‘Disconnected’ event. The way to fix this would be to register just to a single event according to the windows system you are running on, but since we do not care in the Ubuntu One sync daemon, well I left it there so everyone can see it :)

Read more

As most of you know, the Windows files system does not support a number of special characters. To be precise does characters are

:”/\\|?*. As you can imaging this is a problem when syncing between far superior Unix system and Windows. Knowing this, can you please let me know what is wrong/right in this image:

Got it? Lets look closer:

Well, the genius behind this was not me but it was Chipaca, I can tell you, I’m far less imaginative. But this little trick will allow you to sync between Windows and Ubuntu in a far better user friendly way that other sync services do :)

Read more

Sometimes on Linux we take for granted DBus. On the Ubntu One Windows port we have had to deal with the fact that DBus on Windows is not that great and therefore had to write our own IPC between the python code and the c# code. To solve the IPC we have done the following:

Listen to a named pipe from C#

The approach we have followed here is pretty simple, we create a thread pool that will create NamedPipe. The reason for using a threadpool is to avoid the situation in which we only have a single thread dealing with the messages from python and we have a very chatty python developer. The code in c# is very straight forward:

 * Copyright 2010 Canonical Ltd.
 * This file is part of UbuntuOne on Windows.
 * UbuntuOne on Windows is free software: you can redistribute it and/or modify		
 * it under the terms of the GNU Lesser General Public License version 		
 * as published by the Free Software Foundation.		
 * Ubuntu One on Windows is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * GNU Lesser General Public License for more details.	
 * You should have received a copy of the GNU Lesser General Public License	
 * along with UbuntuOne for Windows.  If not, see <>.
 * Authors: Manuel de la Peña <>
using System;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using log4net;
namespace Canonical.UbuntuOne.ProcessDispatcher
    /// <summary>
    /// This oject represents a listener that will be waiting for messages
    /// from the python code and will perform an operation for each messages
    /// that has been recived. 
    /// </summary>
    internal class PipeListener : IPipeListener
        #region Helper strcut
        /// <summary>
        /// Private structure used to pass the start of the listener to the 
        /// different listening threads.
        /// </summary>
        private struct PipeListenerState
            #region Variables
            private readonly string _namedPipe;
            private readonly Action<object> _callback;
            #region Properties
            /// <summary>
            /// Gets the named pipe to which the thread should listen.
            /// </summary>
            public string NamedPipe { get { return _namedPipe; } }
            /// <summary>
            /// Gets the callback that the listening pipe should execute.
            /// </summary>
            public Action<object> Callback { get { return _callback; } }
            public PipeListenerState(string namedPipe, Action<object> callback)
                _namedPipe = namedPipe;
                _callback = callback;
        #region Variables
        private readonly object _loggerLock = new object();
        private ILog _logger;
        private bool _isListening;
        private readonly object _isListeningLock = new object();
        #region Properties
        /// <summary>
        /// Gets the logger to used with the object.
        /// </summary>
        internal ILog Logger
                if (_logger == null)
                    lock (_loggerLock)
                        _logger = LogManager.GetLogger(typeof(PipeListener));
                return _logger;
                _logger = value;
        /// <summary>
        /// Gets if the pipe listener is indeed listening to the pipe.
        /// </summary>
        public bool IsListening
            get { return _isListening; }
            private set
                // we have to lock to ensure that the threads do not screw each
                // other up, this makes a small step of the processing to be sync :(
                lock (_isListeningLock)
                    _isListening = value;
        /// <summary>
        /// Gets and sets the number of threads that will be used to listen to the 
        /// pipe. Each thread will listeng to connections and will dispatch the 
        /// messages when ever they are done.
        /// </summary>
        public int NumberOfThreads { get; set; }
        /// <summary>
        /// Gets and sets the pipe stream factory that know how to generate the streamers used for the communication.
        /// </summary>
        public IPipeStreamerFactory PipeStreamerFactory { get; set; }
        /// <summary>
        /// Gets and sets the action that will be performed with the message of that 
        /// is received by the pipe listener.
        /// </summary>
        public IMessageProcessor MessageProcessor { get; set; }
        #region Helpers
        /// <summary>
        /// Helper method that is used in another thread that will be listening to the possible events from 
        /// the pipe.
        /// </summary>
        private void Listen(object state)
            var namedPipeState = (PipeListenerState)state;
                var threadNumber = Thread.CurrentThread.ManagedThreadId;
                // starts the named pipe since in theory it should not be present, if there is 
                // a pipe already present we have an issue.
                using (var pipeServer = new NamedPipeServerStream(namedPipeState.NamedPipe, PipeDirection.InOut, NumberOfThreads,PipeTransmissionMode.Message,PipeOptions.Asynchronous))
                    Logger.DebugFormat("Thread {0} listenitng to pipe {1}", threadNumber, namedPipeState.NamedPipe);
                    // we wait until the python code connects to the pipe, we do not block the 
                    // rest of the app because we are in another thread.
                    Logger.DebugFormat("Got clien connection in tread {0}", threadNumber);
                        // create a streamer that know the protocol
                        var streamer = PipeStreamerFactory.Create();
                        // Read the request from the client. 
                        var message = streamer.Read(pipeServer);
                        Logger.DebugFormat("Message received to thread {0} is {1}", threadNumber, message);
                        // execute the action that has to occur with the message
                    // Catch the IOException that is raised if the pipe is broken
                    // or disconnected.
                    catch (IOException e)
                        Logger.DebugFormat("Error in thread {0} when reading pipe {1}", threadNumber, e.Message);
                // if we are still listening, we will create a new thread to be used for listening,
                // otherwhise we will not and not lnger threads will be added. Ofcourse if the rest of the
                // threads do no add more than one work, we will have no issues with the pipe server since it
                // has been disposed
                if (IsListening)
                    ThreadPool.QueueUserWorkItem(Listen, namedPipeState);
            catch (PlatformNotSupportedException e)
                // are we running on an OS that does not have pipes (Mono on some os)
                Logger.InfoFormat("Cannot listen to pipe {0}", namedPipeState.NamedPipe);
            catch (IOException e)
                // there are too many servers listening to this pipe.
                Logger.InfoFormat("There are too many servers listening to {0}", namedPipeState.NamedPipe);
        /// <summary>
        /// Starts listening to the different pipe messages and will perform the appropiate
        /// action when a message is received.
        /// </summary>
        /// <param name="namedPipe">The name fof the pipe to listen.</param>
        public void StartListening(string namedPipe)
            if (NumberOfThreads < 0)
                throw new PipeListenerException(
                    "The number of threads to use to listen to the pipe must be at least one.");
            IsListening = true;
            // we will be using a thread pool that will allow to have the different threads listening to 
            // the messages of the pipes. There could be issues if the devel provided far to many threads
            // to listen to the pipe since the number of pipe servers is limited.
            for (var currentThreaCount = 0; currentThreaCount < NumberOfThreads; currentThreaCount++)
                // we add an new thread to listen
                ThreadPool.QueueUserWorkItem(Listen, new PipeListenerState(namedPipe, MessageProcessor.ProcessMessage));
        /// <summary>
        /// Stops listening to the different pipe messages. All the thread that are listening already will 
        /// be forced to stop.
        /// </summary>
        public void StopListening()
            IsListening = false;

Sending messages from python

Once the pipe server is listening in the .Net side we simple have to use the CallNamedPipe method to be able to send messages to .Net. In my case I have used Json as a stupid protocol, ideally you should do something smart like protobuffers.

 call the pipe with the message
        data = win32pipe.CallNamedPipe(pipe_name, 
            data_json, len(data_json), 0 )
    except Exception, e:
        print "Error: C# client is not listening!! %s" % e.message

Read more

It is not a secret that I love Spring.Net, it just makes the development of big application a pleasure. During the port of Ubuntu One to Windows I have been using the framework to initialise the WCF service that we use to provide other .Net applications the ability of communicating with Ubuntu One. Yes, this is our DBus alternative!

The idea behind using WCF is to allow other applications to use the different features that Ubuntu One provides, the very first application that we would like to use this would be Banshee on Windows (I have to start looking into that, but I have too much to do right now). In order to provide this functionality we use named pipes to allow the communication, there are two reasons for this:

  • For an application to host a WCF service that uses a binding besides the named pipe binding requires special permissions. This is clearly a no no for a user application like Ubuntu One.
  • Named pipes are dammed efficient!!! Named pipes on Windows are at the kernel level, cool :)

Initially I though of hosting the WCF services as a Windows services, why not?!?! Once I had this feature implemented, I realized the following. It turns out that while impersonation does get spawn within different threads, this is not the case for processes. This is a major pain in the ass. The main reason for this being a problem is the fact that if an application is executed in a different user space, the different env variables that are used are those of the user executing the code. This means that things like your user roming app dir will not be able to use, plus other security issues.

After realizing that the WCF services could not be hosted on a Windows service, I moved to write a work a round that would do the following:

  1. Configure the WCF services to use named pipes only for the current user.
  2. Start a console application that will host the WCF services.
  3. Start the different WCF clients for Ubuntu One (currently is our clietn app, but should it could be your own!

Although the definition of the solution is simple, we have to work around the issue that up ’til now all our WCF services were defined through configuration and were injected by the IoC. Usually you can change the location of you app domain configuration by using the following code:


In theory wth the above code you can redirect the configuration to a new file, and if you use for example:


you will be able to get the value of your new configuration. Unfortunatly, the Spring.Net IoC uses the ConfigurationManager class which ignores that setting… Now what?

Well, re-writting all the code to not use Spring.Net IoC was not an option because it means changing a lot of work and does mean to move from an application where dependencies are injected to one were we have to manually init all the different objects. After some careful though, I move to use a small CLR detail that I knew to make the AppDomain that executed our code to use the users configuration. The trick is the following, use one AppDomain to start the application. This would be a dummy AppDomain that does not execute any code at all but launches a second AppDomain whose configuration is the correct one and which will execute the actual code.

In case I did not make any sense, here is an example code:

using System;
using Canonical.UbuntuOne.Common.Container;
using Canonical.UbuntuOne.Common.Utils;
using log4net;
namespace Canonical.UbuntuOne.ProcessDispatcher
    static class Program
        private static readonly ILog _logger = LogManager.GetLogger(typeof(Program));
        private static readonly ConfigurationLocator _configLocator = new ConfigurationLocator();
        /// <summary>
        /// This method starts the service.
        /// </summary>
        static void Main()
            _logger.Debug("Redirecting configuration");
            // Setup information for the new appdomain.
            var setup = new AppDomainSetup
                ConfigurationFile = _configLocator.GetCurrentUserDaemonConfiguration()
            // Create the new appdomain with the new config.
            var executionAppDomain = AppDomain.CreateDomain("ServicesAppDomain",
                AppDomain.CurrentDomain.Evidence, setup);
            // Call the write config method in that appdomain.
            executionAppDomain.DoCallBack(() =>
                _logger.Debug("Starting services.");
                // use the IoC to get the implementation of the SyncDaemon service, the IoC will take care of 
                // setting the object correctly.
                ObjectsContainer.Initialize(new SpringContainer());
                var syncDaemonWindowsService = 
                // To run more than one service you have to add them here
                while (true) ;

Well I hope this helps someone else :D

Read more

In one of my previous posts about the port of U1 to Windows I mentioned that setting up the enviroment to build the solution and work on it was a PITA. To solve that I have created a hideous batch script that will set up your devel environment to work on the U1 windows port.

Take a look at the monster:

@ECHO off 
:: Tell the user what the batch does.
ECHO This batch will set up your enviroment do that you can build the Windows Ubuntu One port.
ECHO The following actions will be taken:
ECHO    1. Download and install python 2.6.
ECHO    2. Install easy_install for windows.
ECHO    3. Install python required libraries.
ECHO    4. Install py2exe.
ECHO    5. Install bazaar.
ECHO    6. Install Ubuntu One Protocol
ECHO Quering OS architecture
:: Get the first 3 chars of process identifier
SET ARCH=%processor_identifier:~0,3%
IF NOT %ARCH% == x86 GOTO :X64
:: set the paths for the x86 packages
ECHO Setting Install Congi for arch %ARCH%
SET PYWIN32INSTALLER=pywin32-214.win32-py2.6.exe
ECHO Starting installation
:: ============================================================================
:: Set up temp directory 
:: ============================================================================
ECHO The file that will be used to store the downloaded data is:
:: If the dir does not exist we have not problem and continue
:: otherwhise delete the dir and create it so that we do not
:: have old data present
:: Ask user if he wants to delete the dir, he might not want to
CHOICE /C YN /M "The dir is already present. Do you want to delete it"
:: User does not want to delete, we are not that smart!! bye!
ECHO Please delete the dir manually
ECHO Leaving installation
:: ============================================================================
:: Set up python
:: ============================================================================
ECHO Checking if python2.6 is in the system
:: This is very anoying, FOR /F will work differently depending on the output
:: of reg which is not consistent between os (xp, 7) we have to choose the tokens 
:: according to the os
VER | FIND "XP" > nul
FOR /F "tokens=%PYTHONPATHTOKENS%" %%A IN ('REG QUERY HKLM\Software\Python\PythonCore\2.6\InstallPath /ve') DO @SET PYTHONPATH=%%A
:: donload python and install it
ECHO Download python 2.6
wget.exe -v --output-document=%TEMPFILE%\python.msi %PYTHONDOWNLOAD%
ECHO Installing python...
START /wait msiexec.exe /i %TEMPFILE%\python.msi
:: Set the location of python
:: Let user know we did find python 2.6
ECHO Python is present!
ECHO Python dir is %PYTHONPATH%
:: ============================================================================
:: Setup pywin32 extensions
:: ============================================================================
:: This should be downloaded, but I have issues with wget and sourceforge
ECHO Installing pywin32 python extensions
:: ============================================================================
:: Set up easy_install
:: ============================================================================
ECHO Checking if easy_install is in the system
ECHO Download easy_install for Windows
wget.exe -v --output-document=%TEMPFILE%\easy_install.exe 
ECHO Installing easy_install...
:: set the easy_install path, this is not superb since if the user changed
:: the path of python, we have problems
ECHO easy_install is present!
:: ============================================================================
:: Set up dependencies
:: ============================================================================
ECHO The following dependencies will be installed using easy_install
ECHO    1. zope.interface
ECHO    2. oauth
ECHO    4. boto
ECHO    5. lazr.authentication
ECHO    6. lazr.restfulclient
ECHO    7. lazr.uri
ECHO Installing dependencies
%EASYINSTALLPATH% -Z -U zope.interface oauth boto lazr.authentication lazr.restfulclient lazr.uri 
ECHO Python dependencies have been installed
:: ============================================================================
:: Set up depedencies that cannot be install with easy_install
:: Install py2exe using an msi, the easy_install pacakage fails in systems such
:: as XP
ECHO Downloading  py2exe for Windows
wget.exe -v --output-document=%TEMPFILE%\py2exe.exe "%PY2EXEDOWNLOAD%"
ECHO Installing py2exe
:: It is of extreme importance to install protoc before, otherwhise the protobuf
:: module for python will not be correctly generated
ECHO Downloading protobuf compiler for python
wget.exe -v --output-document=%TEMPFILE%\ %PROTOCDOWNLOAD%
ECHO Extracting protobuf compiler
IF NOT EXIST "%ProgramFiles%\Protoc" MD "%ProgramFiles%\Protoc"
unzip -o %TEMPFILE%\ -d "%ProgramFiles%\Protoc"
:: distutils does not work with the pacakage correctly and we have to unxip ourselves
ECHO Downloading Protobuf for Windows
wget.exe -v --output-document=%TEMPFILE%\ %PROTOBUFDOWNLOAD%
unzip -o %TEMPFILE%\ -d %TEMPFILE%\Protoc
:: distutils.spawn.find_executable is used to find protoc but it does not do a
:: a very good job on windows, although we have installer protoc in the 
:: %ProgramFiles% we are going to copy it to the current location so that 
:: python can find it... lame!
COPY /B "%ProgramFiles%\Protoc\protoc.exe" %TEMPFILE%\Protoc\protobuf-2.3.0\python
ECHO Installing Protobuf for Windows
START /B "%PYTHONPATH%\pycdthon.exe" /D%TEMPFILE%\Protoc\protobuf-2.3.0\python install
:: The from google creates an egg, but py2exe does notlike that, lets extract it
unzip -o %PYTHONPATH%\Lib\site-packages\protobuf-2.3.0-py2.6.egg -d %PYTHONPATH%\Lib\site-packages
:: The msi does not add an entry in the reg therefore we always install :(
ECHO Downloading  pyOpenSSl for Windows
wget.exe -v --output-document=%TEMPFILE%\pyOpenSSL.msi %PYOPENSSLDOWNLOAD%
ECHO Installing pyOpenSSL
START /wait msiexec.exe /i %TEMPFILE%\pyOpenSSL.msi
:: Twisted cannot be installer with easy_install on windows
ECHO Downloading twisted for Windows
wget.exe -v --output-document=%TEMPFILE%\twisted.msi %TWISTEDDOWNLOAD%
ECHO Installing twisted
START /wait msiexec.exe /i %TEMPFILE%\twisted.msi
:: Install xdg.BaseDirectory for Windows
ECHO Installing xdg.BaseDirectory
IF NOT EXIST %PYTHONPATH%\Lib\site-packages\xdg MD %PYTHONPATH%\Lib\site-packages\xdg
COPY %PYTHONPATH%\Lib\site-packages\xdg
ECHO. 2>%PYTHONPATH%\Lib\site-packages\xdg\
:: ============================================================================
:: Set up bazaar
:: ============================================================================
ECHO Checking if bzr is in the system
:: This is tricky since the Program Files dir has a space and does provide 
:: a problem because the delimeters are not correctly set, well is batch :(
FOR /F "tokens=3-6" %%A IN ('REG QUERY HKLM\Software\Bazaar /v InstallPath') DO @SET BZRPATH=%%A %%B
ECHO Downloading bazaar
wget.exe -v --output-document=%TEMPFILE%\bazaar.exe %BZRDOWNLOAD%
ECHO Installing bazaar
:: get the path from reg 
ECHO bzr is present!
ECHO bzr path is %BZRPATH%
:: ============================================================================
:: Set up ubuntuone-storage-protocol
:: ============================================================================
ECHO Branching ubuntuone-storage-protocol
"%BZRPATH%\bzr.exe" branch lp:ubuntuone-storage-protocol  %TEMPFILE%\ubuntuone-storage-protocol
:: distutils.spawn.find_executable is used to find protoc but it does not do a
:: a very good job on windows, although we have installer protoc in the 
:: %ProgramFiles% we are going to copy it to the current location so that 
:: python can find it... lame!
COPY /B "%ProgramFiles%\Protoc\protoc.exe" %TEMPFILE%\ubuntuone-storage-protocol
ECHO Installing ubuntuone-storage-protocol from source
:: Use start, otherwhise we will not be able to set the execution dir and 
:: will complain about not being able to find the correct dirs
:: to install
START /B "%PYTHONPATH%\python.exe" /D%TEMPFILE%\ubuntuone-storage-protocol\ install
:: ubuntuone.logger is in not pacakage, we just copy it from the utils dir
COPY %PYTHONPATH%\Lib\site-packages\ubuntuone
:: ============================================================================
:: Clean up the downloaded data
:: ============================================================================
ECHO Cleaning Temp files
ECHO Installation completed!!

I most confess I felt dirty when I finished, but at least it does the job. I hope that know more people are tempted to compile the solution and give it a go. You can find this script in lp:~mandel/ubuntuone-windows-installer/add_env_set_up

Remember that we do not have yet an official .msi from Canonical, so please is you do find a U1 Windows before the official release, DO NOT TRUST IT.

Read more

On the process on porting Ubuntu One to windows we took a number of decisions that we expect to make peoples live better when installing it.


One of the most important decisions that had to be taken was what to do with python. Most of the code (probably all) of the sync daemon has been written in python and reusing that code is a plus.

In this situation we could have chosen to different paths:

  • Dependency: Add the presence of a python runtime as a dependency. That is either add a bootstrap that installs python, install it in its normal location through a python installer or ask the user to do it for us.
  • Embedded:Use py2exe or pyInstaller to distribute the python binaries so that we do not “require” python to be there.

Both options have their pros an cons but we decide for second one for the following reasons:

  • A user could change the python runtime and brake Ubuntu One
  • More than one runtime could be there.
  • Is a normal user really interested about having python in their machine?

Unfortunately so far I have not managed to use pyInstaller which I’ve been told is smarter than py2exe in terms of creating smaller binary packages (anyone with experience with that is more than welcome to help me).

But Pyhon is not THAT heavy

Indeed, Python is not that heavy and should not make the msi huge. But of course Ubuntu One has a number of dependencies that are not Python alone:

  • Ubuntu SSO: In the last iteration of Ubuntu One the service started using the Ubuntu SSO. This code has been re-writen in C# and in the near future will be exposed by a WCF service so that things such as the Ubuntu One music store could be used on Windows (banshee sounds like a great app to offer on windows)
  • Gnome keyring: This is actually a dependency from Ubuntu SSO, but it has to be taken into account. We needed a place where we could store secrets. I have implemented a small lib that allows to store secrets in the Current User registry key that uses the DPAPI so that the secrets are safe. Later in the cycle I’ll add a WCF service that will allow other apps to store secrets there and might even add an abstraction layer so that it uses the GnomeKeyring code done by NDesk when on Linux.
  • Dependency Injection: I have intially started using WPF but I do not deny the option of using GTK# in the future, or at least have the option. For that I have sued the DI of Spring.Net so that contributors can add their own views. You could even customize your Ubuntu One by modifying the DI and point to a dll that provides a new UI.
  • PDBs are present: Currently PDB files are present and the code is compiled in DEBUG mode, this does make a diff in the size of it.

At the end of the day, all this adds up making the msi quite big. Currently I’m focused on porting all the required code, in the next cycle I’ll make sure you have a nice and small package :D

Read more