Canonical Voices

Posts tagged with 'ubuntu one'

mandel

Yet again Windows has presented me a challenge when trying to work with its file system, this time in the form of lock files. The Ubuntu One client on linux uses pyinotify to be able to listen to the file system events this, for example, allows the daemon to be updating your files when a new version has been created without the direct intervention of the user.

Although Windows does not have pyinotify (for obvious reasons) a developer that wants to perform such a directory monitoring can rely on the ReadDirectoryChangesW function. This function provides a similar behavior but unfortunately the information it provides is limited when compared with the one from pyinotify. On one hand, there are less events you can listen on Windows (IN_OPEN and IN_CLOSE for example are not present) but it also provides very little information by just giving 5 actions back, that is while on Windows you can listen to:

  • FILE_NOTIFY_CHANGE_FILE_NAME
  • FILE_NOTIFY_CHANGE_DIR_NAME
  • FILE_NOTIFY_CHANGE_ATTRIBUTES
  • FILE_NOTIFY_CHANGE_SIZE
  • FILE_NOTIFY_CHANGE_LAST_WRITE
  • FILE_NOTIFY_CHANGE_LAST_ACCESS
  • FILE_NOTIFY_CHANGE_CREATION
  • FILE_NOTIFY_CHANGE_SECURITY

You will only get back 5 values which are integers that represent the action that was performed. YesterdayI decide to see if it was possible to query the Windows Object Manager to see the currently used FILE HANDLES which would returned the open files. My idea was to write such a function and the pool (ouch!) to find when a file was opened or close. The result of such an attempt is the following:

import os
import struct
 
import winerror
import win32file
import win32con
 
from ctypes import *
from ctypes.wintypes import *
from Queue import Queue
from threading import Thread
from win32api import GetCurrentProcess, OpenProcess, DuplicateHandle
from win32api import error as ApiError
from win32con import (
    FILE_SHARE_READ,
    FILE_SHARE_WRITE,
    OPEN_EXISTING,
    FILE_FLAG_BACKUP_SEMANTICS,
    FILE_NOTIFY_CHANGE_FILE_NAME,
    FILE_NOTIFY_CHANGE_DIR_NAME,
    FILE_NOTIFY_CHANGE_ATTRIBUTES,
    FILE_NOTIFY_CHANGE_SIZE,
    FILE_NOTIFY_CHANGE_LAST_WRITE,
    FILE_NOTIFY_CHANGE_SECURITY,
    DUPLICATE_SAME_ACCESS
)
from win32event import WaitForSingleObject, WAIT_TIMEOUT, WAIT_ABANDONED
from win32event import error as EventError
from win32file import CreateFile, ReadDirectoryChangesW, CloseHandle
from win32file import error as FileError
 
# from ubuntuone.platform.windows.os_helper import LONG_PATH_PREFIX, abspath
 
LONG_PATH_PREFIX = '\\\\?\\'
# constant found in the msdn documentation:
# http://msdn.microsoft.com/en-us/library/ff538834(v=vs.85).aspx
FILE_LIST_DIRECTORY = 0x0001
FILE_NOTIFY_CHANGE_LAST_ACCESS = 0x00000020
FILE_NOTIFY_CHANGE_CREATION = 0x00000040
 
# XXX: the following code is some kind of hack that allows to get the opened
# files in a system. The techinique uses an no documented API from windows nt
# that is internal to MS and might change in the future braking our code :(
UCHAR = c_ubyte
PVOID = c_void_p
 
ntdll = windll.ntdll
 
SystemHandleInformation = 16
STATUS_INFO_LENGTH_MISMATCH = 0xC0000004
STATUS_BUFFER_OVERFLOW = 0x80000005L
STATUS_INVALID_HANDLE = 0xC0000008L
STATUS_BUFFER_TOO_SMALL = 0xC0000023L
STATUS_SUCCESS = 0
 
CURRENT_PROCESS = GetCurrentProcess ()
DEVICE_DRIVES = {}
for d in "abcdefghijklmnopqrstuvwxyz":
    try:
        DEVICE_DRIVES[win32file.QueryDosDevice (d + ":").strip ("\x00").lower ()] = d + ":"
    except FileError, (errno, errctx, errmsg):
        if errno == 2:
            pass
        else:
          raise
 
class x_file_handles(Exception):
    pass
 
def signed_to_unsigned(signed):
    unsigned, = struct.unpack ("L", struct.pack ("l", signed))
    return unsigned
 
class SYSTEM_HANDLE_TABLE_ENTRY_INFO(Structure):
    """Represent the SYSTEM_HANDLE_TABLE_ENTRY_INFO on ntdll."""
    _fields_ = [
        ("UniqueProcessId", USHORT),
        ("CreatorBackTraceIndex", USHORT),
        ("ObjectTypeIndex", UCHAR),
        ("HandleAttributes", UCHAR),
        ("HandleValue", USHORT),
        ("Object", PVOID),
        ("GrantedAccess", ULONG),
    ]
 
class SYSTEM_HANDLE_INFORMATION(Structure):
    """Represent the SYSTEM_HANDLE_INFORMATION on ntdll."""
    _fields_ = [
        ("NumberOfHandles", ULONG),
        ("Handles", SYSTEM_HANDLE_TABLE_ENTRY_INFO * 1),
    ]
 
class LSA_UNICODE_STRING(Structure):
    """Represent the LSA_UNICODE_STRING on ntdll."""
    _fields_ = [
        ("Length", USHORT),
        ("MaximumLength", USHORT),
        ("Buffer", LPWSTR),
    ]
 
class PUBLIC_OBJECT_TYPE_INFORMATION(Structure):
    """Represent the PUBLIC_OBJECT_TYPE_INFORMATION on ntdll."""
    _fields_ = [
        ("Name", LSA_UNICODE_STRING),
        ("Reserved", ULONG * 22),
    ]
 
class OBJECT_NAME_INFORMATION (Structure):
    """Represent the OBJECT_NAME_INFORMATION on ntdll."""
    _fields_ = [
        ("Name", LSA_UNICODE_STRING),
    ]
 
class IO_STATUS_BLOCK_UNION (Union):
    """Represent the IO_STATUS_BLOCK_UNION on ntdll."""
    _fields_ = [
        ("Status", LONG),
        ("Pointer", PVOID),
    ]
 
class IO_STATUS_BLOCK (Structure):
    """Represent the IO_STATUS_BLOCK on ntdll."""
    _anonymous_ = ("u",)
    _fields_ = [
        ("u", IO_STATUS_BLOCK_UNION),
        ("Information", POINTER (ULONG)),
    ]
 
class FILE_NAME_INFORMATION (Structure):
    """Represent the on FILE_NAME_INFORMATION ntdll."""
    filename_size = 4096
    _fields_ = [
        ("FilenameLength", ULONG),
        ("FileName", WCHAR * filename_size),
    ]
 
def get_handles():
    """Return all the processes handles in the system atm."""
    system_handle_information = SYSTEM_HANDLE_INFORMATION()
    size = DWORD (sizeof (system_handle_information))
    while True:
        result = ntdll.NtQuerySystemInformation(
            SystemHandleInformation,
            byref(system_handle_information),
            size,
            byref(size)
        )
        result = signed_to_unsigned(result)
        if result == STATUS_SUCCESS:
            break
        elif result == STATUS_INFO_LENGTH_MISMATCH:
            size = DWORD(size.value * 4)
            resize(system_handle_information, size.value)
        else:
            raise x_file_handles("NtQuerySystemInformation", hex(result))
 
    pHandles = cast(
        system_handle_information.Handles,
        POINTER(SYSTEM_HANDLE_TABLE_ENTRY_INFO * \
                system_handle_information.NumberOfHandles)
    )
    for handle in pHandles.contents:
        yield handle.UniqueProcessId, handle.HandleValue
 
def get_process_handle (pid, handle):
    """Get a handle for the process with the given pid."""
    try:
        hProcess = OpenProcess(win32con.PROCESS_DUP_HANDLE, 0, pid)
        return DuplicateHandle(hProcess, handle, CURRENT_PROCESS,
            0, 0, DUPLICATE_SAME_ACCESS)
    except ApiError,(errno, errctx, errmsg):
        if errno in (
              winerror.ERROR_ACCESS_DENIED,
              winerror.ERROR_INVALID_PARAMETER,
              winerror.ERROR_INVALID_HANDLE,
              winerror.ERROR_NOT_SUPPORTED
        ):
            return None
        else:
            raise
 
 
def get_type_info (handle):
    """Get the handle type information."""
    public_object_type_information = PUBLIC_OBJECT_TYPE_INFORMATION()
    size = DWORD(sizeof(public_object_type_information))
    while True:
        result = signed_to_unsigned(
            ntdll.NtQueryObject(
                handle, 2, byref(public_object_type_information), size, None))
        if result == STATUS_SUCCESS:
            return public_object_type_information.Name.Buffer
        elif result == STATUS_INFO_LENGTH_MISMATCH:
            size = DWORD(size.value * 4)
            resize(public_object_type_information, size.value)
        elif result == STATUS_INVALID_HANDLE:
            return None
        else:
            raise x_file_handles("NtQueryObject.2", hex (result))
 
 
def get_name_info (handle):
    """Get the handle name information."""
    object_name_information = OBJECT_NAME_INFORMATION()
    size = DWORD(sizeof(object_name_information))
    while True:
        result = signed_to_unsigned(
            ntdll.NtQueryObject(handle, 1, byref (object_name_information),
            size, None))
        if result == STATUS_SUCCESS:
            return object_name_information.Name.Buffer
        elif result in (STATUS_BUFFER_OVERFLOW, STATUS_BUFFER_TOO_SMALL,
                        STATUS_INFO_LENGTH_MISMATCH):
            size = DWORD(size.value * 4)
            resize (object_name_information, size.value)
        else:
            return None
 
 
def filepath_from_devicepath (devicepath):
    """Return a file path from a device path."""
    if devicepath is None:
        return None
    devicepath = devicepath.lower()
    for device, drive in DEVICE_DRIVES.items():
        if devicepath.startswith(device):
            return drive + devicepath[len(device):]
    else:
        return devicepath
 
def get_real_path(path):
    """Return the real path avoiding issues with the Library a in Windows 7"""
    assert os.path.isdir(path)
    handle = CreateFile(
        path,
        FILE_LIST_DIRECTORY,
        FILE_SHARE_READ | FILE_SHARE_WRITE,
        None,
        OPEN_EXISTING,
        FILE_FLAG_BACKUP_SEMANTICS,
        None
    )
    name = get_name_info(int(handle))
    CloseHandle(handle)
    return filepath_from_devicepath(name)
 
def get_open_file_handles():
    """Return all the open file handles."""
    print 'get_open_file_handles'
    result = set()
    this_pid = os.getpid()
    for pid, handle in get_handles():
        if pid == this_pid:
            continue
        duplicate = get_process_handle(pid, handle)
        if duplicate is None:
            continue
        else:
            # get the type info and name info of the handle
            type = get_type_info(handle)
            name = get_name_info(handle)
            # add the handle to the result only if it is a file
            if type and type == 'File':
                # the name info represents the path to the object,
                # we need to convert it to a file path and then
                # test that it does exist
                if name:
                    file_path = filepath_from_devicepath(name)
                    if os.path.exists(file_path):
                        result.add(file_path)
    return result
 
def get_open_file_handles_under_directory(directory):
    """get the open files under a directory."""
    result = set()
    all_handles = get_open_file_handles()
    # to avoid issues with Libraries on Windows 7 and later, we will
    # have to get the real path
    directory = get_real_path(os.path.abspath(directory))
    print 'Dir ' + directory
    if not directory.endswith(os.path.sep):
        directory += os.path.sep
    for file in all_handles:
        print 'Current file ' + file
        if directory in file:
            result.add(file)
    return result

The above code uses undocumented functions from the ntdll which I supposed Microsoft does not want me to use. An while it works, the solution does no scale since the process of querying the Object Manager is vey expensive and can rocket your CPU if performed several times. Nevertheless the above code works correctly and could be used to write a tools similar to those written by sysinternals.

I hope someone will find a use for the code, in my case it is code that I’ll have to throw away :(

Read more
mandel

In the last post I explained how to set the security attributes of a file on Windows. What naturally follows such a post is explaining how to implement the os.access method that takes into account such settings because the default implementation of python will ignore them. Lets first define when does a user have read access in our use case:

I user has read access if the user sid has read access our the sid of the ‘Everyone’ group has read access.

The above also includes any type of configuration like rw or rx. In order to be able to do this we have to understand how does Windows NT set the security of a file. On Windows NT the security of a file is set by using a bitmask of type DWORD which can be compared to a 32 bit unsigned long in ANSI C, and this is as far as the normal things go, let continue with the bizarre Windows implementation. For some reason I cannot understand the Windows developers rather than going with the more intuitive solution of using a bit per right, they instead, have decided to use a combination of bits per right. For example, to set the read flag 5 bits have to be set, for the write flag they use 6 bits and for the execute 4 bits are used. To make matters more simple the used bitmask overlap, that is if we remove the read flag we will be removing bit for the execute mask, and there is no documentation to be found about the different masks that are used…

Thankfully for use the cfengine project has had to go through this process already and by trial an error discovered the exact bits that provide the read rights. Such a magic number is:

0xFFFFFFF6

Therefore we can easily and this flag to an existing right to remove the read flag. The number also means that the only import bit that we are interested in are bits 0 and 3 which when set mean that the read flag was added. To make matters more complicated the ‘Full Access’ rights does not use such flag. In order to know if a user has the Full Access rights we have to look at bit 28 which if set does represent the ‘Full Access’ flag.

So to summarize, to know if a user has the read flag we have to look at bit 28 to test for the ‘Full Access’ flag, if the ‘Full Access’ was not granted we have to look at bits 0 and 3 and when both of them are set the usre has the read flag, easy right ;) . Now to the practical example, the bellow code does exactly what I just explained using python and the win32api and win32security modules.

from win32api import GetUserName
 
from win32security import (
    LookupAccountName,
    LookupAccountSid,
    GetFileSecurity,
    SetFileSecurity,
    ACL,
    DACL_SECURITY_INFORMATION,
    ACL_REVISION
)
from ntsecuritycon import (
    FILE_ALL_ACCESS,
    FILE_GENERIC_EXECUTE,
    FILE_GENERIC_READ,
    FILE_GENERIC_WRITE,
    FILE_LIST_DIRECTORY
)
 
platform = 'win32'
 
EVERYONE_GROUP = 'Everyone'
ADMINISTRATORS_GROUP = 'Administrators'
 
def _int_to_bin(n):
    """Convert an int to a bin string of 32 bits."""
    return "".join([str((n >> y) & 1) for y in range(32-1, -1, -1)])
 
def _has_read_mask(number):
    """Return if the read flag is present."""
    # get the bin representation of the mask
    binary = _int_to_bin(number)
    # there is actual no documentation of this in MSDN but if bt 28 is set,
    # the mask has full access, more info can be found here:
    # http://www.iu.hio.no/cfengine/docs/cfengine-NT/node47.html
    if binary[28] == '1':
        return True
    # there is no documentation in MSDN about this, but if bit 0 and 3 are true
    # we have the read flag, more info can be found here:
    # http://www.iu.hio.no/cfengine/docs/cfengine-NT/node47.html
    return binary[0] == '1' and binary[3] == '1'
 
def access(path):
    """Return if the path is at least readable."""
    # for a file to be readable it has to be readable either by the user or
    # by the everyone group
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = security_descriptor.GetSecurityDescriptorDacl()
    sids = []
    for index in range(0, dacl.GetAceCount()):
        # add the sid of the ace if it can read to test that we remove
        # the r bitmask and test if the bitmask is the same, if not, it means
        # we could read and removed it.
        ace = dacl.GetAce(index)
        if _has_read_mask(ace[1]):
            sids.append(ace[2])
    accounts = [LookupAccountSid('',x)[0] for x in sids]
    return GetUserName() in accounts or EVERYONE_GROUP in accounts

When I wrote this my brain was in a WTF state so I’m sure that the horrible _int_to_bin function can be exchanged by the bin build in function from python. If you fancy doing it I would greatly appreciate it I cannot take this any longer ;)

Read more
mandel

While working on making the Ubuntu One code more multiplatform I founded myself having to write some code that would set the attributes of a file on Windows. Ideally os.chmod would do the trick, but of course this is windows, and it is not fully supported. According to the python documentation:

Note: Although Windows supports chmod(), you can only set the file’s read-only flag with it (via the stat.S_IWRITE and stat.S_IREAD constants or a corresponding integer value). All other bits are ignored.

Grrrreat… To solve this issue I have written a small function that will allow to set the attributes of a file by using the win32api and win32security modules. This solves partially the issues since 0444 and others cannot be perfectly map to the Windows world. In my code I have made the assumption that using the groups ‘Everyone’, ‘Administrators’ and the user name would be close enough for our use cases.

Here is the code in case anyone has to go through this:

from win32api import MoveFileEx, GetUserName
 
from win32file import (
    MOVEFILE_COPY_ALLOWED,
    MOVEFILE_REPLACE_EXISTING,
    MOVEFILE_WRITE_THROUGH
)
from win32security import (
    LookupAccountName,
    GetFileSecurity,
    SetFileSecurity,
    ACL,
    DACL_SECURITY_INFORMATION,
    ACL_REVISION
)
from ntsecuritycon import (
    FILE_ALL_ACCESS,
    FILE_GENERIC_EXECUTE,
    FILE_GENERIC_READ,
    FILE_GENERIC_WRITE,
    FILE_LIST_DIRECTORY
)
 
EVERYONE_GROUP = 'Everyone'
ADMINISTRATORS_GROUP = 'Administrators'
 
def _get_group_sid(group_name):
    """Return the SID for a group with the given name."""
    return LookupAccountName('', group_name)[0]
 
 
def _set_file_attributes(path, groups):
    """Set file attributes using the wind32api."""
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = ACL()
    for group_name in groups:
        # set the attributes of the group only if not null
        if groups[group_name]:
            group_sid = _get_group_sid(group_name)
            dacl.AddAccessAllowedAce(ACL_REVISION, groups[group_name],
                group_sid)
    # the dacl has all the info of the dff groups passed in the parameters
    security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
    SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)
 
def set_file_readonly(path):
    """Change path permissions to readonly in a file."""
    # we use the win32 api because chmod just sets the readonly flag and
    # we want to have imore control over the permissions
    groups = {}
    groups[EVERYONE_GROUP] = FILE_GENERIC_READ
    groups[ADMINISTRATORS_GROUP] = FILE_GENERIC_READ
    groups[GetUserName()] = FILE_GENERIC_READ
    # the above equals more or less to 0444
    _set_file_attributes(path, groups)

For those who might want to remove the read access from a group, you just have to not pass the group in the groups parameter which would remove the group from the security descriptor.

Read more
mandel

At the moment I am sprinting in Argentina trying to make the Ubuntu One port to Windows better by adding support to the sync daemon used on Linux. While the rest of the guys are focused in accomodating the current code to my “multiplatform” requirements, I’m working on getting a number of missing parts to work on windows. One of this parts is the lack of network manager on Windows.

One of the things we need to know to coninusly sync you files on windows is to get an event when your network is present, or dies. As usual this is far easier on Linux than on Windows. To get this event you have to implement the ISesNetwork interface from COM that will allow your object to register to network status changes. Due to the absolute lack of examples on the net (or how bad google is getting ;) ) I’ve decided to share the code I managed to get working:

"""Implementation of ISesNework in Python."""
 
import logging
import logging.handlers
 
import pythoncom
 
from win32com.server.policy import DesignatedWrapPolicy
from win32com.client import Dispatch
 
# set te logging to store the data in the ubuntuone folder
handler = logging.handlers.RotatingFileHandler('network_manager.log', 
                    maxBytes=400, backupCount=5)
service_logger = logging.getLogger('NetworkManager')
service_logger.setLevel(logging.DEBUG)
service_logger.addHandler(handler)
 
## from EventSys.h
PROGID_EventSystem = "EventSystem.EventSystem"
PROGID_EventSubscription = "EventSystem.EventSubscription"
 
# sens values for the events, this events contain the uuid of the
# event, the name of the event to be used as well as the method name 
# of the method in the ISesNetwork interface that will be executed for
# the event.
 
SUBSCRIPTION_NETALIVE = ('{cd1dcbd6-a14d-4823-a0d2-8473afde360f}',
                         'UbuntuOne Network Alive',
                         'ConnectionMade')
 
SUBSCRIPTION_NETALIVE_NOQOC = ('{a82f0e80-1305-400c-ba56-375ae04264a1}',
                               'UbuntuOne Net Alive No Info',
                               'ConnectionMadeNoQOCInfo')
 
SUBSCRIPTION_NETLOST = ('{45233130-b6c3-44fb-a6af-487c47cee611}',
                        'UbuntuOne Network Lost',
                        'ConnectionLost')
 
SUBSCRIPTION_REACH = ('{4c6b2afa-3235-4185-8558-57a7a922ac7b}',
                       'UbuntuOne Network Reach',
                       'ConnectionMade')
 
SUBSCRIPTION_REACH_NOQOC = ('{db62fa23-4c3e-47a3-aef2-b843016177cf}',
                            'UbuntuOne Network Reach No Info',
                            'ConnectionMadeNoQOCInfo')
 
SUBSCRIPTION_REACH_NOQOC2 = ('{d4d8097a-60c6-440d-a6da-918b619ae4b7}',
                             'UbuntuOne Network Reach No Info 2',
                             'ConnectionMadeNoQOCInfo')
 
SUBSCRIPTIONS = [SUBSCRIPTION_NETALIVE,
                 SUBSCRIPTION_NETALIVE_NOQOC,
                 SUBSCRIPTION_NETLOST,
                 SUBSCRIPTION_REACH,
                 SUBSCRIPTION_REACH_NOQOC,
                 SUBSCRIPTION_REACH_NOQOC2 ]
 
SENSGUID_EVENTCLASS_NETWORK = '{d5978620-5b9f-11d1-8dd2-00aa004abd5e}'
SENSGUID_PUBLISHER = "{5fee1bd6-5b9b-11d1-8dd2-00aa004abd5e}"
 
# uuid of the implemented com interface
IID_ISesNetwork = '{d597bab1-5b9f-11d1-8dd2-00aa004abd5e}'
 
class NetworkManager(DesignatedWrapPolicy):
    """Implement ISesNetwork to know about the network status."""
 
    _com_interfaces_ = [IID_ISesNetwork]
    _public_methods_ = ['ConnectionMade',
                        'ConnectionMadeNoQOCInfo', 
                        'ConnectionLost']
    _reg_clsid_ = '{41B032DA-86B5-4907-A7F7-958E59333010}' 
    _reg_progid_ = "UbuntuOne.NetworkManager"
 
    def __init__(self, connected_cb, disconnected_cb):
        self._wrap_(self)
        self.connected_cb = connected_cb 
        self.disconnected_cb = disconnected_cb
 
    def ConnectionMade(self, *args):
        """Tell that the connection is up again."""
        service_logger.info('Connection was made.')
        self.connected_cb()
 
    def ConnectionMadeNoQOCInfo(self, *args):
        """Tell that the connection is up again."""
        service_logger.info('Connection was made no info.')
        self.connected_cb()
 
    def ConnectionLost(self, *args):
        """Tell the connection was lost."""
        service_logger.info('Connection was lost.')
        self.disconnected_cb() 
 
    def register(self):
        """Register to listen to network events."""
        # call the CoInitialize to allow the registration to run in an other
        # thread
        pythoncom.CoInitialize()
        # interface to be used by com
        manager_interface = pythoncom.WrapObject(self)
        event_system = Dispatch(PROGID_EventSystem)
        # register to listent to each of the events to make sure that
        # the code will work on all platforms.
        for current_event in SUBSCRIPTIONS:
            # create an event subscription and add it to the event
            # service
            event_subscription = Dispatch(PROGID_EventSubscription)
            event_subscription.EventClassId = SENSGUID_EVENTCLASS_NETWORK
            event_subscription.PublisherID = SENSGUID_PUBLISHER
            event_subscription.SubscriptionID = current_event[0]
            event_subscription.SubscriptionName = current_event[1]
            event_subscription.MethodName = current_event[2]
            event_subscription.SubscriberInterface = manager_interface
            event_subscription.PerUser = True
            # store the event
            try:
                event_system.Store(PROGID_EventSubscription, 
                                   event_subscription)
            except pythoncom.com_error as e:
                service_logger.error(
                    'Error registering to event %s', current_event[1])
 
        pythoncom.PumpMessages()
 
if __name__ == '__main__':
    from threading import Thread
    def connected():
        print 'Connected'
 
    def disconnected():
        print 'Disconnected'
 
    manager = NetworkManager(connected, disconnected)
    p = Thread(target=manager.register)
    p.start()

The above code represents a NetworkManager class that will execute a callback according to the event that was raised by the sens subsystem. It is important to note that in the above code the ‘Connected’ event will be fired 3 times since we registered to three different connect events while it will fire a single ‘Disconnected’ event. The way to fix this would be to register just to a single event according to the windows system you are running on, but since we do not care in the Ubuntu One sync daemon, well I left it there so everyone can see it :)

Read more
mandel

As most of you know, the Windows files system does not support a number of special characters. To be precise does characters are

:”/\\|?*. As you can imaging this is a problem when syncing between far superior Unix system and Windows. Knowing this, can you please let me know what is wrong/right in this image:

Got it? Lets look closer:

Well, the genius behind this was not me but it was Chipaca, I can tell you, I’m far less imaginative. But this little trick will allow you to sync between Windows and Ubuntu in a far better user friendly way that other sync services do :)

Read more
mandel

Sometimes on Linux we take for granted DBus. On the Ubntu One Windows port we have had to deal with the fact that DBus on Windows is not that great and therefore had to write our own IPC between the python code and the c# code. To solve the IPC we have done the following:

Listen to a named pipe from C#

The approach we have followed here is pretty simple, we create a thread pool that will create NamedPipe. The reason for using a threadpool is to avoid the situation in which we only have a single thread dealing with the messages from python and we have a very chatty python developer. The code in c# is very straight forward:

/*
 * Copyright 2010 Canonical Ltd.
 * 
 * This file is part of UbuntuOne on Windows.
 * 
 * UbuntuOne on Windows is free software: you can redistribute it and/or modify		
 * it under the terms of the GNU Lesser General Public License version 		
 * as published by the Free Software Foundation.		
 *		
 * Ubuntu One on Windows is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the	
 * GNU Lesser General Public License for more details.	
 *
 * You should have received a copy of the GNU Lesser General Public License	
 * along with UbuntuOne for Windows.  If not, see <http://www.gnu.org/licenses/>.
 * 
 * Authors: Manuel de la Peña <manuel.delapena@canonical.com>
 */
using System;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using log4net;
 
namespace Canonical.UbuntuOne.ProcessDispatcher
{
    /// <summary>
    /// This oject represents a listener that will be waiting for messages
    /// from the python code and will perform an operation for each messages
    /// that has been recived. 
    /// </summary>
    internal class PipeListener : IPipeListener
    {
        #region Helper strcut
 
        /// <summary>
        /// Private structure used to pass the start of the listener to the 
        /// different listening threads.
        /// </summary>
        private struct PipeListenerState
        {
            #region Variables
 
            private readonly string _namedPipe;
            private readonly Action<object> _callback;
 
            #endregion
 
            #region Properties
 
            /// <summary>
            /// Gets the named pipe to which the thread should listen.
            /// </summary>
            public string NamedPipe { get { return _namedPipe; } }
 
            /// <summary>
            /// Gets the callback that the listening pipe should execute.
            /// </summary>
            public Action<object> Callback { get { return _callback; } }
 
            #endregion
 
            public PipeListenerState(string namedPipe, Action<object> callback)
            {
                _namedPipe = namedPipe;
                _callback = callback;
            }
        }
 
        #endregion
 
        #region Variables
 
        private readonly object _loggerLock = new object();
        private ILog _logger;
        private bool _isListening;
        private readonly object _isListeningLock = new object();
 
        #endregion
 
        #region Properties
 
        /// <summary>
        /// Gets the logger to used with the object.
        /// </summary>
        internal ILog Logger
        {
            get
            {
                if (_logger == null)
                {
                    lock (_loggerLock)
                    {
                        _logger = LogManager.GetLogger(typeof(PipeListener));
                    }
                }
                return _logger;
            }
            set
            {
                _logger = value;
            }
        }
 
        /// <summary>
        /// Gets if the pipe listener is indeed listening to the pipe.
        /// </summary>
        public bool IsListening
        {
            get { return _isListening; }
            private set
            {
                // we have to lock to ensure that the threads do not screw each
                // other up, this makes a small step of the processing to be sync :(
                lock (_isListeningLock)
                {
                    _isListening = value;
                }
            }
        }
 
        /// <summary>
        /// Gets and sets the number of threads that will be used to listen to the 
        /// pipe. Each thread will listeng to connections and will dispatch the 
        /// messages when ever they are done.
        /// </summary>
        public int NumberOfThreads { get; set; }
 
        /// <summary>
        /// Gets and sets the pipe stream factory that know how to generate the streamers used for the communication.
        /// </summary>
        public IPipeStreamerFactory PipeStreamerFactory { get; set; }
 
        /// <summary>
        /// Gets and sets the action that will be performed with the message of that 
        /// is received by the pipe listener.
        /// </summary>
        public IMessageProcessor MessageProcessor { get; set; }
 
        #endregion
 
        #region Helpers
 
        /// <summary>
        /// Helper method that is used in another thread that will be listening to the possible events from 
        /// the pipe.
        /// </summary>
        private void Listen(object state)
        {
            var namedPipeState = (PipeListenerState)state;
 
            try
            {
                var threadNumber = Thread.CurrentThread.ManagedThreadId;
                // starts the named pipe since in theory it should not be present, if there is 
                // a pipe already present we have an issue.
                using (var pipeServer = new NamedPipeServerStream(namedPipeState.NamedPipe, PipeDirection.InOut, NumberOfThreads,PipeTransmissionMode.Message,PipeOptions.Asynchronous))
                {
                    Logger.DebugFormat("Thread {0} listenitng to pipe {1}", threadNumber, namedPipeState.NamedPipe);
                    // we wait until the python code connects to the pipe, we do not block the 
                    // rest of the app because we are in another thread.
                    pipeServer.WaitForConnection();
 
                    Logger.DebugFormat("Got clien connection in tread {0}", threadNumber);
                    try
                    {
                        // create a streamer that know the protocol
                        var streamer = PipeStreamerFactory.Create();
                        // Read the request from the client. 
                        var message = streamer.Read(pipeServer);
                        Logger.DebugFormat("Message received to thread {0} is {1}", threadNumber, message);
 
                        // execute the action that has to occur with the message
                        namedPipeState.Callback(message);
                    }
                    // Catch the IOException that is raised if the pipe is broken
                    // or disconnected.
                    catch (IOException e)
                    {
                        Logger.DebugFormat("Error in thread {0} when reading pipe {1}", threadNumber, e.Message);
                    }
 
                }
                // if we are still listening, we will create a new thread to be used for listening,
                // otherwhise we will not and not lnger threads will be added. Ofcourse if the rest of the
                // threads do no add more than one work, we will have no issues with the pipe server since it
                // has been disposed
                if (IsListening)
                {
                    ThreadPool.QueueUserWorkItem(Listen, namedPipeState);
                }
            }
            catch (PlatformNotSupportedException e)
            {
                // are we running on an OS that does not have pipes (Mono on some os)
                Logger.InfoFormat("Cannot listen to pipe {0}", namedPipeState.NamedPipe);
            }
            catch (IOException e)
            {
                // there are too many servers listening to this pipe.
                Logger.InfoFormat("There are too many servers listening to {0}", namedPipeState.NamedPipe);
            }
        }
 
        #endregion
 
        /// <summary>
        /// Starts listening to the different pipe messages and will perform the appropiate
        /// action when a message is received.
        /// </summary>
        /// <param name="namedPipe">The name fof the pipe to listen.</param>
        public void StartListening(string namedPipe)
        {
            if (NumberOfThreads < 0)
            {
                throw new PipeListenerException(
                    "The number of threads to use to listen to the pipe must be at least one.");
            }
            IsListening = true;
            // we will be using a thread pool that will allow to have the different threads listening to 
            // the messages of the pipes. There could be issues if the devel provided far to many threads
            // to listen to the pipe since the number of pipe servers is limited.
            for (var currentThreaCount = 0; currentThreaCount < NumberOfThreads; currentThreaCount++)
            {
                // we add an new thread to listen
                ThreadPool.QueueUserWorkItem(Listen, new PipeListenerState(namedPipe, MessageProcessor.ProcessMessage));
            }
 
        }
 
        /// <summary>
        /// Stops listening to the different pipe messages. All the thread that are listening already will 
        /// be forced to stop.
        /// </summary>
        public void StopListening()
        {
            IsListening = false;
        }
 
    }
}

Sending messages from python

Once the pipe server is listening in the .Net side we simple have to use the CallNamedPipe method to be able to send messages to .Net. In my case I have used Json as a stupid protocol, ideally you should do something smart like protobuffers.

 call the pipe with the message
    try:
        data = win32pipe.CallNamedPipe(pipe_name, 
            data_json, len(data_json), 0 )
    except Exception, e:
        print "Error: C# client is not listening!! %s" % e.message

Read more
mandel

It is not a secret that I love Spring.Net, it just makes the development of big application a pleasure. During the port of Ubuntu One to Windows I have been using the framework to initialise the WCF service that we use to provide other .Net applications the ability of communicating with Ubuntu One. Yes, this is our DBus alternative!

The idea behind using WCF is to allow other applications to use the different features that Ubuntu One provides, the very first application that we would like to use this would be Banshee on Windows (I have to start looking into that, but I have too much to do right now). In order to provide this functionality we use named pipes to allow the communication, there are two reasons for this:

  • For an application to host a WCF service that uses a binding besides the named pipe binding requires special permissions. This is clearly a no no for a user application like Ubuntu One.
  • Named pipes are dammed efficient!!! Named pipes on Windows are at the kernel level, cool :)

Initially I though of hosting the WCF services as a Windows services, why not?!?! Once I had this feature implemented, I realized the following. It turns out that while impersonation does get spawn within different threads, this is not the case for processes. This is a major pain in the ass. The main reason for this being a problem is the fact that if an application is executed in a different user space, the different env variables that are used are those of the user executing the code. This means that things like your user roming app dir will not be able to use, plus other security issues.

After realizing that the WCF services could not be hosted on a Windows service, I moved to write a work a round that would do the following:

  1. Configure the WCF services to use named pipes only for the current user.
  2. Start a console application that will host the WCF services.
  3. Start the different WCF clients for Ubuntu One (currently is our clietn app, but should it could be your own!

Although the definition of the solution is simple, we have to work around the issue that up ’til now all our WCF services were defined through configuration and were injected by the spring.net IoC. Usually you can change the location of you app domain configuration by using the following code:

AppDomain.CurrentDomain.SetData(“APP_CONFIG_FILE”,”c:\\ohad.config);

In theory wth the above code you can redirect the configuration to a new file, and if you use for example:

System.Configuration.ConfigurationSettings.AppSettings["my_setting"]

you will be able to get the value of your new configuration. Unfortunatly, the Spring.Net IoC uses the ConfigurationManager class which ignores that setting… Now what?

Well, re-writting all the code to not use Spring.Net IoC was not an option because it means changing a lot of work and does mean to move from an application where dependencies are injected to one were we have to manually init all the different objects. After some careful though, I move to use a small CLR detail that I knew to make the AppDomain that executed our code to use the users configuration. The trick is the following, use one AppDomain to start the application. This would be a dummy AppDomain that does not execute any code at all but launches a second AppDomain whose configuration is the correct one and which will execute the actual code.

In case I did not make any sense, here is an example code:

using System;
using Canonical.UbuntuOne.Common.Container;
using Canonical.UbuntuOne.Common.Utils;
using log4net;
 
namespace Canonical.UbuntuOne.ProcessDispatcher
{
 
    static class Program
    {
        private static readonly ILog _logger = LogManager.GetLogger(typeof(Program));
        private static readonly ConfigurationLocator _configLocator = new ConfigurationLocator();
 
        /// <summary>
        /// This method starts the service.
        /// </summary>
        static void Main()
        {
 
            _logger.Debug("Redirecting configuration");
 
            // Setup information for the new appdomain.
            var setup = new AppDomainSetup
            {
                ConfigurationFile = _configLocator.GetCurrentUserDaemonConfiguration()
            };
 
            // Create the new appdomain with the new config.
            var executionAppDomain = AppDomain.CreateDomain("ServicesAppDomain",
                AppDomain.CurrentDomain.Evidence, setup);
 
            // Call the write config method in that appdomain.
            executionAppDomain.DoCallBack(() =>
            {
                _logger.Debug("Starting services.");
                // use the IoC to get the implementation of the SyncDaemon service, the IoC will take care of 
                // setting the object correctly.
                ObjectsContainer.Initialize(new SpringContainer());
                var syncDaemonWindowsService = 
                     ObjectsContainer.GetImplementationOf<SyncDaemonWindowsService>();
                // To run more than one service you have to add them here
                syncDaemonWindowsService.Start();
                while (true) ;
            });
 
        }
    }
}

Well I hope this helps someone else :D

Read more
beuno

I am ZOMG very tired from the exciting release for Ubuntu One, but I wanted to highlight a very pleasant surprise.
We launched the Ubuntu One Music Streaming a week ago, and yesterday, while hadn't yet publicly released, we got a patch that adds support to tell last.fm the music you are listing from the Android app.

A big thank you to Scott Ferguson for being so awesome.

Read more
beuno

Matt Griffin has written a great blog post, so I'm just going to echo it:

After over a year’s worth of feedback from users like you and a clear view of where we want to take Ubuntu One in the future, we’ve just made some changes to the Ubuntu One service offering and pricing plans.

For starters, we will no longer offer the 50 GB plan to new subscribers. Everyone will get the basic plan and then have the option to add various ‘add-ons’ of services and storage as needed. But here are the details:

Ubuntu One Basic – available now
This is the same as the current free 2 GB option but with a new name. Users can continue to sync files, contacts, bookmarks and notes for free as part of our basic service and access the integrated Ubuntu One Music Store. We are also extending our platform support to include a Windows client, which will be available in Beta very soon.

Ubuntu One Mobile – available October 7th
Ubuntu One Mobile is our first example of a service that helps you do more with the content stored in your personal cloud. With Ubuntu One Mobile’s main feature – mobile music streaming – users can listen to any MP3 songs in their personal cloud (any owned MP3s, not just those purchased from the Ubuntu One Music Store) using our custom developed apps for iPhone and Android (coming soon to their respective marketplaces). These will be open source and available from Launchpad. Ubuntu One Mobile will also include the mobile contacts sync feature that was launched in Beta for the 10.04 release.

Ubuntu One Mobile is available for $3.99 (USD) per month or $39.99 (USD) per year. Users interested in this add-on can try the service free for 30 days. Ubuntu One Mobile will be the perfect companion to your morning exercise, daily commute, and weekend at the beach – we’re really excited to bring you this service!

Ubuntu One 20-Packs – available now
A 20-Pack is 20 GB of storage for files, contacts, notes, and bookmarks. Users will be able to add multiple 20-Packs at $2.99 (USD) per month or $29.99 (USD) per year each. If you start with Ubuntu One Basic (2 GB) and add 1 20-Pack (20 GB), you will have 22 GB of storage.

All add-ons are available for purchase in multiple currencies – USD, EUR and, recently added, GBP.

Users currently paying for the old 50 GB plan (including mobile contacts sync) can either keep their existing service or switch to the new plans structure to get more value from Ubuntu One at a lower price.

We know that you will enjoy these new add-ons as well as the performance enhancements we’ve made to Ubuntu One in recent months. If you have questions, our recently updated support area is a great place to start. There you’ll find a link to the current status of Ubuntu One services, a link to our frequently updated list of frequently asked questions, and a way to send us a direct message. As always, you can also ping the team on IRC (#ubuntuone in freenode). We welcome your questions, comments and suggestions.

Read more
beuno

After a solid 6 months of work, music streaming is up for public testing!  \o/

Read the full announcement for all the details, and go see the wiki page on how to sign up.

Read more
mandel

In one of my previous posts about the port of U1 to Windows I mentioned that setting up the enviroment to build the solution and work on it was a PITA. To solve that I have created a hideous batch script that will set up your devel environment to work on the U1 windows port.

Take a look at the monster:

@ECHO off 
:: Tell the user what the batch does.
ECHO. 
ECHO This batch will set up your enviroment do that you can build the Windows Ubuntu One port.
ECHO.
ECHO The following actions will be taken:
ECHO    1. Download and install python 2.6.
ECHO    2. Install easy_install for windows.
ECHO    3. Install python required libraries.
ECHO    4. Install py2exe.
ECHO    5. Install bazaar.
ECHO    6. Install Ubuntu One Protocol
ECHO.
ECHO Quering OS architecture
:: Get the first 3 chars of process identifier
SET ARCH=%processor_identifier:~0,3%
IF NOT %ARCH% == x86 GOTO :X64
:: set the paths for the x86 packages
ECHO Setting Install Congi for arch %ARCH%
SET PYWIN32INSTALLER=pywin32-214.win32-py2.6.exe
SET PYTHONDOWNLOAD=http://www.python.org/ftp/python/2.6.6/python-2.6.6.msi
SET EASYINSTALLDOWNLOAD=http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11.win32-py2.6.exe#md5=1509752c3c2e64b5d0f9589aafe053dc
SET PY2EXEDOWNLOAD=http://sourceforge.net/projects/py2exe/files/py2exe/0.6.9/py2exe-0.6.9.win32-py2.6.exe/download
SET PROTOCDOWNLOAD=http://protobuf.googlecode.com/files/protoc-2.3.0-win32.zip
SET PROTOBUFDOWNLOAD=http://protobuf.googlecode.com/files/protobuf-2.3.0.zip
SET PYOPENSSLDOWNLOAD=http://pypi.python.org/packages/2.6/p/pyOpenSSL/pyOpenSSL-0.10.winxp32-py2.6.msi#md5=90920217fb35d76524cab66c8c135cc8
SET TWISTEDDOWNLOAD=http://tmrc.mit.edu/mirror/twisted/Twisted/10.1/Twisted-10.1.0.winxp32-py2.6.msi
SET BZRDOWNLOAD=http://launchpad.net/bzr/2.2/2.2.0/+download/bzr-2.2.0.win32-py2.6.exe
GOTO :STARTINSTALLATION
:X64
SET PYWIN32INSTALLER=pywin32-214.win-amd64-py2.6.exe
SET PYTHONDOWNLOAD=http://www.python.org/ftp/python/2.6.6/python-2.6.6.amd64.msi
SET EASYINSTALLDOWNLOAD=http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11.win32-py2.6.exe#md5=1509752c3c2e64b5d0f9589aafe053dc
SET PY2EXEDOWNLOAD=http://sourceforge.net/projects/py2exe/files/py2exe/0.6.9/py2exe-0.6.9.win64-py2.6.amd64.exe/download
SET PROTOCDOWNLOAD=http://protobuf.googlecode.com/files/protoc-2.3.0-win32.zip
SET PROTOBUFDOWNLOAD=http://protobuf.googlecode.com/files/protobuf-2.3.0.zip
SET PYOPENSSLDOWNLOAD=http://pypi.python.org/packages/2.6/p/pyOpenSSL/pyOpenSSL-0.10.winxp32-py2.6.msi#md5=90920217fb35d76524cab66c8c135cc8
SET TWISTEDDOWNLOAD=http://tmrc.mit.edu/mirror/twisted/Twisted/10.1/Twisted-10.1.0.winxp32-py2.6.msi
SET BZRDOWNLOAD=http://launchpad.net/bzr/2.2/2.2.0/+download/bzr-2.2.0.win32-py2.6.exe
ECHO 64
:STARTINSTALLATION
ECHO Starting installation
 
:: ============================================================================
:: Set up temp directory 
:: ============================================================================
ECHO.
SET TEMPFILE=%TEMP%\ENV_SET_UP
ECHO The file that will be used to store the downloaded data is:
ECHO %TEMPFILE%
:: If the dir does not exist we have not problem and continue
:: otherwhise delete the dir and create it so that we do not
:: have old data present
IF NOT EXIST %TEMPFILE% GOTO :NOTEMPDIR
:: Ask user if he wants to delete the dir, he might not want to
CHOICE /C YN /M "The dir is already present. Do you want to delete it"
IF ERRORLEVEL 1 GOTO :DELETEDIR
:: User does not want to delete, we are not that smart!! bye!
ECHO Please delete the dir manually
ECHO Leaving installation
EXIT
:DELETEDIR
RD /s /Q %TEMPFILE%
:NOTEMPDIR
MD %TEMPFILE%
 
:: ============================================================================
:: Set up python
:: ============================================================================
 
SET PYTHONPATH=""
ECHO Checking if python2.6 is in the system
:GETPYTHONPATH
:: This is very anoying, FOR /F will work differently depending on the output
:: of reg which is not consistent between os (xp, 7) we have to choose the tokens 
:: according to the os
SET PYTHONPATHTOKENS=3
VER | FIND "XP" > nul
IF %ERRORLEVEL% == 0 SET PYTHONPATHTOKENS=4
FOR /F "tokens=%PYTHONPATHTOKENS%" %%A IN ('REG QUERY HKLM\Software\Python\PythonCore\2.6\InstallPath /ve') DO @SET PYTHONPATH=%%A
IF NOT %PYTHONPATH% == "" GOTO :PYTHONPRESENT
:: donload python and install it
ECHO Download python 2.6
wget.exe -v --output-document=%TEMPFILE%\python.msi %PYTHONDOWNLOAD%
ECHO Installing python...
START /wait msiexec.exe /i %TEMPFILE%\python.msi
:: Set the location of python
GOTO :GETPYTHONPATH
:PYTHONPRESENT
:: Let user know we did find python 2.6
ECHO Python is present!
ECHO Python dir is %PYTHONPATH%
 
:: ============================================================================
:: Setup pywin32 extensions
:: ============================================================================
 
:: This should be downloaded, but I have issues with wget and sourceforge
ECHO Installing pywin32 python extensions
START /wait %PYWIN32INSTALLER%
 
:: ============================================================================
:: Set up easy_install
:: ============================================================================
 
ECHO Checking if easy_install is in the system
IF EXIST %PYTHONPATH%\Scripts\easy_install.exe GOTO :EASYINSTALLPRESENT
ECHO Download easy_install for Windows
wget.exe -v --output-document=%TEMPFILE%\easy_install.exe 
ECHO Installing easy_install...
START /wait %TEMPFILE%\easy_install.exe %EASYINSTALLDOWNLOAD%
:EASYINSTALLPRESENT
:: set the easy_install path, this is not superb since if the user changed
:: the path of python, we have problems
SET EASYINSTALLPATH=%PYTHONPATH%\Scripts\easy_install.exe
ECHO easy_install is present!
ECHO Python dir is %EASYINSTALLPATH%
 
:: ============================================================================
:: Set up dependencies
:: ============================================================================
 
 
ECHO The following dependencies will be installed using easy_install
ECHO    1. zope.interface
ECHO    2. oauth
ECHO    4. boto
ECHO    5. lazr.authentication
ECHO    6. lazr.restfulclient
ECHO    7. lazr.uri
ECHO Installing dependencies
%EASYINSTALLPATH% -Z -U zope.interface oauth boto lazr.authentication lazr.restfulclient lazr.uri 
ECHO.
ECHO Python dependencies have been installed
ECHO.
 
:: ============================================================================
:: Set up depedencies that cannot be install with easy_install
::=============================================================================
 
:: Install py2exe using an msi, the easy_install pacakage fails in systems such
:: as XP
ECHO Downloading  py2exe for Windows
wget.exe -v --output-document=%TEMPFILE%\py2exe.exe "%PY2EXEDOWNLOAD%"
ECHO Installing py2exe
%TEMPFILE%\py2exe.exe
 
:: It is of extreme importance to install protoc before, otherwhise the protobuf
:: module for python will not be correctly generated
ECHO Downloading protobuf compiler for python
wget.exe -v --output-document=%TEMPFILE%\protoc_compiler.zip %PROTOCDOWNLOAD%
ECHO Extracting protobuf compiler
IF NOT EXIST "%ProgramFiles%\Protoc" MD "%ProgramFiles%\Protoc"
unzip -o %TEMPFILE%\protoc_compiler.zip -d "%ProgramFiles%\Protoc"
 
 
:: distutils does not work with the pacakage correctly and we have to unxip ourselves
ECHO Downloading Protobuf for Windows
wget.exe -v --output-document=%TEMPFILE%\protoc.zip %PROTOBUFDOWNLOAD%
unzip -o %TEMPFILE%\protoc.zip -d %TEMPFILE%\Protoc
:: distutils.spawn.find_executable is used to find protoc but it does not do a
:: a very good job on windows, although we have installer protoc in the 
:: %ProgramFiles% we are going to copy it to the current location so that 
:: python can find it... lame!
COPY /B "%ProgramFiles%\Protoc\protoc.exe" %TEMPFILE%\Protoc\protobuf-2.3.0\python
ECHO Installing Protobuf for Windows
START /B "%PYTHONPATH%\pycdthon.exe" /D%TEMPFILE%\Protoc\protobuf-2.3.0\python setup.py install
:: The setup.py from google creates an egg, but py2exe does notlike that, lets extract it
unzip -o %PYTHONPATH%\Lib\site-packages\protobuf-2.3.0-py2.6.egg -d %PYTHONPATH%\Lib\site-packages
 
:: The msi does not add an entry in the reg therefore we always install :(
ECHO Downloading  pyOpenSSl for Windows
wget.exe -v --output-document=%TEMPFILE%\pyOpenSSL.msi %PYOPENSSLDOWNLOAD%
ECHO Installing pyOpenSSL
START /wait msiexec.exe /i %TEMPFILE%\pyOpenSSL.msi
 
:: Twisted cannot be installer with easy_install on windows
ECHO Downloading twisted for Windows
wget.exe -v --output-document=%TEMPFILE%\twisted.msi %TWISTEDDOWNLOAD%
ECHO Installing twisted
START /wait msiexec.exe /i %TEMPFILE%\twisted.msi
 
:: Install xdg.BaseDirectory for Windows
ECHO Installing xdg.BaseDirectory
IF NOT EXIST %PYTHONPATH%\Lib\site-packages\xdg MD %PYTHONPATH%\Lib\site-packages\xdg
COPY BaseDirectory.py %PYTHONPATH%\Lib\site-packages\xdg
ECHO. 2>%PYTHONPATH%\Lib\site-packages\xdg\__init__.py
 
:: ============================================================================
:: Set up bazaar
:: ============================================================================
 
ECHO Checking if bzr is in the system
:GETBZRPATH
SET BZRPATH = ""
:: This is tricky since the Program Files dir has a space and does provide 
:: a problem because the delimeters are not correctly set, well is batch :(
FOR /F "tokens=3-6" %%A IN ('REG QUERY HKLM\Software\Bazaar /v InstallPath') DO @SET BZRPATH=%%A %%B
IF NOT BZRPATH == "" GOTO :BZRPRESENT
ECHO Downloading bazaar
wget.exe -v --output-document=%TEMPFILE%\bazaar.exe %BZRDOWNLOAD%
ECHO Installing bazaar
%TEMPFILE%\bazaar.exe
:: get the path from reg 
GOTO :GETBZRPATH
:BZRPRESENT
ECHO bzr is present!
ECHO bzr path is %BZRPATH%
 
 
:: ============================================================================
:: Set up ubuntuone-storage-protocol
:: ============================================================================
 
ECHO Branching ubuntuone-storage-protocol
"%BZRPATH%\bzr.exe" branch lp:ubuntuone-storage-protocol  %TEMPFILE%\ubuntuone-storage-protocol
:: distutils.spawn.find_executable is used to find protoc but it does not do a
:: a very good job on windows, although we have installer protoc in the 
:: %ProgramFiles% we are going to copy it to the current location so that 
:: python can find it... lame!
COPY /B "%ProgramFiles%\Protoc\protoc.exe" %TEMPFILE%\ubuntuone-storage-protocol
ECHO Installing ubuntuone-storage-protocol from source
:: Use start, otherwhise we will not be able to set the execution dir and 
:: setup.py will complain about not being able to find the correct dirs
:: to install
START /B "%PYTHONPATH%\python.exe" /D%TEMPFILE%\ubuntuone-storage-protocol\ setup.py install
:: ubuntuone.logger is in not pacakage, we just copy it from the utils dir
COPY logger.py %PYTHONPATH%\Lib\site-packages\ubuntuone
 
:: ============================================================================
:: Clean up the downloaded data
:: ============================================================================
 
ECHO Cleaning Temp files
RD /s /Q %TEMPFILE%
ECHO Installation completed!!

I most confess I felt dirty when I finished, but at least it does the job. I hope that know more people are tempted to compile the solution and give it a go. You can find this script in lp:~mandel/ubuntuone-windows-installer/add_env_set_up

Remember that we do not have yet an official .msi from Canonical, so please is you do find a U1 Windows before the official release, DO NOT TRUST IT.

Read more
mandel

On the process on porting Ubuntu One to windows we took a number of decisions that we expect to make peoples live better when installing it.

Python

One of the most important decisions that had to be taken was what to do with python. Most of the code (probably all) of the sync daemon has been written in python and reusing that code is a plus.

In this situation we could have chosen to different paths:

  • Dependency: Add the presence of a python runtime as a dependency. That is either add a bootstrap that installs python, install it in its normal location through a python installer or ask the user to do it for us.
  • Embedded:Use py2exe or pyInstaller to distribute the python binaries so that we do not “require” python to be there.

Both options have their pros an cons but we decide for second one for the following reasons:

  • A user could change the python runtime and brake Ubuntu One
  • More than one runtime could be there.
  • Is a normal user really interested about having python in their machine?

Unfortunately so far I have not managed to use pyInstaller which I’ve been told is smarter than py2exe in terms of creating smaller binary packages (anyone with experience with that is more than welcome to help me).

But Pyhon is not THAT heavy

Indeed, Python is not that heavy and should not make the msi huge. But of course Ubuntu One has a number of dependencies that are not Python alone:

  • Ubuntu SSO: In the last iteration of Ubuntu One the service started using the Ubuntu SSO. This code has been re-writen in C# and in the near future will be exposed by a WCF service so that things such as the Ubuntu One music store could be used on Windows (banshee sounds like a great app to offer on windows)
  • Gnome keyring: This is actually a dependency from Ubuntu SSO, but it has to be taken into account. We needed a place where we could store secrets. I have implemented a small lib that allows to store secrets in the Current User registry key that uses the DPAPI so that the secrets are safe. Later in the cycle I’ll add a WCF service that will allow other apps to store secrets there and might even add an abstraction layer so that it uses the GnomeKeyring code done by NDesk when on Linux.
  • Dependency Injection: I have intially started using WPF but I do not deny the option of using GTK# in the future, or at least have the option. For that I have sued the DI of Spring.Net so that contributors can add their own views. You could even customize your Ubuntu One by modifying the DI and point to a dll that provides a new UI.
  • PDBs are present: Currently PDB files are present and the code is compiled in DEBUG mode, this does make a diff in the size of it.

At the end of the day, all this adds up making the msi quite big. Currently I’m focused on porting all the required code, in the next cycle I’ll make sure you have a nice and small package :D

Read more
beuno

We spend a lot of time making sure we're not violating any licenses, and usually work with upstream early on. Charlie Smotherman got confused about how we had implemented music streaming and filed a bug reporting a violation. I'd encourage anyone who even suspects there may be a license violation to report a bug or contact us as soon as possible, but maybe hold off on the inflammatory blog posts  ;)
We've contacted him explaining all this but he seems to not had a chance to update his claims so I'm bringing this up now.

Nobody on the Ubuntu One team commented on any of his blog posts either. Ampache seems like a nice piece of software and even some people on the Ubuntu One team use it. We chose to go with Subsonic clients (we are not using any of the server pieces as it doesn't fit with our infrastructure) because the API seemed to be very nice, the existing clients where very nice to use, and all upstream developers where friendly and happy to help us release the service.

I'm sorry if any feelings got hurt, but there's no need to lash out like that.

For future reference, the whole team hangs out in #ubuntuone on Freenode.

Read more
mandel

As some of you may know I am the person that is currently working on the port of Ubuntu One to Windows. Recently a colleague asked me how to install Ubuntu One in its current alpha state on Windows, and I though that posting the instructions for the adventurous would be a nice thing to do (I might get someone interested to help me too ;) ).

Setting the build environment

Because the .msi that we generate does not have the digital signature of Canonical we do not distribute it yet. But you shall not worry since all the code is open source are you are more than able to compile, test and create the installer by yourself. To do that you have to set up your environment (I should create an msi or script for this….). In order to set everything, follow this steps:

  1. Install python 2.6 for windows and extensions (installer)
  2. Install py2exe
  3. Patch py2exe with this
  4. Add the following implementation of XDG for windows.
  5. Install this python packages:

    • twisted
    • oauth
    • ubuntuone-storage-protocol

    It is important that if you use easy_install to install the packages you need to use the -Z option so that the dependecies are not isntalled as eggs. Py2exe cannot work with eggs and by using the -Z option the egss will be automatically extracted for you.

Creating the .msi

As usual everything in the build process has been automated. To automate the process we have used Nant which will build, test and create the .msi to be used.

A list of the commands can be found in the ubuntu wiki, nevertheless here they are:

bootstrapper
Creates a bootstrapper that will allow to install in the users machine the Ubuntu One solution plus a number of useful applications (Tomboy + GtkSharp)
build
Compiles the different projects that are part of the solution.
clean
Removes the different results from the last compilation/task .
installer
Compiles the application using the build task, runs the unit tests usint the tests task and creates a msi installer that can be used to install Ubuntu One in the users machine (do not confuse with the bootstrapper, it only installes Ubuntu One)
tests
Compiles the solution using the build task and runs the different unit tests. The output of the tests can be found in the test-results dir.

In order to build the msi you will have to execute the following from the root of the directory:

tools\Nant\bin\nant.exe installer

Once you have done that you will be able to find the msi in the install directory that you will be able to use to install the app in your machine.

Installing

Well it is an .msi, so double click ;)

Using

As I mentioned, this is an alpha, an very very early alpha and it means that there is some work to get it running. Currently the most blocking issue is the fact that we do not have an implementation of Ubuntu SSO on windows an therefore we cannot retrieve the OAuth tokens required by Ubuntu One. Here are the instructions to get them:

1. Get credentials from Linux

The first step is to get your credentials from a machine that you already have paired. To do so, launch seahorse (the image might help)

Once you have opened seahorse you should be able to find an UbuntuOne token (if not, you will need to pair your machine to Ubuntu One). Right click on it and selected the properties options which should open a dialog like the following:

At this point simple click on the + sign and select “Show password” so that you can copy paste the Oauth tokens.

2. Set you OAuth in Windows

Currently the OAuth in Windows are read from an env variable. To be able to start syncing in your Windows machine you will have to set the env variable with the tokens you just retrieved from your Linux box. This example will be using Windows XP but it should be the same in other Windows versions.

To access to the env vars in Windows XP right click in “My Computer” and select “Properties”:

This will launch the system properties dialog. Select the “Advance” tab where you will find the option of “Enviroment Variables”:

Once the “Enviroment Variables” dialog is launched you will have to create a new env variable in the “User Variables” section:

The data to be used in the following:

Variable Name
UbuntuOne
Variable value
Your OAuth token from Linux.
Sync
If you did not restart your machine after the installer, do it. In the next boot time you will have the following:

Not all the actions of the menu are yet there, but for sure you can use the “Synchronize Now” option.

How can I help

Well the easiest way to help is to file bugs, secondly join #ubuntuone on freenode and look for mandel (me :D ) and I will happily explain the C# code as well as the python code and the work we have to do. This is not an easy project so do not get scared by the amount of code done so far or were to start, I’m here for that

Happy syncing!

Read more
beuno

One of the questions that took a little while for me to fully understand was a very simple one: why does Ubuntu One exist?

Depending on who you ask, you may get a different answer, but here's my take on it.

Above all, to extend the power of Ubuntu as an environment. Ubuntu One already allows you to many things beyond the basic file sync we started off with, you can keep your contacts from your phone and desktop  (and between other Ubuntu devices) in sync and backed up, notes, bookmarks, all your important files are backed up and synced, you can share them privately or publicly, you can buy music that gets delivered right to your music player, and soon you will be able to stream any of your music to your phone. And this is just today. As the project matures, we are working hard to make it easy for more and more third-party projects to use our platform and out-pace us in ideas and code.
All of this allows Ubuntu to extend its reach into mobile devices and even other operating systems. It feels like integrating into the real world today, not only the world we want to build.

Openness is the next thing on my mind. I know about all the criticisms about the server software not being open, I understand them and I've been through this same process with Launchpad. Right now, Canonical doesn't see a way to fund a 30+ developer team of rockstars, a huge infrastructure and bandwidth usage that is mostly used at no cost and still offer up the code to any competitor who could set up a competing project within minutes. I am sure someday, just like with Launchpad, we will figure it out and I will see all my commits push me up thousands of positions on ohloh. Until then, I'll have to continue working on Wikkid or any of the other 20 projects I use and am interested in, to keep me at a decent ranking.
All that said, the Ubuntu One team releases tons of source code all the time. A lot of the libraries we build are open sourced as soon as we get some time to clean them up and split it out of our source tree. All our desktop clients are open source from the start. On top of that, we work on pieces like desktopcouch, enabling couchdb for the desktop. We even got the chance to work with a closed-source iphone application, iSub, to open source his code so we could base our new streaming client on. We get to pay developers of open source projects on the Android platform as well, to work on improving it so we can deliver a better and more secure experience. We also get a chance to learn to package applications and upload new versions of the libraries we use to the Ubuntu repositories. And hundreds of other small things we do that feel so natural we forget to advertise and be proud of. All of this on Canonical's dime.

Finally, a goal that is dear to my heart. Make Canonical profitable. I have been overwhelmed over and over again by the passion with which Mark personally, and the company as a whole, contributes to making open source be the standard way of developing software in the world. I can understand why it's easy to feel uncomfortable with a privately owned company pursuing a profit while sponsoring an open source project which thousands of people contribute to, but after having sat down in dozens of meetings where everybody there cared about making sure we continue to grow as a community and that open source continues to win over tens of thousands of computers each month, I only worry about Canonical *not* being sustainable and constantly growing.

All these reasons for working on Ubuntu One have been close to my heart for many years now, a long time before I took the final step of investing not only my free time, but my work time, leisure time, and not too seldom, my sleep time,  and started working for Canonical in a very strict sense of the term "full time".

I've spent time working in a few different teams, all of them are interesting, exceptionally skilled and open source is a core part of their lives. Ubuntu One is where I feel I can do the most impact today, and I'm beyond lucky to have given the opportunity to act on it.

Read more
mandel

I have been working for about 2 moths now and after releasing our internal alpha release I have found a very interesting bug. In our port to windows we have decided to try and make your live as nice as possible in an environment as rough as Windows and to do so we allow our port to auto-update to always deliver the latests bug fixes.

Ofcourse to ensure that we are updating you system with the correct data we always perform a check sum of the msi. While our msi is updating to S3 using python, the code that downloads it is C#. Here are the different codes to calcualte the checksum:

    fd = open(filename,'r')
    md5_hash = hashlib.md5(fd.read()).hexdigest()
private static string Checksum(AlgoirhtmsEnum algorithm, string filePath)
        {
            HashAlgorithm hash;
            switch (algorithm)
            {
                case AlgoirhtmsEnum.SHA1:
                    hash = new SHA1Managed();
                    break;
                case AlgoirhtmsEnum.SHA256:
                    hash = new SHA256Managed();
                    break;
                case AlgoirhtmsEnum.SHA384:
                    hash = new SHA384Managed();
                    break;
                case AlgoirhtmsEnum.SHA512:
                    hash = new SHA512Managed();
                    break;
                default:
                    hash = new MD5CryptoServiceProvider();
                    break;
 
            }
            var checksum = "";
            using (var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read))
            {
                var md5 = hash.ComputeHash(stream);
                for (var i = 0; i < md5.Length; i++)
                {
                    checksum += md5[i].ToString("x2").ToLower();
                }
            }
            return checksum;
        }

Believe it or not the hash returned by each piece of code was different. WTF!!!! After a ridiculous time looking at it I managed to spot the issue. If you are using python on windows, unless you use the b option when opening a file, python will convert all the CRLF to LF making the hash to be different, how to fixed this? simply open the file this way:

fd = open(filename,'rb')

Go an figure….

Read more
beuno

A little while back, I hinted at us delivering new features for mobile phones, specifically Android and iPhone. Now that we're past the initial research, architecture and initial implementation phase, I'd like to share one of the new features we'll be releasing in Ubuntu 10.10: music streaming.

In Ubuntu 10.04, we released the music store, and to compliment that, we will be allowing you to stream any music you have in your Ubuntu One account to your iPhone or Android mobile phone. This feature will be bundled as part of the paid plan, although we are planning some re-structuring to that, yet to be announced.

We've chosen to base this new service on free software, and have picked Subsonic clients as our platform, implementing compatible APIs on our servers.
On the iPhone, Ben Baron, who develops the iSub client for that platform, has decided to open source the code for his application, enabling us to build our iPhone as an open source project. We can't thank him enough, for enabling that for us, you should try out iSub, it's an amazing application.

We hope to slowly start opening up the testing of the service before the 10.10 release, but more on that as we make progress.

More on our epic roadmap to 10.10 soon!

Read more
beuno

We should have released the source for the iphone client right after we did the upload to the appstore, but a bunch of bureaucracy and crazy work deadlines postponed this until now.
We're going to be doing some work for the Ubuntu 10.10 release on the iphone client as well as on a new Android client, both clients are going to be open source, like all our other Ubuntu One clients.
We've created the projects on Launchpad, pushed the initial source code for the iphone client, and will start pushing Android as soon as we get out of the exploration stage.

The projects are available at:

iphone:  https://launchpad.net/ubuntuone-ios-client
android: https://launchpad.net/ubuntuone-android-client

Stay tuned for more on our new mobile services!

Read more
beuno

A few months back the Ubuntu One team launched mobile contacts syncing, our first step into the mobile world. After a few initial rocky Beta days of cleaning up some scaling rough edges, it’s been a smooth ride since. It turned out to be a very popular service, which has us excited, and reinforced our eagerness to build more mobile services for Maverick.
While the full roadmap hasn’t been set in stone yet, we’ve had a lot of feedback about offering a separate, feature-rich mobile service at a lower price, as well as integration into Android.
We’ve decided to take on some of these challenges, and are committed to delivering more and more mobile services, some of which we will introduce around the Ubuntu Maverick release in October.

In the meantime, we’ve decided to extend the 30-day trial period for mobile contact sync until the Maverick release, where we will re-instate it as part of a bigger, juicier and with more native integration, mobile package.

This is effective now, so if you’ve signed up for our paid account exclusively for mobile sync, feel free to downgrade to the free plan, we will notify all mobile users before the 30-day trial is turned on again.

As we finish our research and initial development, we will announce the features that will be rolled out and probably open up for testing in our alpha phase to a small group of lucky people.

It seems to be the case every release, but, the future is exciting!

Read more
beuno

We have very exciting and challenging plans for the future of the new web+mobile Ubuntu One team (more on this soon), and we’re looking for an exceptional web engineer to join us.

The summary for this position is:

We are looking for an exceptional engineer to work on Ubuntu One’s web infrastructure with a proven track record for exceptional problem solving and integration into third-party systems. This person should help the team design, build, and deploy web and mobile applications with a high degree of quality and passion. If you’re the type of person who gets excited about delivering cutting-edge technology to hundreds of thousands of users, in a lean and friendly environment, we are looking for you!

If this sounds like you, check out the full job description and send us your CV!

Read more