Canonical Voices

Posts tagged with 'ubuntu'

mandel

I the last few months I have been working on the Ubuntu Download Manager, one of The Big Rocks of August. The u-d-m provide a dbus service that allows applications to request downloads to be performed, to such download requests it adds some nice features that a user on a mobile phone, and probably a desktop, is interested to have. Some of those features are:

  • Apparmor isolation per download. That means that only you application can interact with its downloads.
  • Pause/Resume downloads
  • Autodetect network connection.
  • WIFI only downloads.
  • Hash check support.
  • Allow downloads to be performed while an application has been paused or killed.
  • Group downloads, where a bunch of files are provided and the different downloads are performed as a single atomic operation.

A download might seem a simple action to perform, right? Well, as soon as you start supporting all the above a single download operation becomes a fairly complicated matter. The following is a state machine that identifies the states of a download that would support such features:

Download

As you can see, it is a complicated matter and all these has to be tested and check by the QA team. By providing u-d-m (and later a client library to use approach in C and in the Ubuntu SDK, I’m terribly sorry but I did not have the time to finish it on time for the release) we are helping developers to perform simple downloads with robust code and do not worry about all the corner cases. Performing a download is as simple as requesting it and listen to the different signals. This kind of service is also provided by FirefoxOs, WEbOs and Tizan (but no in IOS or SailFish) but I believe we are doing a better job at exposing a richer API. Of course all this is open source and at least our friend at Jolla (and I really mean friends, I think they are doing an awesome work!!! and competition + collaboration is great).

In the following days I’ll be posting on hot to use the API via C, C++ and DBus.

Read more
mandel

In one of my previous posts about the port of U1 to Windows I mentioned that setting up the enviroment to build the solution and work on it was a PITA. To solve that I have created a hideous batch script that will set up your devel environment to work on the U1 windows port.

Take a look at the monster:

@ECHO off 
:: Tell the user what the batch does.
ECHO. 
ECHO This batch will set up your enviroment do that you can build the Windows Ubuntu One port.
ECHO.
ECHO The following actions will be taken:
ECHO    1. Download and install python 2.6.
ECHO    2. Install easy_install for windows.
ECHO    3. Install python required libraries.
ECHO    4. Install py2exe.
ECHO    5. Install bazaar.
ECHO    6. Install Ubuntu One Protocol
ECHO.
ECHO Quering OS architecture
:: Get the first 3 chars of process identifier
SET ARCH=%processor_identifier:~0,3%
IF NOT %ARCH% == x86 GOTO :X64
:: set the paths for the x86 packages
ECHO Setting Install Congi for arch %ARCH%
SET PYWIN32INSTALLER=pywin32-214.win32-py2.6.exe
SET PYTHONDOWNLOAD=http://www.python.org/ftp/python/2.6.6/python-2.6.6.msi
SET EASYINSTALLDOWNLOAD=http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11.win32-py2.6.exe#md5=1509752c3c2e64b5d0f9589aafe053dc
SET PY2EXEDOWNLOAD=http://sourceforge.net/projects/py2exe/files/py2exe/0.6.9/py2exe-0.6.9.win32-py2.6.exe/download
SET PROTOCDOWNLOAD=http://protobuf.googlecode.com/files/protoc-2.3.0-win32.zip
SET PROTOBUFDOWNLOAD=http://protobuf.googlecode.com/files/protobuf-2.3.0.zip
SET PYOPENSSLDOWNLOAD=http://pypi.python.org/packages/2.6/p/pyOpenSSL/pyOpenSSL-0.10.winxp32-py2.6.msi#md5=90920217fb35d76524cab66c8c135cc8
SET TWISTEDDOWNLOAD=http://tmrc.mit.edu/mirror/twisted/Twisted/10.1/Twisted-10.1.0.winxp32-py2.6.msi
SET BZRDOWNLOAD=http://launchpad.net/bzr/2.2/2.2.0/+download/bzr-2.2.0.win32-py2.6.exe
GOTO :STARTINSTALLATION
:X64
SET PYWIN32INSTALLER=pywin32-214.win-amd64-py2.6.exe
SET PYTHONDOWNLOAD=http://www.python.org/ftp/python/2.6.6/python-2.6.6.amd64.msi
SET EASYINSTALLDOWNLOAD=http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11.win32-py2.6.exe#md5=1509752c3c2e64b5d0f9589aafe053dc
SET PY2EXEDOWNLOAD=http://sourceforge.net/projects/py2exe/files/py2exe/0.6.9/py2exe-0.6.9.win64-py2.6.amd64.exe/download
SET PROTOCDOWNLOAD=http://protobuf.googlecode.com/files/protoc-2.3.0-win32.zip
SET PROTOBUFDOWNLOAD=http://protobuf.googlecode.com/files/protobuf-2.3.0.zip
SET PYOPENSSLDOWNLOAD=http://pypi.python.org/packages/2.6/p/pyOpenSSL/pyOpenSSL-0.10.winxp32-py2.6.msi#md5=90920217fb35d76524cab66c8c135cc8
SET TWISTEDDOWNLOAD=http://tmrc.mit.edu/mirror/twisted/Twisted/10.1/Twisted-10.1.0.winxp32-py2.6.msi
SET BZRDOWNLOAD=http://launchpad.net/bzr/2.2/2.2.0/+download/bzr-2.2.0.win32-py2.6.exe
ECHO 64
:STARTINSTALLATION
ECHO Starting installation
 
:: ============================================================================
:: Set up temp directory 
:: ============================================================================
ECHO.
SET TEMPFILE=%TEMP%\ENV_SET_UP
ECHO The file that will be used to store the downloaded data is:
ECHO %TEMPFILE%
:: If the dir does not exist we have not problem and continue
:: otherwhise delete the dir and create it so that we do not
:: have old data present
IF NOT EXIST %TEMPFILE% GOTO :NOTEMPDIR
:: Ask user if he wants to delete the dir, he might not want to
CHOICE /C YN /M "The dir is already present. Do you want to delete it"
IF ERRORLEVEL 1 GOTO :DELETEDIR
:: User does not want to delete, we are not that smart!! bye!
ECHO Please delete the dir manually
ECHO Leaving installation
EXIT
:DELETEDIR
RD /s /Q %TEMPFILE%
:NOTEMPDIR
MD %TEMPFILE%
 
:: ============================================================================
:: Set up python
:: ============================================================================
 
SET PYTHONPATH=""
ECHO Checking if python2.6 is in the system
:GETPYTHONPATH
:: This is very anoying, FOR /F will work differently depending on the output
:: of reg which is not consistent between os (xp, 7) we have to choose the tokens 
:: according to the os
SET PYTHONPATHTOKENS=3
VER | FIND "XP" > nul
IF %ERRORLEVEL% == 0 SET PYTHONPATHTOKENS=4
FOR /F "tokens=%PYTHONPATHTOKENS%" %%A IN ('REG QUERY HKLM\Software\Python\PythonCore\2.6\InstallPath /ve') DO @SET PYTHONPATH=%%A
IF NOT %PYTHONPATH% == "" GOTO :PYTHONPRESENT
:: donload python and install it
ECHO Download python 2.6
wget.exe -v --output-document=%TEMPFILE%\python.msi %PYTHONDOWNLOAD%
ECHO Installing python...
START /wait msiexec.exe /i %TEMPFILE%\python.msi
:: Set the location of python
GOTO :GETPYTHONPATH
:PYTHONPRESENT
:: Let user know we did find python 2.6
ECHO Python is present!
ECHO Python dir is %PYTHONPATH%
 
:: ============================================================================
:: Setup pywin32 extensions
:: ============================================================================
 
:: This should be downloaded, but I have issues with wget and sourceforge
ECHO Installing pywin32 python extensions
START /wait %PYWIN32INSTALLER%
 
:: ============================================================================
:: Set up easy_install
:: ============================================================================
 
ECHO Checking if easy_install is in the system
IF EXIST %PYTHONPATH%\Scripts\easy_install.exe GOTO :EASYINSTALLPRESENT
ECHO Download easy_install for Windows
wget.exe -v --output-document=%TEMPFILE%\easy_install.exe 
ECHO Installing easy_install...
START /wait %TEMPFILE%\easy_install.exe %EASYINSTALLDOWNLOAD%
:EASYINSTALLPRESENT
:: set the easy_install path, this is not superb since if the user changed
:: the path of python, we have problems
SET EASYINSTALLPATH=%PYTHONPATH%\Scripts\easy_install.exe
ECHO easy_install is present!
ECHO Python dir is %EASYINSTALLPATH%
 
:: ============================================================================
:: Set up dependencies
:: ============================================================================
 
 
ECHO The following dependencies will be installed using easy_install
ECHO    1. zope.interface
ECHO    2. oauth
ECHO    4. boto
ECHO    5. lazr.authentication
ECHO    6. lazr.restfulclient
ECHO    7. lazr.uri
ECHO Installing dependencies
%EASYINSTALLPATH% -Z -U zope.interface oauth boto lazr.authentication lazr.restfulclient lazr.uri 
ECHO.
ECHO Python dependencies have been installed
ECHO.
 
:: ============================================================================
:: Set up depedencies that cannot be install with easy_install
::=============================================================================
 
:: Install py2exe using an msi, the easy_install pacakage fails in systems such
:: as XP
ECHO Downloading  py2exe for Windows
wget.exe -v --output-document=%TEMPFILE%\py2exe.exe "%PY2EXEDOWNLOAD%"
ECHO Installing py2exe
%TEMPFILE%\py2exe.exe
 
:: It is of extreme importance to install protoc before, otherwhise the protobuf
:: module for python will not be correctly generated
ECHO Downloading protobuf compiler for python
wget.exe -v --output-document=%TEMPFILE%\protoc_compiler.zip %PROTOCDOWNLOAD%
ECHO Extracting protobuf compiler
IF NOT EXIST "%ProgramFiles%\Protoc" MD "%ProgramFiles%\Protoc"
unzip -o %TEMPFILE%\protoc_compiler.zip -d "%ProgramFiles%\Protoc"
 
 
:: distutils does not work with the pacakage correctly and we have to unxip ourselves
ECHO Downloading Protobuf for Windows
wget.exe -v --output-document=%TEMPFILE%\protoc.zip %PROTOBUFDOWNLOAD%
unzip -o %TEMPFILE%\protoc.zip -d %TEMPFILE%\Protoc
:: distutils.spawn.find_executable is used to find protoc but it does not do a
:: a very good job on windows, although we have installer protoc in the 
:: %ProgramFiles% we are going to copy it to the current location so that 
:: python can find it... lame!
COPY /B "%ProgramFiles%\Protoc\protoc.exe" %TEMPFILE%\Protoc\protobuf-2.3.0\python
ECHO Installing Protobuf for Windows
START /B "%PYTHONPATH%\pycdthon.exe" /D%TEMPFILE%\Protoc\protobuf-2.3.0\python setup.py install
:: The setup.py from google creates an egg, but py2exe does notlike that, lets extract it
unzip -o %PYTHONPATH%\Lib\site-packages\protobuf-2.3.0-py2.6.egg -d %PYTHONPATH%\Lib\site-packages
 
:: The msi does not add an entry in the reg therefore we always install :(
ECHO Downloading  pyOpenSSl for Windows
wget.exe -v --output-document=%TEMPFILE%\pyOpenSSL.msi %PYOPENSSLDOWNLOAD%
ECHO Installing pyOpenSSL
START /wait msiexec.exe /i %TEMPFILE%\pyOpenSSL.msi
 
:: Twisted cannot be installer with easy_install on windows
ECHO Downloading twisted for Windows
wget.exe -v --output-document=%TEMPFILE%\twisted.msi %TWISTEDDOWNLOAD%
ECHO Installing twisted
START /wait msiexec.exe /i %TEMPFILE%\twisted.msi
 
:: Install xdg.BaseDirectory for Windows
ECHO Installing xdg.BaseDirectory
IF NOT EXIST %PYTHONPATH%\Lib\site-packages\xdg MD %PYTHONPATH%\Lib\site-packages\xdg
COPY BaseDirectory.py %PYTHONPATH%\Lib\site-packages\xdg
ECHO. 2>%PYTHONPATH%\Lib\site-packages\xdg\__init__.py
 
:: ============================================================================
:: Set up bazaar
:: ============================================================================
 
ECHO Checking if bzr is in the system
:GETBZRPATH
SET BZRPATH = ""
:: This is tricky since the Program Files dir has a space and does provide 
:: a problem because the delimeters are not correctly set, well is batch :(
FOR /F "tokens=3-6" %%A IN ('REG QUERY HKLM\Software\Bazaar /v InstallPath') DO @SET BZRPATH=%%A %%B
IF NOT BZRPATH == "" GOTO :BZRPRESENT
ECHO Downloading bazaar
wget.exe -v --output-document=%TEMPFILE%\bazaar.exe %BZRDOWNLOAD%
ECHO Installing bazaar
%TEMPFILE%\bazaar.exe
:: get the path from reg 
GOTO :GETBZRPATH
:BZRPRESENT
ECHO bzr is present!
ECHO bzr path is %BZRPATH%
 
 
:: ============================================================================
:: Set up ubuntuone-storage-protocol
:: ============================================================================
 
ECHO Branching ubuntuone-storage-protocol
"%BZRPATH%\bzr.exe" branch lp:ubuntuone-storage-protocol  %TEMPFILE%\ubuntuone-storage-protocol
:: distutils.spawn.find_executable is used to find protoc but it does not do a
:: a very good job on windows, although we have installer protoc in the 
:: %ProgramFiles% we are going to copy it to the current location so that 
:: python can find it... lame!
COPY /B "%ProgramFiles%\Protoc\protoc.exe" %TEMPFILE%\ubuntuone-storage-protocol
ECHO Installing ubuntuone-storage-protocol from source
:: Use start, otherwhise we will not be able to set the execution dir and 
:: setup.py will complain about not being able to find the correct dirs
:: to install
START /B "%PYTHONPATH%\python.exe" /D%TEMPFILE%\ubuntuone-storage-protocol\ setup.py install
:: ubuntuone.logger is in not pacakage, we just copy it from the utils dir
COPY logger.py %PYTHONPATH%\Lib\site-packages\ubuntuone
 
:: ============================================================================
:: Clean up the downloaded data
:: ============================================================================
 
ECHO Cleaning Temp files
RD /s /Q %TEMPFILE%
ECHO Installation completed!!

I most confess I felt dirty when I finished, but at least it does the job. I hope that know more people are tempted to compile the solution and give it a go. You can find this script in lp:~mandel/ubuntuone-windows-installer/add_env_set_up

Remember that we do not have yet an official .msi from Canonical, so please is you do find a U1 Windows before the official release, DO NOT TRUST IT.

Read more
mandel

On the process on porting Ubuntu One to windows we took a number of decisions that we expect to make peoples live better when installing it.

Python

One of the most important decisions that had to be taken was what to do with python. Most of the code (probably all) of the sync daemon has been written in python and reusing that code is a plus.

In this situation we could have chosen to different paths:

  • Dependency: Add the presence of a python runtime as a dependency. That is either add a bootstrap that installs python, install it in its normal location through a python installer or ask the user to do it for us.
  • Embedded:Use py2exe or pyInstaller to distribute the python binaries so that we do not “require” python to be there.

Both options have their pros an cons but we decide for second one for the following reasons:

  • A user could change the python runtime and brake Ubuntu One
  • More than one runtime could be there.
  • Is a normal user really interested about having python in their machine?

Unfortunately so far I have not managed to use pyInstaller which I’ve been told is smarter than py2exe in terms of creating smaller binary packages (anyone with experience with that is more than welcome to help me).

But Pyhon is not THAT heavy

Indeed, Python is not that heavy and should not make the msi huge. But of course Ubuntu One has a number of dependencies that are not Python alone:

  • Ubuntu SSO: In the last iteration of Ubuntu One the service started using the Ubuntu SSO. This code has been re-writen in C# and in the near future will be exposed by a WCF service so that things such as the Ubuntu One music store could be used on Windows (banshee sounds like a great app to offer on windows)
  • Gnome keyring: This is actually a dependency from Ubuntu SSO, but it has to be taken into account. We needed a place where we could store secrets. I have implemented a small lib that allows to store secrets in the Current User registry key that uses the DPAPI so that the secrets are safe. Later in the cycle I’ll add a WCF service that will allow other apps to store secrets there and might even add an abstraction layer so that it uses the GnomeKeyring code done by NDesk when on Linux.
  • Dependency Injection: I have intially started using WPF but I do not deny the option of using GTK# in the future, or at least have the option. For that I have sued the DI of Spring.Net so that contributors can add their own views. You could even customize your Ubuntu One by modifying the DI and point to a dll that provides a new UI.
  • PDBs are present: Currently PDB files are present and the code is compiled in DEBUG mode, this does make a diff in the size of it.

At the end of the day, all this adds up making the msi quite big. Currently I’m focused on porting all the required code, in the next cycle I’ll make sure you have a nice and small package :D

Read more
mandel

As some of you may know I am the person that is currently working on the port of Ubuntu One to Windows. Recently a colleague asked me how to install Ubuntu One in its current alpha state on Windows, and I though that posting the instructions for the adventurous would be a nice thing to do (I might get someone interested to help me too ;) ).

Setting the build environment

Because the .msi that we generate does not have the digital signature of Canonical we do not distribute it yet. But you shall not worry since all the code is open source are you are more than able to compile, test and create the installer by yourself. To do that you have to set up your environment (I should create an msi or script for this….). In order to set everything, follow this steps:

  1. Install python 2.6 for windows and extensions (installer)
  2. Install py2exe
  3. Patch py2exe with this
  4. Add the following implementation of XDG for windows.
  5. Install this python packages:

    • twisted
    • oauth
    • ubuntuone-storage-protocol

    It is important that if you use easy_install to install the packages you need to use the -Z option so that the dependecies are not isntalled as eggs. Py2exe cannot work with eggs and by using the -Z option the egss will be automatically extracted for you.

Creating the .msi

As usual everything in the build process has been automated. To automate the process we have used Nant which will build, test and create the .msi to be used.

A list of the commands can be found in the ubuntu wiki, nevertheless here they are:

bootstrapper
Creates a bootstrapper that will allow to install in the users machine the Ubuntu One solution plus a number of useful applications (Tomboy + GtkSharp)
build
Compiles the different projects that are part of the solution.
clean
Removes the different results from the last compilation/task .
installer
Compiles the application using the build task, runs the unit tests usint the tests task and creates a msi installer that can be used to install Ubuntu One in the users machine (do not confuse with the bootstrapper, it only installes Ubuntu One)
tests
Compiles the solution using the build task and runs the different unit tests. The output of the tests can be found in the test-results dir.

In order to build the msi you will have to execute the following from the root of the directory:

tools\Nant\bin\nant.exe installer

Once you have done that you will be able to find the msi in the install directory that you will be able to use to install the app in your machine.

Installing

Well it is an .msi, so double click ;)

Using

As I mentioned, this is an alpha, an very very early alpha and it means that there is some work to get it running. Currently the most blocking issue is the fact that we do not have an implementation of Ubuntu SSO on windows an therefore we cannot retrieve the OAuth tokens required by Ubuntu One. Here are the instructions to get them:

1. Get credentials from Linux

The first step is to get your credentials from a machine that you already have paired. To do so, launch seahorse (the image might help)

Once you have opened seahorse you should be able to find an UbuntuOne token (if not, you will need to pair your machine to Ubuntu One). Right click on it and selected the properties options which should open a dialog like the following:

At this point simple click on the + sign and select “Show password” so that you can copy paste the Oauth tokens.

2. Set you OAuth in Windows

Currently the OAuth in Windows are read from an env variable. To be able to start syncing in your Windows machine you will have to set the env variable with the tokens you just retrieved from your Linux box. This example will be using Windows XP but it should be the same in other Windows versions.

To access to the env vars in Windows XP right click in “My Computer” and select “Properties”:

This will launch the system properties dialog. Select the “Advance” tab where you will find the option of “Enviroment Variables”:

Once the “Enviroment Variables” dialog is launched you will have to create a new env variable in the “User Variables” section:

The data to be used in the following:

Variable Name
UbuntuOne
Variable value
Your OAuth token from Linux.
Sync
If you did not restart your machine after the installer, do it. In the next boot time you will have the following:

Not all the actions of the menu are yet there, but for sure you can use the “Synchronize Now” option.

How can I help

Well the easiest way to help is to file bugs, secondly join #ubuntuone on freenode and look for mandel (me :D ) and I will happily explain the C# code as well as the python code and the work we have to do. This is not an easy project so do not get scared by the amount of code done so far or were to start, I’m here for that

Happy syncing!

Read more