Canonical Voices

Posts tagged with 'python'

pitti

I just found out that PyGObject 3.7.91 as released yesterday breaks GEdit plugins. I just pushed out 3.7.91.1 to unbreak this again, sorry about that!

Read more
pitti

I just released a new PyGObject for GNOME 3.7.91. This brings some marshalling fixes, plugs tons of memory leaks, and now raises a Python DeprecationWarning when your code calls a method which is marked as deprecated in the typelib. Please note that Python hides them by default, so if you are interested in those you need to run python with the -Wd option.

Thanks to all contributors!

  • Fix many memory leaks (#675726, #693402, #691501, #510511, #672224, and several more which are detected by our test suite) (Martin Pitt)
  • Dot not clobber original Gdk/Gtk functions with overrides (Martin Pitt) (#686835)
  • Optimize GValue.get/set_value by setting GValue.g_type to a local (Simon Feltman) (#694857)
  • Run tests with G_SLICE=debug_blocks (Martin Pitt) (#691501)
  • Add override helper for stripping boolean returns (Martin Pitt) (#694431)
  • Drop obsolete pygobject_register_sinkfunc() declaration (Martin Pitt) (#639849)
  • Fix marshalling of C arrays with explicit length in signal arguments (Martin Pitt) (#662241)
  • Fix signedness, overflow checking, and 32 bit overflow of GFlags (Martin Pitt) (#693121)
  • gi/pygi-marshal-from-py.c: Fix build on Visual C++ (Chun-wei Fan) (#692856)
  • Raise DeprecationWarning on deprecated callables (Martin Pitt) (#665084)
  • pygtkcompat: Add Widget.window, scroll_to_mark, and window methods (Simon Feltman) (#694067)
  • pygtkcompat: Add Gtk.Window.set_geometry_hints which accepts keyword arguments (Simon Feltman) (#694067)
  • Ship pygobject.doap for autogen.sh (Martin Pitt) (#694591)
  • Fix crashes in various GObject signal handler functions (Simon Feltman) (#633927)
  • pygi-closure: Protect the GSList prepend with the GIL (Olivier Crête) (#684060)
  • generictreemodel: Fix bad default return type for get_column_type (Simon Feltman)

Read more
facundo


Finalmente terminé de armar todo para liberar al mundo la versión traducida al español del tutorial de Python 3.

Realmente había terminado de traducirlo hace un par de semanas. Pero (aunque la generación del HTML fue bastante directo), generar el PDF me trajo bastante problemas.

Por un lado, el PDF generado por inkscape para los headers bonitos que tenemos es incorrecto en algunos detalles, así que tuve que meter un hack en el proyecto pdfrw para evitar que explote con ese problema. Luego habían problemas con páginas en blanco al final de capítulos que tuvieran footnotes, así que metí un workaround en rst2pdf para que reportlab no la cague al generar el PDF. Y dos o tres detalles más, pero menores (gracias Roberto Alsina por toda la ayuda en este bardo). Todos los parches (para poder reproducir la generación de algo copado) están en el README del proyecto.

También tuve que renovar el cómo presentábamos el tutorial, porque ahora tenemos el mismo para las versiones 2 y 3 de Python. Entonces ahora puse que la URL principal del tutorial apunte a un HTML muy bonito (gracias Rodrigo Bistolfi) que te deja elegir si querés ir qué versión del tutorial, tanto en Python 2 o 3, como en su formato: HTML o PDF. También reformulé la página del tutorial dentro del sitio de PyAr.

En fin. Disfruten y pásenlo.

Read more
Michael

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
                     verbosity=0)
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,
                     verbosity=0)

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,
                     verbosity=0)
 

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code

        self.migrate_to_dest()

        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.


Filed under: django, python, testing

Read more
facundo

El árbol fluorescente


En otra edición de "cosas que hice hace tiempo y me resultaron útiles ahora", les presento un proyecto que nació hace cinco años y medio de una charla de PyAr.

Como explico en este post, para jugar un rato con un amigo hice un árbol Trie, que luego de algunas optimizaciones degeneró en algo que llamé "Fucked Trie".

Este árbol para guardar palabras y buscarlas por prefijo de forma muy muy rápida resultó ser lo que necesitaba en el laburo un par de semanas atrás, pero con un cambio: ahora cada palabra tenía que guardar cierta metadata (que luego se obtendría al buscar).

Entonces, agarré el código original, lo modifiqué un poco, y armé este proyecto nuevo que se llama Fluorescent Trie (porque Fucked quedaba muy fuerte para un proyecto, vissste).

Fluorescent trie

Características de este árbol:

- Está pensando para mantenerlo en memoria: ocupa poco, y carga rápido

- Las búsquedas son por prefijo: O sea, entrando con "foo" encuentra "foo" y todo lo que empieza por "foo". No encuentra "grafoo".

- Las búsquedas son extremadamente rápidas (en el orden de los 10-4 segundos).

- Cada palabra tiene un payload que puede ser cualquier cosa.

Si lo necesitan para algo, aprovechen.

Read more
pitti

I just released a new PyGObject for GNOME 3.7.90, with a nice set of bug fixes and some internal code cleanup. Thanks to all contributors!

  • overrides: Fix inconsistencies with drag and drop target list API (Simon Feltman) (#680640)
  • pygtkcompat: Add pygtk compatible GenericTreeModel implementation (Simon Feltman) (#682933)
  • overrides: Add support for iterables besides tuples for TreePath creation (Simon Feltman) (#682933)
  • Prefix __module__ attribute of function objects with gi.repository (Niklas Koep) (#693839)
  • configure.ac: only enable code coverage when available (Jonathan Ballet) (#693328)
  • Correctly set properties on object with statically defined properties (Jonathan Ballet) (#693618)
  • autogen.sh: Use gnome-autogen.sh (Martin Pitt) (#693328)
  • Fix reference leaks with transient floating objects (Simon Feltman) (#687522)

Read more
pitti

I just released a new PyGObject for GNOME 3.7.5. Unfortunately master.gnome.org is out of space right now, so I put the new tarball on my Ubuntu people account for the time being.

This again brings a nice set of memory leak and bug fixes, some more reduction of static bindings, and better support for building under Windows.

Thanks to all contributors!

  • Move various signal methods from static bindings to gi and python (Simon Feltman) (#692918)
  • GLib overrides: Support unpacking ‘maybe’ variants (Paolo Borelli) (#693032)
  • Fix ref count leak when creating pygobject wrappers for input args (Mike Gorse) (#675726)
  • Prefix names of typeless enums and flags for GType registration (Simon Feltman) (#692515)
  • Fix compilation with non-C99 compilers such as Visual C++ (Chun-wei Fan) (#692856)
  • gi/overrides/Glib.py: Fix running on Windows/non-Unix (Chun-wei Fan)
  • Do not immediately initialize Gdk and Gtk on import (Martin Pitt) (#692300)
  • Accept ±inf and NaN as float and double values (Martin Pitt) (#692381)
  • Fix repr() of GLib.Variant (Martin Pitt)
  • Fix gtk-demo for Python 3 (Martin Pitt)
  • Define GObject.TYPE_VALUE gtype constant (Martin Pitt)
  • gobject: Go through introspection on property setting (Olivier Crête) (#684062)
  • Clean up caller-allocated GValues and their memory (Mike Gorse) (#691820)
  • Use GNOME_COMPILE_WARNINGS from gnome-common (Martin Pitt)

Read more
brendandonegan

Part of my duties as a Hardware Certification Engineer is to develop and maintain our test and bug reports. Previously these have been Python scripts containing lots of inline HTML (yuck :/), which I’m ashamed to admit it because I didn’t know any better when I initially wrote them. I’m currently in the middle of refactoring (hopefully not ref**ktoring) the scripts to take advantage of a new API that’s available for querying data from the Certification website and have decided to use Jinja (precisely Jinja2) as the template engine.

So I have a list of systems which may be in different locations and I want to sort them by location. Initially I did this in the Python code which gathered the data, using a simple .sort(key = lambda obj: obj.datacentre). Then the template code used the sorted list and all was well. Later on when reading the Jinja template documentation I discovered a way to sort a list in the template itself. I considered this to be a cleaner, more transparent way to modify the list (somebody looking at the template asking ‘why does it come out in this order’ has the answer right in front of them). So I gave it a try by changing

for hw in {%reported_hardware %}

to

{% for hw in reported_hardware|sort(attribute='datacentre') %}

and was unfortunately confronted with a traceback from the template engine:

Traceback (most recent call last):
...
AttributeError: Hardware instance has no attribute '__getitem__'

This looks like it’s saying that the ‘Hardware’ object (which I’ll introduce in a second) isn’t the type that the template engine was expecting.

Initially I defined the Hardware object like this:

class Hardware:

def __init__(self..., datacentre):
...
self.datacentre = datacentre

My first attempt to fix this involved making datacentre a property:

@property

def datacentre(self):

return self._datacentre

But this didn’t seem to do the trick. Eventually I read in the Python documentation about the property built in working only on new-style classes and this set off the lightbulb

class Hardware(object):

This seemed to be the magic ingredient, and the sort in the template engine started working. I also realised that making datacentre a property wasn’t really necessary, so reverted that change.

Update: I found out that this whole problem is avoided by using Python3.x because all classes there are new-style classes. So the advice above only applies if you’re stuck with Python2.x


Read more
thisisjedimike

Django-group-access is an app for django that controls row-level access to data based on group membership. This was perfect for a few projects we were developing in Canonical. So perfect, in fact, that we wrote django-group-access, and open sourced it.

It does a few nifty things. And these nifty things got us out of a tight spot where we had to redefine access control rules at short notice.

1) Declare access control rules, instead of having to code them in your views.

This means that if you want to control access to instances of MyModel, you just have to declare that MyModel is access controlled. MyModel will automatically pick up all the attributes needed to share it with groups, and have a user take ownership of it. It’s as simple as:

from django_group_access import register
from myapp.models import MyModel

register(MyModel)

2) Optional middleware will automatically filter your querysets so your views do not have to be aware of access control.

In some access control apps (admittedly, ones that are more focussed on providing row-level permissions, rather than row-level access) you have to modify your code and models to be aware of the access control. For example, you might have to do something like this:

def my_view(request):
    records = MyModel.objects.accessible_by_user(request.user).all()

That could be a lot of work if you’re integrating it into an existing app, and prohibitive if you’re integrating into a third party app.

With the automatic filtering in django-group-access, it remains:

def my_view(request):
    records = MyModel.objects.all()

That would give you all the records that the currently logged in user has access to. Plus, at no extra charge, it includes the .unrestricted method to give you unfiltered access to a queryset, like this:

def my_view(request):
    # all the visible records for the current user
    records = MyModel.objects.all()
    # all the records that exist
    all_records = records.unrestricted()

3) It allows you to create a hierarchy of models that control access.

Say you had a Room model, and a House model. If you had access to a Room, you want access to the House too.

With django-group-access, that’s easy.

class House(models.Model):
    address = models.TextField()

class Room(models.Model):
    description = models.TextField()
    house = models.ForeignKey(House)

register(Room)
register(House, control_relation='room')

Now, because django-group-access allows us to declare access rules instead of coding them into the models, and because it does automatic filtering, changing how the visibility of records is worked out becomes as easy as changing how your models are registered, and migrating your schema. (And, of course, changing the test data that your extensive unit and integration tests, set up. That’ll teach you to use test data factories…)

Development on django-group-access is ongoing, with plans for django-access-tng which will include row-level permissions and finer grained access control that doesn’t need to share records with groups to grant access.

But pretend you didn’t read that last paragraph. Our current product is the best. The new one might never come. Buy now to avoid disappointment.


Read more
facundo

CDPedia 0.8


Luego de bastante laburo las últimas semanas, estoy muy contento de anunciar (de mi parte y de Python Argentina) que está en la calle la versión 0.8 de CDPedia, un proyecto que permite acceder a la información de la Wikipedia en castellano sin necesidad de una conexión a Internet.  Se puede descargar libremente de la red y grabar a CDs, DVDs o memorias USB para repartirla sin restricciones.

La CDPedia funciona en cualquier computadora, ya sea que tenga Linux, MacOS o Windows como sistema operativo, y esta última versión incluye contenido de Wikipedia a Diciembre de 2012.

Para mayor información e instrucciones para la descarga, visitar la página del proyecto.

Logo


Esta versión trae las siguientes novedades:

  • Renovamos completamente la generación del ejecutable para Windows, se deberían tener muchísimos menos problemas para usar la CDPedia en ese entorno.
  • Al extraer las páginas de Wikipedia, se les hace un análisis para evitar los artículos vandalizados (en este caso, se baja una versión más antigua, pero correcta).
  • El servidor interno ahora es multiusuario, lo que simplifica enormemente instalar CDPedia en un servidor y que sea accedida desde distintos clientes.
  • Ahora se incluyen en los discos todas las páginas de las categorías, no sólo la primera.
  • Actualizamos el contenido a Diciembre 2012.
  • Varias mejoras a la hora de la generación de discos y tarballs, así como también en la calidad del código.

Si tenés discos disponibles, hacé algunas copias de CDPedia para regalar a tus familiares y amigos, a la biblioteca de tu barrio, o a la escuela de tus hijos. ¡Difundí CDPedia!

Read more
pitti

I just released a new PyGObject, for GNOME 3.7.4 which is due on Wednesday.

This release saw a lot of bug and memory leak fixes again, as well as enabling some more data types such as GParamSpec, boxed list properties, or directly setting string members in structs.

Thanks to all contributors!

Summary of changes (see change log for complete details):

  • Allow setting values through GtkTreeModelFilter (Simonas Kazlauskas) (#689624)
  • Support GParamSpec signal arguments from Python (Martin Pitt) (#683099)
  • pygobject_emit(): Fix cleanup on error (Martin Pitt)
  • Add signal emission methods to TreeModel which coerce the path argument (Simon Feltman) (#682933)
  • Add override for GValue (Bastian Winkler) (#677473)
  • Mark caller-allocated boxed structures as having a slice allocated (Mike Gorse) (#699501)
  • pygi-property: Support boxed GSList/GList types (Olivier Crête) (#684059)
  • tests: Add missing backwards compat methods for Python 2.6 (Martin Pitt) (#691646)
  • Allow setting TreeModel values to None (Simon Feltman) (#684094)
  • Set clean-up handler for marshalled arrays (Mike Gorse) (#691509)
  • Support setting string fields in structs (Vadim Rutkovsky) (#678401)
  • Permit plain integers for “gchar” values (Martin Pitt)
  • Allow single byte values for int8 types (Martin Pitt) (#691524)
  • Fix invalid memory access handling errors when registering an enum type (Mike Gorse)
  • Fix (out) arguments in callbacks (Martin Pitt)
  • Fix C to Python marshalling of struct pointer arrays (Martin Pitt)
  • Don’t let Property.setter() method names define property names (Martin Pitt) (#688971)
  • Use g-i stack allocation API (Martin Pitt) (#615982)
  • pyg_value_from_pyobject: support GArray (Ray Strode) (#690514)
  • Fix obsolete automake macros (Marko Lindqvist) (#691101)
  • Change dynamic enum and flag gtype creation to use namespaced naming (Simon Feltman) (#690455)
  • Fix Gtk.UIManager.add_ui_from_string() override for non-ASCII chars (Jonathan Ballet) (#690329)
  • Don’t dup strings before passing them to type registration functions (Mike Gorse) (#690532)
  • Fix marshalling of arrays of boxed struct values (Carlos Garnacho) (#656312)
  • testhelpermodule.c: Do not unref called method (Martin Pitt)

Read more
facundo


Y digo que es un regalo para las fiestas porque esta versión trae algunas cosas interesantes.

Por un lado, ¡hay un nuevo backend! Cuando actualicen los contenidos van a ver que pueden descargar programas del Banco Audiovisual de Contenidos Universales Argentino (gracias Gonzalo!),

Por otro lado, rediseñamos la GUI en el último sprint. Ahora tenemos un sector a la derecha con la imagen del programa (gracias Diego!) y la descripción (más un botón de acción). Y también hay una cola de descargas, donde se muestra lo que se está descargando, lo que ya terminó (y cómo), y lo que está encolado para después.

Los programas terminados ahora se ponen de otro color, y son fáciles de filtrar porque hay un checkbox que hace que se muestren sólo esos.

También hay un mejor manejo de las cancelaciones, más logging, y otras pequeñas mejoras y correcciones.

Como siempre, las formas de instalarlo, toda la info, y etc, en la página del proyecto.
   
¡¡Feliz fin de año!!

Read more
facundo

Cortando mal


Una de las cosas que repito mucho en mi charla de Entendiendo Unicode es que "siempre, siempre, siempre, hay que procesar los textos estando en unicode, no en bytes, porque se pueden obtener resultados inesperados".

Acabo de encontrar un caso de estos. Lo interesante es que es un caso que no había visto nunca.

Estoy agregando a mi programa Encuentro un nuevo backend: BACUA. Bueno, ya casi está (gracias a la ayuda de Gonzalo Martinez), el tema es que había una página que tenía problemas de unicode. Me puse a investigar, y resulta que el problema es que se estaba decodificando con el encoding incorrecto.

Se intentaba decodificar con UTF-8, pero como fallaba, se decodificaba con otra cosa, y algunas palabras quedaban mal.

Empecé a ver en detalle, y resulta que la página está toda bien codificada en UTF-8, excepto una parte. La linea "molesta" es esta:

'          <h5 class="sinopsis_cat">Los Ludomatic, banda de m\xc3\xbasica infantil exitosa en los a\xc3\xb1os 80, se re\xc3\xbane luego de veinte a\xc3\xb1os para ver que sus vidas no son como lo hab\xc3\xadan imaginado tiempo atr\xc3\xa1s. Toni, Becca, Marco, Lupe y Ren\xc3 ...</h5><br/>\r\n'

Como pueden ver, está toda casi bien, en utf8... por ejemplo, dice "Los Ludomatic, banda de música", y ahí vemos que la "ú" está bien codificada en utf8 como 0xC3 0xBA. El problema está al final, en el último nombre. Seguramente debía decir "Toni, Becca, Marco, Lupe y René", pero está cortado (con el agregado de los tres puntos, para indicar continuación).

Y está cortado mal.

Obviamente, si los que generaron la página hubiesen procesado el texto como unicode, se hubiese cortado antes de la é o después de la é. Pero no, lo manejaron como bytes, donde la é codificada como utf8 es 0xC3 0xA9. Y por mala suerte el corte cayó en el medio de esos dos bytes. Y quedó el 0xC3 suelto, que no es utf8 válido.

Y bueno. Eso. Recuerden: Siempre hay que procesar los textos como Unicode.

Read more
pitti

I just released a new PyGObject, for GNOME 3.7.3 which is due on Wednesday.

This is mostly a bug fix release. There is one API addition, it brings back official support for calling GLib.io_add_watch() with a Python file object or fd as first argument, in addition to the official API which expects a GLib.IOChannel object. These modes were marked as deprecated in 3.7.2 (only).

Thanks to all contributors!

Summary of changes (see change log for complete details):

  • Add support for caller-allocated GArray out arguments (Martin Pitt) (#690041)
  • Re-support calling GLib.io_add_watch with an fd or Python file (Martin Pitt)
  • pygtkcompat: Work around IndexError on large flags (Martin Pitt)
  • Fix pyg_value_from_pyobject() range check for uint (Martin Pitt)
  • Fix tests to work with g-i 1.34.2 (Martin Pitt)
  • Fix wrong refcount for GVariant property defaults (Martin Pitt) (#689267)
  • Fix array arguments on 32 bit (Martin Pitt)
  • Add backwards compatible API for GLib.unix_signal_add_full() (Martin Pitt)
  • Drop MININT64/MAXUINT64 workaround, current g-i gets this right now (Martin Pitt)
  • Fix maximum and minimum ranges of TYPE_(U)INT64 properties (Simonas Kazlauskas) (#688949)
  • Ship pygi-convert.sh in tarballs (Martin Pitt) (#688697)
  • Various added and improved tests (Martin Pitt)

Read more
Michael Hall

UPDATE: A command porting walk-through has beed added to the documentation.

Back around UDS time, I began work on a reboot of Quickly, Ubuntu’s application development tool.  After two months and just short of 4000 lines of code written, I’m pleased to announce that the inner-workings of the new code is very nearly complete!  Now I’ve reached the point where I need your help.

The Recap

First, let me go back to what I said needed to be done last time.  Port from Python 2 to Python 3: Done. Add built-in argument handling: Done. Add meta-data output: Well, not quite.  I’m working on that though, and now I can add it without requiring anything from template authors.

But here are some other things I did get done. Add Bash shell completion: Done.  Added Help command (that works with all other commands): Done.  Created command class decorators: Done.  Support templates installed in any XDG_DATA_DIRS: Done.  Allow template overriding on the command-line: Done.  Started documentation for template authors: Done.

Now it’s your turn

With the core of the Quickly reboot nearly done, focus can now turn to the templates.  At this point I’m reasonably confident that the API used by the templates and commands won’t change (at least not much).  The ‘create’ and ‘run’ commands from the ubuntu-application template have already been ported, I used those to help develop the API.  But that leaves all the rest of the commands that need to be updated (see list at the bottom of this post).  If you want to help make application development in Ubuntu better, this is a great way to contribute.

For now, I want to focus on finishing the port of the ubuntu-application template.  This will reveal any changes that might still need to be made to the new API and code, without disrupting multiple templates.

How to port a Command

The first thing you need to do is understand how the new Quickly handles templates and commands.  I’ve started on some documentation for template developers, with a Getting Started guide that covers the basics.  You can also find me in #quickly in Freenode IRC for help.

Next you’ll need to find the code for the command you want to port.  If you already have the current Quickly installed, you can find them in /usr/share/quickly/templates/ubuntu-application/, or you can bzr branch lp:quickly to get the source.

The commands are already in Python, but they are stand-alone scripts.  You will need to convert them into Python classes, with the code to be executed being called in the run() method.  You can add your class to the ./data/templates/ubuntu-application/commands.py file in the new Quickly branch (lp:quickly/reboot).  Then submit it as a merge proposal against lp:quickly/reboot.

Grab one and go!

So here’s the full list of ubuntu-application template commands.  I’ll update this list with progress as it happens.  If you want to help, grab one of the TODO commands, and start porting.  Email me or ping me on IRC if you need help.

add: TODO
configure: TODO
create: DONE!
debug: TODO
design: TODO
edit: DONE!
license: TODO
package: DONE!
release: TODO
run: DONE!
save: DONE!
share: TODO
submitubuntu: TODO
test: TODO
tutorial: TODO
upgrade: TODO

Read more
Michael Hall

During this latest round of arguing over the inclusion of Amazon search results in the Unity Dash, Alan Bell pointed out the fact that while the default scopes shipped in Ubuntu were made to check the new privacy settings, we didn’t do a very good job of telling third-party developers how to do it.

(Update: I was told a better way of doing this, be sure to read the bottom of the post before implementing it in your own code)

Since I am also a third-party lens developer, I decided to add it to my come of my own code and share how to do it with other lens/scope developers.  It turns out, it’s remarkably easy to do.

Since the privacy setting is stored in DConf, which we can access via the Gio library, we need to include that in our GObject Introspection imports:

from gi.repository import GLib, Unity, Gio

Then, before performing a search, we need to fetch the Unity Lens settings:

lens_settings = Gio.Settings(‘com.canonical.Unity.Lenses’)

The key we are interested in is ’remote-content-search’, and it can have one of two value, ‘all’ or ‘none’.  Since my locoteams-scope performs only remote searches, by calling the API on http://loco.ubuntu.com, if the user has asked that no remote searches be made, this scope will return without doing anything.

And that’s it!  That’s all you need to do in order to make your lens or scope follow the user’s privacy settings.

Now, before we get to the comments, I’d like to kindly point out that this post is about how to check the privacy setting in your lens or scope.  It is not about whether or not we should be doing remote searches in the dash, or how you would rather the feature be implemented.  If you want to pile on to those argument some more, there are dozens of open threads all over the internet where you can do that.  Please don’t do it here.
&nbps;

Update

I wasn’t aware, but there is a PreferencesManager class in Unity 6 (Ubuntu 12.10) that lets you access the same settings:

You should use this API instead of going directly to GSettings/DConf.

Read more
facundo

Charlas repetidas


Les comentaba en un post anterior que había vuelto a dar "Entendiendo Unicode" en las PyCones de Argentina y Venezuela. Charlando con alguien más de este tema, le decía que seguramente era la charla que más veces dí.

Bueno, no, este es el top 5 de todas las charlas que dí:

   13  Introducción a Python
   10  Entendiendo Unicode
    8  Python más rápido que C
    4  Python 3000
    3  Comunidad, anarquía y subversión

Se acumulan las repeticiones.

Y es algo natural, teniendo en cuenta que por año preparo casi siempre una charla nueva pero presento entre media docena y una docena de veces...

Read more
Barry Warsaw

UDS Update #1 - OAuth

For UDS-R for Raring (i.e. Ubuntu 13.04) in Copenhagen, I sponsored three blueprints.  These blueprints represent most of the work I will be doing for the next 6 months, as we're well on our way to the next LTS, Ubuntu 14.04.

I'll provide some updates to the other blueprints later, but for now, I want to talk about OAuth and Python 3.  OAuth is a protocol which allows you to programmatically interact with certain website APIs, in an authenticated manner, without having to provide your website password.  Essentially, it allows you to generate an authorization token which you can use instead, and it allows you to manage and share these tokens with applications, so that you can revoke them if you want, or decide how and which applications to trust to act on your behalf.

A good example of a site that uses OAuth is Launchpad, but many other sites also support OAuth, such as Twitter and Facebook.

There are actually two versions of OAuth out there.  OAuth version 1 is definitely the more prevelent, since it has been around for years, is relatively simple (at least on the client side), and enshrined in RFC 5849.  There are tons of libraries available that support OAuth v1, in a multitude of languages, with Python being no exception.

OAuth v2 is much less common, since it is currently only a draft specification, and has had its share of design-by-committee controversy.  Still, some sites such as Facebook do require OAuth v2.

One of the very earliest Python libraries to support OAuth v1, on both the client and server side, was python-oauth (I'll use the Debian package names in this post), and on the Ubuntu desktop, you'll find lots of scripts and libraries that use python-oauth.  There are major problems with this library though, and I highly recommend not using it.  The biggest problems are that the code is abandoned by its upstream maintainer (it hasn't be updated on PyPI since 2009), and it is not Python 3 compatible.  Because the OAuth v2 draft came after this library was abandoned, it provides no support for the successor specification.

For this reason, one of the blueprints I sponsored was specifically to survey the alternatives available for Python programmers, and make a decision about which one we would officially endorse for Ubuntu.  By "official endorsement" I mean promote the library to other Python programmers (hence this post!) and to port all of our desktop scripts from python-oauth to the agreed upon library.

After some discussion, it was unanimous by the attendees of the UDS session (both in-person and remotely), to choose the python-oauthlib as our preferred library.

python-oauthlib has a lot going for it.  It's Python 3 compatible, has an active upstream maintainer, supports both RFC 5849 for v1, and closely follows the draft for v2.  It's a well-tested, solid library, and it is available in Ubuntu for both Python 2 and Python 3.  Probably the only negative is that the library does not provide any support for the server side.  This is not a major problem for our immediate plans, since there aren't any server applications on the Ubuntu desktop requiring OAuth.  Eventually, yes, we'll need server side support, but we can punt on that recommendation for now.

Another cool thing about python-oauthlib is that it has been adopted by the python-requests library, meaning, if you want to use a modern replacement for the urllib2/httplib2 circus which supports OAuth out of the box, you can just use python-requests, provide the appropriate parameters, and you get request signing for free.

So, as you'll see from the blueprint, there are several bugs linked to packages which need porting to python-oauthlib for Ubuntu 13.04, and I am actively working on them, though contributions, as always, are welcome!  I thought I'd include a little bit of code to show you how you might port from python-oauth to python-oauthlib.  We'll stick with OAuth v1 in this discussion.

The first thing to recognize is that python-oauth uses different, older terminology that predates the RFC.  Thus, you'll see references to a token key and token secret, as well as a consumer key and consumer secret.  In the RFC, and in python-oauthlib, these terms are client key, client secret, resource owner key, and resource owner secret respectively.  After you get over that hump, the rest pretty much falls into place.  As an example, here is a code snippet from the piston-mini-client library which used the old python-oauth library:

class OAuthAuthorizer(object):
    """Authenticate to OAuth protected APIs."""
    def __init__(self, token_key, token_secret, consumer_key, consumer_secret,
                 oauth_realm="OAuth"):
        """Initialize a ``OAuthAuthorizer``.

        ``token_key``, ``token_secret``, ``consumer_key`` and
        ``consumer_secret`` are required for signing OAuth requests.  The
        ``oauth_realm`` to use is optional.
        """
        self.token_key = token_key
        self.token_secret = token_secret
        self.consumer_key = consumer_key
        self.consumer_secret = consumer_secret
        self.oauth_realm = oauth_realm

    def sign_request(self, url, method, body, headers):
        """Sign a request with OAuth credentials."""
        # Import oauth here so that you don't need it if you're not going
        # to use it.  Plan B: move this out into a separate oauth module.
        from oauth.oauth import (OAuthRequest, OAuthConsumer, OAuthToken,
                                 OAuthSignatureMethod_PLAINTEXT)
        consumer = OAuthConsumer(self.consumer_key, self.consumer_secret)
        token = OAuthToken(self.token_key, self.token_secret)
        oauth_request = OAuthRequest.from_consumer_and_token(
            consumer, token, http_url=url)
        oauth_request.sign_request(OAuthSignatureMethod_PLAINTEXT(),
                                   consumer, token)
        headers.update(oauth_request.to_header(self.oauth_realm))


The constructor is pretty simple, and it uses the old OAuth terminology.  The key thing to notice is the way the old API required you to create a consumer, a token, and then a request object, then ask the request object to sign the request.  On top of all the other disadvantages, this isn't a very convenient API.  Let's look at the snippet after conversion to python-oauthlib.

class OAuthAuthorizer(object):
    """Authenticate to OAuth protected APIs."""
    def __init__(self, token_key, token_secret, consumer_key, consumer_secret,
                 oauth_realm="OAuth"):
        """Initialize a ``OAuthAuthorizer``.

        ``token_key``, ``token_secret``, ``consumer_key`` and
        ``consumer_secret`` are required for signing OAuth requests.  The
        ``oauth_realm`` to use is optional.
        """
        # 2012-11-19 BAW: python-oauthlib requires unicodes for its tokens and
        # secrets.  Assume utf-8 values.
        # https://github.com/idan/oauthlib/issues/68
        self.token_key = _unicodeify(token_key)
        self.token_secret = _unicodeify(token_secret)
        self.consumer_key = _unicodeify(consumer_key)
        self.consumer_secret = _unicodeify(consumer_secret)
        self.oauth_realm = oauth_realm

    def sign_request(self, url, method, body, headers):
        """Sign a request with OAuth credentials."""
        # 2012-11-19 BAW: In order to preserve API backward compatibility,
        # convert empty string body to None.  The old python-oauth library
        # would treat the empty string as "no body", but python-oauthlib
        # requires None.
        if not body:
            body = None
        # Import oauthlib here so that you don't need it if you're not going
        # to use it.  Plan B: move this out into a separate oauth module.
        from oauthlib.oauth1 import Client, SIGNATURE_PLAINTEXT
        oauth_client = Client(self.consumer_key, self.consumer_secret,
                              self.token_key, self.token_secret,
                              signature_method=SIGNATURE_PLAINTEXT,
                              realm=self.oauth_realm)
        uri, signed_headers, body = oauth_client.sign(
            url, method, body, headers)
        headers.update(signed_headers)


See how much nicer this is?  You need only create a client object, essentially using all the same bits of information.  Then you ask the client to sign the request, and update the request headers with the signature.  Much easier.

Two important things to note.  If you are doing an HTTP GET, there is no request body, and thus no request content which needs to contribute to the signature.  In python-oauth, you could specify an empty body by using either None or the empty string.  piston-mini-client uses the latter, and this is embodied in its public API.  python-oauthlib however, treats the empty string as a body being present, so it would require the Content-Type header to be set even for an HTTP GET which has no content (i.e. no body).  This is why the replacement code checks for an empty string being passed in (actually, any false-ish value), and coerces that to None.

The second issue is that python-oauthlib requires the keys and secrets to be Unicode objects; they cannot be bytes objects.  In code ported straight from Python 2 however, these values are usually 8-bit strings, and so become bytes objects in Python 3.  python-oauthlib will raise a ValueError during signing if any of these are bytes objects.  Thus the use of the _unicodeify() function to decode these values to unicodes.

def _unicodeify(s):
    if isinstance(s, bytes):
        return s.decode('utf-8')
    return s


The above works in both Python 2 and Python 3.  Of course, we don't know for sure that the bytes values are UTF-8, but it's the only sane encoding to expect, and if a client of piston-mini-client were to be so insane as to use an incompatible encoding (US-ASCII is fine because it's compatible with UTF-8), it would be up to the client to just pass in unicodes in the first place.  At the time of this writing, this is under active discussion with upstream, but for now, it's not too difficult to work around.

Anyway, I hope this helps, and I encourage you to help increase the popularity of python-oauthlib on the Cheeseshop, so that we can one day finally kill off the long defunct python-oauth library.

Read more
pitti

just released a new PyGObject, for GNOME 3.7.2 which is due on Wednesday.

In this version PyGObject went through some major refactoring: Some 5.000 lines of static bindings were removed and replaced with proper introspection and some overrides for backwards compatibility, and the static/GI/overrides code structure was simplified. For the developer this means that you can now use the full GLib API, a lot of which was previously hidden by old and incomplete static bindings; also you can and should now use the officially documented GLib API instead of PyGObject’s static one, which has been marked as deprecated. For PyGObject itself this change means that the code structure is now a lot simpler to understand, all the bugs in the static GLib bindings are gone, and the GLib bindings will not go out of sync any more.

Lots of new tests were written to ensure that the API is backwards compatible, but experience teaches that ther is always the odd corner case which we did not cover. So if your code does not work any more with 3.7.2, please do report bugs.

Another important change is that if you build pygobject from source, it now defaults to using Python 3 if installed. As before, you can build for Python 2 with PYTHON=python2.7 or the new --with-python=python2.7 configure option.

This release also brings several marshalling fixes, docstring improvements, support for code coverage, and other bug fixes.

Thanks to all contributors!

Summary of changes (see changelog for complete details):

  • [API change] Drop almost all static GLib bindings and replace them with proper introspection. This gets rid of several cases where the PyGObject API was not matching the real GLib API, makes the full GLib API available through introspection, and makes the code smaller, easier to maintain. For backwards compatibility, overrides are provided to emulate the old static binding API, but this will throw a PyGIDeprecationWarning for the cases that diverge from the official API (in particular, GLib.io_add_watch() and GLib.child_watch_add() being called without a priority argument). (Martin Pitt, Simon Feltman)
  • [API change] Deprecate calling GLib API through the GObject namespace. This has always been a misnomer with introspection, and will be removed in a later version; for now this throws a PyGIDeprecationWarning.
  • [API change] Do not bind gobject_get_data() and gobject_set_data(). These have been deprecated for a cycle, now dropped entirely. (Steve Frécinaux) (#641944)
  • [API change] Deprecate void pointer fields as general PyObject storage. (Simon Feltman) (#683599)
  • Add support for GVariant properties (Martin Pitt)
  • Add type checking to GVariant argument assignment (Martin Pitt)
  • Fix marshalling of arrays of struct pointers to Python (Carlos Garnacho) (#678620)
  • Fix Gdk.Atom to have a proper str() and repr() (Martin Pitt) (#678620)
  • Make sure g_value_set_boxed does not cause a buffer overrun with GStrvs (Simon Feltman) (#688232)
  • Fix leaks with GValues holding boxed and object types (Simon Feltman) (#688137)
  • Add doc strings showing method signatures for gi methods (Simon Feltman) (#681967)
  • Set Property instance doc string and blurb to getter doc string (Simon Feltman) (#688025)
  • Add GObject.G_MINSSIZE (Martin Pitt)
  • Fix marshalling of GByteArrays (Martin Pitt)
  • Fix marshalling of ssize_t to smaller ints (Martin Pitt)
  • Add support for lcov code coverage, and add a lot of missing GIMarshallingTests and g-i Regress tests. (Martin Pitt)
  • pygi-convert: remove deprecated GLib ? GObject conversions (Jose Rostagno)
  • Add support for overriding GObject.Object (Simon Feltman) (#672727)
  • Add –with-python configure option (Martin Pitt)
  • Do not prefer unversioned “python” when configuring, as some distros have “python” as Python 3. Use Python 3 by default if available. Add –with-python configure option as an alternative to setting $PYTHON, whic is more discoverable. (Martin Pitt)
  • Fix property lookup in class hierarchy (Daniel Drake) (#686942)
  • Move property and signal creation into _class_init() (Martin Pitt) (#686149)
  • Fix duplicate symbols error on OSX (John Ralls)
  • [API add] Add get_introspection_module for getting un-overridden modules (Simon Feltman) (#686828)
  • Work around wrong 64 bit constants in GLib Gir (Martin Pitt) (#685022)
  • Mark GLib.Source.get_current_time() as deprecated (Martin Pitt)
  • Fix OverflowError in source_remove() (Martin Pitt) (#684526)

Read more
Michael Hall

Shortly after the Ubuntu App Showdown earlier this year, Didier Roche and Michael Terry kicked off a series of discussions about a ground-up re-write of Quickly.  Not only would this fix many of the complications app developers experienced during the Showdown competition, but it would also make it easier to write tools around Quickly itself.

Unfortunately, neither Didier nor Michael were going to have much time this cycle to work on the reboot.  We had a UDS session to discuss the initiative, but we were going to need community contributions in order to get it done.

JFDI

I was very excited about the prospects of a Quickly reboot, but knowing that the current maintainers weren’t going to have time to work on it was a bit of a concern.  So much so, that during my 9+ hour flight from Orlando to Copenhagen, I decided to have a go at it myself. Between the flight, a layover in Frankfurt without wifi, and a few late nights in the Bella Sky hotel, I had the start of something promising enough to present during the UDS session.  I was pleased that both Didier and Michael liked my approach, and gave me some very good feedback on where to take it next.  Add another 9+ hour flight home, and I had a foundation on which a reboot can begin.

Where is stands now

My code branch is now a part of the Quickly project on Launchpad, you can grab a copy of it by running bzr branch lp:quickly/reboot.  The code currently provides some basic command-line functionality (including shell completion), as well as base classes for Templates, Projects and Commands.  I’ve begun porting the ubuntu-application template, reusing the current project_root files, but built on the new foundation.  Currently only the ‘create’ and ‘run’ commands have been converted to the new object-oriented command class.

I also have examples showing how this new approach will allow template authors to easily sub-class Templates and Commands, by starting both a port of the ubuntu-cli template, and also creating an ubuntu-git-application template that uses git instead of bzr.

What comes next

This is only the very beginning of the reboot process, and there is still a massive amount of work to be done.  For starters, the whole thing needs to be converted from Python 2 to Python 3, which should be relatively easy except for one area that does some import trickery (to keep Templates as python modules, without having to install them to PYTHON_PATH).  The Command class also needs to gain argument parameters, so they can be easily introspected to see what arguments they can take on the command line.  And the whole thing needs to gain a structured meta-data output mechanism so that non-Python application can still query it for information about available templates, a project’s commands and their arguments.

Where you come in

As I said at the beginning of the post, this reboot can only succeed if it has community contributions.  The groundwork has been laid, but there’s a lot more work to be done than I can do myself.  Our 13.04 goal is to have all of the existing functionality and templates (with the exception of the Flash template) ported to the reboot.  I can use help with the inner-working of Quickly core, but I absolutely need help porting the existing templates.

The new Template and Command classes make this much easier (in my opinion, anyway), so it will mostly be a matter of copy/paste/tweak from the old commands to the new ones. In many cases, it will make sense to sub-class and re-use parts of one Template or Command in another, further reducing the amount of work.

Getting started

If you are interested in helping with this effort, or if you simply want to take the current work for a spin, the first thing you should do is grab the code (bzr branch lp:quickly/reboot).  You can call the quickly binary by running ./bin/quickly from within the project’s root.

Some things you can try are:

./bin/quickly create ubuntu-application /tmp/foo

This will create a new python-gtk project called ‘foo’ in /tmp/foo.  You can then call:

./bin/quickly -p /tmp/foo run

This will run the applicaiton.  Note that you can use -p /path/to/project to make the command run against a specific project, without having to actually be in that directory.  If you are in that directory, you won’t need to use -p (but you will need to give the full path to the new quickly binary).

If you are interested in the templates, they are in ./data/templates/, each folder name corresponds to a template name.  The code will look for a class called Template in the base python namespace for the template (in ubuntu-application/__init__.py for example), which must be a subclass of the BaseTemplate class.  You don’t have to define the class there, but you do need to import it there.  Commands are added to the Template class definition, they can take arguments at the time you define them (see code for examples), and their .run() method will be called when invoked from the command line.  Unlike Templates, Commands can be defined anywhere, with any name, as long as they subclass BaseCommand and are attached to a template.

Read more