Canonical Voices

Posts tagged with 'python'

facundo


Un tema que se ha visto varias veces, tanto en la lista de PyAr como en la vida real, es que los desarrolladores que no estuvieron involucrados en proyectos grandes, o que sólo estuvieron metidos en uno o dos sistemas (más allá el tamaño), no saben muy bien qué estructura, o qué forma, darle a un proyecto nuevo.

Es totalmente comprensible. La estructura a tener depende de muchos factores: de la complejidad del proyecto, de cuan listo lo deja uno para empaquetarlo, de la prolijidad del desarrollador, etc.

Lo notable es que (en mi experiencia) el proyecto aunque nazca pequeño, siempre conviene que esté ordenado. Y que la forma de ordenarlo, qué estructura darle, cambia en función de las necesidades (como decía arriba) pero que siempre es bueno tener alguna.

En función de todo esto es que paso a contarles qué estructura tiene hoy Encuentro. No es la mejor del mundo, pero es la que a mí me funciona en este y otros proyectos. Y es una buena base como para que alguien que no tiene idea sepa para qué lado ir ordenando los tantos.

El código en sí lo pueden ver acá o si tienen instalado bazaar pueden hacer bzr branch lp:encuentro y exploran el código de forma local.

Bueno, a los bifes.


El código "útil" en sí

Tenemos dos archivos y varios directorios...

- test: Este es un script que básicamente ejecuta todas las comprobaciones que necesitamos para asegurarnos que el proyecto está "verde". En el caso de encuentro, corre las pruebas de unidad (las que están en el directorio tests, ver abajo), luego corre un verificador estático de código genérico (pylint) y finalmente otro verificador puntual para pep 8.

- version.txt: La versión del programa. La tengo separada sólo por consistencia: me gusta que esté en un sólo lado así es la misma para todos los que la necesitan (setup.py, para mostrarlo al arrancar, o cuando el usuario pide info del programa, etc).

- bin/: Aquí (normalmente) hay un sólo archivo, con el nombre del proyecto: encuentro. Este es el script de inicio, el que arranca todo el sistema ya sea cuando lo ejecutamos desde el proyecto mismo, desde un tarball descomprimido, e incluso es el que va a parar a /usr/bin cuando se instala. Este es el único que es ejecutable, el resto del sistema son sólo módulos.

- encuentro/: Es el directorio principal del proyecto (por eso el nombre). Acá tenemos todo el código "de producción" del proyecto, con su estructura interna. Por lo pronto, en este mismo directorio están todos los módulos que tienen que ver con el funcionamiento interno de Encuentro.

- encuentro/ui/: Aquí tenemos todo el código que necesitamos para armar la Interfaz del Usuario del programa. También tiene que ver con el funcionamiento interno de Encuentro, pero es sólo el manejo de la interfaz. La separación de qué va aquí o qué va directamente en encuentro/ a veces es complicada.

- encuentro/ui/media/: Todas las imágenes, audios, etc, que necesitamos para que funcione la UI en sí.

- encuentro/logos/: También imágenes, pero que se usan como identificación del programa en sí. Aunque algunas se usan en la parte de UI, están todas acá porque también se usan en otros contextos (por ejemplo, en la instalación del paquete).

- tests/: Los tests de unidad del proyecto, normalmente un montón de archivos cuyo nombre arranca con "test_" pero también pueden haber otros (módulos o no) para dar soporte a las pruebas.


Otros directorios

Estos son directorios puntuales que tengo para Encuentro. Algunos se repiten con otros proyectos, otros no.

- qtreactor: El módulo de integración entre Qt (el framework de interfaz gráfica que estoy usando) y Twisted (una biblioteca asincrónica que uso para trabajar con la red).

- server: Cuando le decimos al programa "local" de Encuentro que actualice los episodios, se baja algunos archivos comprimidos de mi server, con toda la metadata. Estos archivos comprimidos se generan una vez al día a partir de los sitios webs de Encuentro, Conectate, BACUA, etc. El código para realizar todo esto está en este directorio.

- web: Todos los archivos necesarios para montar el sitio web del proyecto.

- windows: Imágenes, configuraciones, y explicaciones necesarias para armar el .exe en Windows y luego armar con eso el instalador final que se distribuye.


Otros archivos

Estos son otros archivos que no tienen demasiada relación entre sí, pero que son importantes en distintos momentos de la vida del proyecto:

- AUTHORS, COPYING: Info legal: cuales son las personas que participaron del proyecto, y la licencia del mismo.

- LEEME.txt, README.txt, AYUDA.txt: Textos de ayuda para la persona que llega por primera vez al proyecto (viéndolo desde los archivos fuente). Está en dos idiomas, pero como Encuentro es inherentemente para personas que hablan castellano, el LEEME es el que tiene la info posta.

- anuncio.txt, pasos_release.txt: Recordatorios y textos preparados para mí (o para el que haga la release del proyecto... que vengo siendo siempre yo, :p).

- pylintrc: Un archivo de configuración para el verificador estático de código que mencionaba arriba.

- setup.py, MANIFEST.in: Script principal de empaquetamiento e instalación, más un archivo que podríamos decir de configuración del mismo.

- encuentro.desktop, source_encuentro.py: Dos archivitos necesarios en sistemas Debian/Ubuntu (al menos). El primero le pasa al sistema info para poner el programa en el menú del usuario, y el otro es usado en caso de que el programa crashee, para informar automáticamente del problema.

Read more
Barry Warsaw

I'm writing a bunch of new code these days for Ubuntu Touch's Image Based Upgrade system.  Think of it essentially as Ubuntu Touch's version of upgrading the phone/tablet (affectionately called phablet) operating system in a bulk way rather than piecemeal apt-gets the way you do it on a traditional Ubuntu desktop or server.  One of the key differences is that a phone has to detour through a reboot in order to apply an upgrade since its Ubuntu root file system is mounted read-only during the user session.

Anyway, those details aren't the focus of this article.  Instead, just realize that because it's a pile of new code, and because we want to rid ourselves of Python 2, at least on the phablet image if not everywhere else in Ubuntu, I am prototyping all this in Python 3, and specifically 3.3.  This means that I can use all the latest and greatest cool stuff in the most recent stable Python release.  And man, is there a lot of cool stuff!

One module in particular that I'm especially fond of is contextlibContext managers are objects implementing the protocol behind the with statement, and they are typically used to guarantee that some resource is cleaned up properly, even in the event of error conditions.  When you see code like this:

with open(somefile) as fp:
    data = fp.read()

you are invoking a context manager.  Python was clever enough to make file objects support the context manager protocol so that you never have to explicitly close the file; that happens automatically when the with statement completes, regardless of whether the code inside the with statement succeeds or raises an exception.

It's also very easy to define your own context managers to properly handle other kinds of resources.  I won't go into too much detail here, because this is all well-established; the with statement has been, er, with us since Python 2.5.

You may be familiar with the contextlib module because of the @contextmanager decorator it provides.  This makes it trivial to define a new context manager without having to deal with all the intricacies of the protocol.  For example, here's how you would implement a context manager that temporarily changes the current working directory:

import os
from contextlib import contextmanager

@contextmanager
def chdir(dir):
    cwd = os.getcwd()
    try:
        os.chdir(dir)
        yield
    finally:
        os.chdir(cwd)

In this example, the yield cedes control back to the body of the with statement, and when that completes, the code after the yield is executed.  Because the yield is wrapped inside a try/finally, it is guaranteed that the original working directory is restored.  You would use this code like so:

with chdir('/tmp'):
    print(os.getcwd())

So far, so good, but this is nothing revolutionary.  Python 3.3 brings additional awesomeness to contextlib by way of the new ExitStack class.

The documentation for ExitStack is a bit dense, and even the examples didn't originally make it clear to me how amazing this new API is.  In my opinion, this is so powerful, it changes completely the way you think about deploying safe code.

So what is an ExitStack?  One way to think about it is as an extensible context manager.  It's used in with statements just like any other context manager:

from contextlib import ExitStack
with ExitStack() as stack:
    # do some magical stuff

Just like any other context manager, the ExitStack's "exit" code is guaranteed to be run at the end of the with statement.  It's the programmable extensibility of the ExitStack where the cool stuff happens.

The first interesting method of an ExitStack you might use is the callback() method.  Let's say for example that in your with statement, you are creating a temporary directory and you want to make sure that temporary directory gets deleted when the with statement exits.  You could do something like this:

import shutil, tempfile
with ExitStack() as stack:
    tempdir = tempfile.mkdtemp()
    stack.callback(shutil.rmtree, tempdir)

Now, when the with statement completes, it calls all of its callbacks, which includes removing the temporary directory.

So, what's the big deal?  Let's say you're actually creating three temporary directories and any of those calls could fail.  To guarantee that all successfully created directories are deleted at the end of the with statement, regardless of whether an exception occurred in the middle, you could do this:

with ExitStack() as stack:
    tempdirs = []
    for i in range(3):
        tempdir = tempfile.mkdtemp()
        stack.callback(shutil.rmtree, tempdir)
        tempdirs.append(tempdir)
    # Do something with the tempdirs

If you knew statically that you wanted three temporary directories, you could set this up with nested with statements, or a single with statement containing multiple backslash-separated targets, but that gets unwieldy very quickly.  And besides, that's impossible if you only know the number of directories you need dynamically at run time.  On the other hand, the ExitStack makes it easy to guarantee everything gets cleaned up and there are no leaks.

That's powerful enough, but it's not all you can do!  Another very useful method is enter_context().

Let's say that you are opening a bunch of files and you want the following behavior: if all of the files open successfully, you want to do something with them, but if any of them fail to open, you want to make sure that the ones that did get open are guaranteed to get closed.  Using ExitStack.enter_context() you can write code like this:

files = []
with ExitStack() as stack:
    for filename in filenames:
        # Open the file and automatically add its context manager to the stack.
        # enter_context() returns the passed in context manager, i.e. the 
        # file object.
        fp = stack.enter_context(open(filename))
        files.append(fp)
    # Capture the close method, but do not call it yet.
    close_all_files = stack.pop_all().close

(Note that the contextlib documentation contains a more efficient, but denser way of writing the same thing.)

So what's going on here?  First, the open(filename) does what it always does of course, it opens the file and returns a file object, which is also a context manager.  However, instead of using that file object in a with statement, we add it to the ExitStack by passing it to the enter_context() method.  For convenience, this method returns the passed in object.

So what happens if one of the open() calls fail before the loop completes?  The with statement will exit as normal and the ExitStack will exit all the context managers it knows about.  In other words, all the files that were successfully opened will get closed.  Thus, in an error condition, you will be left with no open files and no leaked file descriptors, etc.

What happens if the loop completes and all files got opened successfully?  Ah, that's where the next bit of goodness comes into play: the ExitStack's pop_all() method.

pop_all() creates a new ExitStack, and populates it from the original ExitStack, removing all the context managers from the original ExitStack.  So, after stack.pop_all() completes, the original ExitStack, i.e. the one used in the with statement, is now empty.  When the with statement exits, the original ExitStack contains no context managers so none of the files are closed.

Well, then, how do you close all the files once you're done with them?  That's the last bit of magic.  ExitStacks have a .close() method which unwinds all the registered context managers and callbacks and invokes their exit functionality.  So, after you're finally done with all the files and you want to clean everything up, you would just do:

close_all_files()

And that's it.

Hopefully that all makes sense.  I know it took a while to sink in for me, but now that it has, it's clear the enormous power this gives you.  You can write much safer code, in the sense that it's easier to ensure much better guarantees that your resources are cleaned up at the right time.

The real power comes when you have many different disparate resources to clean up for a particular operation.  For example, in the test suite for the Image Based Upgrader, I have a test where I need to create a temporary directory and start an HTTP server in a thread.  Roughly, my code looks like this:

@classmethod
def setUpClass(cls):
    cls._cleaner = ExitStack()
    try:
        cls._serverdir = tempfile.mkdtemp()
        cls._cleaner.callback(shutil.rmtree, cls._serverdir)
        # ...
        cls._stop = make_http_server(cls._serverdir)
        cls._cleaner.callback(cls._stop)
    except:
        cls._cleaner.pop_all().close()
        raise

@classmethod
def tearDownClass(cls):
    cls._cleaner.close()

Notice there's no with statement there at all. :)   This is because the resources must remain open until tearDownClass() is called, unless some exception occurs during the setUpClass().  If that happens, the bare except will ensure that all the context managers are properly closed, leaving the original ExitStack empty.  (The bare except is acceptable here because the exception is re-raised after the resources are cleaned up.)  Even though the exception will prevent the tearDownClass() from being called, it's still safe to do so in case it is called for some odd reason, because the original ExitStack is empty.

But if no exception occurs, the original ExitStack will contain all the context managers that need to be closed, and calling .close() on it in the tearDownClass() does exactly that.

I have one more example from my recent code.  Here, I need to create a GPG context (the details are unimportant), and then use that context to verify the detached signature of a file.  If the signature matches, then everything's good, but if not, then I want to raise an exception and throw away both the data file and the signature (i.e. .asc) file.  Here's the code:

with ExitStack() as stack:
    ctx = stack.enter_context(Context(pubkey_path))
    if not ctx.verify(asc_path, channels_path):
        # The signature did not verify, so arrange for the .json and .asc
        # files to be removed before we raise the exception.
        stack.callback(os.remove, channels_path)
        stack.callback(os.remove, asc_path)
        raise FileNotFoundError


Here we create the GPG context, which itself is a context manager, but instead of using it in a with statement, we add it to the ExitStack.  Then we verify the detached signature (asc_path) of a data file (channels_path), and only arrange to remove those files if the verification fails.  When the FileNotFoundError is raised, the ExitStack in the with statement unwinds, removing both files and closing the GPG context.  Of course, if the signature matches, only the GPG context is closed -- the channels_path and asc_path files are not removed.

You can see how an ExitStack actually functions as a fairly generic resource manager!

To me, this revolutionizes the management of external resources.  The new ExitStack object, and the methods and semantics it exposes, make it so much easier to manage those resources, guaranteeing that they get cleaned up at the right time, once and only once, regardless of whether errors occur or not.

ExitStack takes the already powerful concept of context managers and turns it up to 11.  There's more you can do, and it's worth spending some time reading the contextlib documentation in Python 3.3, especially the examples and recipes.

As I mentioned on Twitter, it's features like this that make using Python 2 seem downright barbaric.

Read more
pitti

Time for the first PyGObject release for GNOME 3.9.x! This release brings the performance optimizations (thanks to Daniel Drake), quite a lot of internal code cleanup, and various bug fixes.

Thanks to all contributors!

  • gtk-demo: Wrap description strings at 80 characters (Simon Feltman) (#698547)
  • gtk-demo: Use textwrap to reformat description for Gtk.TextView (Simon Feltman) (#698547)
  • gtk-demo: Use GtkSource.View for showing source code (Simon Feltman) (#698547)
  • Use correct class for GtkEditable’s get_selection_bounds() function (Mike Ruprecht) (#699096)
  • Test results of g_base_info_get_name for NULL (Simon Feltman) (#698829)
  • Remove g_type_init conditional call (Jose Rostagno) (#698763)
  • Update deps versions also in README (Jose Rostagno) (#698763)
  • Drop compat code for old python version (Jose Rostagno) (#698763)
  • Remove duplicate call to _gi.Repository.require() (Niklas Koep) (#698797)
  • Add ObjectInfo.get_class_struct() (Johan Dahlin) (#685218)
  • Change interpretation of NULL pointer field from None to 0 (Simon Feltman) (#698366)
  • Do not build tests until needed (Sobhan Mohammadpour) (#698444)
  • pygi-convert: Support toolbar styles (Kai Willadsen) (#698477)
  • pygi-convert: Support new-style constructors for Gio.File (Kai Willadsen) (#698477)
  • pygi-convert: Add some support for recent manager constructs (Kai Willadsen) (#698477)
  • pygi-convert: Check for double quote in require statement (Kai Willadsen) (#698477)
  • pygi-convert: Don’t transform arbitrary keysym imports (Kai Willadsen) (#698477)
  • Remove Python keyword escapement in Repository.find_by_name (Simon Feltman) (#697363)
  • Optimize signal lookup in gi repository (Daniel Drake) (#696143)
  • Optimize connection of Python-implemented signals (Daniel Drake) (#696143)
  • Consolidate signal connection code (Daniel Drake) (#696143)
  • Fix setting of struct property values (Daniel Drake)
  • Optimize property get/set when using GObject.props (Daniel Drake) (#696143)
  • configure.ac: Fix PYTHON_SO with Python3.3 (Christoph Reiter) (#696646)
  • Simplify registration of custom types (Daniel Drake) (#696143)
  • pygi-convert.sh: Add GStreamer rules (Christoph Reiter) (#697951)
  • pygi-convert: Add rule for TreeModelFlags (Jussi Kukkonen)
  • Unify interface struct to Python GI marshaling code (Simon Feltman) (#693405)
  • Unify Python interface struct to GI marshaling code (Simon Feltman) (#693405)
  • Unify Python float and double to GI marshaling code (Simon Feltman) (#693405)
  • Unify filename to Python GI marshaling code (Simon Feltman) (#693405)
  • Unify utf8 to Python GI marshaling code (Simon Feltman) (#693405)
  • Unify unichar to Python GI marshaling code (Simon Feltman) (#693405)
  • Unify Python unicode to filename GI marshaling code (Simon Feltman) (#693405)
  • Unify Python unicode to utf8 GI marshaling code (Simon Feltman) (#693405)
  • Unify Python unicode to unichar GI marshaling code (Simon Feltman) (#693405)
  • Fix enum and flags marshaling type assumptions (Simon Feltman)
  • Make AM_CHECK_PYTHON_LIBS not depend on AM_CHECK_PYTHON_HEADERS (Christoph Reiter) (#696648)
  • Use distutils.sysconfig to retrieve the python include path. (Christoph Reiter) (#696648)
  • Use g_strdup() consistently (Martin Pitt) (#696650)
  • Support PEP 3149 (ABI version tagged .so files) (Christoph Reiter) (#696646)
  • Fix stack corruption due to incorrect format for argument parser (Simon Feltman) (#696892)
  • Deprecate GLib and GObject threads_init (Simon Feltman) (#686914)
  • Drop support for Python 2.6 (Martin Pitt)
  • Remove static PollFD bindings (Martin Pitt) (#686795)
  • Drop test skipping due to too old g-i (Martin Pitt)
  • Bump glib and g-i dependencies (Martin Pitt)

Read more
facundo

Actualizaciones


Estos días hice dos releases rápidos.

El primero fue de LauncherPosta, porque resulta que en Ubuntu Raring crasheaba mal. O sea, no tiraba un traceback: crasheaba.

¿Lo peor? Es un problema de la librería PyGtk, de cómo está implementado para Gtk 3. ¿Lo peor más peor de todos los peores? Es por diseño, y les parece bien que crashee a la mierda en lugar de tirar un traceback decente (miren el bug que abrí y lo que me respondieron).

En fin, esto refuerza lo que les decía que Gtk3 me gustaba menos y menos y me estoy pasando a Qt.

BTW, de LauncherPosta liberé la versión 1.0, con el casi único cambio de soportar mejor el toqueteo de la configuración del systray bajo Unity (un pedazo de código que luego les compartiré más separadamente).

El segundo release fue de Enjuewemela.

Hace rato que no sacaba una versión del jueguito. Es que aunque le había hecho un montón de correcciones, había un gran feature que estaba esperando: el replayer.

¿Qué es el replayer? Es un modo de ejecución de Enjuewemela que le decís que te reproduzca un juego anterior (le tenés que pasar el log que generó la jugada), y podés ir viendo exactamente el juego que hiciste, avanzando y retrocediendo con las flechas. Esto más que nada lo hice porque era necesario para poder detectar algunos crashes raros, y porque era divertido de hacer, :)

Los cambios más interesantes para esta versión 0.5, aparte de la habilidad de "re-jugar un log", son:

- Alienta cuando hay múltiples coincidencias
- Cambia el tablero cuando cambia de nivel
- Múltiple fondos
- Correcto manejo de los highscores
- Otras pequeñas mejoras y un montón de correcciones.

La verdad es que estoy un poco harto de Enjuewemela. Hay que ponerle un montón de laburo para "hacerlo más lindo" al juego, y la verdad es que "hacerlo lindo" no es algo que me divierta.

Así que creo que sacaré un gran último feature, y luego creo que lo paso a mantenimiento, porque tengo otros proyectos bastante más interesantes para empujar.

Ya los comentaré por este mismo canal. Stay tuned.

Read more
niemeyer

Last week I was part of a rant with a couple of coworkers around the fact Go handles errors for expected scenarios by returning an error value instead of using exceptions or a similar mechanism. This is a rather controversial topic because people have grown used to having errors out of their way via exceptions, and Go brings back an improved version of a well known pattern previously adopted by a number of languages — including C — where errors are communicated via return values. This means that errors are in the programmer’s face and have to be dealt with all the time. In addition, the controversy extends towards the fact that, in languages with exceptions, every unadorned error comes with a full traceback of what happened and where, which in some cases is convenient.

All this convenience has a cost, though, which is rather simple to summarize:

Exceptions teach developers to not care about errors.

A sad corollary is that this is relevant even if you are a brilliant developer, as you’ll be affected by the world around you being lenient towards error handling. The problem will show up in the libraries that you import, in the applications that are sitting in your desktop, and in the servers that back your data as well.

Raymond Chen described the issue back in 2004 as:

Writing correct code in the exception-throwing model is in a sense harder than in an error-code model, since anything can fail, and you have to be ready for it. In an error-code model, it’s obvious when you have to check for errors: When you get an error code. In an exception model, you just have to know that errors can occur anywhere.

In other words, in an error-code model, it is obvious when somebody failed to handle an error: They didn’t check the error code. But in an exception-throwing model, it is not obvious from looking at the code whether somebody handled the error, since the error is not explicit.
(…)
When you’re writing code, do you think about what the consequences of an exception would be if it were raised by each line of code? You have to do this if you intend to write correct code.

That’s exactly right. Every line that may raise an exception holds a hidden “else” branch for the error scenario that is very easy to forget about. Even if it sounds like a pointless repetitive task to be entering that error handling code, the exercise of writing it down forces developers to keep the alternative scenario in mind, and pretty often it doesn’t end up empty.

It isn’t the first time I write about that, and given the controversy that surrounds these claims, I generally try to find one or two examples that bring the issue home. So here is the best example I could find today, within the pty module of Python’s 3.3 standard library:

def spawn(argv, master_read=_read, stdin_read=_read):
    """Create a spawned process."""
    if type(argv) == type(''):
        argv = (argv,)
    pid, master_fd = fork()
    if pid == CHILD:
        os.execlp(argv[0], *argv)
    (...)

Every time someone calls this logic with an improper executable in argv there will be a new Python process lying around, uncollected, and unknown to the application, because execlp will fail, and the process just forked will be disregarded. It doesn’t matter if a client of that module catches that exception or not. It’s too late. The local duty wasn’t done. Of course, the bug is trivial to fix by adding a try/except within the spawn function itself. The problem, though, is that this logic looked fine for everybody that ever looked at that function since 1994 when Guido van Rossum first committed it!

Here is another interesting one:

$ make clean
Sorry, command-not-found has crashed! Please file a bug report at:

https://bugs.launchpad.net/command-not-found/+filebug

Please include the following information with the report:

command-not-found version: 0.3
Python version: 3.2.3 final 0
Distributor ID: Ubuntu
Description:    Ubuntu 13.04
Release:        13.04
Codename:       raring
Exception information:

unsupported locale setting
Traceback (most recent call last):
  File "/.../CommandNotFound/util.py", line 24, in crash_guard
    callback()
  File "/usr/lib/command-not-found", line 69, in main
    enable_i18n()
  File "/usr/lib/command-not-found", line 40, in enable_i18n
    locale.setlocale(locale.LC_ALL, '')
  File "/usr/lib/python3.2/locale.py", line 541, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

That’s a pretty harsh crash for the lack of locale data in a system-level application that is, ironically, supposed to tell users what packages to install when commands are missing. Note that at the top of the stack there’s a reference to crash_guard. This function has the intent of catching all exceptions right at the edge of the call stack, and displaying a detailed system specification and traceback to aid in fixing the problem.

Such “parachute catching” is a fairly common pattern in exception-oriented programming and tends to give developers the false sense of having good error handling within the application. Rather than actually guarding the application, though, it’s just a useful way to crash. The proper thing to have done in the case above would be to print a warning, if at all, and then let the program run as usual. This would have been achieved by simply wrapping that one line as in:

try:
    locale.setlocale(locale.LC_ALL, '')
except Exception as e:
    print("Cannot change locale:", e)

Clearly, it was easy to handle that one. The problem, again, is that it was very natural to not do it in the first place. In fact, it’s more than natural: it actually feels good to not be looking at the error path. It’s less code, more linear, and what’s left is the most desired outcome.

The consequence, unfortunately, is that we’re immersing ourselves in a world of brittle software and pretty whales. Although more verbose, the error result style builds the correct mindset: does that function or method have a possible error outcome? How is it being handled? Is that system-interacting function not returning an error? What is being done with the problem that, of course, can happen?

A surprising number of crashes and plain misbehavior is a result of such unconscious negligence.

Read more
facundo

Migrando Encuentro a PyQt


Este no es un post sobre Encuentro precisamente, sino sobre la experiencia de migrar Encuentro a Qt.

O, mejor dicho, a PyQt. ¿Qué es PyQt? Sencillo: una capa de unión para poder usar Qt desde Python. ¿Y qué es Qt? Qt es una biblioteca multiplataforma para desarrollar aplicaciones con interfaz gráfica. En otras palabras, una biblioteca para hacer las ventanas, botones, y todo eso que arma la interfaz gráfica de un programa de escritorio.

Con esa descripciones no tendríamos diferencia entre PyQt/Qt y PyGtk/Gtk, que es lo que usaba Encuentro hasta ahora. Entonces, ¿por qué migrar?

Son varias las razones... pero principalmente porque empaquetar PyGtk en un .exe es un dolor de muelas, y eso llevó a que la última versión que corre en windows es la que no funciona porque cambió todo el backend web (cuando los videos pasaron de ser hosteados por Encuentro a estar en Conectate). En otras palabras: la última versión de Encuentro que corre en windows no sirve para nada, y básicamente es culpa de Gtk.

Otras razón de menor importancia es que no me gustó como Gtk evoluciona. El futuro del framework es Gtk3, y ya estuve tirando código para usarlo, y lo que usé me gustó menos que Gtk2, así que me pareció un buen momento de cambiar. Finalmente, es una buena excusa para aprender Qt, ;)

Qt

En fin. La migración ya está terminada, pude hacer en Qt todo lo que tenía que hacer en función de la interfaz de Encuentro. ¿Qué me pareció? Bueno, las sensaciones son varias.

Me gustó Qt, mucho más cuadradito, más pytónico especialmente en la versión 4 que es la que yo estoy usando. Aunque la mayoría del código es muy similar, hay varias cosas que son más sencillas que en Gtk, aunque no todas, y hay bordes que limar.

(En este punto quiero aclarar que en ningún punto usé Qt Creator, el constructor gráfico de interfaces, sino que hice todo todo a mano, lo cual me permitió meterme bien adentro del framework y aprender mucho de su estructura subyacente.)

Un ejemplo de borde sencillo: no se puede saber si una señal está conectada o no. Entonces, por ejemplo, yo tengo un botón que muta de función, y a veces tiene que tener una señal conectada, y a veces otra (para que al hacer click haga una cosa u otra; en particular en el contexto de Encuentro: que el botón dispare la descarga del episodio, o la reproducción). Cuando el contexto cambia y se hace la revisión del estado del botón, no puedo decirle que desconecte cualquier señal que tenga, o preguntar qué señal tiene y desconectarla, tengo que (a mano) guardar en algún lado la señal que había conectado antes para desconectarla y conectar la nueva que corresponda.

Un ejemplo de algo complicado de hacer en Qt (que en Gtk es trivial): QTreeWidget no soporta HTML en el texto. Esto es, la habilidad de insertar tags para cambiar el tipo de texto (en el caso de Encuentro, yo lo necesito para resaltar en amarillo el fondo de las letras que coinciden con lo que el usuario ingresó en el campo de filtrar). Finalmente lo pude hacer, adaptando un ejemplo que Roberto Alsina encontró en la web, pero lo hace más lento, le agrega pequeños glitches que aunque no me joden, no deberían estar, y me mete a mí código oscurísimo que no es ni cerca de fácil de debuguear.

Por último, la integración con Twisted no es trivial. Hay cosas que en Encuentro están hechas con Twisted que podrían hacerse con herramientas más propias de Qt, sí, pero en este caso de migración, ya estaban hechas en Twisted y mi idea era aprovecharlas. Pero tuve que meter en el proyecto todo un módulo de integración y levantar la aplicación y cerrarla de una manera no trivial (y que me costó tiempo y sudor hacer que funcione correctamente, especialmente la parte de cerrar la aplicación, porque tuve que apagar los hilos de Twisted a mano).

La conclusión es que Qt me gustó bastante, y aunque extraño algunas cositas de Gtk, seguramente mis nuevos proyectos estarán usando PyQt.

Read more
niemeyer

This weekend the proper environment settled out for sorting a pet peeve that shows up every once in a while when coding: writing logic that interacts with other applications in the system via their stdin and stdout streams is often more involved than it should be, which seems pretty ironic when sitting in front of a Unix-like system.

Rather than going over the trouble of setting up pipes and hooking them up in a custom way, often applications end up just delegating the job to /bin/sh, which is not ideal for a number of reasons: argument formatting isn’t straightforward, injecting custom application-defined logic is hard, which means even simple tasks that might be easily achieved by the language end up shelling out to further external applications, and so on.

In an attempt to address that, I’ve spent some time working on an experimental Go package that is being released today: pipe.

I hope you like it as well, and please drop me a note if you find any issues.

Read more
niemeyer

There are a number of common misconceptions in software development surrounding the idea of concurrency. This has been coming for decades, and some of the issues have just been reinforced one more time in an otherwise interesting post in LinkedIn’s engineering blog that recommends their development framework.

Such issues may be observed throughout the post, but can be elucidated via this short paragraph:

As we saw with the Scala and JavaScript examples above, for very simple cases, the Evented (asynchronous) code is generally more complicated than Threaded (synchronous) code. However, in most real-world scenarios, you’ll have to make several I/O calls, and to make them fast, you’ll need to do them in parallel.

At a glance, this may look like a sane proposition. There’s agreement that an asynchronous API or framework is one that does not block the flow of execution when faced with a task that has a long or non-predictable deadline, and this coding style is harder for human beings to get right. For example, if you see code such as:

data = read(filename)

There’s less brain work to process and build on it than so called asynchronous logic such as:

read(filename, callback)

It’s also true that there are important interfaces that follow the asynchronous style to prevent resource waste. Some of these exist in the kernel I/O API.

So what’s the issue, then?

There are a few. The first one is the statement that to make I/O scale you have to do it in parallel. That’s clearly not true. Scalable I/O requires your program to not waste an irresponsible amount of memory and CPU per operation. This may be achieved with simple concurrent techniques, and concurrency is not parallelism.

This drives to the next point, which is the strong association between synchronous programming and threads. You can have synchronous programming, and its simplified mental model, without operating system threads. This can be done by having a compiler and runtime that is mindful about performance and resource consumption, building on the efficient interfaces to implement its abstractions.

These ideas have also been covered in this paper from 2003, including benchmark results that debunk the performance myth. What seems most interesting about this paper is that it theorizes such a compiler and runtime that would allow “overcom[ing] limitations in current threads packages and improv[ing] safety, programmer productivity, and performance”, by using techniques such as dynamic stack growth, stack moving, cheaper synchronization, and compile-time data race detection.

That exact mix, including all of the properties described in the paper, are available today in the Go language. You can have synchronous programming, concurrency, parallelism, and performance. We live in the future.

Read more
pitti

I just pushed out a new python-dbusmock release 0.6.

Calling a method on the mock now emits a MethodCalled signal on the org.freedesktop.DBus.Mock interface. In some cases this is easier to track than parsing the mock’s log or using GetMethodCalls. Thanks to Lars Uebernickel for this.

DBusMockObject.AddTemplate() and DBusTestCase.spawn_server_template() can now load local templates from your own project by specifying a path to a *.py file as template name. Thanks to Lucas De Marchi for this feature.

I also wrote a quite comprehensive template for systemd’s logind. It stubs out the power management functionality as well as user/seat/session objects, and is convincing enough for loginctl. Some bits like AttachDevice is missing, as this sounds unlikely to be required for D-BUS mock tests, but please let me know if you need anything else.

The mock processes now terminate automatically if their connected D-BUS goes down, as advertised in the documentation.

You can get the new tarball from Launchpad, and I uploaded it to Debian experimental now.

Enjoy!

Read more
pitti

I just released a new PyGObject for GNOME 3.7.92. This fixes a couple of crashes and marshalling errors again, but most importantly got a change to automatically mute the PyGIDeprecationWarnings for stable versions. Please run pythonX.X with the -Wd option to still be able to see them.

We got through all our bugs that were milestoned for GNOME 3.8 and don’t want to or plan to introduce any major behavioural change at this point, so barring catastrophes this is what will be in GNOME 3.8.0.

Thanks to all contributors!

  • Fix stack smasher when marshaling enums as a vfunc return value (Simon Feltman) (#637832)
  • Change base class of PyGIDeprecationWarning based on minor version (Simon Feltman) (#696011)
  • autogen.sh: Source gnome-autogen to fix out of source builddir (Alban Browaeys) (#694889)
  • pygtkcompat: Make gdk.Window.get_geometry return tuple of 5 (Simon Feltman)
  • pygtkcompat: Initialize hint to zero in set_geometry_hints (Simon Feltman)
  • Remove incorrect bounds check with property helper flags (Simon Feltman)
  • Fix crash when setting property of type object to an incorrect type (Simon Feltman) (#695420)
  • Remove skipping of object property tests (Simon Feltman) (#695420)
  • Give more informative error when setting property to incorrect type (Simon Feltman) (#695420)

Read more
pitti

I just found out that PyGObject 3.7.91 as released yesterday breaks GEdit plugins. I just pushed out 3.7.91.1 to unbreak this again, sorry about that!

Read more
pitti

I just released a new PyGObject for GNOME 3.7.91. This brings some marshalling fixes, plugs tons of memory leaks, and now raises a Python DeprecationWarning when your code calls a method which is marked as deprecated in the typelib. Please note that Python hides them by default, so if you are interested in those you need to run python with the -Wd option.

Thanks to all contributors!

  • Fix many memory leaks (#675726, #693402, #691501, #510511, #672224, and several more which are detected by our test suite) (Martin Pitt)
  • Dot not clobber original Gdk/Gtk functions with overrides (Martin Pitt) (#686835)
  • Optimize GValue.get/set_value by setting GValue.g_type to a local (Simon Feltman) (#694857)
  • Run tests with G_SLICE=debug_blocks (Martin Pitt) (#691501)
  • Add override helper for stripping boolean returns (Martin Pitt) (#694431)
  • Drop obsolete pygobject_register_sinkfunc() declaration (Martin Pitt) (#639849)
  • Fix marshalling of C arrays with explicit length in signal arguments (Martin Pitt) (#662241)
  • Fix signedness, overflow checking, and 32 bit overflow of GFlags (Martin Pitt) (#693121)
  • gi/pygi-marshal-from-py.c: Fix build on Visual C++ (Chun-wei Fan) (#692856)
  • Raise DeprecationWarning on deprecated callables (Martin Pitt) (#665084)
  • pygtkcompat: Add Widget.window, scroll_to_mark, and window methods (Simon Feltman) (#694067)
  • pygtkcompat: Add Gtk.Window.set_geometry_hints which accepts keyword arguments (Simon Feltman) (#694067)
  • Ship pygobject.doap for autogen.sh (Martin Pitt) (#694591)
  • Fix crashes in various GObject signal handler functions (Simon Feltman) (#633927)
  • pygi-closure: Protect the GSList prepend with the GIL (Olivier Crête) (#684060)
  • generictreemodel: Fix bad default return type for get_column_type (Simon Feltman)

Read more
facundo


Finalmente terminé de armar todo para liberar al mundo la versión traducida al español del tutorial de Python 3.

Realmente había terminado de traducirlo hace un par de semanas. Pero (aunque la generación del HTML fue bastante directo), generar el PDF me trajo bastante problemas.

Por un lado, el PDF generado por inkscape para los headers bonitos que tenemos es incorrecto en algunos detalles, así que tuve que meter un hack en el proyecto pdfrw para evitar que explote con ese problema. Luego habían problemas con páginas en blanco al final de capítulos que tuvieran footnotes, así que metí un workaround en rst2pdf para que reportlab no la cague al generar el PDF. Y dos o tres detalles más, pero menores (gracias Roberto Alsina por toda la ayuda en este bardo). Todos los parches (para poder reproducir la generación de algo copado) están en el README del proyecto.

También tuve que renovar el cómo presentábamos el tutorial, porque ahora tenemos el mismo para las versiones 2 y 3 de Python. Entonces ahora puse que la URL principal del tutorial apunte a un HTML muy bonito (gracias Rodrigo Bistolfi) que te deja elegir si querés ir qué versión del tutorial, tanto en Python 2 o 3, como en su formato: HTML o PDF. También reformulé la página del tutorial dentro del sitio de PyAr.

En fin. Disfruten y pásenlo.

Read more
Michael

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
                     verbosity=0)
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,
                     verbosity=0)

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,
                     verbosity=0)
 

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code

        self.migrate_to_dest()

        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.


Filed under: django, python, testing

Read more
Michael

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
                     verbosity=0)
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,
                     verbosity=0)

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,
                     verbosity=0)
 

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code

        self.migrate_to_dest()

        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.


Filed under: django, python, testing

Read more
facundo

El árbol fluorescente


En otra edición de "cosas que hice hace tiempo y me resultaron útiles ahora", les presento un proyecto que nació hace cinco años y medio de una charla de PyAr.

Como explico en este post, para jugar un rato con un amigo hice un árbol Trie, que luego de algunas optimizaciones degeneró en algo que llamé "Fucked Trie".

Este árbol para guardar palabras y buscarlas por prefijo de forma muy muy rápida resultó ser lo que necesitaba en el laburo un par de semanas atrás, pero con un cambio: ahora cada palabra tenía que guardar cierta metadata (que luego se obtendría al buscar).

Entonces, agarré el código original, lo modifiqué un poco, y armé este proyecto nuevo que se llama Fluorescent Trie (porque Fucked quedaba muy fuerte para un proyecto, vissste).

Fluorescent trie

Características de este árbol:

- Está pensando para mantenerlo en memoria: ocupa poco, y carga rápido

- Las búsquedas son por prefijo: O sea, entrando con "foo" encuentra "foo" y todo lo que empieza por "foo". No encuentra "grafoo".

- Las búsquedas son extremadamente rápidas (en el orden de los 10-4 segundos).

- Cada palabra tiene un payload que puede ser cualquier cosa.

Si lo necesitan para algo, aprovechen.

Read more
pitti

I just released a new PyGObject for GNOME 3.7.90, with a nice set of bug fixes and some internal code cleanup. Thanks to all contributors!

  • overrides: Fix inconsistencies with drag and drop target list API (Simon Feltman) (#680640)
  • pygtkcompat: Add pygtk compatible GenericTreeModel implementation (Simon Feltman) (#682933)
  • overrides: Add support for iterables besides tuples for TreePath creation (Simon Feltman) (#682933)
  • Prefix __module__ attribute of function objects with gi.repository (Niklas Koep) (#693839)
  • configure.ac: only enable code coverage when available (Jonathan Ballet) (#693328)
  • Correctly set properties on object with statically defined properties (Jonathan Ballet) (#693618)
  • autogen.sh: Use gnome-autogen.sh (Martin Pitt) (#693328)
  • Fix reference leaks with transient floating objects (Simon Feltman) (#687522)

Read more
pitti

I just released a new PyGObject for GNOME 3.7.5. Unfortunately master.gnome.org is out of space right now, so I put the new tarball on my Ubuntu people account for the time being.

This again brings a nice set of memory leak and bug fixes, some more reduction of static bindings, and better support for building under Windows.

Thanks to all contributors!

  • Move various signal methods from static bindings to gi and python (Simon Feltman) (#692918)
  • GLib overrides: Support unpacking ‘maybe’ variants (Paolo Borelli) (#693032)
  • Fix ref count leak when creating pygobject wrappers for input args (Mike Gorse) (#675726)
  • Prefix names of typeless enums and flags for GType registration (Simon Feltman) (#692515)
  • Fix compilation with non-C99 compilers such as Visual C++ (Chun-wei Fan) (#692856)
  • gi/overrides/Glib.py: Fix running on Windows/non-Unix (Chun-wei Fan)
  • Do not immediately initialize Gdk and Gtk on import (Martin Pitt) (#692300)
  • Accept ±inf and NaN as float and double values (Martin Pitt) (#692381)
  • Fix repr() of GLib.Variant (Martin Pitt)
  • Fix gtk-demo for Python 3 (Martin Pitt)
  • Define GObject.TYPE_VALUE gtype constant (Martin Pitt)
  • gobject: Go through introspection on property setting (Olivier Crête) (#684062)
  • Clean up caller-allocated GValues and their memory (Mike Gorse) (#691820)
  • Use GNOME_COMPILE_WARNINGS from gnome-common (Martin Pitt)

Read more
brendandonegan

Part of my duties as a Hardware Certification Engineer is to develop and maintain our test and bug reports. Previously these have been Python scripts containing lots of inline HTML (yuck :/), which I’m ashamed to admit it because I didn’t know any better when I initially wrote them. I’m currently in the middle of refactoring (hopefully not ref**ktoring) the scripts to take advantage of a new API that’s available for querying data from the Certification website and have decided to use Jinja (precisely Jinja2) as the template engine.

So I have a list of systems which may be in different locations and I want to sort them by location. Initially I did this in the Python code which gathered the data, using a simple .sort(key = lambda obj: obj.datacentre). Then the template code used the sorted list and all was well. Later on when reading the Jinja template documentation I discovered a way to sort a list in the template itself. I considered this to be a cleaner, more transparent way to modify the list (somebody looking at the template asking ‘why does it come out in this order’ has the answer right in front of them). So I gave it a try by changing

for hw in {%reported_hardware %}

to

{% for hw in reported_hardware|sort(attribute='datacentre') %}

and was unfortunately confronted with a traceback from the template engine:

Traceback (most recent call last):
...
AttributeError: Hardware instance has no attribute '__getitem__'

This looks like it’s saying that the ‘Hardware’ object (which I’ll introduce in a second) isn’t the type that the template engine was expecting.

Initially I defined the Hardware object like this:

class Hardware:

def __init__(self..., datacentre):
...
self.datacentre = datacentre

My first attempt to fix this involved making datacentre a property:

@property

def datacentre(self):

return self._datacentre

But this didn’t seem to do the trick. Eventually I read in the Python documentation about the property built in working only on new-style classes and this set off the lightbulb

class Hardware(object):

This seemed to be the magic ingredient, and the sort in the template engine started working. I also realised that making datacentre a property wasn’t really necessary, so reverted that change.

Update: I found out that this whole problem is avoided by using Python3.x because all classes there are new-style classes. So the advice above only applies if you’re stuck with Python2.x


Read more
facundo

CDPedia 0.8


Luego de bastante laburo las últimas semanas, estoy muy contento de anunciar (de mi parte y de Python Argentina) que está en la calle la versión 0.8 de CDPedia, un proyecto que permite acceder a la información de la Wikipedia en castellano sin necesidad de una conexión a Internet.  Se puede descargar libremente de la red y grabar a CDs, DVDs o memorias USB para repartirla sin restricciones.

La CDPedia funciona en cualquier computadora, ya sea que tenga Linux, MacOS o Windows como sistema operativo, y esta última versión incluye contenido de Wikipedia a Diciembre de 2012.

Para mayor información e instrucciones para la descarga, visitar la página del proyecto.

Logo


Esta versión trae las siguientes novedades:

  • Renovamos completamente la generación del ejecutable para Windows, se deberían tener muchísimos menos problemas para usar la CDPedia en ese entorno.
  • Al extraer las páginas de Wikipedia, se les hace un análisis para evitar los artículos vandalizados (en este caso, se baja una versión más antigua, pero correcta).
  • El servidor interno ahora es multiusuario, lo que simplifica enormemente instalar CDPedia en un servidor y que sea accedida desde distintos clientes.
  • Ahora se incluyen en los discos todas las páginas de las categorías, no sólo la primera.
  • Actualizamos el contenido a Diciembre 2012.
  • Varias mejoras a la hora de la generación de discos y tarballs, así como también en la calidad del código.

Si tenés discos disponibles, hacé algunas copias de CDPedia para regalar a tus familiares y amigos, a la biblioteca de tu barrio, o a la escuela de tus hijos. ¡Difundí CDPedia!

Read more