Canonical Voices

niemeyer

One of my “official side projects” is the Go language driver for MongoDB, started a few years back while looking for a way to store data conveniently from the Go language while leaving aside some of the problems we have mapping code into table-based approaches.

Nowadays this is used in projects at Canonical, in the MongoDB tooling itself, and also in some of my own personal projects. For the personal server-side projects I’ve been using docker containers to conveniently deploy the database tooling, but this week when I went to update some of my older images pulled from the docker hub I found that the docker installed on that server was a bit too old, so it was time to update the servers.

But that got me wondering: what if I replaced all those containers by snaps? This would allow me to keep the convenience and safety of the isolation, while making a lot of things simpler. Unlike docker, snaps make the installed tooling more easily accessible to the host system (bins in the search path, processes as actual children from shell and systemd, etc), and can even use system resources directly assuming interfaces allow it (e.g. home files).

So I got into that, and perhaps ended up overdoing it a little bit. Based on my experience testing the driver, I really appreciate having all versions available for playing with, rather than just the latest one. This is how my local development system looks like right now:

$ snap list | grep 'Name\|mongo'
Name                  Version               Rev     Developer   Notes
mongo22               2.2.7                 1       niemeyer    -
mongo24               2.4.14                1       niemeyer    -
mongo26               2.6.12                1       niemeyer    -
mongo30               3.0.12                1       niemeyer    -
mongo32               3.2.7                 1       niemeyer    -
mongo33               3.3.9                 1       niemeyer    -

These are backed by upstream tarballs (snapcraft downloaded them pre-built from mongodb.com), and are all installed, running, and with tooling available for playing with. They are also published to the snap store which means you can easily make use of them locally as well. If you want to do that, here is a crash course on snaps and on how I packaged the database together specifically.

If you’re using Ubuntu, update to release 16.04 which has snaps working out of the box. For other distributions have a look at snapcraft.io to see what command must be run for ensuring it is available.

Then, pick your version and install it. For example, if you want to play with the features just announced at MongoDB World this week, you want the unstable version 3.3.9:

$ snap install mongo33
93.59 MB / 93.59 MB [============================] 100.00 % 1.33 MB/s 

Name     Version  Rev  Developer  Notes
mongo33  3.3.9    1    niemeyer   -

After that you already have the tooling available in your path (assuming you have /snap/bin there), and the daemon started. So go ahead and fire the client to talk to the database:

$ mongo33
MongoDB shell version: 3.3.9
connecting to: 127.0.0.1:3317/test
MongoDB server version: 3.3.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
Server has startup warnings: 
(...) 
(...) ** NOTE: This is a development version (3.3.9) of MongoDB.
(...) **       Not recommended for production.
> 

Note that the server started on the non-standard port 33017 to allow multiple versions to be running together. The pattern is that the snap mongoNN will run on port NN017.

The tools installed also follow a similar pattern. Where upstream uses mongo, mongod, mongodump, etc, the snaps will have them under /snap/bin as mongo33, mongo33.d, mongo33.dump, and so on.

The systemd unit, if you want to interact with it for restarting, improving, debugging, etc, is named snap.mongo33.mongod.service, as usual for any snap that contains a daemon (snap.<snap name>.<app name>.service).

The data for the process that runs under systemd lives in the the standard writable area that the confinement system opens up for snaps in general – in this specific case /var/snap/mongo33/common/. The common directory is reused untouched on updates across snap revisions, which implies snapd won’t copy the data over when doing such updates. This compromises slightly the update safety, but is worth it for bulk data that we don’t want to copy on every snap refresh.

So this ended up quite nicely. Next up is the Go server code itself, which will be packed as a snap too, for deploying it into the servers in a similar way. The exact details for how these snaps are being built are publicly available too.

If you want a hand doing something similar, we have some helpful people at #snappy on FreeNode and also snapcraft@lists.snapcraft.io.

@gniemeyer

Read more
niemeyer

Over the last several months there has been noticeable and growing pain associated with the evolving integration tests around snapd, and given the project goal of being a cross-distribution platform, we are very keen on solving this problem appropriately so that stability is guaranteed everywhere.

With that mindset a more focused effort was made over the last few weeks to produce a tool that can get the project out of those problems, and onto a runway of more pleasant stability. Despite the short amount of time, I’m very happy about the Spread project which resulted from this effort.

Spread is not Jenkins or Travis, and is not a language or library either. Spread is a tool that will very conveniently ship your code to one or more systems, in parallel, and then offer the right set of options so you can run whatever you need to run to make sure the logic is working, and drive it all from the local system. That implies you can run Spread inside Travis, Jenkins, or your terminal, in a similar way to how your unit tests work.

Here is a short list of interesting facts about Spread:

  • Full-system tests with on demand machine allocation.
  • Multi-backend with Linode and LXD (for local runs) out of the box for now.
  • Multi-language since it can run arbitrary remote code.
  • Agent-less and driven via embedded ssh (kudos to Go team).
  • Convenient harness with project+backend+suite+test prepare and restore scripts.
  • Variants feature for test duplication without copy & paste.
  • Great debugging support – add -debug and stop with a shell inside every failure.
  • Reuse of servers – server allocation is fast, but not allocating is faster.
  • Reasonable test outputs with the shell’s +x mode on failures.
  • … and so forth.

This is all well documented, so I’ll just provide one example here to offer a real taste of how the system feels like.

This is spread.yaml, put in the project root to define the basics:

project: spread

backends:
    lxd:
        systems:
            - ubuntu-16.04
            - ubuntu-14.04

path: /home/test

prepare: |
    echo Entering project...
restore: |
    echo Leaving project...

suites:
    tests/: 
        summary: Integration tests
        prepare: |
            echo Entering suite...
        restore: |
            echo Leaving suite...

The suite name is also the path under which the tests are found.

Then, this is tests/hello/task.yaml:

summary: Greet the world
prepare: |
    echo "Entering task..."
restore: |
    echo "Leaving task..."
environment:
    FOO/a: one
    FOO/b: two
execute: |
    echo "Hello world!"
    [ $FOO = one ] || exit 1

The outcome should be almost obvious (intended feature :-). The one curious detail here is the FOO/a and FOO/b environment variables. This is how to introduce variants, which means this one test will in fact become two: first with FOO=one, and then with FOO=two. Now consider that such environment variables can be defined at any level – project, backend, suite, and task – and imagine how easy it is to test small variations without any copy & paste. After cascading takes place (project→backend→suite→task) all environment variables using a given variant key will be present at once on the same execution.

Now let’s try to run this configuration, including the -debug flag so we get a shell on the failures. Note how with a single test we get four different jobs, two variants over two systems, with the variant b failing as instructed:

$ spread -debug

2016/06/11 19:09:27 Allocating lxd:ubuntu-14.04...
2016/06/11 19:09:27 Allocating lxd:ubuntu-16.04...
2016/06/11 19:09:41 Waiting for LXD container to have an address...
2016/06/11 19:09:43 Waiting for LXD container to have an address...
2016/06/11 19:09:44 Allocated lxd:ubuntu-14.04.
2016/06/11 19:09:44 Connecting to lxd:ubuntu-14.04...
2016/06/11 19:09:48 Allocated lxd:ubuntu-16.04.
2016/06/11 19:09:48 Connecting to lxd:ubuntu-16.04...
2016/06/11 19:09:52 Connected to lxd:ubuntu-14.04.
2016/06/11 19:09:52 Sending project data to lxd:ubuntu-14.04...
2016/06/11 19:09:53 Connected to lxd:ubuntu-16.04.
2016/06/11 19:09:53 Sending project data to lxd:ubuntu-16.04...

2016/06/11 19:09:54 Error executing lxd:ubuntu-14.04:tests/hello:b :
-----
+ echo Hello world!
Hello world!
+ [ two = one ]
+ exit 1
-----

2016/06/11 19:09:54 Starting shell to debug...

lxd:ubuntu-14.04 ~/tests/hello# echo $FOO
two
lxd:ubuntu-14.04 ~/tests/hello# cat /etc/os-release | grep ^PRETTY
PRETTY_NAME="Ubuntu 14.04.4 LTS"
lxd:ubuntu-14.04 ~/tests/hello# exit
exit

2016/06/11 19:09:55 Error executing lxd:ubuntu-16.04:tests/hello:b :
-----
+ echo Hello world!
Hello world!
+ [ two = one ]
+ exit 1
-----

2016/06/11 19:09:55 Starting shell to debug...

lxd:ubuntu-16.04 ~/tests/hello# echo $FOO
two
lxd:ubuntu-16.04 ~/tests/hello# cat /etc/os-release | grep ^PRETTY
PRETTY_NAME="Ubuntu 16.04 LTS"
lxd:ubuntu-16.04 ~/tests/hello# exit
exit


2016/06/11 19:10:33 Discarding lxd:ubuntu-14.04 (spread-129)...
2016/06/11 19:11:04 Discarding lxd:ubuntu-16.04 (spread-130)...
2016/06/11 19:11:05 Successful tasks
2016/06/11 19:11:05 Aborted tasks: 0
2016/06/11 19:11:05 Failed tasks: 2
    - lxd:ubuntu-14.04:tests/hello:b
    - lxd:ubuntu-16.04:tests/hello:b
error: unsuccessful run

This demonstrates many of the stated goals (parallelism, clarity, convenience, debugging, …) while running on a local system. Running on a remote system is just as easy by using an appropriate backend. The snapd project on GitHub, for example, is hooked up on Travis to run Spread and then ship its tests over to Linode. Here is a real run output with the initial tests being ported, and a basic smoke test.

If you like what you see, by all means please go ahead and make good use of it.

We’re all for more stability and sanity everywhere.

@gniemeyer

Read more
niemeyer

As much anticipated, this week Ubuntu 16.04 LTS was released with integrated support for snaps on classic Ubuntu.

Snappy 2.0 is a modern software platform, that includes the ability to define rich interfaces between snaps that control their security and confinement, comprehensive observation and control of system changes, completion and undoing of partial system changes across restarts/reboots/crashes, macaroon-based authentication for local access and store access, preliminary development mode, a polished filesystem layout and CLI experience, modern sequencing of revisions, and so forth.

The previous post in this series described the reassuring details behind how snappy does system changes. This post will now cover Snappy interfaces, the mechanism that controls the confinement and integration of snaps with other snaps and with the system itself.

A snap interface gives one snap the ability to use resources provided by another snap, including the operating system snap (ubuntu-core is itself a snap!). That’s quite vague, and intentionally so. Software interacts with other software for many reasons and in diverse ways, and Snappy is a platform that has to mediate all of that according to user needs.

In practice, though, the mechanism is straightforward and pleasant to deal with. Without any snaps in the system, there are no interfaces available:

% sudo snap interfaces
error: no interfaces found

If we install the ubuntu-core snap alone (done implicitly when the first snap is installed), we can already see some interface slots being provided by it, but no plugs connected to them:

% sudo snap install ubuntu-core
75.88 MB / 75.88 MB [=====================] 100.00 % 355.56 KB/s 

% snap interfaces
Slot                 Plug
:firewall-control    -
:home                -
:locale-control      -
(...)
:opengl              -
:timeserver-control  -
:timezone-control    -
:unity7              -
:x11                 -

The syntax is <snap>:<slot> and <snap>:<plug>. The lack of a snap name is a shorthand notation for slots and plugs on the operating system snap.

Now let’s install an application:

% sudo snap install ubuntu-calculator-app
120.01 MB / 120.01 MB [=====================] 100.00 % 328.88 KB/s 

% snap interfaces
Slot                 Plug
:firewall-control    -
:home                -
:locale-control      -
(...)
:opengl              ubuntu-calculator-app
:timeserver-control  -
:timezone-control    -
:unity7              ubuntu-calculator-app
:x11                 -

At this point the application should work fine. But let’s instead see what happens if we take away one of these interfaces:

% sudo snap disconnect \
             ubuntu-calculator-app:unity7 ubuntu-core:unity7 

% /snap/bin/ubuntu-calculator-app.calculator
QXcbConnection: Could not connect to display :0

The application installed depends on unity7 to be able to display itself properly, which is itself based on X11. When we disconnected the interface that gave it permission to be accessing these resources, the application was unable to touch them.

The security minded will observe that X11 is not in fact a secure protocol. A number of system abuses are possible when we hand an application this permission. Other interfaces such as home would give the snap access to every non-hidden file in the user’s $HOME directory (those that do not start with a dot), which means a malicious application might steal personal information and send it over the network (assuming it also defines a network plug).

Some might be surprised that this is the case, but this is a misunderstanding about the role of snaps and Snappy as a software platform. When you install software from the Ubuntu archive, that’s a statement of trust in the Ubuntu and Debian developers. When you install Google’s Chrome or MongoDB binaries from their respective archives, that’s a statement of trust in those developers (these have root on your system!). Snappy is not eliminating the need for that trust, as once you give a piece of software access to your personal files, web camera, microphone, etc, you need to believe that it won’t be using those allowances maliciously.

The point of Snappy’s confinement in that picture is to enable a software ecosystem that can control exactly what is allowed and to whom in a clear and observable way, in addition to the same procedural care that we’ve all learned to appreciate in the Linux world, not instead of it. Preventing people from using all relevant resources in the system would simply force them to use that same software over less secure mechanisms instead of fixing the problem.

And what we have today is just the beginning. These interfaces will soon become much richer and more fine grained, including resource selection (e.g. which serial port?), and some of them will disappear completely in favor of more secure choices (Unity 8, for instance).

These are exciting times for Ubuntu and the software world.

@gniemeyer

Read more
niemeyer

As announced last Saturday, Snappy Ubuntu Core 2.0 has just been tagged and made its way into the archives of Ubuntu 16.04, which is due for the final release in the next days. So this is a nice time to start covering interesting aspects of what is being made available in this release.

A good choice for the first post in this series is talking about how snappy performs changes in the system, as that knowledge will be useful in observing and understanding what is going on in your snappy platform. Let’s start with the first operation you will likely do when first interacting with the snappy platform — install:

% sudo snap install ubuntu-calculator-app
120.01 MB / 120.01 MB [===============================] 100.00 % 1.45 MB/s

This operation is traditionally done on analogous systems in an ephemeral way. That is, the software has either a local or a remote database of options to install, and once the change is requested the platform of choice will start acting on it with all state for the modification kept in memory. If something doesn’t go so well, such as a reboot or even a crash, the modification is lost.. in the best case. Besides being completely lost, it might also be partially applied to the system, with some files spread through the filesystem, and perhaps some of the involved hooks run. After the restart, the partial state remains until some manual action is taken.

Snappy instead has an engine that tracks and controls such changes in a persistent manner. All the recent changes, pending or not, may be observed via the API and the command line:

% snap changes
ID   Status  ...  Summary
1    Done    ...  Install "ubuntu-calculator-app" snap

(the spawn and ready date/time columns have been hidden for space)

The output gives an overview of what happened recently in the system, whether pending or not. If one of these changes is unintendedly interrupted for whatever reason, the daemon will attempt to continue the requested change at the next opportunity.

Continuing is not always possible, though, because there are external factors that such a change will generally depend upon (the snap being available, the system state remaining similar, etc). In those cases, the change will fail, and any relevant modifications performed on the system while attempting to accomplish the defined goal will be undone.

Because such partial states are possible and need to be handled properly by the system, changes are in fact broken down into finer grained tasks which are also tracked and observable while in progress or after completion. Using the change ID obtained in the former command, we can get a better picture of what that changed involved:

% snap changes 1
Status ...  Summary
Done   ...  Download snap "ubuntu-core" from channel "stable"
Done   ...  Mount snap "ubuntu-core"
Done   ...  Copy snap "ubuntu-core" data
Done   ...  Setup snap "ubuntu-core" security profiles
Done   ...  Make snap "ubuntu-core" available
Done   ...  Download snap "ubuntu-calculator-app"
Done   ...  Mount snap "ubuntu-calculator-app"
Done   ...  Copy snap "ubuntu-calculator-app" data
Done   ...  Setup snap "ubuntu-calculator-app" security profiles
Done   ...  Make snap "ubuntu-calculator-app" available

(the spawn and ready date/time columns have been hidden for space)

Here we can observe an interesting implementation detail of the snappy integration into Ubuntu: the ubuntu-core snap is at the moment ~80MB, and contains the software bundled with the snappy platform itself. Instead of having it pre-installed, it’s only pulled in when the first snap is installed.

Another interesting implementation detail that surfaces here is the fact snaps are in fact mounted rather than copied into the system as traditional packaging systems do, and they’re mounted read-only. That means the operation of having the content of a snap in the filesystem is instantaneous and atomic, and so is removing it. There are no partial states for that specific aspect, and the content cannot be modified.

Coming back into the task list, we can see above that all the tasks that the change involved are ready and did succeed, as expected from the earlier output we had seen for the change itself. Being more of an introspection view, though, this tasks view will often also show logs and error messages for the individual tasks, whether in progress or not.

The following view presents a similar change but with an error due to an intentionally corrupted system state that snappy could not recover from (path got a busy mountpoint hacked in):

% sudo snap install xkcd-webserver
[\] Make snap "xkcd-webserver" available to the system
error: cannot perform the following tasks:
- Make snap "xkcd-webserver" available to the system
  (symlink 13 /snap/xkcd-webserver/current: file exists)

% sudo snap changes 2
Status  ...  Summary
Undone  ...  Download snap "xkcd-webserver" from channel "stable"
Undone  ...  Mount snap "xkcd-webserver"
Undone  ...  Copy snap "xkcd-webserver" data
Undone  ...  Setup snap "xkcd-webserver" security profiles
Error   ...  Make snap "xkcd-webserver" available to the system

.................................................................
Make snap "xkcd-webserver" available to the system

2016-04-20T14:14:30-03:00 ERROR symlink 13
    /snap/xkcd-webserver/current: file exists

Note how reassuring that report looks. It says exactly what went wrong, at which stage of the process, and it also points out that all the prior tasks that previously succeeded had their modifications undone. The security profiles were removed, the mount point was taken down, and so on.

This sort of behavior is to be expected of modern operating systems, and is fundamental when considering systems that should work unattended. Whether in a single execution or across restarts and reboots, changes either succeed or they don’t, and the system remains consistent, reliable, observable, and secure.

In the next blog post we’ll see details about the interfaces feature in snappy, which controls aspects of confinement and integration between snaps.

@gniemeyer

Read more
niemeyer

Ubuntu and Snappy community, it’s time to celebrate!

After another intense week and a long Saturday focused on observing and fine tuning the user experience, the development team is proud to announce that Snappy 2.0 has been tagged. As has been recently announced, this release of Snappy Ubuntu Core will be available inside Ubuntu proper, extending it with new capabilities in a seamless manner.

This is an important moment for the project, as it materializes most of the agreements that were made over the past year, and does so with the promise of stability. So you may trust that the important external APIs of the project (filesystem layout, snap format, REST API, etc) will not change from now on.

The features that went into this release are way too rich for me to describe in this post, but you may expect us to be covering the many interesting aspects of Snappy 2.0 in the coming weeks. Rich interfaces between snaps that control security and confinement, comprehensive observation and control of system changes, completion and undoing of partial system changes across restarts/reboots/crashes, macaroon-based authentication for local access and store access, preliminary development mode, a polished filesystem layout and CLI experience, modern sequencing of revisions, and so forth.

Still, the most remarkable aspect about this release to me is that it is a solid foundation. This release exports APIs and is constructed in a way to be proud of, and together with this team I will be delighted to spend the foreseeable future building a platform the world has never seen.

As a final note, I can’t thank the development team enough for the dedication they have put into the project over the past year, and specially over these last two weeks. You were the make it or break it of this project, and you made it.

Thank you!

@gniemeyer

Read more
niemeyer

mgo r2016.02.04

This is one of the most packed releases of the mgo driver for Go in recent times. There are new features, important fixes, and relevant
internal reestructuring to support the on-going server improvements.

As usual for the driver, compatibility is being preserved both with old applications and with old servers, so updating should be a smooth experience.

Release r2016.02.04 of mgo includes the following changes which were requested, proposed, and performed by a great community.

Enjoy!

Exposed access to individual bulk error cases

Accessing the individual errors obtained while attempting a set of bulk operations is now possible via the new mgo.BulkError error type and its Cases method which returns a slice of mgo.BulkErrorCases which are properly indexed according to the operation order used. There are documented server limitations for MongoDB version 2.4 and older.

This change completes the bulk API. It can now perform optimized bulk queries when communicating with recent servers (MongoDB 2.6+), perform the same operations in older servers using compatible but less performant options, and in both cases provide more details about obtained errors.

Feature first requested by pjebs.

New fields in CollectionInfo

The CollectionInfo type has new fields for dealing with the recently introduced document validation MongoDB feature, and also the storage engine-specific options.

Features requested by nexcode and pjebs.

New Find and GetMore command support

MongoDB is moving towards replacing the old wire protocol with a command-based based implementation, and every recent release introduced changes around that. This release of mgo introduces support for the find and getMore commands which were added to MongoDB 3.2. These are exercised whenever querying the database or iterating over queried results.

Previous server releases will continue to use the classical mechanism, and the two approaches should be compatible. Please report any issues in that regard.

Do not fallback to Monotonic mode improperly

Recent driver changes adapted the Pipe.Iter, Collection.Indexes, and Database.CollectionNames methods to work with recent server releases. These changes also introduced a bug that could cause the driver to talk to a secondary server improperly, when that operation was the first operation performed on the session. This has been fixed.

Problem reported by Sundar.

Fix crash in new bulk update API

The new methods introduced in the bulk update API in the last release were crashing when a connection error occurred.

Fix contributed by Maciej Galkowski.

Enable TCP keep-alives for all connections

As requested by developers, TCP keep-alives are now enabled for all connections. No timing is specified, so the default operating system setting will be used.

Feature requested by Hunor Kovács, Berni Varga, and Martin Garton.

ChangeInfo.Updated now behaves as documented

The ChangeInfo.Updated field is documented to report the number of documents that were changed, but in fact that was not possible in old releases of the driver, since the server did not provide that information. Instead, the server only reported the number of documents matched by the selection document.

This has been fixed, so starting with MongoDB 2.6, the driver will behave as documented, and inform the number of documents that were indeed updated. This is related to the next driver change:

New ChangeInfo.Matched field

The new ChangeInfo.Matched field will report the number of documents that matched the selection document, whether the performed change was a removal, an update, or an upsert.

Feature requested by Žygimantas and other list members.

ObjectId now supports TextMarshaler/TextUnmarshaler

ObjectId now knows how to marshal/unmarshal itself as text in hex format when using its encoding.TextMarshaler and TextUnmarshaler interfaces.

Contributed by Jack Spirou.

Created GridFS index is now unique

The index on {“files_id”, “n”} automatically created for GridFS chunks when a file write completes now enforces the uniqueness of the key.

Contributed by Wisdom Omuya.

Use SIGINT in dbtest.DBServer

The dbtest.DBServer was stopping the server with SIGKILL, which would not give it enough time for a clean shutdown. It will now stop it with SIGINT.

Contributed by Haijun Wang.

Ancient field tag logic dropped

The very old field tag format parser, in use several years back even before Go 1 was released, was still around in the code base for no benefit.

This has been removed by Alexandre Cesaro.

Documentation improvements

Documentation improvements were contributed by David Glasser, Ryan Chipman, and Shawn Smith.

Fixed BSON skipping of incorrect slice types

The BSON parser was collapsing when an array value was unmarshaled into an existing field that was not of an appropriate type for such values. This has been fixed so that the the bogus field is ignored and the value skipped.

Fix contributed by Gabriel Russel.

Read more
niemeyer

Another stable release of the mgo Go driver for MongoDB hits the shelves, and this one brings some important improvements and fixes. As usual, it also remains fully compatible with prior releases in the v2 series.

Please read along for the complete list of changes.


New read preference modes

In addition to the traditional Strong, Monotonic, and Eventual modes, the driver now supports all read preference modes defined by the MongoDB specification:

  • PrimaryMode – Talk only to the replica set primary. That’s the same as mgo.Strong.
  • PrimaryPreferredMode – Talk to one of the closest secondaries if the primary is not available.
  • Secondary – Talk to one of the closest replica set secondaries.
  • SecondaryPreferredMode – Talk to the primary if a secondary is not available.
  • NearestMode – Talk to one of the closest servers, whether primary or secondary.

See Session.SetMode for details on how to switch modes.

bson.NewObjectId random initial counter

The bson.NewObjectId method will now uses a random initial counter value, as defined by the specification.

Documentation improvements

Various documentation improvements have been made, as suggested or submitted by contributors.

Bulk API improvements

Several improvements have been made to the Bulk API, including support for update and upsert operations, error reporting improvements, and a more efficient implementation based on write commands where that’s supported (MongoDB 2.6+).

Custom index name support

Indexes created by Collection.EnsureIndex may now declare a custom name during creation, if convenient. The Collection.DropIndexName method was also added to support the dropping of indexes by name.

Collection.Indexes method fixes

The Collection.Indexes method was returning results that lacked some of the information that was input into the Collection.EnsureIndexe method during creation. This has been fixed.

Problem reported by Louisa Berger.

Introduced Index.Minf/Maxf

The Min and Max fields currently offered by the Index type for tuning of geographical indexes were incorrectly assumed to be integers. Unfortunately these types cannot be changed without a backwards incompatible modification, so two new Minf and Maxf fields were introduced with the float64 type. Despite still working, the old fields are now obsolete.

Problem reported by Louisa Berger.

Name resolution fixed for Go 1.5

Go 1.5 modified the behavior of address resolution in a way that breaks the procedure implemented by mgo to resolve names within a given time span. This was addressed and now both IPv4 and IPv6 servers should be working correctly. This change was also applied as a hot fix to the previous release of the driver, to ensure developers could make use of the newly released compiler with mgo.

Issue reported and collaborated around by several contributors.

Read more
niemeyer

mgo r2015.05.29

Another release of mgo hits the shelves, just in time for the upcoming MongoDB World event.

A number of of relevant improvements have landed since the last stable release:


New package for having a MongoDB server in test suites

The new gopkg.in/mgo.v2/dbtest package makes it comfortable to plug a real MongoDB server into test suites. Its simple interface consists of a handful of methods, which together allow obtaining a new mgo session to the server, wiping all existent data, or stopping it altogether once the suite is done. This design encourages an efficient use of resources, by only starting the server if necessary, and then quickly cleaning data across runs instead of restarting the server.

See the documentation for more details.

(UPDATE: The type was originally testserver.TestServer and was renamed dbtest.DBServer to improve readability within test suites. The old package and name remain working for the time being, to avoid breakage)

Full support for write commands

This release includes full support for the write commands first introduced in MongoDB 2.6. This was done in a compatible way, both in the sense that the driver will continue to use the wire protocol to perform writes on older servers, and also in the sense that the public API has not changed.

Tests for the new code path have been successfully run against MongoDB 2.6 and 3.0. Even then, as an additional measure to prevent breakage of existent applications, in this release the new code path will be enabled only when interacting with MongoDB 3.0+. The next stable release should then enable it for earlier releases as well, after some additional real world usage took place.

New ParseURL function

As perhaps one of the most requested features of all times, there’s now a public ParseURL function which allows code to parse a URL in any of the formats accepted by Dial into a DialInfo value which may be provided back into DialWithInfo.

New BucketSize field in mgo.Index

The new BucketSize field in mgo.Index supports the use of indexes of type geoHaystack.

Contributed by Deiwin Sarjas.

Handle Setter and Getter interfaces in slice types

Slice types that implement the Getter and/or Setter interfaces will now be custom encoded/decoded as usual for other types.

Problem reported by Thomas Bouldin.

New Query.SetMaxTime method

The new Query.SetMaxTime method enables the use of the special $maxTimeMS query parameter, which constrains the query to stop after running for the specified time. See the method documentation for details.

Feature implemented by Min-Young Wu.

New Query.Comment method

The new Query.Comment method may be used to annotate queries for further analysis within the profiling data.

Feature requested by Mike O’Brien.

sasl sub-package moved into internal

The sasl sub-package is part of the implementation of SASL support in mgo, and is not meant to be accessed directly. For that reason, that package was moved to internal/sasl, which according to recent Go conventions is meant to explicitly flag that this is part of mgo’s implementation rather than its public API.

Improvements in txn’s PurgeMissing

The PurgeMissing logic was improved to work better in older server versions which retained all aggregation pipeline results in memory.

Improvements made by Menno Smits.

Fix connection statistics bug

Change prevents the number of slave connections from going negative on a particular case.

Fix by Oleg Bulatov.

EnsureIndex support for createIndexes command

The EnsureIndex method will now use the createIndexes command where available.

Feature requested by Louisa Berger.

Support encoding byte arrays

Support encoding byte arrays in an equivalent way to byte slices.

Contributed by Tej Chajed.

Read more
niemeyer

MongoDB 3.0 (previously known as 2.8) is right around the block, and it’s time to release a few fixes and improvements on the mgo driver for Go to ensure it works fine on that new major server version. Compatibility is being preserved both with old applications and with old servers, so updating should be a smooth experience.

Release r2015.01.24 of mgo includes the following changes:


Support ReplicaSetName in DialInfo

DialInfo now offers a ReplicaSetName field that may contain the name of the MongoDB replica set being connected to. If set, the cluster synchronization routines will prevent communication with any server that does not report itself as part of that replica set.

Feature implemented by Wisdom Omuya.

MongoDB 3.0 support for collection and index listing

MongoDB 3.0 requires the use of commands for listing collections and indexes, and may report long results via cursors that must be iterated over. The CollectionNames and Indexes methods were adapted to support both the old and the new cases.

Introduced Collection.NewIter method

In the last few releases of MongoDB, a growing number of low-level database commands are returning results that include an initial set of documents and one or more cursor ids that should be iterated over for obtaining the remaining documents. Such results defeated one of the goals in mgo’s design: developers should be able to walk around the convenient pre-defined static interfaces when they must, so they don’t have to patch the driver when a feature is not yet covered by the convenience layer.

The introduced NewIter method solves that problem by enabling developers to create normal iterators by providing the initial batch of documents and optionally the cursor id for obtaining the remaining documents, if any.

Thanks to John Morales, Daniel Gottlieb, and Jeff Yemin, from MongoDB Inc, for their help polishing the feature.

Improved JSON unmarshaling of ObjectId

bson.ObjectId can now be unmarshaled correctly from an empty or null JSON string, when it is used as a field in a struct submitted for unmarshaling by the json package.

Improvement suggested by Jason Raede.

Remove GridFS chunks if file insertion fails

When writing a GridFS file, the chunks that hold the file content are written into the database before the document representing the file itself is inserted. This ensures the file is made visible to concurrent readers atomically, when it’s ready to be used by the application. If writing a chunk fails, the call to the file’s Close method will do a best effort to clean up previously written chunks. This logic was improved so that calling Close will also attempt to remove chunks if inserting the file document itself failed.

Improvement suggested by Ed Pelc.

Field weight support for text indexing

The new Index.Weights field allows providing a map of field name to field weight for fine tuning text index creation, as described in the MongoDB documentation.

Feature requested by Egon Elbre.

Fixed support for $** text index field name

Support for the special $** field name, which enables the indexing of all document fields, was fixed.

Problem reported by Egon Elbre.

Consider only exported fields on omitempty of structs

The implementation of bson’s omitempty feature was also considering the value of non-exported fields. This was fixed so that only exported fields are taken into account, which is both in line with the overall behavior of the package, and also prevents crashes in cases where the field value cannot be evaluated.

Fix potential deadlock on Iter.Close

It was possible for Iter.Close to deadlock when the associated server was concurrently detected unavailable.

Problem investigated and reported by John Morales.

Return ErrCursor on server cursor timeouts

Attempting to iterate over a cursor that has timed out at the server side will now return mgo.ErrCursor.

Feature implemented by Daniel Gottlieb.

Support for collection repairing

The new Collection.Repair method returns an iterator that goes over all recovered documents in the collection, in a best-effort manner. This is most useful when there are damaged data files. Multiple copies of the same document may be returned by the iterator.

Feature contributed by Mike O’Brien.

Read more
niemeyer

It’s somewhat ironic that just as Ubuntu readies itself for the starting wave of smart connected devices, my latest hardware hack was in fact a disconnected one. In my defense, it’s quite important for these smart devices to preserve a convenient physical interface with the user, so this one was a personal lesson on that.

The device hacked was a capsule-based coffee machine which originally had just a manual handle for on/off operation. This was both boring to use and unfortunate in terms of the outcome being somewhat unpredictable given the variations in amount of water through the capsule. While the manufacturer does offer a modern version of the same machine with an automated system, buying a new one wouldn’t be nearly as satisfying.

So the first act was to take the machine apart and see how it basically worked. To my surprise, this one model is quite difficult to take apart, but it was doable without any visible damage. Once in, the machine was “enhanced” with an external barrel connector that can command the operation of the machine:

Open Coffee Machine

The connector wire was soldered to the right spots, routed away from the hot components, and includes a relay that does the operation safely without bridging the internal circuit into the external world. The proper way to do that would have been with an optocoupler, but without one at hand a relay should do.

With the external connector in place, it was easy to evolve the controlling circuit without bothering with the mechanical side of it. The current version is based on an atmega328p MCU that sits inside a small box exposing a high-quality LED bargraph and a single button that selects the level, turns on the machine on long press, and cancels if pressed again before the selected level is completed:

The MCU stays on 24/7, and when unused goes back into a deep sleep mode consuming only a few microamps from an old laptop battery cell that sits within the same box.

Being a for-fun exercise, the controlling logic was written in assembly to get acquainted with the details of that MCU. The short amount of code is available if you are curious.

Read more
niemeyer

After the updates to gopkg.in itself, it’s time for gopkg.in/yaml.v2 to receive some attention. The following improvements are now available in the yaml package:

Support for omitempty on struct values

The omitempty attribute can now be used in tags of fields with a struct type. In those cases, the given field and its value only become part of the generated yaml document if one or more of the fields exported by the field type contain non-empty values, according to the usual conventions for omitempty .

For instance, considering these two types:

type TypeA struct {
        Maybe TypeB `yaml:",omitempty"` 
}

type TypeB struct {
        N int
}

the yaml package would only serialize the Maybe mapping into the generated yaml document if its N field was non-zero.

Support for inlined maps

The yaml package was previously able to handle the inlining of structs. For example, in the following snippet TypeB would be handled as if its fields were part of TypeA during serialization or deserialization:

type TypeA struct {
        Field TypeB `yaml:",inline"`
}

This convention for inlining differs from the standard json package, which inlines anonymous fields instead of considering such an attribute. That difference is mainly historical: the base of the yaml package was copied from mgo’s bson package, which had this convention before the standard json package supported any inlining at all.

Now the support for inlining maps, previously available in the bson package, is also being copied over. In practice, it allows unmarshaling a yaml document such as

a: 1
b: 2
c: 3

into a type that looks like

type T struct {
        A int
        Rest map[string]int `yaml:",inline"`
}

and obtaining in the resulting Rest field the value map[string]int{“b”: 2, “c”: 3} , while field A is set to 1 as usual. Serializing out that resulting value would reverse the process, and generate the original document including the extra fields.

That’s a convenient way to read documents with a partially known structure and manipulating them in a non-destructive way.

Bug fixes

A few problems were also fixed in this release. Most notably:

  • A spurious error was reported when custom unmarshalers handled errors internally by retrying. Reported a few times and fixed by Brian Bland.

  • An empty yaml list ([]) is now decoded into a struct field as an empty slice instead of a nil one. Reported by Christian Neumann.

  • Unmarshaling into a struct with a slice field would append to it instead of overwriting. Reported by Dan Kinder.

  • Do not use TextMarshaler interface on types that implement yaml.Getter. Reported by Sam Ghods.

Read more
niemeyer

This post provides the background for a deliberate and important decision in the design of gopkg.in that people often wonder about: while the service does support full versions in tag and branch names (as in “v1.2” or “v1.2.3”), the URL must contain only the major version (as in “gopkg.in/mgo.v2”) which gets mapped to the best matching version in the repository.

As will be detailed, there are multiple reasons for that behavior. The critical one is ensuring all packages in a build tree that depend on the same API of a given dependency (different majors means different APIs) may use the exact same version of that dependency. Without that, an application might easily get multiple copies unnecessarily and perhaps incorrectly.

Consider this example:

  • Application A depends on packages B and C
  • Package B depends on D 3.0.1
  • Package C depends on D 3.0.2

Under that scenario, when someone executes go get on application A, two independent copies of D would be embedded in the binary. This happens because both B and C have exact control of the version in use. When everybody can pick their own preferred version, it’s easy to end up with multiple of these.

The current gopkg.in implementation solves that problem by requiring that both B and C necessarily depend on the major version which defines the API version they were coded against. So the scenario becomes:

  • Application A depends on packages B and C
  • Package B depends on D 3.*
  • Package C depends on D 3.*

With that approach, when someone runs go get to import the application it would get the newest version of D that is still compatible with both B and C (might be 3.0.3, 3.1, etc), and both would use that one version. While by default this would just pick up the most recent version, the package might also be moved back to 3.0.2 or 3.0.1 without touching the code. So the approach in fact empowers the person assembling the binary to experiment with specific versions, and gives package authors a framework where the default behavior tends to remain sane.

This is the most important reason why gopkg.in works like this, but there are others. For example, to encode the micro version of a dependency on a package, the import paths of dependent code must be patched on every single minor release of the package (internal and external to the package itself), and the code must be repositioned in the local system to please the go tool. This is rather inconvenient in practice.

It’s worth noting that the issues above describe the problem in terms of minor and patch versions, but the problem exists and is intensified when using individual source code revision numbers to refer to import paths, as it would be equivalent in this context to having a minor version on every single commit.

Finally, when you do want exact control over what builds, godep may be used as a complement to gopkg.in. That partnership offers exact reproducibility via godep, and gives people stable APIs they can rely upon over longer periods with gopkg.in. Good match.

Read more
niemeyer

Improvements on gopkg.in

Early last year the gopkg.in service was introduced with the goal of encouraging Go developers to establish strategies that enable existent software to remain working while package APIs evolve. After the initial discussions and experimentation that went into defining the (simple) design and feature set of the service, it’s great to see that the approach is proving reasonable in practice, with steady growth in usage. Meanwhile, the service has been up and unchanged for several months while we learned more about which areas needed improvement.

Now it’s time to release some of these improvements:

Source code links

Thanks to Gary Burd, godoc.org was improved to support custom source code links, which means all packages in gopkg.in can now properly reference, for any given package version, the exact location of functions, methods, structs, etc. For example, the function name in the documentation at gopkg.in/mgo.v2#Dial is clickable, and redirects to the correct source code line in GitHub.

Unstable releases

As detailed in the gopkg.in documentation, a major version must not have any breaking changes done to it so that dependent packages can rely on the exposed API once it goes live. Often, though, there’s a need to experiment with the upcoming changes before they are ready for prime time, and while small packages can afford to have that testing done locally, it’s usual for non-trivial software to have external validation with experienced developers before the release goes fully public.

To support that scenario properly, gopkg.in now allows the version string in exposed branch and tag names to be suffixed with “-unstable”. For example:

Such unstable versions are hidden from the version list in the package page, except for the specific version being looked at, and their use in released software is also explicitly discouraged via a warning message.

For the package to work properly during development, any imports (both internal and external to the package) must be modified to import the unstable version. While doing that by hand is easy, thanks to Roger Peppe’s govers there’s a very convenient way to do that.

For example, to use mgo.v2-unstable, run:

govers gopkg.in/mgo.v2-unstable

and to go back:

govers gopkg.in/mgo.v2

Repositories with no master branch

Some people have opted to omit the traditional “master” branch altogether and have only versioned branches and tags. Unfortunately, gopkg.in did not accept such repositories as valid. This was fixed.

These changes are all live right now at gopkg.in.

Read more
niemeyer

Last night I did a trivial yet surprisingly satisfying hardware hack, of the kind that can only be accomplished when the brain is in holiday mode. Our son has that very simple airplane toy, which turned out to become one of his most loved ones, enough to have deserved wheel repairs before. He’s also reportedly a fan of all kinds of light-emitting or reflecting objects (including the sun, and specially the moon). So the idea sparkled: how easily can that airplane get a blinking led?

With an attiny85, a CR2032 battery, a LED, and half an hour of soldering work, this was the result:

The code loaded in the chip is small enough to be listed here, and it gets away with blinking without waking up the main CPU clock:

    ; Set inverse OC1B pin as output for the led.
    sbi _SFR_IO_ADDR(DDRB), 3

    ; Enable timer TC1 with PCK/16k prescaling (attiny85 p.89)
    ldi r18, (1<<CS10)|(1<<CS11)|(1<<CS12)|(1<<CS13)
    out _SFR_IO_ADDR(TCCR1), r18

    ; Set OC1B on compare match (250), clear on 0x00 (attiny85 p.86,90)
    ldi r18, (1<<PWM1B) | (1<<COM1B0)
    out _SFR_IO_ADDR(GTCCR), r18
    ldi r18, 250
    out _SFR_IO_ADDR(OCR1B), r18

    ; Set the sleep mode to idle (attiny85 p.39).
    ldi r18, (1<<SE)
    out _SFR_IO_ADDR(MCUCR), r18

    ; Shutdown unnecessary MCU modules (attiny85 p.38)
    ldi r18, (1<<PRTIM0)|(1<<PRUSI)|(1<<PRADC)
    out _SFR_IO_ADDR(PRR), r18

    sleep
    rjmp .-4

The power consumption in the idle mode plus the blinks should keep the coin battery running for a couple of weeks, at least. A vibration sensor would improve that significantly, by enabling the MCU to go into powerdown mode and be awaken externally, but I don’t have a sensor at hand that is small enough.

This is the assembly, and the final result:

Toy Airplane Hack

He’s enjoying it. :-)

Read more
niemeyer

mgo r2014.10.12

A new release of the mgo MongoDB driver for Go is out, packed with contributions and features. But before jumping into the change list, there’s a note in the release of MongoDB 2.7.7 a few days ago that is worth celebrating:

New Tools!
– The MongoDB tools have been completely re-written in Go
– Moved to a new repository: https://github.com/mongodb/mongo-tools
– Have their own JIRA project: https://jira.mongodb.org/browse/TOOLS

So far this is part of an unstable release of the MongoDB server, but it implies that if the experiment works out every MongoDB server release will be carrying client tools developed in Go and leveraging the mgo driver. This extends the collaboration with MongoDB Inc. (mgo is already in use in the MMS product), and some of the features in release r2014.10.12 were made to support that work.

The specific changes available in this release are presented below. These changes do not introduce compatibility issues, and most of them are new features.

Fix in txn package

The bug would be visible as an invariant being broken, and the transaction application logic would panic until the txn metadata was cleaned up. The bug does not cause any data loss nor incorrect transactions to be silently applied. More stress tests were added to prevent that kind of issue in the future.

Debug information contributed by the juju team at Canonical.

MONGODB-X509 auth support

The MONGODB-X509 authentication mechanism, which allows authentication via SSL client certificates, is now supported.

Feature contributed by Gabriel Russel.

SCRAM-SHA-1 auth support

The MongoDB server is changing the default authentication protocol to SCRAM-SHA-1. This release of mgo defaults to authenticating over SCRAM-SHA-1 if the server supports it (2.7.7 and later).

Feature requested by Cailin Nelson.

GSSAPI auth on Windows too

The driver can now authenticate with the GSSAPI (Kerberos) mechanism on Windows using the standard operating system support (SSPI). The GSSAPI support on Linux remains via the cyrus-sasl library.

Feature contributed by Valeri Karpov.

Struct document ids on txn package

The txn package can now handle documents that use struct value keys.

Feature contributed by Jesse Meek.

Improved text index support

The EnsureIndex family of functions may now conveniently define text indexes via the usual shorthand syntax ("$text:field"), and Sort can use equivalent syntax ("$textScore:field") to inject the text indexing score.

Feature contributed by Las Zenow.

Support for BSON’s deprecated DBPointer

Although the BSON specification defines DBPointer as deprecated, some ancient applications still depend on it. To enable the migration of these applications to Go, the type is now supported.

Feature contributed by Mike O’Brien.

Generic Getter/Setter document types

The Getter/Setter interfaces are now respected when unmarshaling documents on any type. Previously they would only be respected on maps and structs.

Feature requested by Thomas Bouldin.

Improvements on aggregation pipelines

The Pipe.Iter method will now return aggregation results using cursors when possible (MongoDB 2.6+), and there are also new methods to tweak the aggregation behavior: Pipe.AllowDiskUse, Pipe.Batch, and Pipe.Explain.

Features requested by Roman Konz.

Decoding into custom bson.D types

Unmarshaling will now work for types that are slices of bson.DocElem in an equivalent way to bson.D.

Feature requested by Daniel Gottlieb.

Indexes and CommandNames via commands

The Indexes and CollectionNames methods will both attempt to use the new command-based protocol, and fallback to the old method if that doesn’t work.

GridFS default chunk size

The default GridFS chunk size changed from 256k to 255k, to ensure that the total document size won’t go over 256k with the additional metadata. Going over 256k would force the reservation of a 512k block when using the power-of-two allocation schema.

Performance of bson.Raw decoding

Unmarshaling data into a bson.Raw will now bypass the decoding process and record the provided data directly into the bson.Raw value. This significantly improves the performance of dumping raw data during iteration.

Benchmarks contributed by Kyle Erf.

Performance of seeking to end of GridFile

Seeking to the end of a GridFile will now not read any data. This enables a client to find the size of the file using only the io.ReadSeeker interface with low overhead.

Improvement contributed by Roger Peppe.

Added Query.SetMaxScan method

The SetMaxScan method constrains the server to only scan the specified number of documents when fulfilling the query.

Improvement contributed by Abhishek Kona.

Added GridFile.SetUploadDate method

The SetUploadDate method allows changing the upload date at file writing time.

Read more
niemeyer

The qml package is right now one of the best choices for creating graphic applications under the Go language. Part of the reason why this is true comes from the convenience of QML, a high-level domain-specific language that allows describing visual components, events, animations, and content in general in a succinct and pleasing way. The integration of such a language with Go means having both a good mechanism for describing visual content, and a good platform for doing general development under, which can range from simple data manipulation to involved OpenGL content rendering.

On the practical side, one of the implications of using such a language partnership is that every Go qml application will have some sort of resource content to deal with, carrying the QML logic. Such content may be loaded either from files on disk, or from strings in memory. Loading from a file means the content may be organized in multiple files that directly reference each other without changing the Go application, and may be updated and tested without rebuilding. Loading from a string in memory means the content needs to be self-contained, but results in a standalone binary (linking aside – still depends on Qt libraries).

There’s a well known trick to have both benefits at once, though, and the basic solution has already been made available in multiple packages: have the content on disk, and use an external tool to pack it up into a Go file that is built into the binary when the content is updated. Unfortunately, this trick alone is not enough with the qml package, because the QML engine needs to know what resources are available and where so that the right thing happens when it sees a directory being imported or an image path being referenced.

To solve that problem, the qml package has been enhanced with functionality that leverages the existing Qt resource system to pack content into the binary itself. Rather than using the upstream C++ and XML-based resource compiler, though, a new resource packer was implemented inside the qml package and made available both under a friendly Go API, and as a tool that follows common Go idioms.

The help text for the genqrc tool describes it in detail:

Usage: genqrc [options] <subdir1> [<subdir2> ...]

The genqrc tool packs all resource files under the provided subdirectories into
a single qrc.go file that may be built into the generated binary. Bundled files
may then be loaded by Go or QML code under the URL "qrc:///some/path", where
"some/path" matches the original path for the resource file locally.

Starting with Go 1.4, this tool may be conveniently run by the "go generate"
subcommand by adding a line similar to the following one to any existent .go
file in the project (assuming the subdirectories ./code/ and ./images/ exist):

    //go:generate genqrc code images

Then, just run "go generate" to update the qrc.go file.

During development, the generated qrc.go can repack the filesystem content at
runtime to avoid the process of regenerating the qrc.go file and rebuilding the
application to test every minor change made. Runtime repacking is enabled by
setting the QRC_REPACK environment variable to 1:

    export QRC_REPACK=1

This does not update the static content in the qrc.go file, though, so after
the changes are performed, genqrc must be run again to update the content that
will ship with built binaries.

The tool may be installed via go get as usual:

go get gopkg.in/qml.v1/cmd/genqrc

and once the qrc.go file has been generated, the main qml file may be
loaded with logic equivalent to:

component, err := engine.LoadFile("qrc:///path/to/file.qml")

The loaded file can in turn reference any other content that was bundled
into the Go binary.

For a better picture, this example demonstrates the use of the tool.

Read more
niemeyer

There were a few long standing issues in the yaml.v1 package which were being postponed so that they could be done at once in a single incompatible change, and the time has come: yaml.v2 is now available.

Besides these incompatible changes, other compatible fixes and improvements were performed in that push, and those were also applied to the existing yaml.v1 package so that dependent applications benefit immediately and without modifications.

The subtopics below outline exactly what changed, and how to adapt existent code when necessary.

Type errors

With yaml.v1, decoding a YAML value that is not compatible with the Go value being unmarshaled into will silently drop the offending value without notice. In many cases continuing with degraded behavior by ignoring the problem is intended, but this was the one and only option.

In yaml.v2, these problems will cause a *yaml.TypeError to be returned, containing helpful information about what happened. For example:

yaml: unmarshal errors:
  line 3: cannot unmarshal !!str `foo` into int

On such errors the decoding process still continues until the end of the YAML document, so ignoring the TypeError will produce logic equivalent to the old yaml.v1 behavior.

New Marshaler and Unmarshaler interfaces

The way that yaml.v1 allowed custom types to implement marshaling and unmarshaling of YAML content was slightly confusing and troublesome. For example, considering a CustomType with a keys field:

type CustomType struct {
        keys map[string]int
}

and supposing the goal is to unmarshal this YAML map into it:

custom:
    a: 1
    b: 2
    c: 3

With yaml.v1, one would need to implement logic similar to the following for that:

func (v *CustomType) SetYAML(tag string, value interface{}) bool {
        if tag == "!!map" {
                m := value.(map[interface{}]interface{})
                // ... iterate/validate/convert key/value pairs 
        }
        return goodValue
}

This is too much trouble when the package can easily do those conversions internally already. To fix that, in yaml.v2 the Getter and Setter interfaces are both gone and were replaced by the Marshaler and Unmarshaler interfaces.

Using the new mechanism, the example above would be implemented as follows:

func (v *CustomType) UnmarshalYAML(unmarshal func(interface{}) error) error {
        return unmarshal(&v.keys)
}

Custom-ordered maps

By default both yaml.v1 and yaml.v2 will marshal keys in a stable order which is increasing within the same type and arbitrarily defined across types. So marshaling is already performed in a sensible order, but it cannot be customized in yaml.v1, and there’s also no way to tell which order the map was originally in, as some applications require.

To fix that, there is a new pair of types that support preserving the order of map keys both when marshaling and unmarshaling: MapSlice and MapItem.

Such an ordered map literal would look like:

m := yaml.MapSlice{{"c", 3}, {"b", 2}, {"a", 1}}

The MapSlice type may be used for variables going in and out of the yaml package, or in struct fields, map values, or anywhere else sensible.

Binary values

Strings in YAML must be valid UTF-8 or UTF-16 (with a byte order mark for the latter), and for binary data the specification defines a standard !!binary tag which represents the raw data encoded (encrypted?) as base64. This is now supported both in yaml.v1 and yaml.v2, transparently. That is, any string value that is not valid UTF-8 will be base64-encoded and appropriately tagged so that it roundtrips as the same string. Short strings are inlined, while long ones are automatically broken into several lines and represented in a proper style. For example:

one: !!binary gICA
two: !!binary |
  gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI
  CAgICAgICAgICAgICAgICAgICAgICAgICAgICA

Multi-line strings

Any string that contains new-line characters (‘\n’) will now be encoded using the literal style. For example, a value that would before be encoded as:

key: "line 1\nline 2\nline 3\n"

is now encoded by both yaml.v1 and yaml.v2 as:

key: |
  line 1
  line 2
  line 3

Other improvements

Besides these major changes, some assorted minor improvements were also performed:

  • Better handling of top-level “null”s (issue #35)
  • Marshal base 60 floats quoted for YAML 1.1 compatibility (issue #34)
  • Better error on invalid map keys (issue #25)
  • Allow non-ASCII characters in plain strings (issue #11).
  • Do not catch unrelated panics by mistake (commit a6dc653f)

For obtaining the yaml.v1 improvements:

go get -u gopkg.in/yaml.v1

For updating to yaml.v2, adapt the code as necessary given the points above, replace the import path, and run:

go get -u gopkg.in/yaml.v2

Read more
niemeyer

As detailed in the preliminary release of qml.v1 for Go a couple of weeks ago, my next task was to finish the improvements in its OpenGL API. Good progress has happened since then, and the new API is mostly done and available for experimentation. At the same time, there’s still work to do on polishing edges and on documenting the extensive API. This blog post aims to present the improvements made, their internal design, and also to invite help for finishing the pending details.

Before diving into the new, let’s first have a quick look at how a Go application using OpenGL might look like with qml.v0. This is an excerpt from the old painting example:

import (
        "gopkg.in/qml.v0"
        "gopkg.in/qml.v0/gl"
)

func (r *GoRect) Paint(p *qml.Painter) {
        width := gl.Float(r.Int("width"))
        height := gl.Float(r.Int("height"))
        gl.Enable(gl.BLEND)
        // ...
}

The example imports both the qml and the gl packages, and then defines a Paint method that makes use of the GL types, functions, and constants from the gl package. It looks quite reasonable, but there are a few relevant shortcomings.

One major issue in the current API is that it offers no means to tell even at a basic level what version of the OpenGL API is being coded against, because the available functions and constants are the complete set extracted from the gl.h header. For example, OpenGL 2.0 has GL_ALPHA and GL_ALPHA4/8/12/16 constants, but OpenGL ES 2.0 has only GL_ALPHA. This simplistic choice was a good start, but comes with a number of undesired side effects:

  • Many trivial errors that should be compile errors fail at runtime instead
  • When the code does work, the developer is not sure about which API version it is targeting
  • Symbols for unsupported API versions may not be available for linking, even if unused

That last point also provides a hint of another general issue: portability. Every system has particularities for how to load the proper OpenGL API entry points. For example, which libraries should be linked with, where they are in the local system, which entry points they support, etc.

So this is the stage for the improvements that are happening. Before detailing the solution, let’s have a look at the new painting example in qml.v1, that makes use of the improved API:

import (
        "gopkg.in/qml.v1"
        "gopkg.in/qml.v1/gl/2.0"
)

func (r *GoRect) Paint(p *qml.Painter) {
        gl := GL.API(p)
        width := float32(r.Int("width"))
        height := float32(r.Int("height"))
        gl.Enable(GL.BLEND)
        // ...
}

With the new API, rather than importing a generic gl package, a version-specific gl/2.0 package is imported under the name GL. That choice of package name allows preserving familiar OpenGL terms for both the functions and the constants (gl.Enable and GL.BLEND, for example). Inside the new Paint method, the gl value obtained from GL.API holds only the functions that are defined for the specific OpenGL API version imported, and the constants in the GL package are also constrained to those available in the given version. Any improper references become build time errors.

To support all the various OpenGL versions and profiles, there are 23 independent packages right now. These packages are of course not being hand-built. Instead, they are generated all at once by a tool that gathers information from various sources. The process can be tersely described as:

  1. A ragel-based parser processes Qt’s qopenglfunctions_*.h header files to collect version-specific functions
  2. The Khronos OpenGL Registry XML is parsed to collect version-specific constants
  3. A number of tweaks defined in the tool’s code is applied to the state
  4. Packages are generated by feeding the state to text templates

Version-specific functions might also be extracted from the Khronos Registry, but there’s a good reason to use information from the Qt headers instead: Qt already solved the portability issue. It works in several platforms, and if somebody is using QML successfully, it means Qt is already using that system’s OpenGL capabilities. So rather than designing a new mechanism to solve the same problem, the qml package now leverages Qt for resolving all the GL function entry points and the linking against available libraries.

Going back to the example, it also demonstrates another improvement that comes with the new API: plain types that do not carry further meaning such as gl.Float and gl.Int were replaced by their native counterparts, float32 and int32. Richer types such as Enum were preserved, and as suggested by David Crawshaw some new types were also introduced to represent entities such as programs, shaders, and buffers. The custom types are all available under the common gl/glbase package that all version-specific packages make use of.

So this is all working and available for experimentation right now. What is left to do is almost exclusively improving the list of function tweaks with two goals in mind, which will be highlighted below as those are areas where help would be appreciated, mainly due to the footprint of the API.

Documentation importing

There are a few hundred functions to document, but a large number of these are variations of the same function. The previous approach was to simply link to the upstream documentation, but it would be much better to have polished documentation attached to the functions themselves. This is the new documentation for MultMatrixd, for example. For now the documentation is being imported manually, but the final process will likely consist of some automation and some manual polishing.

Function polishing

The standard C OpenGL API can often be translated automatically (see BindBuffer or BlendColor), but in other cases the function prototype has to be tweaked to become friendly to Go. The translation tool already has good support for defining most of these tweaks independently from the rest of the machinery. For example, the following logic changes the ShaderSource function from its standard from into something convenient in Go:

name: "ShaderSource",
params: paramTweaks{
        "glstring": {rename: "source", retype: "...string"},
        "length":   {remove: true},
        "count":    {remove: true},
},
before: `
        count := len(source)
        length := make([]int32, count)
        glstring := make([]unsafe.Pointer, count)
        for i, src := range source {
                length[i] = int32(len(src))
                if len(src) > 0 {
                        glstring[i] = *(*unsafe.Pointer)(unsafe.Pointer(&src))
                } else {
                        glstring[i] = unsafe.Pointer(uintptr(0))
                }
        }
`,

Other cases may be much simpler. The MultMatrixd tweak, for instance, simply ensures that the parameter has the proper length, and injects the documentation:

name: "MultMatrixd",
before: `
        if len(m) != 16 {
                panic("parameter m must have length 16 for the 4x4 matrix")
        }
`,
doc: `
        multiplies the current matrix with the provided matrix.
        ...
`,

and as an even simpler example, CreateProgram is tweaked so that it returns a glbase.Program instead of the default uint32.

name:   "CreateProgram",
result: "glbase.Program",

That kind of polishing is where contributions would be most appreciated right now. One valid way of doing this is picking a range of functions and importing and polishing their documentation manually, and while doing that keeping an eye on required tweaks that should be performed on the function based on its documentation and prototype.

If you’d like to help somehow, or just ask questions or report your experience with the new API, please join us in the project mailing list.

Read more
niemeyer

Announcing qml v1 for Go

After a few weeks of slow progress on the qml package for Go, action is starting again.

The first important change is the release of the new v1 API, which is motivated by conversations and debugging sessions we’ve had at GopherCon back in April. The problem being solved is that Mac OS requires its graphic activities to be held in the first thread in the running process, and that’s incompatible with the API provided by the qml package in v0.

In practice, code in v0 looked like this:

func main() {
        qml.Init(nil)
        engine := qml.NewEngine()
        ...
}

This interface implies that logic must continue running in the initial goroutine after the GUI initialization is performed by qml.Init. Internally qml.Init will spawn a goroutine which will take over a process thread to hold the main GUI loop, and this works fine in most OSes, and used to work okay in Mac OS too, probably by chance (there’s a chance the thread locked up is the first one), but recently it started to fail more consistently and enabled the problem to be more clearly spotted and solved.

The solution requires a minor incompatible change. In v1, the initialization procedure takes the following form:

func main() {
        err := qml.Run(run)
        ...
}

func run() error {
        engine := qml.NewEngine()
        ...
}

This subtle change allows the qml package to deterministically take over the initial process thread for running the GUI loop, and is all that needs to change for an application to be initialized under the new API.

This issue also affects testing, but qml.Run isn’t suitable in those cases because the “go test” machinery doesn’t provide an appropriate place for calling it. As a good workaround, the GUI loop may be setup from an init function within any *_test.go file with:

func init() { qml.SetupTesting() }

With that in place, all tests can assume the GUI loop is running and may then use the qml package functionality arbitrarily.

Besides these changes related to initialization, the v1 API will also hold a new GL interface that is still being worked on. The intention was to have that interface ready by the announcement date, but developing the new API in an isolated branch is hindering collaboration on the new package, and is also unfortunate for people that depend on the Mac OS fixes, so v1 is being released with an unstable gl package under the path "gopkg.in/qml.v1/work-in-progress/gl". This package will suffer significant changes in the coming weeks, so it’s best to avoid publishing any applications that depend on it before it is moved to the final import path.

The package source code, documentation, and import path are referenced from gopkg.in, and to talk about any of these changes please join the mailing list.

Thanks to Andrew Gerrand for the help debugging and testing the new initialization procedure.

Read more
niemeyer

This is a special release of mgo, the Go driver for MongoDB. Besides the several new features, this release marks the change of the Go package import path to gopkg.in, after years using the current one based on a static file that lives at labix.org. Note that the package API is still not changing in any backwards incompatible way, though, so it is safe to replace in-use import paths right away. Instead, the change is being done for a few other reasons:

  • gopkg.in is more reliable, offering a geographically distributed replicated deployment with automatic failover
  • gopkg.in has a nice landing page with pointers to source code, API documentation, versions available, etc.
  • gopkg.in is backed by git and github.com, which is one of the most requested changes over the years it has been hosted with Bazaar at Launchpad

So, from now on the import path to use when using and retrieving the package should be:

Starting with this release, the source code is also maintained and updated only on GitHub at:

The old repository and import path will remain up for the time being, to ensure existing applications continue to work, but it’s already holding out-of-date code.

In terms of changes, the r2014.07.21 release brings the following news:

Socket pool size limit may be changed

A new Session.SetPoolLimit was added for changing the number of sockets that may be in use for the desired server before the session will block waiting for an available one. The value defaults to 4096 if unset.

Note that the driver actually had the ability to limit concurrent sockets for a long time, and the hardcoded default was already 4096 sockets per server. The reason why this wasn’t exposed via the API is because people commonly delegate to the database driver the management of concurrency for the whole application, and this is a bad design. Instead, this limit must be set to cover any expected workload of the application, and application concurrency should be controlled “at the door”, by properly restricting used resources (goroutines, etc) before they are even created.

This feature was requested by a number of people, with the most notable threads being with Travis Reader, from IronIO, and the feature is finally being added after Tyler Bunnel, from Stretchr.com, asked for the ability to increase that limit. Thanks to both, and everybody else who chimed in as well.

TCP name resolution times out after 10 seconds

Previous releases had no driver-enforced timeout.

Reported by Cailin Nelson, MongoDB.

json.Number is marshalled as a number

The encoding/json package in the standard library allows numbers to be held as a string with the json.Number type so that code can take into account their visual representation for differentiating between integers and floats (a non-standard behavior). The mgo/bson package will now recognize those values properly, and marshal them as BSON int64s or float64s based on the number representation.

Patch by Min-Young Wu, Facebook.

New GridFile.Abort method for canceling uploads

When called, GridFile.Abort cancels an upload in progress so that closing the file being written to will drop all uploaded chunks rather than atomically making the file available.

Feature requested by Roger Peppe, Canonical.

GridFile.Close drops chunks on write errors

Previously, a write error would prevent the

File">

File">GridFile file from being created, but would leave already written chunks in the database. Now those chunks are dropped when the file is closed.

Support for PLAIN (LDAP) authentication

The driver can now authenticate against MongoDB servers set up using the PLAIN mechanism, which enables authentication against LDAP servers as documented.

Feature requested by Cailin Nelson, MongoDB.

Preliminary support for Bulk API

The ability to execute certain operations in bulk mode is being added to the 2.6 release of the MongoDB server. This allows queuing up inserts, updates, and removals to be sent in batches rather than one by one.

This release includes an experimental API to support that, including compatibility with previous version of the MongoDB server for the added features. The functionality is exposed via the Collection.Bulk method, which returns a value with type *mgo.Bulk that contains the following methods at the moment:

  • Bulk.Insert – enqueues one or more insert operations
  • Bulk.Run – runs all operations in the queue
  • Bulk.Unordered – use unordered mode, so latter operations may proceed when prior ones failed

Besides preparing for the complete bulk API with more methods, this preliminary change adds support for the ContinueOnError wire protocol flag of the insert operation in a way that will remain compatible with the upcoming bulk operation commands, via the unordered mode.

The latter feature was requested by Chandra Sekar, and justifies the early release of the API.

Various compatibility improvements in user handling for 2.6

The MongoDB 2.6 release includes a number of changes in user handling functionality. The existing methods have been internally changed to preserve compatibility to the extent possible. Certain features, such as the handling of the User.UserSource field, cannot be preserved and will cause an error to be reported when used against 2.6.

Wake up nonce waiters on socket death

Problem reported and diagnosed by John Morales, MongoDB.

Don’t burn CPU if no masters are found and FailFast is set

Problem also reported by John Morales, MongoDB.

Stop Iter.Next if failure happens at get-more issuing time

Problem reported by Daniel Gottlieb, MongoDB.

Various innocuous race detector reports fixed

Running the test suite with the race detector enabled would raise various issues due to global variable modifications that are only done and only accessible to the test suite itself. These were fixed and the race detector now runs cleanly over the test suite.

Thanks to everybody that contributed to this release.

Read more