Canonical Voices

What Blogging in the Wind talks about

rvr

In the last years I've been using an iPhone and I recently wanted to transfer the contact list to my phone with Ubuntu Touch. These are the steps I followed.

What is required:

  • iPhone.
  • Mac OSX.
  • Ubuntu Desktop.
  • Ubuntu Touch device.

How to do it:

  1. Export Contacts from the iPhone.
  2. Transfer the contact file to Ubuntu Touch.
  3. Import the contact file.

Let's go step by step.

Exporting the Contact list

There are different ways to export the Contact list from the iPhone to elsewhere. One option is to install an application like My Contacts Backup, and then to transfer the file containing the contacts to the computer. This option is preferred because avoids the need of a Mac desktop.

In my case, I already had my iPhone contacts synchronized with iCloud (option that I don't remember to have activated) and had the MacBook Air at hand. So these are the steps that I followed. First, the iCloud account must be setup to synchronize Contacts. Open Contacts in OSX, go to application menu and press to Accounts...

Screen Shot 2014-12-19 at 15.57.11Then select iCloud and introduce your account credentials.

Screen Shot 2014-12-19 at 15.57.21Finally, enable Contacts synchronization.

Screen Shot 2014-12-19 at 15.59.15After sync is done, the contacts are available in the application. Next step is to export all the list to a vCard file. To do that, select all contacts.

Screen Shot 2014-12-19 at 11.42.39

And export them.

Screen Shot 2014-12-19 at 11.43.07

Give a name to the file, and we are done in OSX.

Screen Shot 2014-12-19 at 11.43.41

The /Users/<user>/Documents/phone-contacts.vcf is a text file in vCard format that contains all the details of our contacts. Use a USB key to copy that file and copy it to an Ubuntu desktop.

Re-formatting

Once the file is in Ubuntu desktop, we need to tweak it a little bit, since the import tool in Ubuntu Touch doesn't like the vcf file as exported directly by OSX. The problem is that each contact entry in the file is not separated by a new line, so it must be added afterward. Open a Terminal and type this command:

Screenshot from 2014-12-19 16:38:49

In text:

$ cat phone-contacts.vcf | sed -e "s/END:VCARD/END:VCARD\n/" > phone-contacts-touch.vcf

A new (and correct) file will be created, phone-contacts-touch.vcf.

Transfer to Ubuntu Touch

Now connect the Ubuntu Touch device to Ubuntu desktop and install some packages:

Screenshot from 2014-12-19 16:49:19

$ sudo apt-get install phablet-tools android-tools-adb

In the Ubuntu Touch phone, go to System Settings > About this phone > Developer Mode and  enable the developer mode. Connect the phone to Ubuntu via USB and type this command to transfer the file to Ubuntu Touch:

Screenshot from 2014-12-19 16:56:31

$ adb push phone-contacts-touch.vcf /home/phablet/Documents

And log into Ubuntu Touch:

$ phablet-shell

Importing the contacts

Screenshot from 2014-12-19 17:01:58

And now, the final step that will import the contacts. That will be done using SyncEvolution tool that comes by default in Ubuntu Touch.

Screenshot from 2014-12-19 17:04:52

$ syncevolution --import ~/Documents/phone-contacts-touch.vcf backend=evolution-contacts database=Personal

And that's it. Now, all the contacts are imported and available in the Address Book app in Ubuntu Touch. Enjoy!

Wrap up

Of course, with some development effort this task could be much easier. A path that I don't know whether it is possible are direct calls to iCloud API's from Ubuntu Touch, but I doubt it. I've took a quick look to  libimobiledevice library. Many years ago there was a tool, python-idevicesync, that was based on it and provided a contact export feature. It's now out of sync with the library API. Another good step would be an Ubuntu Touch app that could read vCard files, and avoid the need to call SyncEvolution using the command line. I'm sure the community will be looking into this features soon.

Read more
rvr

For Mobile World Capital's blog, I was asked to write an article about the impact of free software in the history of mobile. This is it:

«Today Apple and Google dominate the market of mobile operating systems. In a few years, these companies have gained privileged positions in the telecommunications market, overshadowing the terminal manufacturers. Motorola Mobility is now in the hands of Google. But, ironically, the computer industry owes much to the phone industry. Unintentionally, AT&T created the operating system that is today ruling the world, from mobile phones to supercomputers. Or, if you will, the key programming language of the last 40 years. While doing so, they also planted the seed of the free software».

Continue at  Mobile and open source: how Bell Labs (AT&T) created the system of the world.

PS: This is the English translation, it's also available in Catalan and the original, in Spanish.

Read more
rvr

In my daily work at Canonical I use VM's quite often to test Ubuntu Web Apps and new browser releases in different environments. I use a MacBook Pro 15" (mid 2012) as the main computer, currently running Ubuntu 13.04. This computer had rEFIt to boot and the default OSX system. Of the 500 GB, just 75 were dedicated to Linux, so I was forced to delete VM's or backup them in an external hard disk to re-use them. Finally, I decided to buy a SSD, which are quite cheap nowadays.

The SSD I bought was a Samsung 840 250 GB (not Pro version, which has some additional features). It costs 179 €.

This are the steps I followed to move my Ubuntu setup from a HD to a SSD.

  1. Burn a DVD with Ubuntu. I reused an old 12.04 disk, the -mac version.
  2. Boot Ubuntu from the DVD.
  3. Attach an external USB drive. This is used to (backup and) copy the partition from the HD to the SSD.
  4. Run gparted as root to copy the Linux partition (in my case, an ext4).

To copy the partition, I resized the USB drive partitions to live room to the HD's Linux partition, which is 75 GB. Then, in gparted I selected the partition from the HD, selected "Copy" from the partition menu and then "Paste" it in the spare space in the USB drive. This step took +40 minutes. At this point, the Linux partition is available in the external disk.

After that, I switched off the computer and replaced the HD with the SSD. Follow the link to see how. Now, the second part: to move and setup the system to the solid state disk. The SSD disk was blank, so it needed proper configuration.

  1. Boot Ubuntu from the DVD.
  2. Attach the external USB drive.
  3. Run gparted.
  4. Create a partition table in the SSD. I used GUID Partition Table (gpt) format, the one the original HD uses.
  5. Partition the SDD.
    • Create an EFI partition. The first partition has FAT32 format, 200 MB in size, "EFI" as label and "grub_boot" flag. 
    • Create the swap partition. At the end of the disk, I created a 10 GB "linux-swap" partition.
    • The rest of the disk will be available for the main Linux partition.
  6. Copy the Linux partition to the SSD. In gparted, select the Linux partition in the USB drive, copy and paste it in the SSD. This takes +45 minutes.
  7. Resize the Linux partition to fill the entire disk and flag it as "boot".

Congratulations! The Linux partition is now copied bit-by-bit in the SSD. However, it cannot boot. The reasons are: rEFIt (the bootloader) is not installed; the Linux partitions  are not properly configured. To do that, we need to modify the file /etc/fstab in the Linux partition (SSD).

And this is the final third step: setup the system to properly boot. In my SSD, /dev/sda1 is the EFI partition, /dev/sda2 the Linux partition and /dev/sda3 the swap partition. You may have different setup. /etc/fstab must be changed to reflect this addresses. From a terminal:

$ sudo mkdir /media/root 

$ sudo mount /dev/sda2 /media/root

The Linux partition is accesible in the directory /media/root/. The file /etc/fstab/ of the Linux partition can be edited now at /media/root/etc/fstab/

$ gksu gedit /media/root/etc/fstab

This is the original content:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda5 during installation
UUID=24acabd4-2fcb-49aa-9fb3-ce9e657d4465 / ext4 errors=remount-ro,user_xattr 0 1
# swap was on /dev/sda7 during installation
UUID=c40532ab-5716-47bc-9b95-3672b834c6a2 none swap sw 0 0

Here, the modifications:

# / was on /dev/sda5 during installation
/dev/sda2 / ext4 discard,errors=remount-ro,user_xattr 0 1
# swap was on /dev/sda7 during installation
/dev/sda3 none swap sw 0 0

The blkid command can be run as root to find out the UUID strings of the devices and use them instead.

Then, I downloaded the compiled version of rEFInd, which is a fork of the rEFIt bootloader and able to run from Linux, Mac and Windows.

$ cd /media/root/root/

$ sudo wget http://downloads.sourceforge.net/project/refind/0.6.11/refind-bin-0.6.11.zip

This is almost done. We now "log" into the Linux partition (SSD) to apply the changes and setup the bootloaders. In order to do that successfully, the /proc and /dev from the live DVD are mounted to the Linux partition and the EFI partition (this is needed by rEFInd).

$ sudo mkdir /media/root/boot/efi

$ sudo mount -B /proc /media/root/proc

$ sudo mount -B /dev /media/root/dev

$ sudo mount /dev/sda1 /media/root/boot/efi

$ sudo chroot /media/root/

Now we're "logged" in the Linux partition as the root user. This is when GRUB and rEFInd are installed in the SSD to be able to boot Linux from the Mac.

# mount -t sysfs sysfs /sys

# cd /root/

# unzip refind-bin-0.6.11.zip

# cd refind-bin-0.6.11

# ./install-bin.sh --esp --alldrivers

# grub-install /dev/sda

# update-grub

# exit

And that's all. The system should be able to boot the Linux partition from the SSD.

Some caveats and open questions:

  • I installed rEFInd first, without GRUB. rEFInd and the system wasn't able to boot.
  • For some reasons, rEFInd in my system is much slower than rEFIt. It takes around 40 seconds to show up.
  • After installing GRUB, I'm not able to mount the EFI partition. Now has an unkown partition format.
  • Does GRUB really needs rEFInd?

References

Read more
rvr

Happy 32,013!

8330375713_884426e531_z
Look carefully at the picture. More than 1,000 generations separate us from that hand. This person lived 30,000 years ago in southern France. The prehistoric painters of Chauvet Cave transport us to a distant and different time. The ice glaciers covered Europe and Neanderthals were close to extinction. Represented on the walls of the cave are other species of large mammals, also extinct. In light of the embers, skillfull hands like that drew rhinos in combat, megaloceros, cave bears, aurochs and mammoths. A landslide sealed and preserved this cave in pristine condition, until its recent discovery.

Look at the palm of your hand. Look at the picture again. It is a human hand. Troglodytes are often portraited as a crude and inferior species. But when we watch what they achieved, this prejudice is impossible to sustain. Despite the time that separates us, the drawings demonstrate great expertise, imagination and abstraction, and an artistic quality recognizeable as distinctly human. Anyone, from any culture, is able to see himself reflected in the drawings. We are descendants of these cave painters. Genetically, a baby born today would be almost identical to a baby born at that time. What differentiates us is not in the genes.

The world has changed since then. Glaciers have retreated from Europe. The climate is warmer. Animals and plants were domesticated. Hands like those created bricks. With bricks, towns were build. In towns, organized societies fluorished. Social complexity exceeded individual capacity. To account for large quantities became a need. And art gave way to knowledge: 5,500 years ago, marks were made on clay tablets to represent numbers. From there, and in a few hundred years, writing was invented. Thanks to the written symbols, accurate information could be transmitted to the future. And this is what made us modern. Our civilization is supported by the slow accumulation of written knowledge. How to make a building. How to cure a disease. How the universe began. The knowledge to be meaningful, to be valuable, must be shared.

Now put your hand over the image. Think about what you share with her. The length of the fingers. The chin. The sense of warmth. And now think all the possibilities that yours has, that that hand never had. To play the piano. To solve an equation. To write a poem.

My wishes for 2013: Learn. Teach. Create. Share. Transcend. Be human.

Read more
rvr

Transoceanic flights can be boring, even with an iPad to watch movies and play Angry Birds. In my last trip to Boston for a meeting at Canonical's offices, I missed a lot a small computer to play with in the airplane. The 15 inches of the MacBook Pro are good to be productive, but not suitable for the small airline seat. I own a somewhat outdated 9" Acer Aspire One, one of the first netbooks to be released. Unfortunately, it was not working properly.

I bought this Aspire One AOA110 near three years ago. I used it primarly as a portable computer and at hme as the device of choice to surf and read documentation (then, it came the iPad). Some months ago I upgraded the aging operating system, the rpm-based Linux distro called Linpus with Ubuntu Lucid. However, I discovered an issue with the battery: it won't work. The netbook could only be used with with the power cord. I thought it was a hopeless case.

However, it seems that Aspire One battery issue is a common and known problem, which only requires a BIOS update. Acer provides the firmware update in its site, but the programs to execute are for DOS/Windows. Here are the instructions to upgrade it using Linux:

  1. Download an up-to-date BIOS firmware.
  2. Uncompress the downloaded ZIP file. 
  3. Download an ISO image of FreeDOS, an open source version of MS-DOS. I recommend this FreeDOS 1.1 image.
  4. Burn the ISO image to a USB key (beware to backup the USB key contents, burning hte ISO image will delete its contents). I used "dmesg" to find the USB key device (/dev/sdb) and then typed the command below (again, be careful to identify the correct /dev/ or you can destroy your data).

    $ sudo dd if=FreeDOS-1.1-USB-Boot.img of=/dev/sdb

  5. Unplug and plug again the USB key, now it will be mounted.
  6. Reboot and you're done.

This way, I rescued the netbook. Of course, it feels slow compared to more recent computers, but its days may not be over yet :)

Read more
rvr

Fernando Tricas always has interesting things to say. In a recent post he talks about The life of links and digital content (Spanish):

«We tend to assume that digital [content] is forever. But anyone who accumulates enough information also knows that sometimes its difficult to find it, in other cases it breaks and, of course, there is a non-zero probability that things go wrong when hosted by third-party services. It is an old topic here, remember Will we have all this information in the future? . The topic resurfaces as news in the light of Currently charged by the article that can be read at A Year After the Egyptian Revolution, 10% of Its Social Media Documentation Is Already Gone».

In the comments, Anónima said: «Given a time t and an interval Δt, the larger Δt, the more likely is that all information in a time t-Δt you want to find is gone». This sounded like an statement to check, Thus, I decided to do an experiment with del.icio.us' bookmarks.

In delicious.com/rvr I have archived around 4000 links from 2004. So, I downloaded the backup file, an HTML file with all links and metadata (date, title, tags). I developed a python script to process this file: go through the links and save its current status (whether the link is alive or not). With another script, the status were processed to generate the statistics. These are the results:

Captura de pantalla 2012-04-12 a la(s) 01.02.39

As can be seen, there is a correlation between the age of the links and the probability of being dead. For the 10% who cited the Egyptian revolution, in the case of my delicious, we must go back three years ago (2009). But at 6 years from now, a quarter of the links are now defunct. Of course, the sample is very small shouldn't be representative. It would be interesting to compare it with other accounts and to extend the time span: How many links are still alive after 10 or 15 years? Is it the same with information stored in other media? Are all this death links resting in peace in a forgotten Google's cache disk?

I imagine that sometime in the future, librarians will begin to worry not only to digitize remote past documents, but also to preserve those of the present.

In case you are interested, the code to generate such data is available at github.com/vrruiz/delicious-death-links. The spreadsheet is also available in Google Docs .

Read more
rvr

My new job: Ubuntu

Captura de pantalla 2012-01-23 a la(s) 01.09.21

New year, new life. Today was my first one at Canonical. Starting today, I am spending my time to two of my passions: Linux and free software. Canonical is the company behind Ubuntu, "Linux for human beings". This project, created by Mark Shuttleworth, wanted to create the most friendly Linux distribution for the average user. Its popularity backs this vision.

Footer_logo
My love for Unix systems, like Linux, is inherited. When I was a kid, we had computers at home. One of them had a hard disk partition with Xenix, Microsoft's Unix. But my first serious contact with Unix was at university: dumb terminals and graphic stations with HP/UX. It was there where I discovered Linux, 17 years ago. The staff at University of Las Palmas de Gran Canaria were truly free software pioneers in Spain. They allowed the germination of many initiatives: the Spanish Linux mailing list (l-linux@calvo.teleco.ulpgc.es), the Canary Islands Linux User Group, the first university Libre Software Office... and a great pool of hackers. So, it can't be a surprise that I'm very happy to join a company whose core values are so tied to open source and collaboration.

My position at Canonical is Quality Engineer, on the Product Strategy department. I will be working closely with Álvaro López, Robert Carr and other excellent developers. And -although not in the same team-, my brother Alberto Ruiz is also working in that department. 

Exciting times ahead to learn, share and enjoy!

Read more
rvr

In the first post we described what WebKit and JavaScriptCore is, how to program a simple WebKitGTK+ application and how to extend the JavaScript functionality with a dumb (empty) class. In the second post, we extended JavaScript to enable desktop notifications. In the third post,  WebKit/JavaScriptCore were extended to access UPower properties using D-Bus. And in the fourth post, WebKit/JavaScriptCore were extended to use JavaScript callbacks for UPower methods.

In this article we are introducing another way to extend JavaScriptCore: Seed.

 

Seed: A JavaScriptCore engine

In previous articles we've seen the power of JavaScriptCore. However, its features are tied to the WebKit engine. Wouldn't be great if we could run standalone JavaScript programs using JavaScriptCore? That's exactly what Seed does. Seed is a JavaScript interpreter, able to run pure JavaScript programs. It uses WebKitGTK+, not to browse, but to run JavaScript code.

Indeed, Seed comes in two flavors:

  • As a command line interpreter (/usr/bin/seed).
  • As a C library (libseed).

As a command line, Seed is like a Python or Perl interpreter: a script is loaded and executed. However, outside the web environment, JavaScript is pretty much useless. That's why, as we did in our previous examples, Seed adds support for some native libraries. And specifically, Seed can be used to program GTK+ applications using JavaScript, thanks to its GObject Introspection/JavaScript bridge.

libseed, the C library, allows third-parties to easily build scripting capabilities for their programs. It adds a thin layer to JavaScriptCore API. The functions and methods available in JSC are usually available in libseed. This mean we can define new features and access JavaScript objects from C.

 

A simple Seed script

To install Seed in Ubuntu, type:

$ sudo apt-get install seed libseed-gtk3-dev

This is a simple GTK+ program, which just shows a window with a button.

#!/usr/bin/seed

/* Imports GTK+ libraries */
Gtk = imports.gi.Gtk;

/* Inits GTK+ */
Gtk.init(null, null);

/* Creates window */
var window = new Gtk.Window();

/* Terminates program if window is closed */
window.signal.hide.connect(Gtk.main_quit);

/* Creates button */
var button = new Gtk.Button();
button.set_label('Hello world');

/* Terminates program if button is clicked */
button.signal.clicked.connect(Gtk.main_quit);

/* Add button to window */
window.add(button);

/* Shows */
window.set_default_size(100,100);
window.show_all();

/* Main loop */
Gtk.main();

To run it:

$ seed gtk.js

This shows:

Pantallazo del 2011-11-14 01:48:08

WebViews and Seed

In our previous articles, we've extended JavaScript features manually, programming C methods to create JS classes. Those classes were available to our Web Views. But Seed already provides a lot of libraries. In the next example, we'll see how to use libseed to extend WebKit.

Seed is built upon JavaScriptCore, which is a core element of WebKit. In order to expose libseed features in a browser, we need to connect Seed with WebKit. How to do that? When a WebView is created, a JavaScriptCore's Context Object is created. This Context Object stores the status of the JavaScript engine. Generally, Seed creates is own Context Object, but there is function to use an already created context. Using this function, Seed will populate the WebView's context with Seed's libraries.

This the source code of our program which exposes Seed libraries to a WebKitGTK+ web view.

#include <stdio.h>
#include <gtk/gtk.h>
#include <webkit/webkit.h>
#include <seed.h>
#include <JavaScriptCore/JavaScript.h>

SeedEngine *engine;

static void window_object_cleared_cb(WebKitWebView  *web_view,
                                WebKitWebFrame *frame,
                                gpointer        context,
                                gpointer        arg3,
                                gpointer        user_data)

{
    JSGlobalContextRef jsContext = webkit_web_frame_get_global_context(frame);
    engine = seed_init_with_context (NULL, NULL, jsContext);
}

static void destroy_cb(GtkWidget* widget, gpointer data)
{
    g_free (engine);
    gtk_main_quit ();
}

static GtkWidget* main_window;
static WebKitWebView* web_view;

static GtkWidget* create_browser()
{
    GtkWidget* scrolled_window = gtk_scrolled_window_new (NULL, NULL);
    gtk_scrolled_window_set_policy (GTK_SCROLLED_WINDOW (scrolled_window), GTK_POLICY_AUTOMATIC, GTK_POLICY_AUTOMATIC);

    web_view = WEBKIT_WEB_VIEW (webkit_web_view_new ());
    gtk_container_add (GTK_CONTAINER (scrolled_window), GTK_WIDGET (web_view));

    g_signal_connect (G_OBJECT (web_view), "window-object-cleared", G_CALLBACK(window_object_cleared_cb), web_view);

    return scrolled_window;
}

static GtkWidget* create_window()
{
    GtkWidget* window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    gtk_window_set_default_size (GTK_WINDOW (window), 500, 500);
    g_signal_connect (G_OBJECT (window), "destroy", G_CALLBACK (destroy_cb), NULL);
    return window;
}

int main (int argc, char* argv[]) {
    gtk_init (&argc, &argv);
    if (!g_thread_supported())
       g_thread_init (NULL);

    GtkWidget* vbox = gtk_vbox_new(FALSE, 0);
    gtk_box_pack_start(GTK_BOX(vbox), create_browser (), TRUE, TRUE, 0);

    main_window = create_window();
    gtk_container_add(GTK_CONTAINER (main_window), vbox);

    gchar* uri = (gchar*) "file://webkit-seed.html";
    webkit_web_view_load_uri(web_view, uri);

    gtk_widget_grab_focus (GTK_WIDGET (web_view));
    gtk_widget_show_all (main_window);
    gtk_main ();

    return 0;
}

To compile it, type:

$ gcc -o webkit-seed webkit-seed.c `pkg-config --cflags --libs webkitgtk-3.0 seed`

And this is the webkit-seed.html file:

<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>

    <script type="text/javascript">
    var sqlite = imports.sqlite;
    sql = new sqlite.Database('/tmp/webkit_seed_test.db')
    sql.exec('CREATE TABLE test (id INTEGER PRIMARY KEY, data TEXT)');
    sql.close();
    </script>

</body>
</html>

To run the program:

$ webkit-seed

This time, the browser will appear, but the fun action will be in the /tmp directory. There we'll found a new SQLite 3 database, with a two-field table.

Of course, Seed gives access to a lot of sensible libraries. Safety would be at risk if we let any JavaScript program to run this way, but it also opens a whole world of possibilities!

P.S.: Of course, I tried to run the GTK+ example inside the WebView. Unfortunately, it didn't work :-(

Read more
rvr

In the previous articles dedicated to WebKit, we've used the WebKitGTK+ in plain C. Thanks to GObject and GLib libraries,  doing bindings for others languages is quite easy. And of course, bindings for WebKitGTK+ exists for Python, one of my favorite programming languages. Python is ideal for rapid prototyping and/or fast development.

To use WebKitGTK+ in Python and Ubuntu, python-webkit library must be installed:

$ sudo apt-get install python-webkit

This is the source code of a simple WebKit browser in PyGTK+.

#!/usr/bin/env python
import pygtk
import gtk
import webkit

# Creates the GTK+ app
self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
self.scrolled_window = gtk.ScrolledWindow()
# Creates a WebKit view
self.web_view = webkit.WebView()
self.scrolled_window.add(self.web_view)
self.window.add(self.scrolled_window)
# Sets the window size
self.window.set_default_size(1024, 800)
# Loads this URL
self.web_view.load_uri('http://rvr.typepad.com/')
gtk.main()

Using Python we get rid of the manual memory management. Now, let's use WebKit and Python to build a tool.

Facebook and desktop applications: OAuth 2.0.

Facebook has a programmable API, so application can interact with user data. Probably you know Farmville, the popular Facebook game. Farmville uses Facebook API to access to our data. For applications to be able to interact with our data, developers must register the application in Facebook and the each user grants permission access to the app.

This protocol to obtain user access is called OAuth 2.0, and is has become a popular standard among many popular web sites. The workflow is:

  1. The application launchs a browser and load FB's OAuth dialog, passing as arguments the app key and a redirection URL: https://www.facebook.com/dialog/oauth?client_id=YOUR_APP_ID&redirect_uri=...
  2. The user authenticates himself in Facebook.
  3. The user accepts granting access to the application.
  4. Facebook returns an access token to a callback URL.

The problem with this workflow is that desktop applications don't usually run a web server, so it's difficult to get this access token. How can we do it? Using WebKit!

Facebook states that the callback URL can be set to https://www.facebook.com/connect/login_success.html which will receive the access token as a parameter (login_sucess.html#access_token=...). We are going to use WebKit to identify this URLs using this callback:

self.web_view.connect('load-committed', self._load_committed_cb) # Load page

The 'load-commited' signal is emmited by the WebKit view when a page is about to be loaded. In the _load_commited_cb function, the page URL is checked to get Facebook's application access token. This token is finally saved to a file and the browser is closed.

This is the source code:

#!/usr/bin/env python
import pygtk
import gtk
import webkit
import urllib
import urlparse

FB_TOKEN_FILE = 'access_token.txt'

class Browser:
    """ Creates a web browser using GTK+ and WebKit to authorize a
        desktop application in Facebook. It uses OAuth 2.0.
        Requires the Facebook's Application ID. The token is then
        saved to FB_TOKEN_FILE.
    """

    def __init__(self, app_key, scope='offline_access'):
        """ Constructor. Creates the GTK+ app and adds the WebKit widget
            @param app_key Application key ID (Public).

            @param scope A string list of permissions to ask for. More at
            http://developers.facebook.com/docs/reference/api/permissions/
        """
        self.token = ''
        self.token_expire = ''
        self.scope = scope
        # Creates the GTK+ app
        self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
        self.scrolled_window = gtk.ScrolledWindow()
        # Creates a WebKit view
        self.web_view = webkit.WebView()
        self.scrolled_window.add(self.web_view)
        self.window.add(self.scrolled_window)
        # Connects events
        self.window.connect('destroy', self._destroy_event_cb) # Close window
        self.web_view.connect('load-committed', self._load_committed_cb) # Load page
        self.window.set_default_size(1024, 800)
        # Loads the Facebook OAuth page
        self.web_view.load_uri(
            'https://www.facebook.com/dialog/oauth?client_id=%s&redirect_uri=%s&response_type=token&scope=%s' % (urllib.quote(app_key), urllib.quote('https://www.facebook.com/connect/login_success.html'), urllib.quote(self.scope))
            )

    def _load_committed_cb(self, web_view, frame):
        """ Callback. The page is about to be loaded. This event is captured
            to intercept the OAuth 2.0 redirection, which includes the
            access token.

            @param web_view A reference to the current WebKitWebView.

            @param frame A reference to the main WebKitWebFrame.
        """
        # Gets the current URL to check whether is the one of the redirection
        uri = frame.get_uri()
        parse = urlparse.urlparse(uri)
        if (hasattr(parse, 'netloc') and hasattr(parse, 'path') and
            hasattr(parse, 'fragment') and parse.netloc == 'www.facebook.com' and
            parse.path == '/connect/login_success.html' and parse.fragment):
            # Get token from URL
            params = urlparse.parse_qs(parse.fragment)
            self.token = params['access_token'][0]
            self.token_expire = params['expires_in'][0] # Should be equal to 0, don't expire
            # Save token to file
            token_file = open(FB_TOKEN_FILE, 'w')
            token_file.write(self.token)
            token_file.close()
            print "Authentication done. Access token available at %s" % (FB_TOKEN_FILE)
            gtk.main_quit() # Finish

    def _destroy_event_cb(self, widget):
        """ Callback for close window. Closes the application. """
        return gtk.main_quit()

    def authorize(self):
        """ Runs the app. """
        self.window.show_all()
        gtk.main()

if (__name__ == '__main__'):
    # Creates the browser
    browser = Browser(app_key='XXXXXXXXXXX', scope='offline_access,read_stream')
    # Launch browser window
    browser.authorize()
    # Token available?
    print "Token: %s" % (browser.token)

Easy, isn't it?

The complete source code is available at github.com/vrruiz/FacebookAuthBrowser

Read more
rvr

In the first post we described what WebKit and JavaScriptCore is, how to program a simple WebKitGTK+ application and how to extend the JavaScript functionality with a dumb (empty) class. In the second post, we extended JavaScript to enable desktop notifications. In the third post,  WebKit/JavaScriptCore were extended to access UPower properties using D-Bus.

 

Callbacks

As explained in the previous article, D-Bus is object oriented, and interfaces can expose objects with their methods, properties and events. We used D-Bus to access battery status, and extended WebKit/JavaScript Core to read it. In this post we are going to connect D-Bus signals with JavaScript callback functions. What we want to do is execute custom JavaScript code after a D-Bus signal has been emitted (and received).

D-Bus signals are events that clients can subscribe to. As explained, UDisks interface emits signals everytime a new external drive is attached to the system, and Linux desktops capture this events to launch file managers (and other applications).  UPower.Device also emits signals, and we are going to capture Changed().

It's easy to connect functions using  D-Bus Glib bindings. First, we add a signal to the proxy object, which is a  way to indicate the type of the arguments. Changed() doesn't has any parameter, so the call is this:

    dbus_g_proxy_add_signal (proxy, "Changed", G_TYPE_INVALID);

And then we connect the D-Bus signal with a callback function (battery_changed_cb).

    dbus_g_proxy_connect_signal(proxy,
                                "Changed",
                                G_CALLBACK(battery_changed_cb),
                                ref,
                                (GClosureNotify) g_free);

This callback function will receive two parameters, proxy and ref, with references to its correspondent JavaScriptCore context and global object.

So, when UPower updates the battery status, it emits a Changed() signal. This signal will be received by our JavaScript Battery() objects. What happens then? battery_changed_cb will check whether Battery has a property called onChange and whether is of type function. So, if Battery.onChange stores a function, battery_changed_cb will call it. You'll see it more clearly in the JavaScript file.

Above is the source code of our previous D-Bus program. Remember that is based on that of the previous example.

#include <stdlib.h>
#include <dbus/dbus.h>
#include <dbus/dbus-glib.h>
#include <gtk/gtk.h>
#include <webkit/webkit.h>
#include <JavaScriptCore/JavaScript.h>

DBusGConnection *conn;
DBusGProxy *proxy;
DBusGProxy *properties_proxy;

typedef struct {
    JSContextRef context;
    JSObjectRef object;
} RefContextObject;

/* Callback for UPower.Device.Changed() */
static void battery_changed_cb(DBusGProxy *_proxy, RefContextObject *ref) 
{
    g_message("Battery changed");
    
    /* Get onChange property */
    JSStringRef string_onchange;
    string_onchange = JSStringCreateWithUTF8CString("onChange");
    JSValueRef func = JSObjectGetProperty(ref->context, ref->object, string_onchange, NULL);
    JSObjectRef function = JSValueToObject(ref->context, func, NULL);
    JSStringRelease(string_onchange);
    
    if (!JSObjectIsFunction(ref->context, function)) {
         g_message("JSObject is not function or is not set");
         return;
    }
        
    JSValueRef result = JSObjectCallAsFunction(ref->context, // The execution context to use
                                               function, // The JSObject to call as a function.
                                               ref->object, // The object to use as "this," or NULL to use the global object as "this."
                                               0, //  An integer count of the number of arguments in arguments.
                                               NULL, // A JSValue array of arguments to pass to the function. Pass NULL if argumentCount is 0.
                                               NULL); // A pointer to a JSValueRef in which to store an exception, if any. Pass NULL if you do not care to store an exception.
        
}

/* Class initialize */
static void battery_init_cb(JSContextRef ctx,
                            JSObjectRef object)
{
    GError *error = NULL;
    
    conn = dbus_g_bus_get (DBUS_BUS_SYSTEM, &error);

    if (conn == NULL)
    {
        g_printerr ("Failed to open connection to bus: %s\n", error->message);
        g_error_free (error);
        return;
    }

    /* Create a proxy object for "org.freedesktop.UPower" */
    proxy = dbus_g_proxy_new_for_name (conn,
                                       "org.freedesktop.UPower",
                                       "/org/freedesktop/UPower/devices/battery_BAT0",
                                       "org.freedesktop.UPower.Device");
    if (proxy == NULL)
    {
        g_printerr ("Failed to create proxy object\n");
        return;
    }
    
    /* Create a proxy object for "org.freedesktop.UPower.Properties" */
    properties_proxy = dbus_g_proxy_new_from_proxy (proxy,
                                                    "org.freedesktop.DBus.Properties",
                                                    dbus_g_proxy_get_path (proxy));
    if (properties_proxy == NULL)
    {
        g_printerr ("Failed to create proxy object\n");
        return;
    }

    /* Connect callback to signal org.freedestop.UPower.Device.Changed() */
    RefContextObject *ref = g_new(RefContextObject, 1);
    ref->context = ctx;
    ref->object = object;
    
    dbus_g_proxy_add_signal (proxy, "Changed", G_TYPE_INVALID);
    dbus_g_proxy_connect_signal(proxy,
                                "Changed",
                                G_CALLBACK(battery_changed_cb),
                                ref,
                                (GClosureNotify) g_free);
  
    error = NULL;
}

/* Class constructor */
static JSObjectRef battery_constructor_cb(JSContextRef context,
                                          JSObjectRef constructor,
                                          size_t argumentCount,
                                          const JSValueRef arguments[],
                                          JSValueRef *exception)
{
    return constructor;
}                                       

/* Class finalize */
static void battery_destroy_cb(JSObjectRef object)
{
    /* Ends Battery */
    if (proxy != NULL) g_object_unref (proxy);
    if (properties_proxy != NULL) g_object_unref (properties_proxy);
}

static gboolean proxy_property_value(char *property,
				                     GValue *get_value,
                                     GError **error)
{
    /* Call ListNames method, wait for reply */
    return dbus_g_proxy_call (properties_proxy, "Get", error,
                              G_TYPE_STRING, "/org/freedesktop/UPower/devices/battery_BAT0",
                              G_TYPE_STRING, property,
                              G_TYPE_INVALID,
                              G_TYPE_VALUE, get_value,
                              G_TYPE_INVALID);
}

static JSValueRef proxy_double_value(JSContextRef context,
                                     char *property,
                                     size_t argumentCount)
{
    GError *error = NULL;
    GValue get_value = {0, };
  
    if (argumentCount == 0) {
        /* Call method, wait for reply */
        if (!proxy_property_value(property, &get_value, &error))
        {
            g_printerr ("Error: %s\n", error->message);
            g_error_free (error);
            return JSValueMakeUndefined(context);
        }
        
        gdouble value = g_value_get_double(&get_value);
        g_value_unset(&get_value);
        return JSValueMakeNumber(context, value);
    }
    
    return JSValueMakeUndefined(context);
}

static JSValueRef proxy_uint64_value(JSContextRef context,
                                     char *property,
                                     size_t argumentCount)
{
    GError *error = NULL;
    GValue get_value = {0, };
  
    if (argumentCount == 0) {
        /* Call method, wait for reply */
        if (!proxy_property_value(property, &get_value, &error))
        {
            g_printerr ("Error: %s\n", error->message);
            g_error_free (error);
            return JSValueMakeUndefined(context);
        }
        
        guint64 value = g_value_get_uint64(&get_value);
        g_value_unset(&get_value);
        return JSValueMakeNumber(context, value);
    }
    
    return JSValueMakeUndefined(context);
}

static JSValueRef proxy_boolean_value(JSContextRef context,
                                     char *property,
                                     size_t argumentCount)
{
    GError *error = NULL;
    GValue get_value = {0, };
  
    if (argumentCount == 0) {
        /* Call method, wait for reply */
        if (!proxy_property_value(property, &get_value, &error))
        {
            g_printerr ("Error: %s\n", error->message);
            g_error_free (error);
            return JSValueMakeUndefined(context);
        }
        
        gboolean value = g_value_get_boolean(&get_value);
        g_value_unset(&get_value);
        return JSValueMakeBoolean(context, value);
    }
    
    return JSValueMakeUndefined(context);
}

/* Battery.capacity method callback implementation */
static JSValueRef battery_capacity_cb(JSContextRef context,
                                      JSObjectRef function,
                                      JSObjectRef thisObject,
                                      size_t argumentCount,
                                      const JSValueRef arguments[],
                                      JSValueRef *exception)
{
    return proxy_double_value(context, "Capacity", argumentCount);
}

/* Battery.percentage method callback implementation */
static JSValueRef battery_percentage_cb(JSContextRef context,
                                        JSObjectRef function,
                                        JSObjectRef thisObject,
                                        size_t argumentCount,
                                        const JSValueRef arguments[],
                                        JSValueRef *exception)
{
    return proxy_double_value(context, "Percentage", argumentCount);
}

/* Battery.voltage method callback implementation */
static JSValueRef battery_voltage_cb(JSContextRef context,
                                     JSObjectRef function,
                                     JSObjectRef thisObject,
                                     size_t argumentCount,
                                     const JSValueRef arguments[],
                                     JSValueRef *exception)
{
    return proxy_double_value(context, "Voltage", argumentCount);
}

/* Battery.updateTime method callback implementation */
static JSValueRef battery_update_time_cb(JSContextRef context,
                                         JSObjectRef function,
                                         JSObjectRef thisObject,
                                         size_t argumentCount,
                                         const JSValueRef arguments[],
                                         JSValueRef *exception)
{
    return proxy_uint64_value(context, "UpdateTime", argumentCount);
}

/* Battery.PowerSupply method callback implementation */
static JSValueRef battery_power_supply_cb(JSContextRef context,
                                          JSObjectRef function,
                                          JSObjectRef thisObject,
                                          size_t argumentCount,
                                          const JSValueRef arguments[],
                                          JSValueRef *exception)
{
    return proxy_boolean_value(context, "PowerSupply", argumentCount);
}

/* Class method declarations */
static const JSStaticFunction battery_staticfuncs[] =
{
    { "capacity", battery_capacity_cb, kJSPropertyAttributeReadOnly },
    { "percentage", battery_percentage_cb, kJSPropertyAttributeReadOnly },
    { "voltage", battery_voltage_cb, kJSPropertyAttributeReadOnly },
    { "updateTime", battery_update_time_cb, kJSPropertyAttributeReadOnly },
    { "powerSupply", battery_power_supply_cb, kJSPropertyAttributeReadOnly },
    { NULL, NULL, 0 }
};

static const JSClassDefinition notification_def =
{
    0,                     // version
    kJSClassAttributeNone, // attributes
    "Battery",             // className
    NULL,                  // parentClass
    NULL,                  // staticValues
    battery_staticfuncs,   // staticFunctions
    battery_init_cb,       // initialize
    battery_destroy_cb,    // finalize
    NULL,                  // hasProperty
    NULL,                  // getProperty
    NULL,                  // setProperty
    NULL,                  // deleteProperty
    NULL,                  // getPropertyNames
    NULL,                  // callAsFunction
    battery_constructor_cb, // callAsConstructor
    NULL,                  // hasInstance  
    NULL                   // convertToType
};

/* Callback - JavaScript window object has been cleared */
static void window_object_cleared_cb(WebKitWebView  *web_view,
                                     WebKitWebFrame *frame,
                                     gpointer        context,
                                     gpointer        window_object,
                                     gpointer        user_data)

{
    /* Add classes to JavaScriptCore */
    JSClassRef classDef = JSClassCreate(&notification_def);
    JSObjectRef classObj = JSObjectMake(context, classDef, context);
    JSObjectRef globalObj = JSContextGetGlobalObject(context);
    JSStringRef str = JSStringCreateWithUTF8CString("Battery");
    JSObjectSetProperty(context, globalObj, str, classObj, kJSPropertyAttributeNone, NULL);
}

/* Destroy callback */
static void destroy(GtkWidget *widget,
                    gpointer   data )
{
    gtk_main_quit();
}

int
main (int argc, char* argv[])
{
    /* Initialize the widget set */
    gtk_init (&argc, &argv);
    
    /* Create the window widgets */
    GtkWidget *main_window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    GtkWidget *scrolled_window = gtk_scrolled_window_new (NULL, NULL);
    
    /* Create the WebKit Web View widget */
    GtkWidget *web_view = webkit_web_view_new ();
    
    /* Connect the window object cleared event with callback */
    g_signal_connect (G_OBJECT (web_view), "window-object-cleared", G_CALLBACK(window_object_cleared_cb), web_view);


    /* Place the WebKitWebView in the GtkScrolledWindow */
    gtk_container_add (GTK_CONTAINER (scrolled_window), web_view);
    gtk_container_add (GTK_CONTAINER (main_window), scrolled_window);

    /* Connect the destroy window event with destroy function */
    g_signal_connect (G_OBJECT (main_window), "destroy", G_CALLBACK (destroy), NULL);
    
    /* Open webpage */
    webkit_web_view_load_uri (WEBKIT_WEB_VIEW (web_view), "file://webkit-05.html");

    /* Create the main window */
    gtk_window_set_default_size (GTK_WINDOW (main_window), 800, 600);
    
    /* Show the application window */
    gtk_widget_show_all (main_window);

    /* Enter the main event loop, and wait for user interaction */
    gtk_main ();
    
    /* The user lost interest */
    return 0;
}

To compile it on Ubuntu:

$ gcc webkit-05.c -o webkit.c `pkg-config --cflags --libs webkitgtk-3.0 dbus-glib-1`

Above, the corresponding HTML file webkit-05.html.

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
  <h1>Extending JavaScript with WebKit. Battery class.</h1>
  <div id="battery-status"></div>
  <script type="text/javascript">
    var display = function () {
        var div = document.getElementById("battery-status");
        var battery_status = "Capacity: " + Battery.capacity() + "<br>";
        battery_status += "Percentage: " + Battery.percentage() + "%<br>";
        battery_status += "Voltage: " + Battery.voltage() + "<br>";
        battery_status += "Update Time: " + Battery.updateTime() + "<br>";
        battery_status += "Power supply: " + Battery.powerSupply() + "<br>";
        div.innerHTML = battery_status;
    }
  
    var battery = new Battery();
    battery.onChange = display;
    display();
  </script>
</body>
</html>

var display stores a reference to a function. This function fills the empty <div id=battery-status> with the actual battery status. Finally, a Battery() object is created and battery.onChange is defined.

If we run our program

$ ./webkit-05

when UPower.Device.Changed() is emitted, our JavaScript function will be called and the battery status, automatically updated in the web page.

Pantallazo del 2011-11-02 19:15:45
As before, the complete source code is available at github.com/vrruiz/WebKit-JavaScriptCore-Extensions.

Read more
rvr

In the first post we described what WebKit and JavaScriptCore is, how to program a simple WebKitGTK+ application and how to extend the JavaScript functionality with a dumb (empty) class. In the second post, we extended JavaScript to enable desktop notifications.

In this third tutorial we are going to extend WebKit/JavaScriptCore to enable access to system information, specifically, battery status. We'll do this using D-Bus.

 

What's D-Bus?

Today's desktop environments, like GNOME and KDE, quickly respond to USB events: when an external drive is connected, a file navigator appears. How does it work?

GNOME and KDE provide a standard system for interprocess communication, called D-Bus. D-Bus communications is object oriented: applications expose objects with their methods, properties and events, and they are remotely available.

There are GUI applications to play with D-Bus. One of them is D-Feet. To install it on Ubuntu, type:

$ sudo apt-get install d-feet

In D-Feet we can connect to two default buses: the session bus and the system bus. Regular desktop applications use the session bus, tied to the user running the current desktop session. The system bus has a lower level and used by services in the operating system. This screenshot shows D-Feet connected to the session bus:

Pantallazo del 2011-10-27 20:58:36

And finally the enigma is solved: if the desktop wants to know when a new disk is attached to the computer, it just can do it using D-Bus and UDisks interface: actually, the operating system uses the system bus to communicate disk events.


D-Bus and battery status: UPower.

In the screenshot above, on the right we can see the interfaces, methods and properties exposed by UPower in the system bus.  For example, the list of power devices (batteries and power line) is available using the EnumerateDevices() method, which returns the device list as and array of objects. Another interface, UPower.Device, gives us access to the battery status, and we'll use it to build our custom JavaScript class.

In order to connect to D-Bus, we use the Glib's D-Bus bindings. To install the development libraries, type:

$ sudo apt-get install libdbus-glib-1-dev

The actual code is this:

#include <stdlib.h>
#include <dbus/dbus.h>
#include <dbus/dbus-glib.h>
#include <gtk/gtk.h>
#include <webkit/webkit.h>
#include <JavaScriptCore/JavaScript.h>

DBusGConnection *conn;
DBusGProxy *proxy;
DBusGProxy *properties_proxy;

/* Class initialize */
static void battery_init_cb(JSContextRef ctx,
                            JSObjectRef object)
{
    GError *error = NULL;
    
    /* Connection to the system bus */ 
    conn = dbus_g_bus_get (DBUS_BUS_SYSTEM, &error);

    if (conn == NULL)
    {
        g_printerr ("Failed to open connection to bus: %s\n", error->message);
        g_error_free (error);
        return;
    }

    /* Create a proxy object for "org.freedesktop.UPower" */
    proxy = dbus_g_proxy_new_for_name (conn,
                                       "org.freedesktop.UPower",
                                       "/org/freedesktop/UPower/devices/battery_BAT0",
                                       "org.freedesktop.UPower.Device.Properties");
    if (proxy == NULL)
    {
        g_printerr ("Failed to create proxy object\n");
        return;
    }

    /* Creates a proxy using an existing proxy as a template */
    properties_proxy = dbus_g_proxy_new_from_proxy (proxy,
                                                    "org.freedesktop.DBus.Properties",
                                                    dbus_g_proxy_get_path (proxy));
    if (properties_proxy == NULL)
    {
        g_object_unref (proxy)
        g_printerr ("Failed to create proxy object\n");
        return;
    }
  
    error = NULL;
}

/* Class finalize */
static void battery_destroy_cb(JSObjectRef object)
{
    /* Ends Battery. Free allocated memory. */
    if (proxy != NULL) g_object_unref (proxy);
    if (properties_proxy != NULL) g_object_unref (properties_proxy);
}

static gboolean proxy_property_value(char *property,
				                     GValue *get_value,
                                     GError **error)
{
    /* Call Get method, wait for reply */
    return dbus_g_proxy_call (properties_proxy, "Get", error,
                              G_TYPE_STRING, "/org/freedesktop/UPower/devices/battery_BAT0",
                              G_TYPE_STRING, property,
                              G_TYPE_INVALID,
                              G_TYPE_VALUE, get_value,
                              G_TYPE_INVALID);
}

static JSValueRef proxy_double_value(JSContextRef context,
                                     char *property,
                                     size_t argumentCount)
{
    /* Calls to UPower to get a double value */
    
    GError *error = NULL;
    GValue get_value = {0, };
  
    if (argumentCount == 0) {
        /* Get property value */
        if (!proxy_property_value(property, &get_value, &error))
        {
            g_printerr ("Error: %s\n", error->message);
            g_error_free (error);
            return JSValueMakeUndefined(context);
        }
        
        /* Convert value to double */
        gdouble value = g_value_get_double(&get_value);
        g_value_unset(&get_value);
        return JSValueMakeNumber(context, value);
    }
    
    return JSValueMakeUndefined(context);
}

static JSValueRef proxy_uint64_value(JSContextRef context,
                                     char *property,
                                     size_t argumentCount)
{
    /* Calls to UPower to get a uint64 value */
    GError *error = NULL;
    GValue get_value = {0, };
  
    if (argumentCount == 0) {
        /* Get property value */
        if (!proxy_property_value(property, &get_value, &error))
        {
            g_printerr ("Error: %s\n", error->message);
            g_error_free (error);
            return JSValueMakeUndefined(context);
        }
        
        /* Convert value to uint64 */
        guint64 value = g_value_get_uint64(&get_value);
        g_value_unset(&get_value);
        return JSValueMakeNumber(context, value);
    }
    
    return JSValueMakeUndefined(context);
}

static JSValueRef proxy_boolean_value(JSContextRef context,
                                     char *property,
                                     size_t argumentCount)
{
    /* Calls to UPower to get a boolean value */
    
    GError *error = NULL;
    GValue get_value = {0, };
  
    if (argumentCount == 0) {
        /* Call method, wait for reply */
        if (!proxy_property_value(property, &get_value, &error))
        {
            g_printerr ("Error: %s\n", error->message);
            g_error_free (error);
            return JSValueMakeUndefined(context);
        }
        
        /* Convert value to boolean */
        gboolean value = g_value_get_boolean(&get_value);
        g_value_unset(&get_value);
        return JSValueMakeBoolean(context, value);
    }
    
    return JSValueMakeUndefined(context);
}

/* Battery.capacity method callback implementation */
static JSValueRef battery_capacity_cb(JSContextRef context,
                                      JSObjectRef function,
                                      JSObjectRef thisObject,
                                      size_t argumentCount,
                                      const JSValueRef arguments[],
                                      JSValueRef *exception)
{
    /* Get battery capacity status */
    return proxy_double_value(context, "Capacity", argumentCount);
}

/* Battery.percentage method callback implementation */
static JSValueRef battery_percentage_cb(JSContextRef context,
                                        JSObjectRef function,
                                        JSObjectRef thisObject,
                                        size_t argumentCount,
                                        const JSValueRef arguments[],
                                        JSValueRef *exception)
{
    /* Get battery percentage status */
    return proxy_double_value(context, "Percentage", argumentCount);
}

/* Battery.voltage method callback implementation */
static JSValueRef battery_voltage_cb(JSContextRef context,
                                     JSObjectRef function,
                                     JSObjectRef thisObject,
                                     size_t argumentCount,
                                     const JSValueRef arguments[],
                                     JSValueRef *exception)
{
    /* Get battery voltage status */
    return proxy_double_value(context, "Voltage", argumentCount);
}

/* Battery.updateTime method callback implementation */
static JSValueRef battery_update_time_cb(JSContextRef context,
                                         JSObjectRef function,
                                         JSObjectRef thisObject,
                                         size_t argumentCount,
                                         const JSValueRef arguments[],
                                         JSValueRef *exception)
{
    /* Get battery update time */
    return proxy_uint64_value(context, "UpdateTime", argumentCount);
}

/* Battery.PowerSupply method callback implementation */
static JSValueRef battery_power_supply_cb(JSContextRef context,
                                          JSObjectRef function,
                                          JSObjectRef thisObject,
                                          size_t argumentCount,
                                          const JSValueRef arguments[],
                                          JSValueRef *exception)
{
    /* Get battery power supply */
    return proxy_boolean_value(context, "PowerSupply", argumentCount);
}

/* Class method declarations */
static const JSStaticFunction battery_staticfuncs[] =
{
    { "capacity", battery_capacity_cb, kJSPropertyAttributeReadOnly },
    { "percentage", battery_percentage_cb, kJSPropertyAttributeReadOnly },
    { "voltage", battery_voltage_cb, kJSPropertyAttributeReadOnly },
    { "updateTime", battery_update_time_cb, kJSPropertyAttributeReadOnly },
    { "powerSupply", battery_power_supply_cb, kJSPropertyAttributeReadOnly },
    { NULL, NULL, 0 }
};

static const JSClassDefinition notification_def =
{
    0,                     // version
    kJSClassAttributeNone, // attributes
    "Battery",             // className
    NULL,                  // parentClass
    NULL,                  // staticValues
    battery_staticfuncs,   // staticFunctions
    battery_init_cb,       // initialize
    battery_destroy_cb,    // finalize
    NULL,                  // hasProperty
    NULL,                  // getProperty
    NULL,                  // setProperty
    NULL,                  // deleteProperty
    NULL,                  // getPropertyNames
    NULL,                  // callAsFunction
    NULL,                  // callAsConstructor
    NULL,                  // hasInstance  
    NULL                   // convertToType
};

/* Callback - JavaScript window object has been cleared */
static void window_object_cleared_cb(WebKitWebView  *web_view,
                                     WebKitWebFrame *frame,
                                     gpointer        context,
                                     gpointer        window_object,
                                     gpointer        user_data)

{
    /* Add classes to JavaScriptCore */
    JSClassRef classDef = JSClassCreate(&notification_def);
    JSObjectRef classObj = JSObjectMake(context, classDef, context);
    JSObjectRef globalObj = JSContextGetGlobalObject(context);
    JSStringRef str = JSStringCreateWithUTF8CString("Battery");
    JSObjectSetProperty(context, globalObj, str, classObj, kJSPropertyAttributeNone, NULL);
}

/* Destroy callback */
static void destroy(GtkWidget *widget,
                    gpointer   data )
{
    gtk_main_quit();
}

int
main (int argc, char* argv[])
{
    /* Initialize the widget set */
    gtk_init (&argc, &argv);
    
    /* Create the window widgets */
    GtkWidget *main_window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    GtkWidget *scrolled_window = gtk_scrolled_window_new (NULL, NULL);
    
    /* Create the WebKit Web View widget */
    GtkWidget *web_view = webkit_web_view_new ();
    
    /* Connect the window object cleared event with callback */
    g_signal_connect (G_OBJECT (web_view), "window-object-cleared", G_CALLBACK(window_object_cleared_cb), web_view);


    /* Place the WebKitWebView in the GtkScrolledWindow */
    gtk_container_add (GTK_CONTAINER (scrolled_window), web_view);
    gtk_container_add (GTK_CONTAINER (main_window), scrolled_window);

    /* Connect the destroy window event with destroy function */
    g_signal_connect (G_OBJECT (main_window), "destroy", G_CALLBACK (destroy), NULL);
    
    /* Open webpage */
    webkit_web_view_load_uri (WEBKIT_WEB_VIEW (web_view), "file://webkit-04.html");

    /* Create the main window */
    gtk_window_set_default_size (GTK_WINDOW (main_window), 800, 600);
    
    /* Show the application window */
    gtk_widget_show_all (main_window);

    /* Enter the main event loop, and wait for user interaction */
    gtk_main ();
    
    /* The user lost interest */
    return 0;
}

This program creates a new JavaScript class, called Battery, with some useful methods, like Battery.percentage(), Battery.capacity() and Battery.powerSupply(). When the class is initiated, a system bus connection is created, which is used later by the methods to obtain actual battery values.

This is webkit-04.html, a simple web file which uses the newly created Battery class:

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
  <h1>Extending JavaScript with WebKit. Battery class.</h1>
  <script type="text/javascript">
    document.write("Capacity: " + Battery.capacity() + "<br>");
    document.write("Percentage: " + Battery.percentage() + "%<br>");
    document.write("Voltage: " + Battery.voltage() + "<br>");
    document.write("Update Time: " + Battery.updateTime() + "<br>");
    document.write("Power supply: " + Battery.powerSupply() + "<br>");
  </script>
</body>
</html>

To compile and run the program, type:

$ gcc -o webkit-04 webkit-04.c `pkg-config --cflags --libs webkitgtk-3.0 dbus-glib-1`

$ ./webkit-04

The program, if running on a laptop with batteries, will display something like this:

Pantallazo del 2011-10-27 21:28:55

Great, isn't it!? :-)

The source code is available at github.com/vrruiz/WebKit-JavaScriptCore-Extensions.

As we've seen, D-Bus is a powerful system. Mixing JavaScript with D-Bus gives us incredible possibilities. See you in the next post!

Read more
rvr

In recent years, JavaScript has become a very popular language. Google has promoted rich web applications which compete with desktop programs: Gmail, Google Maps and Google Docs are rivals of Outlook, Google Earth and Office. Today HTTP, HTML, CSS and JavaScript are key technologies in which companies are heavily investing so they can develop even more powerful web applications,  i.e. HTML5 features enable off-line web applications (i.e. store local data).

JavaScript has a restricted programming model in order to meet security concerns. But, wouldn't be fun if we could extend JavaScript to create wonderful programs which mix desktop and web technologies? Absolutely! In order to do that, we'll use WebKit and Gtk+.

The source code of this tutorial is available at github.com/vrruiz/WebKit-JavaScriptCore-Extensions.

 

WebKit

Chances are that you're using a browser which comes with WebKit. This open source technology was originally developed by KDE, and forked by Apple. WebKit powers Apple's Safari, Google's Chrome and many other browsers.

WebKit is to browsers what an engine is to cars: many bodyworks can carry the same engine model. WebKit provides the basic functions to download, parse, run and display web pages. However, WebKit doesn't provide a user interface to introduce URL address, change settings, navigation buttons, etc. That's the developer's job.

WebKit is multiplatform. Interesting to us is that WebKit is extensible and provides ways to interact with its JavaScript default engine, JavaScriptCore.

Currently, KDE, Gnome and MacOS X support WebKit, to provide HTML views inside desktop applications. WebKit has been ported to many platforms (Mac, Linux, Windows), many SDK's (Cocoa, Gtk, Qt) and many languages.  One of this ports is WebKitGTK+.

 

WebKitGTK+

GTK+ is multiplatform graphical toolkit, a set of graphical libraries to program desktop applications (the popular Linux desktop enviroment GNOME is built upon GTK+). WebKitGTK+ allows GTK+ applications to display web pages using WebKit. Originally, WebKit is programmed in C++, but WebKitGTK+ has a C interface using GObject (part of GLib, which "enables" object-oriented programming in plain C). 

To install GTK+ and WebKitGTK+ development files in Ubuntu do this in the command line:

$ sudo apt-get install gnome-devel libwebkitgtk-3.0-dev

 

A simple web view.

This program creates a GTK+ application. The main window has 800 x 600 pixels, and contains a scrolled window which finally holds the web view. The web view displays this blog.

#include <gtk/gtk.h>
#include <webkit/webkit.h>

/* Destroy callback */
static void destroy_cb( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit();
}

int
main (int argc, char* argv[])
{
    /* Initialize the widget set */
    gtk_init (&argc, &argv);
    
    /* Create the window widgets */
    GtkWidget *main_window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    GtkWidget *scrolled_window = gtk_scrolled_window_new (NULL, NULL);
    
    /* Create the WebKit Web View widget */
    GtkWidget *web_view = webkit_web_view_new ();

    /* Place the WebKitWebView in the GtkScrolledWindow */
    gtk_container_add (GTK_CONTAINER (scrolled_window), web_view);
    gtk_container_add (GTK_CONTAINER (main_window), scrolled_window);

    /* Connect the destroy window event with destroy function */
    g_signal_connect (G_OBJECT (main_window), "destroy", G_CALLBACK (destroy_cb), NULL);
    
    /* Open webpage */
    webkit_web_view_load_uri (WEBKIT_WEB_VIEW (web_view), "http://rvr.typepad.com/");

    /* Create the main window */
    gtk_window_set_default_size (GTK_WINDOW (main_window), 800, 600);
    
    /* Show the application window */
    gtk_widget_show_all (main_window);

    /* Enter the main event loop, and wait for user interaction */
    gtk_main ();
    
    /* The user lost interest */
    return 0;
}

To compile and run this program, type:

$ gcc webkit-01.c -o webkit-01 `pkg-config --cflags --libs webkitgtk-3.0`

$ ./webkit-01

An application with a web view will appear.

Your own browser in just a minute. Easy, isn't it?

 

Interacting with JavaScriptCore.

What's great about WebKit is that the JavaScript engine can be extended to support custom functions. PhoneGap SDK actually uses this feature to provide mobile developers access to low-level OS features via JavaScript (i.e. accelerometer).

As previously stated, JavaScriptCore is WebKit's JavaScript engine, at it provides an API to extend JavaScript and add new classes. Basically, for each class we need to provide callbacks for the constructor and destructor class, and a list of class methods and a callback for each of them.

Next is the source code of a bare JavaScript class declaration. It does nothing, except to print a messages in the console when the class is initialized and the constructor method called.

#include <gtk/gtk.h>
#include <webkit/webkit.h>
#include <JavaScriptCore/JavaScript.h>

/* Class initialize */
static void class_init_cb(JSContextRef ctx,
                          JSObjectRef object)
{
    g_message("Custom class initialize.");
}

/* Class finalize */
static void class_finalize_cb(JSObjectRef object)
{
    g_message("Custom class finalize.");
}


/* Class constructor. Called at "new CustomClass()" */
JSObjectRef class_constructor_cb(JSContextRef ctx,
                                 JSObjectRef constructor,
                                 size_t argumentCount,
                                 const JSValueRef arguments[],
                                 JSValueRef* exception)
{
    g_message("Custom class constructor");
}

static const JSClassDefinition class_def =
{
    0,                     // version
    kJSClassAttributeNone, // attributes
    "CustomClass",         // className
    NULL,                  // parentClass
    NULL,                  // staticValues
    NULL,                  // staticFunctions
    class_init_cb,         // initialize
    class_finalize_cb,     // finalize
    NULL,                  // hasProperty
    NULL,                  // getProperty
    NULL,                  // setProperty
    NULL,                  // deleteProperty
    NULL,                  // getPropertyNames
    NULL,                  // callAsFunction
    class_constructor_cb,  // callAsConstructor
    NULL,                  // hasInstance  
    NULL                   // convertToType
};

/* Callback - JavaScript window object has been cleared */
static void window_object_cleared_cb(WebKitWebView  *web_view,
                                     WebKitWebFrame *frame,
                                     gpointer        context,
                                     gpointer        window_object,
                                     gpointer        user_data)

{
    /* Add classes to JavaScriptCore */
    JSClassRef classDef = JSClassCreate(&class_def);
    JSObjectRef classObj = JSObjectMake(context, classDef, context);
    JSObjectRef globalObj = JSContextGetGlobalObject(context);
    JSStringRef str = JSStringCreateWithUTF8CString("CustomClass");
    JSObjectSetProperty(context, globalObj, str, classObj, kJSPropertyAttributeNone, NULL);
}

/* Destroy callback */
static void destroy(GtkWidget *widget,
                    gpointer   data )
{
    gtk_main_quit();
}

int
main (int argc, char* argv[])
{
    /* Initialize the widget set */
    gtk_init (&argc, &argv);
    
    /* Create the window widgets */
    GtkWidget *main_window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    GtkWidget *scrolled_window = gtk_scrolled_window_new (NULL, NULL);
    
    /* Create the WebKit Web View widget */
    GtkWidget *web_view = webkit_web_view_new ();
    
    /* Connect the window object cleared event with callback */
    g_signal_connect (G_OBJECT (web_view), "window-object-cleared", G_CALLBACK(window_object_cleared_cb), web_view);

    /* Place the WebKitWebView in the GtkScrolledWindow */
    gtk_container_add (GTK_CONTAINER (scrolled_window), web_view);
    gtk_container_add (GTK_CONTAINER (main_window), scrolled_window);

    /* Connect the destroy window event with destroy function */
    g_signal_connect (G_OBJECT (main_window), "destroy", G_CALLBACK (destroy), NULL);
    
    /* Open webpage */
    webkit_web_view_load_uri (WEBKIT_WEB_VIEW (web_view), "file://webkit-class.html");

    /* Create the main window */
    gtk_window_set_default_size (GTK_WINDOW (main_window), 800, 600);
    
    /* Show the application window */
    gtk_widget_show_all (main_window);

    /* Enter the main event loop, and wait for user interaction */
    gtk_main ();
    
    /* The user lost interest */
    return 0;
}

This program loads a local file, webkit-02.html, which only defines a CustomClass object.

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
  <h1>Extending JavaScript with WebKit. Custom Class.</h1>
  <script type="text/javascript">
    custom = new CustomClass();
  </script>
</body>
</html>

To compile this, type:

$ gcc webkit-02.c -o webkit-02 `pkg-config --cflags --libs webkitgtk-3.0`

The execution of this page will run our callback methods.

$ ./webkit-02

** Message: Custom class initialize.

** Message: Custom class constructor.

Of course, our custom class do nothing, except to print messages.

In the next post, we'll see add some real features to JavaScript. Enjoy!

Read more
rvr

WebKit: Extending JavaScript (II)

In the previous post we described what WebKit and JavaScriptCore is, how to program a simple WebKitGTK+ application and how to extend the JavaScript functionality with a dumb class. Now is time to show add some real features.

A class for desktop notifications.

GNOME and KDE come with a desktop notification support, so programs display pop-up windows in when certain events happen (i.e. new track in the music player). The library which allows this is libnotify. It declares NotifyNotification, a GObject which represents the pop-up notification. It contains a text and other fields. 

To install libnotify in Ubuntu, type:

$ sudo apt-get install libnotify-dev

In the code below, we declare a Notification class in JavaScript, with a constructor and a method called notify to be able to send notification pop-ups from a web page.

#include <gtk/gtk.h>
#include <webkit/webkit.h>
#include <libnotify/notify.h>
#include <JavaScriptCore/JavaScript.h>

/* Class initialize */
static void notification_init_cb(JSContextRef ctx,
                                 JSObjectRef object)
{
    /* Inits notify */
    notify_init("Notification");
}

/* Notification.notify method callback implementation */
static JSValueRef notification_notify_cb(JSContextRef context,
                       JSObjectRef function,
                       JSObjectRef thisObject,
                       size_t argumentCount,
                       const JSValueRef arguments[],
                       JSValueRef *exception)
{
    /* At least, one argument must be received */
    if (argumentCount == 1 && JSValueIsString(context, arguments[0])) {
        /* Converts JSValue to char */
        size_t len;
        char *cstr;
        JSStringRef jsstr = JSValueToStringCopy(context, arguments[0], NULL);
        len = JSStringGetMaximumUTF8CStringSize(jsstr);
        cstr = g_new(char, len);
        JSStringGetUTF8CString(jsstr, cstr, len);

        /* Creates a new NotifyNotification. */
        NotifyNotification *notification = notify_notification_new(cstr, NULL, NULL);
        
        /* Sets the timeout of the notification. */
        notify_notification_set_timeout(notification, 3000);
        
        /* Sets the urgency level of this notification. */
        notify_notification_set_urgency(notification, NOTIFY_URGENCY_NORMAL);
        
        /* Tells the notification server to display the notification on the screen. */
        GError *error = NULL;
        notify_notification_show(notification, &error);
        
        g_object_unref(G_OBJECT(notification));
        g_free(cstr);
        
        JSStringRelease(jsstr);
    }
    
    return JSValueMakeUndefined(context);
}

/* Class method declarations */
static const JSStaticFunction notification_staticfuncs[] =
{
    { "notify", notification_notify_cb, kJSPropertyAttributeReadOnly },
    { NULL, NULL, 0 }
};

static const JSClassDefinition notification_def =
{
    0,                     // version
    kJSClassAttributeNone, // attributes
    "Notification",        // className
    NULL,                  // parentClass
    NULL,                  // staticValues
    notification_staticfuncs, // staticFunctions
    notification_init_cb,  // initialize
    NULL,                  // finalize
    NULL,                  // hasProperty
    NULL,                  // getProperty
    NULL,                  // setProperty
    NULL,                  // deleteProperty
    NULL,                  // getPropertyNames
    NULL,                  // callAsFunction
    NULL,                  // callAsConstructor
    NULL,                  // hasInstance  
    NULL                   // convertToType
};

/* Callback - JavaScript window object has been cleared */
static void window_object_cleared_cb(WebKitWebView  *web_view,
                                     WebKitWebFrame *frame,
                                     gpointer        context,
                                     gpointer        window_object,
                                     gpointer        user_data)

{
    /* Add classes to JavaScriptCore */
    JSClassRef classDef = JSClassCreate(&notification_def);
    JSObjectRef classObj = JSObjectMake(context, classDef, context);
    JSObjectRef globalObj = JSContextGetGlobalObject(context);
    JSStringRef str = JSStringCreateWithUTF8CString("Notification");
    JSObjectSetProperty(context, globalObj, str, classObj, kJSPropertyAttributeNone, NULL);
}

/* Destroy callback */
static void destroy(GtkWidget *widget,
                    gpointer   data )
{
    gtk_main_quit();
}

int
main (int argc, char* argv[])
{
    /* Initialize the widget set */
    gtk_init (&argc, &argv);
    
    /* Create the window widgets */
    GtkWidget *main_window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
    GtkWidget *scrolled_window = gtk_scrolled_window_new (NULL, NULL);
    
    /* Create the WebKit Web View widget */
    GtkWidget *web_view = webkit_web_view_new ();
    
    /* Connect the window object cleared event with callback */
    g_signal_connect (G_OBJECT (web_view), "window-object-cleared", G_CALLBACK(window_object_cleared_cb), web_view);

    /* Place the WebKitWebView in the GtkScrolledWindow */
    gtk_container_add (GTK_CONTAINER (scrolled_window), web_view);
    gtk_container_add (GTK_CONTAINER (main_window), scrolled_window);

    /* Connect the destroy window event with destroy function */
    g_signal_connect (G_OBJECT (main_window), "destroy", G_CALLBACK (destroy), NULL);
    
    /* Open webpage */
    webkit_web_view_load_uri (WEBKIT_WEB_VIEW (web_view), "file://webkit-03.html");

    /* Create the main window */
    gtk_window_set_default_size (GTK_WINDOW (main_window), 800, 600);
    
    /* Show the application window */
    gtk_widget_show_all (main_window);

    /* Enter the main event loop, and wait for user interaction */
    gtk_main ();
    
    /* The user lost interest */
    return 0;
}

In bold is highlighted the code that declares the class method to make it accessible in JavaScript. Class methods  are first implemented using static JSValueRef functions. Then, they are added to an array of class methods, static const JSStaticFunction notification_staticfuncs[], which also defines the actual method names in JavaScript (in the first array argument, the string). Finally, this array of method definitions is passed as argument to the JSClassDefinition.

To compile the program, type:

$ gcc -o webkit-03 webkit-03.c `pkg-config --cflags --libs webkitgtk-3.0 libnotify`

This program loads the following file, webkit-03.html. This HTML code shows a text area and a button. When the button is pressed, the text is send as a desktop notification using the JavaScriptCore extension.

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
  <h1>Extending JavaScript with WebKit. Notification class.</h1>
  <script type="text/javascript">
    function send_notification() {
        var notification_text = document.getElementById("notification-text").value;
        Notification.notify(notification_text);
    }
  </script>
  <form id="form">
    <div><textarea id="notification-text" rows="5" cols="40"></textarea></div>
    <div><button type="button" onclick="send_notification();">Send notification</button></div>
  </form>
</body>
</html>

When the program is run, this is what is displayed in the desktop:

Pantallazo del 2011-10-25 23:28:07
The source code is available at github.com/vrruiz/WebKit-JavaScriptCore-Extensions.

Read more
rvr

Summary

  • Brief analysis of 150,000 photographs from Flickr in the province of Malaga.
  • It identifies the profile and preferences of tourists.

Last Saturday, I  was in Malaga. I was invited by Sonia Blanco and the Universidad Internacional de Andalucia to participate in workshop on Tourism and Social Networks. Sonia is professor at the University of Malaga, and one of the oldest bloggers in the Spanish blogosphere. Sonia asked me to present the analysis Fernando Tricas and myself did about Flickr photos and the Canary Islands (2009-2010), and I gladly accepted. I wanted to bring an update, so we got to work to make a short presentation with data from the province of Malaga. And that's what is shown below.

Video

Last Thursday, with the presentation already made, Fernando passed me an interesting link, a visualization by the Wall Street Journal that shows the density of a week of Foursquare check-ins in New York . If the WSJ could do it, so do we ;)  We already had the data and the map algorithms, so generated the maps by months and joined them to build the animation.

The video below shows the density of photographs taken in the province of Malaga from 2004 to 2010. Blue colors are areas where they make some pictures, and the red areas have made many pictures. There are areas with many photographs, places of touristic interest. And of course, there are months where the activity is higher and lower. 

Data

The video is just a bit of whole presented analysis. Full version is available below.

As you may know, Flickr is a popular photo-sharing service with 5 billion of hosted images and 86 million unique visitors. Flickr has social networking features, since it allows to make contacts. Flickr can play a role in the promotion of tourist destinations, as it is one of the main sources of images on the Internet. But to us, Flickr is a huge source of data: Which are the most photogenic places? Who are taking pictures there? These and other questions can answered using data mining.

For this study we obtained the metadata of 175,000 photographs (62,000 geolocated), 7,900 photographers and 1,470,000 tags (47,000 unique). All these pictures were either marked by the tag "malaga" or GPS coordinates were inside the province of Malaga.

Analysis

Below are the five most relevant slides: the tag cloud, the number of photos and photographers by months, the top 10 countries of the geolocated photographers, the group of tags and heatmaps of the geolocated images.

  • Turismo-malaga-11
  • Turismo-malaga-17
  • Turismo-malaga-13
  • Turismo-malaga-15
  • Turismo-malaga-20
  • Turismo-malaga-20
Turismo-malaga-20

According to those who share photos on Flickr about Malaga, we can conclude that:

  • The high season in Málaga is August (also, in April there is a Holy Week-effect.
  • Users come mainly from UK, USA, Italy, Germany, Madrid and Andalusia. (USA is probably overrepresented compared to real visitors).
  • They are interested in photography, beaches, festivals, fairs, nature, sea, birds, sky, parks.
  • Pictures are taken mainly in Málaga (capital), Ronda, Barcenilla and Benalmadena.

The full presentation slides show more features, such as geolocated photographs by countries. It is interesting to compare these data with the previous study on the Canaries. A more detailed analysis can be done, but the roundtable had limited time. This sneak peek shows the potential of social networking and geolocation services for market research. If you have any questions, ask in the comments!

The presentation and images have a Creative Commons Attribution-Share Alike license.

Finally, my gratitude to the organization of the UNIA for the invitation and hospitality, to Daniel Cerdan for suggesting the title of the post and Fernando Tricas for his unconditional support.

Read more
rvr

Abstract.

  • The Cablegate set is composed of +250,000 diplomatic cables.
  • The total number sent by Embassies and Secretary of State is guessed.

One of the biggest mysteries in astrophysics is the dark matter. Dark matter can not be seen, it doesn't shine nor reflects light. But we infer its existence because dark matter weights, and modifies the path of stars and galaxies. Cablegate has its own dark matter.

According to WikiLeaks, 251,287 communications compose the Cablegate. But what is the real volume of cables between the Embassies and Secretary of State? Can we guess it? The answer is yes, there is a simple way to know it. Using the methodology explained below, the total number of communications between Embassies and the Secretary of State is guessed.

This are the results.

The dark matter of the Embassies.

20101224cablegate-darkmatter.001Between 2005-2009, more than 400,000 non leaked cables are identified. In this case, the uncertainty is larger than with just one embassy due to the small number or released cables. The sum increased by 50% in just one week.

Curiously, the average size of the 1800 published cables is 12 KB. If this average is representative of the whole set, something I doubt, the total size of the 250,000 messages would be 350 MB.

Secretary of State.

In addition to embassies' communications, Cablegate has some cables from the Secretary of State. This messages are often quite interesting, because they request information or send commands to the embassies (eg 09STATE106750).

20101224cablegate-darkmatter.002In 2005 and 2006 there is no released cable, and therefore the sum cannot be estimated. But between 2007 and 2009, the volume of cables sent by the Secretary of State is remarkable (so big, that I doubted that the record number was an ordinal number and not a more sophisticated identifier). Compare this graph with the one of the embassies. 2007 show more cables from the Secretary than all Embassies combined, but beware, because this trend can be reversed with better data.

This results are available in Google Docs.

Madrid Embassy.

This is the chart for Madrid Embassy, which ranks seventh in the number of leaked cables.

20101224cablegate-darkmatter.003Between 2004-2009, the existence of at least 17,000 dispatches sent from Madrid can be deduced. In the same period, there are just 3500 leaked cables. The graph shows the breakdown by year. 2007 is leaked in a high percentage, the oppositat in 2004 and 2005. Also, the number of communications decreases progressively (Why? Maybe other networks are used instead of SIPRNet). The complete table is available in Google Docs.

Cablegate Dark Matter Howto

The Guardian published a text file with dates, source and tags of the 250,000 diplomatic cables included in the Cablegate. The content of this messages are being slowly released. (Using this short descriptions, I did an analysis of the messages related to Spain -tagged as SP-, and suggested the existence of communications related to the 2004 Madrid bombings and the Spaniard Internet Law. Later, El País published this cables, confirming the suspicions).

To infer the volume of communications the methodology is quite simple. Each cable has an identifier. For example, 04MADRID893 summaries the Madrid bombing on March 11th, 2004. This identifier can be broken into three parts:

  • 04: Current year (2004).
  • MADRID: Origin (the Embassy in Madrid)
  • 893: Record number?

What's that record number? Let's investigate. There are some cables sent on December 2004 from Madrid Embassy, as 04MADRID4887 (dated December 29, 2004). Its record number is "4887". Another message sent on February has ID 04MADRID527, record number "527". Looking to others cables dated on January, seems obvious that the record number starts at 1 and goes up, one by one, through the year. The record number is a simple ordinal value. Thanks to this simple rule, and reading the last cables of Madrid Embassy on December 2004, we know it sent ~4900 cables that year alone.

Ideally, the last cable of the year from each Embassy would be available, but the Cablegate data is not complete. Just fraction of the leaked messages has been published so far and those last cables of the year may not be leaked in Cablegate anyway. But, as can be seen in the graphics, this method allows to do an approximation.

The code used for the calculations is available at github (cablegate-sp) and has a BSD license.

Out of sight, out of mind.

One month after the first cable release, only two thousand messages has been published. At this rate it will take a decade to release all Cablegate content. Maybe not all messages are as relevant as those released so far, eg boring messages about visas. But if WikiLeaks has raised such a stir with just 2000 cables, I cannot imagine which other secrets remain in those thousands unfiltered (although top-secret cables use other networks).

Anyway, I'm sure there is still a lot of data mining job to do with the cables.

(Spanish version of this article: Cablegate: Lo que no está en WikiLeaks).

PS (December 30th, 2010): Ricardo Estalmán linked to this entry on Wikipedia about the German tank problem during World War II:

«Suppose one is an Allied intelligence analyst during World War II, and one has some serial numbers of captured German tanks. Further, assume that the tanks are numbered sequentially from 1 to N. How does one estimate the total number of tanks?»

The Cablegate case is quite similar. I will update the estimation with the formula cited in the above article, as soon as possible (Xmas days!).

Read more