Canonical Voices

What Blogging in the Wind talks about

Posts tagged with 'tech'

rvr

Fernando Tricas always has interesting things to say. In a recent post he talks about The life of links and digital content (Spanish):

«We tend to assume that digital [content] is forever. But anyone who accumulates enough information also knows that sometimes its difficult to find it, in other cases it breaks and, of course, there is a non-zero probability that things go wrong when hosted by third-party services. It is an old topic here, remember Will we have all this information in the future? . The topic resurfaces as news in the light of Currently charged by the article that can be read at A Year After the Egyptian Revolution, 10% of Its Social Media Documentation Is Already Gone».

In the comments, Anónima said: «Given a time t and an interval ?t, the larger ?t, the more likely is that all information in a time t-?t you want to find is gone». This sounded like an statement to check, Thus, I decided to do an experiment with del.icio.us' bookmarks.

In delicious.com/rvr I have archived around 4000 links from 2004. So, I downloaded the backup file, an HTML file with all links and metadata (date, title, tags). I developed a python script to process this file: go through the links and save its current status (whether the link is alive or not). With another script, the status were processed to generate the statistics. These are the results:

Captura de pantalla 2012-04-12 a la(s) 01.02.39

As can be seen, there is a correlation between the age of the links and the probability of being dead. For the 10% who cited the Egyptian revolution, in the case of my delicious, we must go back three years ago (2009). But at 6 years from now, a quarter of the links are now defunct. Of course, the sample is very small shouldn't be representative. It would be interesting to compare it with other accounts and to extend the time span: How many links are still alive after 10 or 15 years? Is it the same with information stored in other media? Are all this death links resting in peace in a forgotten Google's cache disk?

I imagine that sometime in the future, librarians will begin to worry not only to digitize remote past documents, but also to preserve those of the present.

In case you are interested, the code to generate such data is available at github.com/vrruiz/delicious-death-links. The spreadsheet is also available in Google Docs .

Read more
rvr

Summary

  • Brief analysis of 150,000 photographs from Flickr in the province of Malaga.
  • It identifies the profile and preferences of tourists.

Last Saturday, I  was in Malaga. I was invited by Sonia Blanco and the Universidad Internacional de Andalucia to participate in workshop on Tourism and Social Networks. Sonia is professor at the University of Malaga, and one of the oldest bloggers in the Spanish blogosphere. Sonia asked me to present the analysis Fernando Tricas and myself did about Flickr photos and the Canary Islands (2009-2010), and I gladly accepted. I wanted to bring an update, so we got to work to make a short presentation with data from the province of Malaga. And that's what is shown below.

Video

Last Thursday, with the presentation already made, Fernando passed me an interesting link, a visualization by the Wall Street Journal that shows the density of a week of Foursquare check-ins in New York . If the WSJ could do it, so do we ;)  We already had the data and the map algorithms, so generated the maps by months and joined them to build the animation.

The video below shows the density of photographs taken in the province of Malaga from 2004 to 2010. Blue colors are areas where they make some pictures, and the red areas have made many pictures. There are areas with many photographs, places of touristic interest. And of course, there are months where the activity is higher and lower. 

Data

The video is just a bit of whole presented analysis. Full version is available below.

As you may know, Flickr is a popular photo-sharing service with 5 billion of hosted images and 86 million unique visitors. Flickr has social networking features, since it allows to make contacts. Flickr can play a role in the promotion of tourist destinations, as it is one of the main sources of images on the Internet. But to us, Flickr is a huge source of data: Which are the most photogenic places? Who are taking pictures there? These and other questions can answered using data mining.

For this study we obtained the metadata of 175,000 photographs (62,000 geolocated), 7,900 photographers and 1,470,000 tags (47,000 unique). All these pictures were either marked by the tag "malaga" or GPS coordinates were inside the province of Malaga.

Analysis

Below are the five most relevant slides: the tag cloud, the number of photos and photographers by months, the top 10 countries of the geolocated photographers, the group of tags and heatmaps of the geolocated images.

  • Turismo-malaga-11
  • Turismo-malaga-17
  • Turismo-malaga-13
  • Turismo-malaga-15
  • Turismo-malaga-20
  • Turismo-malaga-20
Turismo-malaga-20

According to those who share photos on Flickr about Malaga, we can conclude that:

  • The high season in Málaga is August (also, in April there is a Holy Week-effect.
  • Users come mainly from UK, USA, Italy, Germany, Madrid and Andalusia. (USA is probably overrepresented compared to real visitors).
  • They are interested in photography, beaches, festivals, fairs, nature, sea, birds, sky, parks.
  • Pictures are taken mainly in Málaga (capital), Ronda, Barcenilla and Benalmadena.

The full presentation slides show more features, such as geolocated photographs by countries. It is interesting to compare these data with the previous study on the Canaries. A more detailed analysis can be done, but the roundtable had limited time. This sneak peek shows the potential of social networking and geolocation services for market research. If you have any questions, ask in the comments!

The presentation and images have a Creative Commons Attribution-Share Alike license.

Finally, my gratitude to the organization of the UNIA for the invitation and hospitality, to Daniel Cerdan for suggesting the title of the post and Fernando Tricas for his unconditional support.

Read more
rvr

Abstract.

  • The Cablegate set is composed of +250,000 diplomatic cables.
  • The total number sent by Embassies and Secretary of State is guessed.

One of the biggest mysteries in astrophysics is the dark matter. Dark matter can not be seen, it doesn't shine nor reflects light. But we infer its existence because dark matter weights, and modifies the path of stars and galaxies. Cablegate has its own dark matter.

According to WikiLeaks, 251,287 communications compose the Cablegate. But what is the real volume of cables between the Embassies and Secretary of State? Can we guess it? The answer is yes, there is a simple way to know it. Using the methodology explained below, the total number of communications between Embassies and the Secretary of State is guessed.

This are the results.

The dark matter of the Embassies.

20101224cablegate-darkmatter.001Between 2005-2009, more than 400,000 non leaked cables are identified. In this case, the uncertainty is larger than with just one embassy due to the small number or released cables. The sum increased by 50% in just one week.

Curiously, the average size of the 1800 published cables is 12 KB. If this average is representative of the whole set, something I doubt, the total size of the 250,000 messages would be 350 MB.

Secretary of State.

In addition to embassies' communications, Cablegate has some cables from the Secretary of State. This messages are often quite interesting, because they request information or send commands to the embassies (eg 09STATE106750).

20101224cablegate-darkmatter.002In 2005 and 2006 there is no released cable, and therefore the sum cannot be estimated. But between 2007 and 2009, the volume of cables sent by the Secretary of State is remarkable (so big, that I doubted that the record number was an ordinal number and not a more sophisticated identifier). Compare this graph with the one of the embassies. 2007 show more cables from the Secretary than all Embassies combined, but beware, because this trend can be reversed with better data.

This results are available in Google Docs.

Madrid Embassy.

This is the chart for Madrid Embassy, which ranks seventh in the number of leaked cables.

20101224cablegate-darkmatter.003Between 2004-2009, the existence of at least 17,000 dispatches sent from Madrid can be deduced. In the same period, there are just 3500 leaked cables. The graph shows the breakdown by year. 2007 is leaked in a high percentage, the oppositat in 2004 and 2005. Also, the number of communications decreases progressively (Why? Maybe other networks are used instead of SIPRNet). The complete table is available in Google Docs.

Cablegate Dark Matter Howto

The Guardian published a text file with dates, source and tags of the 250,000 diplomatic cables included in the Cablegate. The content of this messages are being slowly released. (Using this short descriptions, I did an analysis of the messages related to Spain -tagged as SP-, and suggested the existence of communications related to the 2004 Madrid bombings and the Spaniard Internet Law. Later, El País published this cables, confirming the suspicions).

To infer the volume of communications the methodology is quite simple. Each cable has an identifier. For example, 04MADRID893 summaries the Madrid bombing on March 11th, 2004. This identifier can be broken into three parts:

  • 04: Current year (2004).
  • MADRID: Origin (the Embassy in Madrid)
  • 893: Record number?

What's that record number? Let's investigate. There are some cables sent on December 2004 from Madrid Embassy, as 04MADRID4887 (dated December 29, 2004). Its record number is "4887". Another message sent on February has ID 04MADRID527, record number "527". Looking to others cables dated on January, seems obvious that the record number starts at 1 and goes up, one by one, through the year. The record number is a simple ordinal value. Thanks to this simple rule, and reading the last cables of Madrid Embassy on December 2004, we know it sent ~4900 cables that year alone.

Ideally, the last cable of the year from each Embassy would be available, but the Cablegate data is not complete. Just fraction of the leaked messages has been published so far and those last cables of the year may not be leaked in Cablegate anyway. But, as can be seen in the graphics, this method allows to do an approximation.

The code used for the calculations is available at github (cablegate-sp) and has a BSD license.

Out of sight, out of mind.

One month after the first cable release, only two thousand messages has been published. At this rate it will take a decade to release all Cablegate content. Maybe not all messages are as relevant as those released so far, eg boring messages about visas. But if WikiLeaks has raised such a stir with just 2000 cables, I cannot imagine which other secrets remain in those thousands unfiltered (although top-secret cables use other networks).

Anyway, I'm sure there is still a lot of data mining job to do with the cables.

(Spanish version of this article: Cablegate: Lo que no está en WikiLeaks).

PS (December 30th, 2010): Ricardo Estalmán linked to this entry on Wikipedia about the German tank problem during World War II:

«Suppose one is an Allied intelligence analyst during World War II, and one has some serial numbers of captured German tanks. Further, assume that the tanks are numbered sequentially from 1 to N. How does one estimate the total number of tanks?»

The Cablegate case is quite similar. I will update the estimation with the formula cited in the above article, as soon as possible (Xmas days!).

Read more