Canonical Voices

UbuntuTouch

在前面的文章中,我们实现了如何使用Content Hub来import我们的图片等。在今天的例程中,我们来介绍如何使用ContentHub来import我们所需要的电话本中的contact。电话本Contacts的许多的API可以在地址http://developer.ubuntu.com/api/apps/qml/sdk-15.04/QtContacts/找到。


为了说明问题,在今天的应用中,我们也使用了特殊的policy contacts。这个policy是为了能够在我们的应用中显示已有的电话本信息,虽然我们在使用ContentHub import时并不需要。contacts目前是受限制的policy。在运行我们的应用时,我们可以强制让它在手机上执行。


        ContactModel {
            id: model
            autoUpdate: true
        }

        Component.onCompleted:  {
            var list = model.availableManagers;

            console.log("The " + list.length + " ContactModel managers are: ");
            for (var item in list ) {
                console.log("manager[ " + item + " ]: " + list[item] );
            }
        }

我们可以通过上面的方法来得到所有已经在手机中有的manager。


qml: The 3 ContactModel managers are: 
qml: manager[ 0 ]: invalid
qml: manager[ 1 ]: galera
qml: manager[ 2 ]: memory

目前在我们的手机中,我们有以上的显示的managers。这里的galera是我们的手机电话本所需要的manager。如果需要读取手机中的电话本,我们可以需要用到这个。memory是存在于手机内存中的。我们可以通过它来import一个.vcf格式的电话本。这样,我们可以轻松地通过ContactModel来翻译解析我们的数据。


为了显示galera中的数据,我们使用如下的代码:


        ContactModel {
            id: phoneContact
            autoUpdate: true

            manager: "galera"
        }

       ...

        ListView {
            id: contactView
            height: parent.height *.45

            anchors {
                left: parent.left
                right: parent.right
                top: parent.top
            }

            model: phoneContact

            delegate: ListItem.Subtitled {
                text: contact.name.firstName
                subText: contact.phoneNumber.number
            }
        }


为了显示我们我们import过来的.vcf格式的contact,我们使用如下的方法:


        ContactModel {
            id: contactModel
            autoUpdate: true

            manager: "memory"
        }

        ...

        ListView {
            id: importedView
            height: parent.height - contactView.height -
                    buttons.height - importedText.height

            anchors {
                left: parent.left
                right: parent.right
                top: importedText.bottom
                bottom: buttons.bottom
            }

            delegate: ListItem.Subtitled {
                text: contact.name.firstName
                subText: contact.phoneNumber.number
            }
        }

       ...

        Connections {
            target: root.activeTransfer ? root.activeTransfer : null
            onStateChanged: {
                if (root.activeTransfer.state === ContentTransfer.Charged) {
                    importItems = root.activeTransfer.items;

                    console.log("length: " + importItems.length);

                    for ( var i = 0; i < importItems.length; i ++ ) {
                        console.log("imported url: " + importItems[i].url);
                        console.log("imported text: " + importItems[i].text);
                    }

                    contactModel.importContacts(importItems[0].url, ["Sync"])

                    // Now dump the data
                    var contacts = contactModel.contacts;

                    console.log("length: " + contacts.length );
                    for ( var contact in contacts ) {
                        console.log("contact[ " + contact + "]: " + contacts[contact].name);
                    }

                    importedView.model = contactModel;

                }
            }
        }


我们可以通过动态设置我们的ListView model的方法来显示我们从ContentHub import过来的contacts。


运行我们的代码:


  


  


所有的源码在: git clone https://gitcafe.com/ubuntu/contact-importer.git

作者:UbuntuTouch 发表于2015/8/25 11:57:56 原文链接
阅读:161 评论:0 查看评论

Read more
UbuntuTouch

对于一些应用场景来说,TCP/IP连接是唯一的一种通讯的协议。对于我们的QML应用来说,我们可以使用WebSocket来建立一个双工的(full-duplex)的TCP/IP连接。在今天的例程中,我们将来介绍如何使用WebSocket来建立这种连接,并实现通信。


首先,我们得import我们需要的模块:

import Qt.WebSockets 1.0

然后,我们使用它:


import QtQuick 2.0
import Ubuntu.Components 1.1
import QtQuick.Layouts 1.1
import Qt.WebSockets 1.0

/*!
    \brief MainView with a Label and Button elements.
*/

MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "websocket.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    // Removes the old toolbar and enables new features of the new header.
    useDeprecatedToolbar: false

    width: units.gu(60)
    height: units.gu(85)

    function appendMessage(msg) {
        var length = output.length;
        output.insert(length, msg + "\r\n");
    }

    Page {
        id: page
        title: i18n.tr("websocket")

        WebSocket {
            id: socket
            url: input.text
            onTextMessageReceived: {
                console.log("something is received!");
                appendMessage("received: " + message);
            }

            onStatusChanged: if (socket.status == WebSocket.Error) {
                                 console.log("Error: " + socket.errorString)
                             } else if (socket.status == WebSocket.Open) {
                                 appendMessage("sending \"Hello world\"");
                                 socket.sendTextMessage("Hello World")
                             } else if (socket.status == WebSocket.Closed) {
                                 appendMessage("Socket closed");
                             }
            active: true
        }

        Column {
            anchors.fill: parent
            spacing: units.gu(1)

            RowLayout {
                id: top
                width: parent.width

                TextField {
                    id: input
                     Layout.minimumWidth: page.width *.7
                    text: "ws://echo.websocket.org"
                }

                Button {
                    id: get
                    text: "Get"
                    onClicked: {
                        socket.active = true
                        socket.sendTextMessage("Nice to meet you!")
                    }
                }
            }

            TextArea {
                id: output
                width: parent.width
                height: page.height - top.height - units.gu(1)
            }
        }
    }
}

在上面的代码中:


       WebSocket {
            id: socket
            url: input.text
            onTextMessageReceived: {
                console.log("something is received!");
                appendMessage("received: " + message);
            }

            onStatusChanged: if (socket.status == WebSocket.Error) {
                                 console.log("Error: " + socket.errorString)
                             } else if (socket.status == WebSocket.Open) {
                                 appendMessage("sending \"Hello world\"");
                                 socket.sendTextMessage("Hello World")
                             } else if (socket.status == WebSocket.Closed) {
                                 appendMessage("Socket closed");
                             }
            active: true
        }

我们从input.text中得到url。当active为真时,建立起Socket的连接。我们可以在onStatusChhanged中得到这个变化。当我们把active设为假时,安全套接字将被自动断开。在例程中,我们使用了一个公共的网站


ws://echo.websocket.org


每当我们向这个地址发送信息时,就会得到和发送信息一模一样的信息(是一个echo服务器)。我们可以通过onTextMessageReceived来得到这个信息。


运行我们的应用:




整个应用的源码在:git clone https://gitcafe.com/ubuntu/websocket.git

作者:UbuntuTouch 发表于2015/8/26 13:58:13 原文链接
阅读:199 评论:0 查看评论

Read more
UbuntuTouch

[原]如何得到Ubuntu手机上的IP地址

在这篇文章中,我们介绍如何在Ubuntu QML应用中得到手机上的IP地址。


我们在我们的main.cpp中加入一些代码来得到IP地址:


main.cpp


#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QQuickView>
#include <QNetworkInterface>
#include <QQmlContext>

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);

    QList<QHostAddress> list = QNetworkInterface::allAddresses();
    QStringList datalist;

    for(int nIter = 0; nIter < list.count(); nIter++) {
        qDebug() << list[ nIter ].toString();
        datalist.append(list[ nIter ].toString());
    }

    QQuickView view;
    view.setSource(QUrl(QStringLiteral("qrc:///Main.qml")));
    view.setResizeMode(QQuickView::SizeRootObjectToView);

    QQmlContext *ctxt = view.rootContext();
    ctxt->setContextProperty("myModel", QVariant::fromValue(datalist));

    view.show();
    return app.exec();
}


Main.qml


import QtQuick 2.0
import Ubuntu.Components 1.1

/*!
    \brief MainView with a Label and Button elements.
*/

MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "ipaddress.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    // Removes the old toolbar and enables new features of the new header.
    useDeprecatedToolbar: false

    width: units.gu(60)
    height: units.gu(85)

    Page {
        title: i18n.tr("ipaddress")

        Column {
            spacing: units.gu(2)
            anchors {
                margins: units.gu(2)
                fill: parent
            }

            Label {
                id: label
                objectName: "label"
                fontSize: "large"

                text: i18n.tr("IP addresses: ")
            }

            ListView {
                width: parent.width
                height: parent.height - label.height
                model: myModel
                delegate: Text {
                    text: modelData
                }
            }

        }
    }
}


运行我们的应用:



这是我们在利用wifi时显示的IP地址。

我们的源码在: git clone https://gitcafe.com/ubuntu/ipaddress.git


作者:UbuntuTouch 发表于2015/8/26 15:05:55 原文链接
阅读:241 评论:0 查看评论

Read more
UbuntuTouch

[原]如何在Ubuntu平台上使用Bluetooth

截止到目前为止,在Ubuntu平台上,Bluetooth还没有被正式支持。对于一些hacker级的开发者来说,可能等不急。我们在这里提供一个方法让大家来尝尝鲜。最终的平台会支持Bluetooth。而且相应的库也将会安装到平台之中去。


为了能够在手机中运行Bluetooth的应用,我们必须做如下的动做:




上面的句子:

$ sudo mount -o rw,remount /

是为了让我们的image变为可读写的状态,以便我们在下面安装我们的Qt Bluetooth库

我们通过命令:


$ sudo apt-get install qml-module-qtbluetooth


来安装相应的Qt Bluetooth的库。


由于目前Bluetooth没有被完全支持,所以没有相应的security policy和它对应。在我们的测试应用中,我们必须适应“unconfined” template:


bluetooth.apparmor

{
    "policy_groups": [
        "networking",
        "webview",
        "connectivity"
    ],
    "policy_version": 1.3,
    "template": "unconfined"
}


Scanner.qml


import QtQuick 2.0
import QtBluetooth 5.2

Item {
    id: top

    property BluetoothService currentService

    BluetoothDiscoveryModel {
        id: btModel
        running: true
        discoveryMode: BluetoothDiscoveryModel.DeviceDiscovery
        onDiscoveryModeChanged: console.log("Discovery mode: " + discoveryMode)
        onServiceDiscovered: console.log("Found new service " + service.deviceAddress + " " + service.deviceName + " " + service.serviceName);
        onDeviceDiscovered: console.log("New device: " + device)
        onErrorChanged: {
                switch (btModel.error) {
                case BluetoothDiscoveryModel.PoweredOffError:
                    console.log("Error: Bluetooth device not turned on"); break;
                case BluetoothDiscoveryModel.InputOutputError:
                    console.log("Error: Bluetooth I/O Error"); break;
                case BluetoothDiscoveryModel.InvalidBluetoothAdapterError:
                    console.log("Error: Invalid Bluetooth Adapter Error"); break;
                case BluetoothDiscoveryModel.NoError:
                    break;
                default:
                    console.log("Error: Unknown Error"); break;
                }
        }
   }

    Rectangle {
        id: busy

        width: top.width * 0.7;
        anchors.horizontalCenter: parent.horizontalCenter
        anchors.top: top.top;
        height: text.height*1.2;
        radius: 5
        color: "#1c56f3"
        visible: btModel.running

        Text {
            id: text
            text: "Scanning"
            font.bold: true
            font.pointSize: 20
            anchors.centerIn: parent
        }

        SequentialAnimation on color {
            id: busyThrobber
            ColorAnimation { easing.type: Easing.InOutSine; from: "#1c56f3"; to: "white"; duration: 1000; }
            ColorAnimation { easing.type: Easing.InOutSine; to: "#1c56f3"; from: "white"; duration: 1000 }
            loops: Animation.Infinite
        }
    }

    ListView {
        id: mainList
        width: top.width
        anchors.top: busy.bottom
        anchors.bottom: buttonGroup.top
        anchors.bottomMargin: 10
        anchors.topMargin: 10
        clip: true

        model: btModel
        delegate: Rectangle {
            id: btDelegate
            width: parent.width
            height: column.height + 10

            property bool expended: false;
            clip: true
            Image {
                id: bticon
                source: "images/default.png";
                width: bttext.height;
                height: bttext.height;
                anchors.top: parent.top
                anchors.left: parent.left
                anchors.margins: 5
            }

            Column {
                id: column
                anchors.left: bticon.right
                anchors.leftMargin: 5
                Text {
                    id: bttext
                    text: deviceName ? deviceName : name
                    font.family: "FreeSerif"
                    font.pointSize: 16
                }

                Text {
                    id: details
                    function get_details(s) {
                        if (btModel.discoveryMode == BluetoothDiscoveryModel.DeviceDiscovery) {
                            //We are doing a device discovery
                            var str = "Address: " + remoteAddress;
                            return str;
                        } else {
                            var str = "Address: " + s.deviceAddress;
                            if (s.serviceName) { str += "<br>Service: " + s.serviceName; }
                            if (s.serviceDescription) { str += "<br>Description: " + s.serviceDescription; }
                            if (s.serviceProtocol) { str += "<br>Protocol: " + s.serviceProtocol; }
                            return str;
                        }
                    }
                    visible: opacity !== 0
                    opacity: btDelegate.expended ? 1 : 0.0
                    text: get_details(service)
                    font.family: "FreeSerif"
                    font.pointSize: 14
                    Behavior on opacity {
                        NumberAnimation { duration: 200}
                    }
                }
            }
            Behavior on height { NumberAnimation { duration: 200} }

            MouseArea {
                anchors.fill: parent
                onClicked: btDelegate.expended = !btDelegate.expended
            }
        }
        focus: true
    }

    Row {
        id: buttonGroup
        property var activeButton: devButton

        anchors.bottom: parent.bottom
        anchors.horizontalCenter: parent.horizontalCenter
        anchors.bottomMargin: 5
        spacing: 10

        Button {
            id: fdButton
            width: top.width/3*0.9
            //mdButton has longest text
            height: mdButton.height
            text: "Full Discovery"
            onClicked: btModel.discoveryMode = BluetoothDiscoveryModel.FullServiceDiscovery
        }
        Button {
            id: mdButton
            width: top.width/3*0.9
            text: "Minimal Discovery"
            onClicked: btModel.discoveryMode = BluetoothDiscoveryModel.MinimalServiceDiscovery
        }
        Button {
            id: devButton
            width: top.width/3*0.9
            //mdButton has longest text
            height: mdButton.height
            text: "Device Discovery"
            onClicked: btModel.discoveryMode = BluetoothDiscoveryModel.DeviceDiscovery
        }
    }

}


关于BluetoothDiscoveryModel的介绍,开发者可以直接参阅Qt文档做更一步的认识。


运行我们的应用:


  


整个应用的源码在:git clone https://gitcafe.com/ubuntu/bluetooth.git

作者:UbuntuTouch 发表于2015/8/27 10:40:20 原文链接
阅读:59 评论:0 查看评论

Read more
UbuntuTouch

[原]如何固定你的Ubuntu应用的方向

在这篇文章中,我们将介绍如何固定一个Ubuntu应用的方向。固定应用的方向对有些游戏应用来说,非常有用。这样可以让游戏专注于一个方向的布局,比如开车的游戏!


在Ubuntu应用中,我们可以通过如下的flag:


MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "usermetrics.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    // Removes the old toolbar and enables new features of the new header.
    useDeprecatedToolbar: false

    width: units.gu(60)
    height: units.gu(85)
   ...
}

这里有一个automaticOrientation标志位。由于一些原因,目前还是不能正常工作,虽然我们可以设置它为false。


目前我发现一个更加简单的办法,就是直接修改项目的.desktop文件:


[Desktop Entry]
Name=fixedorientationapp
Exec=qmlscene $@ Main.qml
Icon=fixedorientationapp.png
Terminal=false
Type=Application
X-Ubuntu-Touch=true
X-Ubuntu-Supported-Orientations=landscape


我们在上面添加了X-Ubuntu-Supported-Orientations=landscape。这样我们的应用就只有在landscape模式下。


当我们把项目的.desktop文件修改为:


[Desktop Entry]
Name=fixedorientationapp
Exec=qmlscene $@ Main.qml
Icon=fixedorientationapp.png
Terminal=false
Type=Application
X-Ubuntu-Touch=true
X-Ubuntu-Supported-Orientations=portrait

当我们运行我们的应用时,我们的应用只有在portrait模式下运行。不可以更改。



整个应用的代码: git clone https://gitcafe.com/ubuntu/fixedorientationapp.git



作者:UbuntuTouch 发表于2015/8/28 16:20:53 原文链接
阅读:147 评论:0 查看评论

Read more
UbuntuTouch

我们可以通过UserMetric的API发布消息到我们手机的欢迎页面(手机的锁屏界面)。在锁屏界面中,我们可以双击中间的圆圈,来循环播放我们手机发布的消息。如下图所示,

我们发布了“Useermetric messages received: 4”个消息。




我们需要使用的模块是:

import UserMetrics 0.1

我们可以使用如下的方法得到它所有的API:


liuxg@liuxg:~$ qmlplugindump import UserMetrics 0.1
QQmlComponent: Component is not ready
liuxg@liuxg:~$ qmlplugindump  UserMetrics 0.1
import QtQuick.tooling 1.1

// This file describes the plugin-supplied types contained in the library.
// It is used for QML tooling purposes only.
//
// This file was auto-generated by:
// 'qmlplugindump UserMetrics 0.1'

Module {
    Component {
        name: "Metric"
        prototype: "QObject"
        exports: ["Metric 0.1"]
        exportMetaObjectRevisions: [0]
        Property { name: "name"; type: "string" }
        Property { name: "format"; type: "string" }
        Property { name: "emptyFormat"; type: "string" }
        Property { name: "domain"; type: "string" }
        Property { name: "minimum"; type: "double" }
        Property { name: "maximum"; type: "double" }
        Method {
            name: "increment"
            Parameter { name: "amount"; type: "double" }
        }
        Method { name: "increment" }
        Method {
            name: "update"
            Parameter { name: "value"; type: "double" }
        }
    }
}


我们的应用非常简单:


Main.qml


import QtQuick 2.0
import Ubuntu.Components 1.1
import UserMetrics 0.1

/*!
    \brief MainView with a Label and Button elements.
*/

MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "usermetrics.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    // Removes the old toolbar and enables new features of the new header.
    useDeprecatedToolbar: false

    width: units.gu(60)
    height: units.gu(85)

    Page {
        title: i18n.tr("usermetrics")

        Metric {
            id: incomingMessagesMetric
            name: "usermetric"
            format: i18n.tr("Usermetric messages received today: %1")
            emptyFormat: i18n.tr("No Usermetric messages received today.")
            domain: applicationName
        }

        Column {
            anchors.centerIn: parent
            spacing: units.gu(2)

            Button {
                text: i18n.tr("Increase")

                onClicked: {
                    console.log("Going to increase the metric!");
                    incomingMessagesMetric.increment()
                }
            }

            Button {
                text: i18n.tr("Decrease")

                onClicked: {
                    console.log("Going to increase the metric bye 2!");
                    incomingMessagesMetric.increment(-2)
                }
            }

            Button {
                text: i18n.tr("Decrease")

                onClicked: {
                    console.log("Going to update to 3 to the metric!");
                    incomingMessagesMetric.update(3)
                }
            }

        }
    }
}


为了使用UserMetric,我们必须在我们的应用中使用usermetrics security policy:

usermetrics.apparmor


{
    "policy_groups": [
        "networking",
        "webview",
        "usermetrics"
    ],
    "policy_version": 1.3
}


运行我们的应用:




每当我们点击increase按钮后,我们的欢迎界面数据将增加一个:




整个应用的源码在:git clone https://gitcafe.com/ubuntu/usermetrics.git


作者:UbuntuTouch 发表于2015/8/28 16:51:41 原文链接
阅读:150 评论:0 查看评论

Read more
UbuntuTouch

[原]利用Qt.locale显示本地化数据

我们知道对于一些应用来说,我们可以根据语言的选择来显示不同的数据格式,比如时间,金钱等。在今天的例程中,我们来展示如何Qt.locale根据不同的语言选择来帮助我们显示不同格式的数据。


import QtQuick 2.0
import Ubuntu.Components 1.1

/*!
    \brief MainView with a Label and Button elements.
*/

MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "locale.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    // Removes the old toolbar and enables new features of the new header.
    useDeprecatedToolbar: false

    width: units.gu(60)
    height: units.gu(85)

    Page {
        title: i18n.tr("Locale")

        Rectangle {
            id: root
            anchors.fill: parent
            color: "lightgray"

            property string locale: view.currentItem.locale

            Text {
                id: title
                text: "Select locale:"
            }

            Rectangle {
                id: chooser
                anchors.top: title.bottom
                anchors.topMargin: 5
                width: parent.width-10
                x: 5
                height: parent.height/2 - 10
                color: "#40300030"
                ListView {
                    id: view
                    clip: true
                    focus: true
                    anchors.fill: parent
                    model: [
                        "en_US",
                        "en_GB",
                        "fi_FI",
                        "de_DE",
                        "ar_SA",
                        "hi_IN",
                        "zh_CN",
                        "th_TH",
                        "fr_FR",
                        "nb_NO",
                        "sv_SE"
                    ]
                    delegate: Text {
                        property string locale: modelData
                        height: units.gu(3)
                        width: view.width
                        text: Qt.locale(modelData).name + " ("+ Qt.locale(modelData).nativeCountryName + "/" + Qt.locale(modelData).nativeLanguageName + ")"
                        MouseArea {
                            anchors.fill: parent
                            onClicked: view.currentIndex = index

                        }
                    }
                    highlight: Rectangle {
                        height: 30
                        color: "#60300030"
                    }
                }
            }

            Rectangle {
                color: "white"
                anchors.top: chooser.bottom
                anchors.topMargin: 5
                anchors.bottom: parent.bottom
                anchors.bottomMargin: 5
                x: 5; width: parent.width - 10

                Column {
                    anchors.fill: parent
                    spacing: 5
                    Text {
                        property var date: new Date()
                        text: "Date: " + date.toLocaleDateString(Qt.locale(root.locale))
                    }
                    Text {
                        property var date: new Date()
                        text: "Time: " + date.toLocaleTimeString(Qt.locale(root.locale))
                    }
                    Text {
                        property var dow: Qt.locale(root.locale).firstDayOfWeek
                        text: "First day of week: " + Qt.locale(root.locale).standaloneDayName(dow)
                    }
                    Text {
                        property var num: 10023823
                        text: "Number: " + num.toLocaleString(Qt.locale(root.locale))
                    }
                    Text {
                        property var num: 10023823
                        text: "Currency: " + num.toLocaleCurrencyString(Qt.locale(root.locale))
                    }
                }
            }
        }
    }
}


在例程中,注意下面的写法:


delegate: Text {
                        property string locale: modelData
                        height: units.gu(3)
                        width: view.width
                        text: Qt.locale(modelData).name + " ("+ Qt.locale(modelData).nativeCountryName + "/" + Qt.locale(modelData).nativeLanguageName + ")"
                        MouseArea {
                            anchors.fill: parent
                            onClicked: view.currentIndex = index

                        }
                    }

我们可以通过定义一个property locale来得到当前的Item的modelData。这样在delegate的外面,我们通过


 property string locale: view.currentItem.locale

来得到当前列表项中的modelData,进而在程序的其它部分进行引用!

  


整个项目的源码在:git clone https://gitcafe.com/ubuntu/locale.git



作者:UbuntuTouch 发表于2015/8/31 15:12:33 原文链接
阅读:203 评论:0 查看评论

Read more
UbuntuTouch

在今天的这篇文章中,我们将介绍在Ubuntu平台中如何使用页面布局自动适配不同的屏幕尺寸,从而使得同一个应用在不同屏幕尺寸上使得我们的应用显示更加合理。更确切地说我们在不同的屏幕尺寸的设备中不需要修改我们的代码。这对于为了Ubuntu平台的convergence非常有用。本文的英文出处“Adaptive page layouts made easy”。




这种自适应布局对有些应用非常有用。比如对于一个信息的应用来说,如果运行在PC或平板电脑上,在屏幕的左边我们可以显示所有信息的列表,而在屏幕的右边显示所选对话的的conversation的细节。而对于一个小的屏幕的设备,比如说手机来说,在开始的页面只能显示信息的列表,而当一个信息被选中后,信息的详细信息将被显示在同样的屏幕尺寸的另外一个页面。等我们看完信息后,我们可以通过返回键重新返回到列表的页面。在Ubuntu的设计中,我们不需要为这两种情况分别发布我们的应用。我们只需要使用我们Ubuntu平台提供的自适应页面布局就可以完全搞定。


   




详细的视频可以观看视频“Ubuntu 手机邮件软件”。


特别需要指出的是,为了完成这个功能,我们必须在我们的应用中使用AdaptivePageLayout。目前该API只限于Ubuntu.Components 1.3。目前这个API还处于开发中,并在不断地完善。我们可以在Wily (15.10)中体会这个功能。如果你的桌面电脑是vivid (Ubuntu 15.04)的话,你可以通过添加如下的PPA来得到最新的Ubuntu.Component 1.3:


$add-apt-repository ppa:ci-train-ppa-service/stable-phone-overlay
$apt-get update
$apt-get upgrade

这样你就可以先体验这个功能了。如果你想在你的手机上体验的话,你也需要把手机的软件刷成15.10或以上。


AdaptivePageLayout


AdaptivePageLayout是一个Item,并具有一下的属性和方法:

  • property Page primaryPage
  • function addPageToCurrentColumn(sourcePage, newPage)
  • function addPageToNextColumn(sourcePage, newPage)
  • function removePages(page)

为了完全理解AdaptivePageLayout是如何工作的。想象一下AdaptivePageLayout在内部管理了无数个可能显示到你屏幕的虚拟的Column。并不是所有的虚拟的Column都是可见的。在默认的情况下,依赖于你的AdaptivePageLayout的尺寸的大小,只有一个或两个Column是可见的。一个页面可以被加入到一个虚拟的Column中。

被定义为primaryPage的页面将被显示在最左边的Column里。其它的Column里将是空的。

为了在第一页面显示另外一个Page,我们可以调用addPageToCurrentColumn(),并把当前页面primaryPage作为参数传人。这样显示的页面将会显示在同样的页面位置(同样的Column),并在header的位置出现一个back的按钮来关掉页面,并返回到上一个页面中。这种情况和我们以前的PageStack是没有任何的差别的。这在我以前的文章“Ubuntu OS上的QML应用框架”中有详细的介绍。





如上所示,当我们的屏幕尺寸为100x70 gu时,我们可以看出来在header部分出现一个返回的按钮以关掉当前页面,并返回。

当我们的屏幕尺寸为60x85 gu时:

  

就像我们之前所讲的,它和我们以前说说的PageStack是没有任何差别的。


接下来,我们调用addPageToNextColumn,并传人和上面一样的参数,我们就会发现不同的显示:

当我们的尺寸为100x70 gu时,



上面点击“Add page right”:

 onClicked: layout.addPageToNextColumn(rootPage, rightPage)

我们很容易看出来,rightPage显示在primaryPage的右边,它并没有覆盖我们左边的primaryPage。

当我们把我们的尺寸改为60x85 gu时:

  

上面也是在点击“Add page right”时显示的画面。除了我们唯一修改的尺寸外,我们并没有修改其它的任何代码。就像上面图上显示的那样,由于屏幕尺寸的限制,rightPage覆盖了primaryPage,并在header部分显示一个返回的按钮。

我们可以在运行我们应用的情况下,在Desktop上通过拖拽的方式改变应用的宽度,我们可以看到应用的布局发生的改变。


import QtQuick 2.4
import Ubuntu.Components 1.3

/*!
    \brief MainView with a Label and Button elements.
*/

MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "adaptivepagelayout.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    width: units.gu(100)
    height: units.gu(70)

    AdaptivePageLayout {
        id: layout
        anchors.fill: parent
        primaryPage: rootPage

        Page {
            id: rootPage
            title: i18n.tr("Root page")

            Rectangle {
                id: rect
                anchors.fill: parent
                color: "red"
                border.color: "green"
                border.width: units.gu(1)

                Component.onCompleted: {
                    console.log("rect size: " + rect.width + " " + rect.height)
                }
            }

            Column {
                anchors {
                    top: parent.top
                    left: parent.left
                    margins: units.gu(1)
                }
                spacing: units.gu(1)

                Button {
                    text: "Add page left"
                    onClicked: layout.addPageToCurrentColumn(rootPage, leftPage)
                }
                Button {
                    text: "Add page right"
                    onClicked: layout.addPageToNextColumn(rootPage, rightPage)
                }
                Button {
                    text: "Add sections page right"
                    onClicked: layout.addPageToNextColumn(rootPage, sectionsPage)
                }
            }

            Component.onCompleted: {
                console.log("Initial rootpage size: " + rootPage.width + " " + rootPage.height)
            }

            onWidthChanged: {
                console.log("rootPage width changed: " + rootPage.width)
            }
        }

        Page {
            id: leftPage
            title: i18n.tr("First column")

            Rectangle {
                anchors {
                    fill: parent
                    margins: units.gu(2)
                }
                color: UbuntuColors.orange

                Button {
                    anchors.centerIn: parent
                    text: "right"
                    onTriggered: layout.addPageToNextColumn(leftPage, rightPage)
                }
            }

            Component.onCompleted: {
                console.log("Initial leftPage size: " + leftPage.width + " " + leftPage.height)
            }

            onWidthChanged: {
                console.log("leftPage width changed: " + leftPage.width)
            }
        }

        Page {
            id: rightPage
            title: i18n.tr("Second column")

            Rectangle {
                anchors {
                    fill: parent
                    margins: units.gu(2)
                }
                color: UbuntuColors.green

                Button {
                    anchors.centerIn: parent
                    text: "Another page!"
                    onTriggered: layout.addPageToCurrentColumn(rightPage, sectionsPage)
                }
            }

            Component.onCompleted: {
                console.log("Initial rightPage size: " + rightPage.width + " " + rightPage.height)
            }

            onWidthChanged: {
                console.log("rightPage width changed: " + rightPage.width)
            }
        }

        Page {
            id: sectionsPage
            title: i18n.tr("Page with sections")
            head.sections.model: [i18n.tr("one"), i18n.tr("two"), i18n.tr("three")]

            Rectangle {
                anchors {
                    fill: parent
                    margins: units.gu(2)
                }
                color: UbuntuColors.blue
            }

            onWidthChanged: {
                console.log("sectionsPage width changed: " + sectionsPage.width)
            }
        }
    }
}


整个项目的源码在:git clone https://gitcafe.com/ubuntu/adaptivepagelayout.git



作者:UbuntuTouch 发表于2015/9/1 11:11:46 原文链接
阅读:39 评论:0 查看评论

Read more
UbuntuTouch

在今天的这篇文章中,我们将介绍如何读取一个CSV文件,并使用一个table进行展示数据。我们知道在Ubuntu平台中目前没有移植TableView。那么我们怎么来展示一个Table的数据呢? 答案是使用我们的ListItem。关于ListItem的更多的描述,大家可以参阅文章“浅叙Ubuntu.Components 1.2中的ListItem控件”。


首先,在Ubuntu桌面上,我们可以使用我们的LibreOffice来创建一个我们的数据表:




我们可以通过“Save as”菜单把我们的表格存为一个CSV的文件格式:




CSV格式的文件实际上是以分号分开的格式文件:


张三,1981.1.1,男,北京市,街道胡同号234567
李四,1982.2.2,男,南京市,街道胡同号234567
王红,1990.5.20,女,深圳市,街道胡同号234567
王五兰,2000.10.1,男,成都市,街道胡同号234567
陈建东,1985.11.23,男,上海市,街道胡同号234567

我们可以创建一个最简单的qmlproject应用,并把我们刚才创建好的csv文件考入到项目的根目录中。


为了读取这个文件,我们定义一个自己的model来供我们的ListView来使用:


AppModel.qml


import QtQuick 2.0

Item {
    property string source: ""

    property ListModel model : ListModel { id: jsonModel }
    property alias count: jsonModel.count

    function createPerson(r) {
        return {
                "name": r[0],
                "birthday":r[1],
                "sex":r[2],
                "city":r[3],
                "address":r[4]
               };
    }

    onSourceChanged: {
        console.log("Loading " + source)

        var xhr = new XMLHttpRequest;
        xhr.open("GET", source);
        xhr.onreadystatechange = function() {
            if (xhr.readyState == XMLHttpRequest.DONE) {
                var doc = xhr.responseText;
                console.log("doc: " + doc);

                 var records = xhr.responseText.split('\n');

                console.log("length: " + records.length)

                for ( var i = 0; i < records.length; i++ ) {
                    console.log("record: " + i + " "  + records[i]);

                    var r = records[i].split(',');
                    if ( r.length === 5 )
                        jsonModel.append(createPerson(r));
                }
            }
        }

       xhr.send();
    }
}

当我们的source被设置或发生改变时,onSourceChanged将被自动调用。我们使用XMLHttpRequest来读取本地的文件,并把解释好的数据放入到我们的ListModel中。


一旦有了数据,我们就可以通过我们的ListView来显示为我们的数据。尽管我们没有TableView这样的API供我们使用,我们可以通过ListItem来完成我们的相应的功能。

Main.qml


import QtQuick 2.0
import Ubuntu.Components 1.2

/*!
    \brief MainView with a Label and Button elements.
*/

MainView {
    // objectName for functional testing purposes (autopilot-qt5)
    objectName: "mainView"

    // Note! applicationName needs to match the "name" field of the click manifest
    applicationName: "tableview.liu-xiao-guo"

    /*
     This property enables the application to change orientation
     when the device is rotated. The default is false.
    */
    //automaticOrientation: true

    // Removes the old toolbar and enables new features of the new header.
    //    useDeprecatedToolbar: false

    width: units.gu(60)
    height: units.gu(85)

    Page {
        title: i18n.tr("tableview")

        AppModel {
            id: appmodel
            source: "file.csv"
        }

        ListView {
            id: listview
            anchors.fill: parent
            model: appmodel.model

            delegate: ListItem {
                id: del
                width: listview.width
                height: layout.childrenRect.height + units.gu(1)

                Rectangle {
                    anchors.fill: parent
                    color: index%2 == 0 ?  "Azure" : "Beige"
                }


                Column {
                    id: layout
                    width: listview.width
                    spacing: units.gu(1)

                    Item {
                        width: parent.width
                        height: name.height
                        Label { id: name; x: units.gu(1); text: model.name }
                        Label { id: birthday; x: del.ListView.view.width/4; text: model.birthday }
                        Label { id: sex; x: listview.width/2; text: model.sex }
                        Label { id: city; x: listview.width*3/4; text: model.city }
                    }

                    Label {
                        text: model.address
                        fontSize: "large"
                        anchors.right: parent.right
                        anchors.rightMargin: units.gu(1)
                    }
                }
            }
        }
    }
}



我们的显示也非常地简单。我们直接在ListItem中使用了一个二行的显示。第一行是四个Label,分别用于显示我们的数据表中的数据。同时我们在第二行中显示地址信息。这在标准的TableView中是没法实现的。从某种程度上讲,我们的ListItem是非常灵活的。

运行为我们的应用:



  


显然,我们使用ListView也同样很好地展示了我们的数据。

整个项目的源码在: git clone https://gitcafe.com/ubuntu/tableview.git
作者:UbuntuTouch 发表于2015/9/1 15:02:53 原文链接
阅读:25 评论:0 查看评论

Read more
beuno

Now that we've open sourced the code for Ubuntu One filesync, I thoughts I'd highlight some of the interesting challenges we had while building and scaling the service to several million users.

The teams that built the service were roughly split into two: the foundations team, who was responsible for the lowest levels of the service (storage and retrieval of files, data model, client and server protocol for syncing) and the web team, focused on user-visible services (website to manage files, photos, music streaming, contacts and Android/iOS equivalent clients).
I joined the web team early on and stayed with it until we shut it down, so that's where a lot of my stories will be focused on.

Today I'm going to focus on the challenge we faced when launching the Photos and Music streaming services. Given that by the time we launched them we had a few years of experience serving files at scale, our challenge turned out to be in presenting and manipulating the metadata quickly to each user, and be able to show the data in appealing ways to users (showing music by artist, genre and searching, for example). Photos was a similar story, people tended to have many thousands of photos and songs and we needed to extract metadata, parse it, store it and then be able to present it back to users quickly in different ways. Easy, right? It is, until a certain scale  :)
Our architecture for storing metadata at the time was about 8 PostgreSQL master databases where we sharded metadata across (essentially your metadata lived on a different DB server depending on your user id) plus at least one read-only slave per shard. These were really beefy servers with a truck load of CPUs, more than 128GB of RAM and very fast disks (when reading this, remember this was 2009-2013, hardware specs seem tiny as time goes by!).  However, no matter how big these DB servers got, given how busy they were and how much metadata was stored (for years, we didn't delete any metadata, so for every change to every file we duplicated the metadata) after a certain time we couldn't get a simple listing of a user's photos or songs (essentially, some of their files filtered by mimetype) in a reasonable time-frame (less than 5 seconds). As it grew we added caches, indexes, optimized queries and code paths but we quickly hit a performance wall that left us no choice but a much feared major architectural change. I say much feared, because major architectural changes come with a lot of risk to running services that have low tolerance for outages or data loss, whenever you change something that's already running in a significant way you're basically throwing out most of your previous optimizations. On top of that as users we expect things to be fast, we take it for granted. A 5 person team spending 6 months to make things as you expect them isn't really something you can brag about in the middle of a race with many other companies to capture a growing market.
In the time since we had started the project, NoSQL had taken off and matured enough for it to be a viable alternative to SQL and seemed to fit many of our use cases much better (webscale!). After some research and prototyping, we decided to generate pre-computed views of each user's data in a NoSQL DB (Cassandra), and we decided to do that by extending our existing architecture instead of revamping it completely. Given our code was pretty well built into proper layers of responsibility we hooked up to the lowest layer of our code,-database transactions- an async process that would send messages to a queue whenever new data was written or modified. This meant essentially duplicating the metadata we stored for each user, but trading storage for computing is usually a good trade-off to make, both in cost and performance. So now we had a firehose queue of every change that went on in the system, and we could build a separate piece of infrastructure who's focus would only be to provide per-user metadata *fast* for any type of file so we could build interesting and flexible user interfaces for people to consume back their own content. The stated internal goals were: 1) Fast responses (under 1 second), 2) Less than 10 seconds between user action and UI update and 3) Complete isolation from existing infrastructure.
Here's a rough diagram of how the information flowed throw the system:

U1 Diagram

It's a little bit scary when looking at it like that, but in essence it was pretty simple: write each relevant change that happened in the system to a temporary table in PG in the same transaction that it's written to the permanent table. That way you get transactional guarantees that you won't loose any data on that layer for free and use PG's built in cache that keeps recently added records cheaply accessible.
Then we built a bunch of workers that looked through those rows, parsed them, sent them to a persistent queue in RabbitMQ and once it got confirmation it was queued it would delete it from the temporary PG table.
Following that we took advantage of Rabbit's queue exchange features to build different types of workers that processes the data differently depending on what it was (music was stored differently than photos, for example).
Once we completed all of this, accessing someone's photos was a quick and predictable read operation that would give us all their data back in an easy-to-parse format that would fit in memory. Eventually we moved all the metadata accessed from the website and REST APIs to these new pre-computed views and the result was a significant reduction in load on the main DB servers, while now getting predictable sub-second request times for all types of metadata in a horizontally scalable system (just add more workers and cassandra nodes).

All in all, it took about 6 months end-to-end, which included a prototype phase that used memcache as a key/value store.

You can see the code that wrote and read from the temporary PG table if you branch the code and look under: src/backends/txlog/
The worker code, as well as the web ui is still not available but will be in the future once we finish cleaning it up to make it available. I decided to write this up and publish it now because I believe the value is more in the architecture rather than the code itself   :)

Read more
Colin Ian King

Identifying Suspend/Resume delays

The Intel SuspendResume project aims to help identify delays in suspend and resume.  After seeing it demonstrated by Len Brown (Intel) at this years Linux Plumbers conference I gave it a quick spin and was delighted to see how easy it is to use.

The project has some excellent "getting started" documentation describing how to configure a system and run the suspend resume analysis script which should be read before diving in too deep.

For the impatient, one can do try it out using the following:

git clone https://github.com/01org/suspendresume.git
cd suspendresume
sudo ./analyze_suspend.py


..and manually resume once after the machine has completed a successful suspend.

This will create a directory containing dumps of the kernel log and ftrace output as well as an html web page that one can read into your favourite web browser to view the results.  One can zoom in/out of the web page to drill down and see where the delays are occurring, an example from the SuspendResume project page is shown below:

example webpage (from https://01.org/suspendresume)

It is a useful project, kudos to Intel for producing it.  I thoroughly recommend using it to identify the delays in suspend/resume.

Read more
April Wang

因为黑客松活动,在炎炎夏日的8月里第一次飞到了深圳,这个潮湿闷热的城市让我“大跌眼镜”(不停冒汗,眼镜真的一直往下掉)。 活动现场是在位于福田区华强北商圈中的华强创客中心,不论是平日里还是周末,路边楼下仿佛永远都是人流涌动热闹非凡。华强北创客中心是由华强集团倾力打造,中国第一个为创业者提供一站式服务的综合型创新创业生态平台。一期建筑面积有5000㎡,位于华强广场B座7楼的空中花园,堪称是华强北闹市中的一片室外桃园。整体设计布满了类似街头艺术的graffiti式画作,置身其中就能感受到它灵感激发的能量,黑客松选这里自然是理所当然。

 

Canonical一直坚信激励创新的最佳方式就是将他们需要的技术给到创新者的手中,这次深圳黑客松除了Ubuntu手机操作系统平台之外,我们还带了Ubuntu Snappy Core, 一款安全易用的智能硬件操作系统技术。针对这个最新技术, 我们在活动TechTalk环节详细讲解了如何通过KVM来做开发的上手介绍。错过的同学可以在这里下载文档参看视频。而参加活动的同学们通过将Ubuntu手机平台和Snappy技术相结合将会获得特别IoT奖项。所以这场活动的亮点和看点更加有趣。 

22日的上午10点半,倒计时开始,黑客松正式进入hacking时段。不吃不喝不停不休的30个小时之后。

呵呵, 开玩笑了,一定是有吃有喝有玩有乐了,而且还有夜宵火锅,台式足球。

既然是场hackathon,重头戏当然还是这场hacking party产出的作品了。下面我就挑几组现场做了作品和大家分享。

QML Git-OSC是由开源中国团队开发的一款基于QML的Ubuntu手机应用,有了它程序猿攻城狮们可以直接通过Ubuntu手机端访问查看保存在自己在Git@OSC上的Repo详情和代码了。作为一款为写代码人群定制的应用,这组团队成功获得了最佳上手奖- 樱桃机械键盘。

iFace是第二组出场的demo作品,展示队员Mago用含蓄幽默的方式介绍了一款,在“这个看脸的时代”,用你的颜值当工具的应用。通过脸部验证登录,上网约聊。这么懂时代,能聊的队伍很不意外的拿走了由优麒麟赞助的能说会道奖-一款便携音箱。

这场黑客松中最吸引眼球的团队E Minor(E小调),在30个小时的黑客松内产出了两款作品:LibreOffice Impress Remote 和让我从第一天就在期待的Project MrRobot。Impress Remote作品正如其名,可以让你的Ubuntu手机即刻变成你Impress文档的remote,简单但超级实用。Project Mr Robot是一款由Ubuntu手机操控超萌Rapiro机器人的应用,通过这款应用你可以通过语音,按键和摇手机来操控它。两款作品的代码也完全开源,感兴趣的同学们可以在这里这里分别找到他们的代码。这支团队,轻松拿下我们的最佳颜值奖(Ubuntu双肩背包)和最佳极客奖(由华硕公司特别赞助的移动便携投影仪);这里也特别感谢为这组颁奖,也是我们现场评审之一的美女评审秦夏鸿女士(灵游科技的副总裁)。此外Project Robot在活动结束第二天就已经被Softpedia点名报道了。

IoT Ranger是一款专为电脑牵挂强迫症人群定制的应用。它为Ubuntu手机用户提供了一个随时监测家里电脑运行状态的应用。这款基于cordava的Ubuntu手机应用,巧妙使用运行在kvm环境下的网络服务框架,成功的将Ubuntu Snappy Core技术和Ubuntu手机应用开发相结合。绝对是这次黑客松中当之不愧的IoT特别奖项作品,自然拿下了我们精心准备的Beaglebone Black。

活动现场展示的作品还有几组我就不在这里一一介绍了, 感兴趣的同学们可以后续在Ubuntu开发者网站(cn.developer.ubuntu.com)的黑客松页面找到每组作品介绍。在短短的30个小时内,我们见证了如此之多的精彩,不禁就已经开始期待下一次了。 希望在后面的日子里,每个团队都能实现作品的成功部署,在外面的世界里成功立足。 代码写的辛苦,但是能和兴趣相投的人一起通宵畅聊应该是最过瘾的事了。在此献上活动现场制作的文化衫照片和大家分享。

最后要再次感谢这次活动的特约赞助商华硕,除了特别极客奖之外,还为大家提供了丰盛的签到奖和现场Demo奖;感谢线上线下的协办单位和论坛平台让这场黑客松成为可能(Git@OSC, SegmentFault,开源中国开源社Linux伊甸园Linux中国Linuxtoy.orgQTCN开发网, Meego南极圈深圳开放创新实验室SegmentFault腾讯开放平台优麒麟中芬设计园),感谢现场评审团队,还有让这场活动分外精彩的场地赞助华强北创客中心,为大家提供分外精彩的场地赞助,轻松愉快的氛围激发大家无限的创作灵感。

 

Read more
facundo

Hostería sede del PyCamp


Entre las fotos que saqué del PyCamp de hace un par de semanas está esta, que me gustó tanto que la pongo acá aparte, un poco más grande...

Hostería sede del PyCamp 2015 en La Serranita

Es la hostería donde fue sede el evento (donde dormíamos y trabajábamos... las comidas fueron en otro lugar). Una construcción en múltiples niveles muy muy linda.

Más fotos del PyCamp acá.

Read more
Joseph Salisbury

Meeting Minutes

IRC Log of the meeting.

Meeting minutes.

Agenda

20150825 Meeting Agenda


Release Metrics and Incoming Bugs

Release metrics and incoming bug data can be reviewed at the following link:

  • http://kernel.ubuntu.com/reports/kt-meeting.txt


Status: CVE’s

The current CVE status can be reviewed at the following link:

  • http://kernel.ubuntu.com/reports/kernel-cves.html


Status: Wily Development Kernel

We have rebased our Wily master-next branch to the latest upstream
v4.2-rc8 and uploaded to our ~canonical-kernel-team PPA. The fglrx DKMS
package is still failing to build with this latest kernel. We are
actively investigating to get this resolved.
—–
Important upcoming dates:

  • https://wiki.ubuntu.com/WilyWerewolf/ReleaseSchedule
    Thurs Aug 27 – Beta 1 (~2 days away)
    Thurs Sep 24 – Final Beta (~4 weeks away)
    Thurs Oct 8 – Kernel Freeze (~6 weeks away)
    Thurs Oct 15 – Final Freeze (~7 weeks away)
    Thurs Oct 22 – 15.10 Release (~8 weeks away)


Status: Stable, Security, and Bugfix Kernel Updates – Precise/Trusty/Utopic/Vivid

Status for the main kernels, until today:

  • Precise – Verification & Testing
  • Trusty – Verification & Testing
  • lts-Utopic – Verification & Testing
  • Vivid – Verification & Testing

    Current opened tracking bugs details:

  • http://kernel.ubuntu.com/sru/kernel-sru-workflow.html
    For SRUs, SRU report is a good source of information:
  • http://kernel.ubuntu.com/sru/sru-report.html

    Schedule:

    cycle: 16-Aug through 05-Sep
    ====================================================================
    14-Aug Last day for kernel commits for this cycle
    15-Aug – 22-Aug Kernel prep week.
    23-Aug – 29-Aug Bug verification & Regression testing.
    30-Aug – 05-Sep Regression testing & Release to -updates.


Open Discussion or Questions? Raise your hand to be recognized

No open discussion.

Read more
Tristram Oaten

Publishing Vanilla

We’ve got a new CSS framework at Canonical, named Vanilla. My colleague Ant has a great write-up introducing Vanilla. Essentially it’s a CSS microframework powered by Sass. The build process consists of two steps, an open source build, and a private build.

Open Source Build

While there are inevitably componants that need to be kept private (keys, tokens, etc.) being Canonical, we want to keep much of the build in the open, in addition to the code. We wanted the build to be as automated and close to CI/CD principles as possible. Here’s what happens:

Committing to our github repository kicks off a travis build that runs gulp tests, which include sass-lint. And we also use david-dm.org to make sure our npm dependencies are up to date. All of these have nice badges we can link to right from our github page, so the first thing people see is the heath of our project. I really like this, it keeps us honest, and informs the community.

Not everything can be done with travis, however, as publishing Vanilla to npm, updating our project page and demo site require some private credentials. For the confidential build, we use Jenkins. (formally Hudson, a java-based build management system.).

Private Build with Jenkins

Our Jenkins build does a few things:

  1. Increment the package.json version number
  2. npm publish (package)
  3. Build Sass with npm install
  4. Upload css to our assets server
  5. Update Sassdoc
  6. Update demo site with new CSS

Robin put this functionality together in a neat bash script: publish.sh.

We use this script in a Jenkins build that we kick off with a few parameters, point, minor and major to indicate the version to be updated in package.json. This allows our devs push-button releases on the fly, with the same build, from bugfixes all the way up to stable releases (1.0.0)

After less than 30 seconds, our demo site, which showcases framework elements and their usage, is updated. This demo is styled with the latest version of Vanilla, and also serves as documentation and a test of the CSS. We take advantage of github’s html publishing feature, Github Pages. Anyone can grab – or even hotlink – the files on our release page.

The Future

It’d be nice for the regression test (which we currently just eyeball) to be automated, perhaps with a visual diff tool such as PhantomCSS or a bespoke solution with Selenium.

Wrap-up

Vanilla is ready to hack on, go get it here and tell us what you think! (And yes, you can get it in colours other than Ubuntu Orange)

Read more
facundo


Como casi siempre a Córdoba, fuí y volví en micro (porque en el colectivo en general duermo pasablemente bien, entonces aprovecho la noche, y no pierdo medio día "para viajar"). Esta vez, por otro lado, fuí un día antes, porque tenía que hacer un trámite en Córdoba Capital, así que estuve el jueves hospedado en la casa de Nati y Matías, trabajando durante el día, jugando juegos de mesa durante la noche.

El viernes a la mañana hicimos el viaje hasta La Serranita con los chicos. Llegamos a media mañana, y ahí se dió la situación de siempre, que es muchas veces esperada: saludar y abrazar a viejos amigos que uno no puede ver tan seguido (y a cuatro o cinco nuevos que uno todavía no conoce :) ).

El grupo recién reunido, charlando sobre las propuestas

Como quedaron planeadas las actividades


El lugar estuvo muy bueno. Quizás me podría quejar que el salón principal era demasiado ajustado, y que las comidas eran en una hostería a cuatro cuadras de distancia, pero el resto estuvo más que bien. No sólo las instalaciones (habitaciones, parque, quincho, etc, etc), sino la atención humana. Un lujo.

Hasta buena internet tuvimos en este PyCamp, ya que estábamos en la red vecinal abierta que Nico Echaniz y amigos montaron en La Quintana y ciudades aledañas. Eso sí, notamos que cuando teníamos problemas con lo que era comunicaciones el tema estaba en el router que estábamos usando (y eso que terminamos poniendo uno muy bueno). Decidimos que había que invertir en hardware un poco más pro (no algo "de uso hogareño bueno" sino algo "profesional")... veremos cuanto cuesta, pero creo que vamos a gastar unos mangos ahí, ya que nos queda no sólo para el PyCamp sino para otros eventos.

Una terraza dos niveles más arriba que la sala de laburo

Pequeño parque en uno de los niveles

A nivel proyectos y Python: lo de siempre... se hacen mil cosas, desde laburar en proyectos estables hasta delirar con cosas nuevas, se mejoran cosas arrancadas de antes, se agregan funcionalidades, se empiezan proyectos que se terminan en esos cuatro días, se arrancan cosas que luego duran mucho tiempo, etc... pero lo más importante no pasa por ahí.

El núcleo del evento es humano. Charlar con gente que conocés de siempre, y podés delirar ideas, proyectos nuevos, o simplemente charlar. Y conocer gente nueva. Pibes que están haciendo cosas locas o no, con laburos copados o no, con vidas interesantes o no. Pero charlar, compartir tiempo, ver como las otras personas encaran un proyecto, qué aportan, como ayudarlos, como transmitirles experiencias.

El programar Python es casi una excusa para que todo lo otro suceda. Y digo "casi" porque sí, claro, lo que se programa y hace está buenísimo también :D

En el comedor, almorzando

En la sala principal de laburo (no era grande, pero no era la única)

En ese aspecto, yo estuve principalmente con dos proyectos. Por un lado filesync server, recientemente liberado open source, con un cambio muy grande que empecé el jueves mismo estando en la casa de Nati y continué intermitentemente durante los cuatro días de PyCamp.

El otro proyecto en el que invertí mucho tiempo es fades que desarrollo principalmente con Nico. Es que se enganchó mucha gente que le gustaba lo que fades ofrece, y aportaron un montón de ideas buenísimas. ¡Y no sólo ideas! También código, branches que mergeamos o que todavía tenemos que revisar. Ya iremos metiendo todo, y queremos hacer un release en las próximas semanas. Estén atentos, porque fades ahora hace cosas que te vuela la peluca :D

Pero no sólo trabajé en eso. También porté Tritcask a que trabaje simultaneamente con Python 2 y Python 3 (arranqué sólo con esto, pero el 70% del laburo lo hicimos juntos con SAn). Y estuvimos buscando cómo hacer para detectar cuanto de un subtítulo matchea con las voces de un video, de manera de poder determinar si está bien sincronizado o no. Y estuve haciendo algo de código asincrónico usando asyncio. Y estuve charlando con SAn, DiegoM, Bruno y Nico Echaniz sobre una especie de Repositorio Federado de Contenido Educativo. Y estuve ayudando a gente a trabajar en Python mismo durante un cortito Python Bug Day (Jairo solucionó un issue y medio!!).

Camino al río

Recorriendo la vera del río, saltando de piedra en piedra

El mejor asado de un PyCamp, ever

Y tomé sol. Y tuve en mis manos una espada de verdad por primera vez. Y caminé por el costado del río saltando de piedra en piedra. Y comí un asadazo (entre el centenar de kilos de comida que ingeríamos por día por persona). Y conocí La Serranita. Y charlé mil. Y usé un sistema de realidad virtual. Y jugué a muchos juegos de mesa.

Y abracé amigos.

Read more
Daniel Holbach

In the flurry of uploads for the C++ ABI transition and other frantic work (Thursday is Feature Freeze day) this gem maybe went unnoticed:

snapcraft (0.1) wily; urgency=low

  * Initial release

What this means? If you’re on wily, you can easily try out snapcraft and get started turning software into snaps. We have some initial docs available on the developer site which should help you find your way around.

This is a 0.1 release, so there are bugs and there might be bigger changes coming your way, but there will also be more docs, more plugins and more good stuff in general. If you’re curious, you might want to sign up for the daily build (just add the ppa:snappy-dev/snapcraft-daily PPA).

Here’s a brilliant example of what snapcraft can do for you: packaging a Java app was never this easy.

If you’re more into client apps, check out Ted’s article on how to create a QML snap.

As you can easily see: the future is on its way and upstreams and app developer will have a much easier time sharing their software.

As I said above: snapcraft is still a 0.1 release. If you want to let us know your feedback and find bugs or propose merges, you can find snapcraft in Launchpad.

Read more
Robin Winslow

pre {font-size: 1em; margin-bottom: 0.75em; padding: 0.75em} code {padding-left: 0.5em; padding-right: 0.5em} pre code {padding: 0; display: block;}

I recently tried to setup OpenID for one of our sites to support authentication with login.ubuntu.com, and it took me much longer than I’d anticipated because our site is behind a reverse-proxy.

My problem

I was trying to setup OpenID with the django-openid-auth plugin. Normally our sites don’t include absolute links (https://example.com/hello-world) back to themselves, because relative URLs (/hello-world) work perfectly well, so normally Django doesn’t need to know the domain name that it’s hosted it.

However, when authenticating with OpenID, our website needs to send the user off to login.ubuntu.com with a callback url so that once they’re successfully authenticed they can be directed back to our site. This means that the django-openid-auth needs to ask Django for an absolute URL to send off to the authenticator (e.g. https://example.com/openid/complete).

The problem with proxies

In our setup, the Django app is served with a light Gunicorn server behind an Apache front-end which handles HTTPS negotiation:

User <-> Apache <-> Gunicorn (Django)

(There’s actually an additional HAProxy load-balancer in between, which I thought was complicating matters, but it turns out HAProxy was just passing through requests absolutely untouched and so was irrelevant to the problem.)

Apache was setup as a reverse-proxy to Django, meaning that the user only ever talks to Apache, and Apache goes off to get the response from Django itself, with Django’s local network IP address – e.g. 10.0.0.3.

It turns out this is the problem. Because Apache, and not the user directly, is making the request to Django, Django sees the request come in at http://10.0.0.3/openid/login rather than https://example.com/openid/login. This meant that django-openid-auth was generating and sending the wrong callback URL of http://10.0.0.3/openid/complete to login.ubuntu.com.

How Django generates absolute URLs

django-openid-auth uses HttpRequest.build_absolute_uri which in turn uses HttpRequest.get_host to retrieve the domain. get_host then normally uses the HTTP_HOST header to generate the URL, or if it doesn’t exist, it uses the request URL (e.g.: http://10.0.0.3/openid/login).

However, after inspecting the code for get_host I discovered that if and only if settings.USE_X_FORWARDED_HOST is True then Django will look for the X-Forwarded-Host header first to generate this URL. This is the key to the solution.

Solving the problem – Apache

In our Apache config, we were initially using mod_rewrite to forward requests to Django.

RewriteEngine On
RewriteRule ^/?(.*)$ http://10.0.0.3/$1 [P,L]

However, when proxying with this method Apache2 doesn’t send the X_Forwarded_Host header that we need. So we changed it to use mod_proxy:

ProxyPass / http://10.0.0.3/
ProxyPassReverse / http://10.0.0.3/

This then means that Apache will send three headers to Django: X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Server, which will contain the information for the original request.

In our case the Apache frontend used HTTPS protocol, whereas Django was only using so we had to pass that through as well by manually setting Apache to pass an X-Forwarded-Proto to Django. Our eventual config changes looked like this:

<VirtualHost *:443>
    ...
    RequestHeader set X-Forwarded-Proto 'https' env=HTTPS

    ProxyPass / http://10.0.0.3/
    ProxyPassReverse / http://10.0.0.3/
    ...
</VirtualHost>

This meant that Apache now passes through all the information Django needs to properly build absolute URLs, we just need to make Django parse them properly.

Solving the problem – Django

By default, Django ignores all X-Forwarded headers. As mentioned earlier, you can set get_host to read the X-Forwarded-Host header by setting USE_X_FORWARDED_HOST = True, but we also needed one more setting to get HTTPS to work. These are the settings we added to our Django settings.py:

# Setup support for proxy headers
USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

After changing all these settings, we now have Apache passing all the relevant information (X-Forwarded-Host, X-Forwarded-Proto) so that Django is now able to successfully generate absolute URLs, and django-openid-auth now works a charm.

Read more
Prakash

Taiwanese firm Foxconn’s decision to invest a whopping USD five billion in India has caused unease in China as it marks the first top international firm opting for India amid a slowdown in the Chinese economy.

“Foxconn chooses India over China for new plant,” read the headline in state-run china.org.cn while carrying the news of the Taiwanese electronic giant signing up to set up a big plant in Maharashtra.

“Foxconn’s latest India investment represents the leading electronic product maker’s intention to profit from the world’s fastest expanding market of smartphones. Foxconn, famous for making parts for Apple, will reportedly produce Xiaomi phones in the new factory, a rumour that Foxconn authorities did not clarify or comment,” it said.

Read More: http://www.financialexpress.com/article/industry/companies/foxconn-shift-to-india-causes-concerns-in-china/116976/

Read more
Michi Henning

A Fast Thumbnailer for Ubuntu

Over the past few months, James Henstridge, Xavi Garcia Mena, and I have implemented a fast and scalable thumbnailing service for Ubuntu and Ubuntu Touch. This post explains how we did it, and how we achieved our performance and reliability goals.

Introduction

On a phone as well as the desktop, applications need to display image thumbnails for various media, such as photos, songs, and videos. Creating thumbnails for such media is CPU-intensive and can be costly in bandwidth if images are retrieved over the network. In addition, different types of media require the use of different APIs that are non-trivial to learn. It makes sense to provide thumbnail creation as a platform API that hides this complexity from application developers and, to improve performance, to cache thumbnails on disk.

This article explains the requirements we had and how we implemented a thumbnailer service that is extremely fast and scalable, and robust in the face of power loss or crashes.

Requirements

We had a number of requirements we wanted to meet in our implementation.

  • Robustness
    In the event of a crash, the implementation must guarantee the integrity of on-disk data structures. This is particularly important on a phone, where we cannot expect the user to perform manual recovery (such as cleaning up damaged files). Because batteries can run out at any time, integrity must be guaranteed even in the face of power loss.
  • Scalability
    It is common for people to store many thousands of songs and photos on a device, so the cache must scale to at least tens of thousands of records. Thumbnails can range in size from a few kilobytes to well over a megabyte (for “thumbnails” at full-screen resolution), so the cache must deal efficiently with large records.
  • Re-usability
    Persistent and reliable on-disk storage of arbitrary records (ranging in size from a few bytes to potentially megabytes) is a common application requirement, so we did not want to create a cache implementation that is specific to thumbnails. Instead, the disk cache is provided as a stand-alone C++ API that can be used for any number of other purposes, such as a browser or HTTP cache, or to build an object file cache similar to ccache.
  • High performance
    The performance of the thumbnailer directly affects the user experience: it is not nice for the customer to look at “please wait a while” icons in, say, an image gallery while thumbnails are being loaded one by one. We therefore had to have a high-performance implementation that delivers cached thumbnails quickly (on the order of a millisecond per thumbnail on an Arm CPU). An efficient implementation also helps to conserve battery life.
  • Location independence and extensibility
    Canonical runs an image server at dash.ubuntu.com that provides album and artist artwork for many musicians and bands. Images from this server are used to display artwork in the music player for media that contains ID3 tags, but does not embed artwork in the media file. The thumbnailer must work with embedded images as well as remote images, and it must be possible to extend it for new types of media without unduly disturbing the existing code.
  • Low bandwidth consumption
    Mobile phones typically come with data caps, so the cache has to be frugal with network bandwidth.
  • Concurrency and isolation
    The implementation has to allow concurrent access by multiple applications, as well as concurrent access from a single implementation. Besides needing to be thread-safe, this means that a request for a thumbnail that is slow (such as downloading an image over the network) must not delay other requests.
  • Fault tolerance
    Mobile devices lose network access without warning, and users can add corrupt media files to their device. The implementation must be resilient to partial failures, such as incomplete network replies, dropped connections, and bad image data. Moreover, the recovery strategy for such failures must conserve battery and avoid repeated futile attempts to create thumbnails from media that cannot be retrieved or contains malformed data.
  • Security
    The implementation must ensure that applications cannot see (or, worse, overwrite) each other’s thumbnails or coerce the thumbnailer into delivering images from files that an application is not allowed to read.
  • Asynchronous API
    The customers of the thumbnailer are applications that are written in QML or Qt, which cannot block in the UI thread. The thumbnailer therefore must provide a non-blocking API. Moreover, the application developer should be able to get the best possible performance without having to use threads. Instead, concurrency must be internal to the implementation (which is able to put threads to use intelligently where they make sense), instead of the application throwing threads at the problem in the hope that it might make things faster when, in fact, it might just add overhead.
  • Monitoring
    The effectiveness of a cache cannot be assessed without statistics to show hit and miss rates, evictions, and other basic performance data, so it must provide a way to extract this information.
  • Error reporting
    When something goes wrong with a system service, typically the only way to learn about the problem is to look at log messages. In case of a failure, the implementation must leave enough footprints behind to allow someone to diagnose a failure after the fact with some chance of success.
  • Backward compatibility
    This project was a rewrite of an earlier implementation. Rather than delivering a “big bang” piece of software and potentially upsetting existing clients, we incrementally changed the implementation such that existing applications continued to work. (The only pre-existing interface was a QML interface that required no change.)

System architecture

Here is a high-level overview of the main system components.

A Fast Thumbnailer for UbuntuExternal API

To the outside world, the thumbnailer provides two APIs.

One API is a QML plugin that registers itself as an image provider for QQuickAsyncImageProvider. This allows the caller to to pass a URI that encodes a query for a local or remote thumbnail at a particular size; if the URI matches the registered provider, QML transfers control to the entry points in our plugin.

The second API is a Qt API that provides three methods:

QSharedPointer<Request> getThumbnail(QString const& filePath,
                                     QSize const& requestedSize);
QSharedPointer<Request> getAlbumArt(QString const& artist,
                                    QString const& album,
                                    QSize const& requestedSize);
QSharedPointer<Request> getArtistArt(QString const& artist,
                                     QString const& album,
                                     QSize const& requestedSize);

The getThumbnail() method extracts thumbnails from local media files, whereas getAlbumArt() and getArtistArt() retrieve artwork from the remote image server. The returned Request object provides a finished signal, and methods to test for success or failure of the request and to extract a thumbnail as a QImage. The request also provides a waitForFinished() method, so the API can be used synchronously.

Thumbnails are delivered to the caller in the size they are requested, subject to a (configurable) 1920-pixel limit. As an escape hatch, requests with width and height of zero deliver artwork at its original size, even if it exceeds the 1920-pixel limit. The scaling algorithm preserves the original aspect ratio and never scales up from the original, so the returned thumbnails may be smaller than their requested size.

DBus service

The thumbnailer is implemented as a DBus service with two interfaces. The first interface provides the server-side implementation of the three methods of the external API; the second interface is an administrative interface that can deliver statistics, clear the internal disk caches, and shut down the service. A simple tool, thumbnailer-admin, allows both interfaces to be called from the command line.

To conserve resources, the service is started on demand by DBus and shuts down after 30 seconds of idle time.

Image extraction

Image extraction uses an abstract base class. This interface is independent of media location and type. The actual image extraction is performed by derived implementations that download images from the remote server, extract them from local image files, or extract them from local streaming media files. This keeps knowledge of image location and encoding out of the main caching and error handling logic, and allows us to support new media types (whether local or remote) by simply adding extra derived implementations.

Image extraction is asynchronous, with currently three implementations:

  • Image downloader
    To retrieve artwork from the remote image server, the service talks to an abstract base class with asynchronous download_album() and download_artist() methods. This allows multiple downloads to run concurrently and makes it easy to add new local or remote image providers without disturbing the code for existing ones. A class derived from that abstract base implements a REST API with QNetworkAccessManager to retrieve images from dash.ubuntu.com.
  • Photo extractor
    The photo extractor is responsible for delivering images from local image files, such as JPEG or PNG files. It simply delegates that work to the image converter and scaler.
  • Audio and video thumbnail extractor
    To extract thumbnails from audio and video files, we use GStreamer. Due to reliability problems with some codecs that can hang or crash, we delegate the task to a separate vs-thumb executable. This shields the service from failures and also allows us to run several GStreamer pipelines concurrently without a crash of one pipeline affecting the others.

Image converter and scaler

We use a simple Image class with a synchronous interface to convert and scale different image formats to JPEG. The implementation uses Gdk-Pixbuf, which can handle many different input formats and is very efficient.

For JPEG source images, the code checks for the presence of EXIF data using libexif and, if it contains a thumbnail that is at least as large as the requested size, scales the thumbnail from the EXIF data. (For images taken with the camera on a Nexus 4, the original image size is 3264×1836, with an embedded EXIF thumbnail of 512×288. Scaling from the EXIF thumbnail is around one hundred times faster than scaling from the full-size image.)

Disk cache

The thumbnailer service optimizes performance and conserves bandwidth and battery by adopting a layered caching strategy.

Two-level caching with failure lookup

Internally, the service uses three separate on-disk caches:

  • Full-size cache
    This cache stores images that are expensive to retrieve (images that are remote or are embedded in audio and video files) at original resolution (scaled down to a 1920-pixel bounding box if the original image is larger). The default size of this cache is 50 MB, which is sufficient to hold around 400 images at 1920×1080 resolution. Images are stored in JPEG format (at a 90% quality setting).
  • Thumbnail cache
    This cache stores thumbnails at the size that was requested by the caller, such as 512×288. The default size of this cache is 100 MB, which is sufficient to store around 11,000 thumbnails at 512×288, or around 25,000 thumbnails at 256×144.
  • Failure cache
    The failure cache stores the keys for images that could not be extracted because of a failure. For remote images, this means that the server returned an authoritative answer “no such image exists”, or that we encountered an unexpected (non-authoritative) failure, such as the server not responding or a DNS lookup timing out. For local images, it means either that the image data could not be processed because it is damaged, or that an audio file does not contain embedded artwork.

The full-size cache exists because it is likely that an application will request thumbnails at different sizes for the same image. For example, when scrolling through a list of songs that shows a small thumbnail of the album cover beside each song, the user is likely to select one of the songs to play, at which point the media player will display the same cover in a larger size. By keeping full-size images in a separate (smallish) cache, we avoid performing an expensive extraction or download a second time. Instead, we create additional thumbnails by scaling them from the full-size cache (which uses an LRU eviction policy).

The thumbnail cache stores thumbnails that were previously retrieved, also using LRU eviction. Thumbnails are stored as JPEG at the default quality setting of 75%, at the actual size that was requested by the caller. Storing JPEG images (rather than, say, PNG) saves space and increases cache effectiveness. (The minimal quality loss from compression is irrelevant for thumbnails). Because we store thumbnails at the size they are actually needed, we may have several thumbnails for the same image in the cache (each thumbnail at a different size). But applications typically ask for thumbnails in only a small number of sizes, and ask for different sizes for the same image only rarely. So, the slight increase in disk space is minor and amply repaid by applications not having to scale thumbnails after they receive them from the cache, which saves battery and achieves better performance overall.

Finally, the failure cache is used to stop futile attempts to repeatedly extract a thumbnail when we know that the attempt will fail. It uses LRU eviction with an expiry time for each entry.

Cache lookup algorithm

When asked for a thumbnail at a particular size, the lookup and thumbnail generation proceed as follows:

  1. Check if a thumbnail exists in the requested size in the thumbnail cache. If so, return it.
  2. Check if a full-size image for the thumbnail exists in the full-size cache. If so, scale the new thumbnail from the full-size image, add the thumbnail to the thumbnail cache, and return it.
  3. Check if there is an entry for the thumbnail in the failure cache. If so, return an error.
  4. Attempt to download or extract the original image for the thumbnail. If the attempt fails, add an entry to the failure cache and return an error.
  5. If the original image was delivered by the remote server or was extracted locally from streaming media, add it to the full-size cache.
  6. Scale the thumbnail to the desired size, add it to the thumbnail cache, and return it.

Note that these steps represent only the logical flow of control for a particular thumbnail. The implementation executes these steps concurrently for different thumbnails.

Designing for performance

Apart from fast on-disk caches (see below), the thumbnailer must make efficient use of I/O bandwidth and threads. This not only means making things fast, but also to not unnecessarily waste resources such as threads, memory, network connections, or file descriptors. Provided that enough requests are made to keep the service busy, we do not want it to ever wait for a download or image extraction to complete while there is something else that could be done in the mean time, and we want it to keep all CPU cores busy. In addition, requests that are slow (because they require a download or a CPU-intensive image extraction) must not block requests that are queued up behind them if those requests would result in cache hits that could be returned immediately.

To achieve a high degree of concurrency without blocking on long-running operations while holding precious resources, the thumbnailer uses a three-phase lookup algorithm:

  1. In phase 1, we look at the caches to determine if we have a hit or an authoritative miss. Phase 1 is very fast. (It takes around a millisecond to return a thumbnail from the cache on a Nexus 4.) However, cache lookup can briefly stall on disk I/O or require a lot of CPU to extract and scale an image. To get good performance, phase 1 requests are passed to a thread pool with as many threads as there are CPU cores. This allows the maximum number of lookups to proceed concurrently.
  2. Phase 2 is initiated if phase 1 determines that a thumbnail requires download or extraction, either of which can take on the order of seconds. (In case of extraction from local media, the task is CPU intensive; in case of download, most of the time is spent waiting for the reply from the server.) This phase is scheduled asynchronously from an event loop. This minimizes task switching and allows large numbers of requests to be queued while only using a few bytes for each request that is waiting in the queue.
  3. Phase 3 is really a repeat of phase 1: if phase 2 produces a thumbnail, it adds it to the cache; if phase 2 does not produce a thumbnail, it creates an entry in the failure cache. By simply repeating phase 1, the lookup then results in either a thumbnail or an error.

If phase 2 determines that a download or extraction is required, that work is performed concurrently: the service schedules several downloads and extractions in parallel. By default, it will run up to two concurrent downloads, and as many concurrent GStreamer pipelines as there are CPUs. This ensures that we use all of the available CPU cores. Moreover, download and extraction run concurrently with lookups for phase 1 and 3. This means that, even if a cache lookup briefly stalls on I/O, there is a good chance that another thread can make use of the CPU.

Because slow operations do not block lookup, this also ensures that a slow request does not stall requests for thumbnails that are already in the cache. In other words, it does not matter how many slow requests are in progress: requests that can be completed quickly are indeed completed quickly, regardless of what is going on elsewhere.

Overall, this strategy works very well. For example, with sufficient workload, the service achieves around 750% CPU utilization on an 8-core desktop machine, while still delivering cache hits almost instantaneously. (On a Nexus 4, cache hits take a little over 1 ms while concurrent extractions or downloads are in progress.)

A re-usable persistent cache for C++

The three internal caches are implemented by a small and flexible C++ API. This API is available as a separate reusable PersistentStringCache component (see persistent-cache-cpp) that provides a persistent store of arbitrary key–value pairs. Keys and values can be binary, and entries can be large. (Megabyte-sized values do not present a problem.)

The implementation uses leveldb, which provides a very fast NoSQL database that scales to multi-gigabyte sizes and provides integrity guarantees. In particular, if the calling process crashes, all inserts that completed at the API level will be intact after a restart. (In case of a power failure or kernel crash, a few buffered inserts can be lost, but the integrity of the database is still guaranteed.)

To use a cache, the caller instantiates it with a path name, a maximum size, and an eviction policy. The eviction policy can be set to either strict LRU (least-recently-used) or LRU with an expiry time. Once a cache reaches its maximum size, expired entries (if any) are evicted first and, if that does not free enough space for a new entry, entries are discarded in least-recently-used order until enough room is available to insert a new record. (In all other respects, expired entries behave like entries that were never added.)

A simple get/put API allows records to be retrieved and added, for example:

auto c = core::PersistentStringCache::open(
    “my_cache”, 100 * 1024 * 1024, core::CacheDiscardPolicy::lru_only);
// Look for an entry and add it if there is a cache miss.
string key = "Bjarne";
auto value = c->get(key);
if (value) {
    cout << key << ″: ″ << *value << endl;
} else {
    value = "C++ inventor";  // Provide a value for the key. 
    c->put(key, *value);     // Insert it.
}

Running this program prints nothing on the first run, and “Bjarne: C++ inventor” on all subsequent runs.

The API also allows application-specific metadata to be added to records, provides detailed statistics, supports dynamic resizing of caches, and offers a simple adapter template that makes it easy to store complex user-defined types without the need to clutter the code with explicit serialization and deserialization calls. (In a pinch, if iteration is not needed, the cache can be used as a persistent map by setting an impossibly large cache size, in which case no records are ever evicted.)

Performance

Our benchmarks indicate good performance. (Figures are for an Intel Ivy Bridge i7-3770k 3.5 GHz machine with a 256 GB SSD.) Our test uses 60-byte string keys. Values are binary blobs filled with random data (so they are not compressible), 20 kB in size with a standard deviation of 7,000, so the majority of values are 13–27 kB in size. The cache size is 100 MB, so it contains around 5,000 records.

Filling the cache with 100 MB of records takes around 2.8 seconds. Thereafter, the benchmark does a random lookup with an 80% hit probability. In case of a cache miss, it inserts a new random record, evicting old records in LRU order to make room for the new one. For 100,000 iterations, the cache returns around 4,800 “thumbnails” per second, with an aggregate read/write throughput of around 93 MB/sec. At 90% hit rate, we see twice the performance at around 7,100 records/sec. (Writes are expensive once the cache is full due to the need to evict entries, which requires updating the main cache table as well as an index.)

Repeating the test with a 1 GB cache produces identical timings so (within limits) performance remains constant for large databases.

Overall, performance is restricted largely by the bandwidth to disk. With a 7,200 rpm disk, we measured around one third of the performance with an SSD.

Recovering from errors

The overall design of the thumbnailer delivers good performance when things work. However, our implementation has to deal with the unexpected, such as network requests that do not return responses, GStreamer pipelines that crash, request overload, and so on. What follows is a partial list of steps we took to ensure that things behave sensibly, particularly on a battery-powered device.

Retry strategy

The failure cache provides an effective way to stop the service from endlessly trying to create thumbnails that, in an earlier attempt, returned an error.

For remote images, we know that, if the server has (authoritatively) told us that it has no artwork for a particular artist or album, it is unlikely that artwork will appear any time soon. However, the server may be updated with more artwork periodically. To deal with this, we add an expiry time of one week to the entries in the failure cache. That way, we do not try to retrieve the same image again until at least one week has passed (and only if we receive a request for a thumbnail for that image again later).

As opposed to authoritative answers from the image server (“I do not have artwork for this artist.”), we can also encounter transient failures. For example, the server may currently be down, or there may be some other network-related issue. In this case, we remember the time of the failure and do not try to contact the remote server again for two hours. This conserves bandwidth and battery power.

The device may also disconnected from the network, in which case any attempt to retrieve a remote image is doomed. Our implementation returns failure immediately on a cache miss for a remote image if no network is present or the device is in flight mode. (We do not add an entry to the failure cache in this case).

For local files, we know that, if an attempt to get a thumbnail for a particular file has failed, future attempts will fail as well. This means that the only way for the problem to get fixed is by modifying or replacing the actual media file. To deal with this, we add the inode number, modification time, and inode modification time to the key for local images. If a user replaces, say, a music file with a new one that contains artwork, we automatically pick up the new version of the file because its key has changed; the old version will eventually fall out of the cache.

Download and extraction failures

We monitor downloads and extractions for timely completion. (Timeouts for downloads and extractions can be configured separately.) If the server does not respond within 10 seconds, we abandon the attempt and treat it it as a transient network error. Similarly, the vs-thumb processes that extract images from audio and video files can hang. We monitor these processes and kill them if they do not produce a result within 10 seconds.

Database corruption

Assuming an error-free implementation of leveldb, database corruption is impossible. However, in practice, an errant command could scribble over the database files. If leveldb detects that the database is corrupted, the recovery strategy is simple: we delete the on-disk cache and start again from scratch. Because the cache contents are ephemeral anyway, this is fine (other than slower operation until the working set of thumbnails makes it into the cache again).

Dealing with backlog

The asynchronous API provided by the service allows an application to submit an unlimited number of requests. Lots of requests happen if, for example, the user has inserted a flash card with thousands of photos into the device and then requests a gallery view for the collection. If the service’s client-side API blindly forwards requests via DBus, this causes a problem because DBus terminates the connection once there are more than around 400 outstanding requests.

To deal with this, we limit the number of outstanding requests to 200 and send another request via DBus only when an earlier request completes. Additional requests are queued in memory. Because this happens on the client side, the number of outstanding requests is limited only by the amount of memory that is available to the client.

A related problem arises if a client submits many requests for a thumbnail for the same image. This happens when, for example, the user looks at a list of tracks: tracks that belong to the same album have the same artwork. If artwork needs to be retrieved from the remote server, naively forwarding cache misses for each thumbnail to the server would end up re-downloading the same image several times.

We deal with this by maintaining an in-memory map of all remote download requests that are currently in progress. If phase 1 reports a cache miss, before initiating a download, we add the key for the remote image to the map and remove it again once the download completes. If more requests for the same image encounter a cache miss while the download for the original request is still in progress, the key for the in-progress download is still in the map, and we hold additional requests for the same image until the download completes. We then schedule the held requests as usual and create their thumbnails from the image that was cached by the first request.

Security

The thumbnailer runs with normal user privileges. We use AppArmor’s aa_query_label() function to verify that the calling client has read access to a file it wants a thumbnail for. This prevents one application from accessing thumbnails produced by a different application, unless both applications can read the original file. In addition, we place the entire service under an AppArmor profile to ensure that it can write only to its own cache directory.

Conclusion

Overall, we are very pleased with the overall design and performance of the thumbnailer. Each component has a clearly defined role with a clean interface, which made it easy for us to experiment and to refine the design as we went along. The design is extensible, so we can support additional media types or remote data sources without disturbing the existing code.

We used threads sparingly and only where we saw worthwhile concurrency opportunities. Using asynchronous interfaces for long-running operations kept resource usage to a minimum and allowed us to take advantage of I/O interleaving. In turn, this extracts the best possible performance from the hardware.

The thumbnailer now runs on Ubuntu Touch and is used by the gallery, camera, and music apps, as well as for all scopes that display media thumbnails.

This article has been originally published on Michi Henning's blog.

Read more