Canonical Voices

mandel

I have had a nice riddle given to me by Tim-Erwin within the comments of Bitten in the ass by ReadDirectoryChangesW and multithreading which I really appreciate. His problem was with the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import win32file
import win32con
import win32event
import pywintypes
 
FILE_LIST_DIRECTORY = 0x0001
folder = r"C:\some folder to watch"
 
handle = win32file.CreateFile (folder, FILE_LIST_DIRECTORY,
              win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
              None, win32con.OPEN_EXISTING,
              win32con.FILE_FLAG_BACKUP_SEMANTICS, None)
buffer = win32file.AllocateReadBuffer(1024)
overlapped = pywintypes.OVERLAPPED()
overlapped.hEvent = win32event.CreateEvent(None, 0, 0, None)
flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME |\
                  win32con.FILE_NOTIFY_CHANGE_DIR_NAME |\
                  win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |\
                  win32con.FILE_NOTIFY_CHANGE_SIZE |\
                  win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |\
                  win32con.FILE_NOTIFY_CHANGE_SECURITY
 
win32file.ReadDirectoryChangesW(handle, buffer, False, flags, overlapped)
print "never reached until a change is done in the folder"
win32event.WaitForSingleObject(overlapped.hEvent, 5000)

As Tim-Erwin said in his comments, the above code was not being blocked in the WaitForSingleObject call but in the ReadDirecotryChangesW one. This is clearly my fault because in the blog post I did not give a very important detail that is needed to get the code working async. In order for the above to work correctly it is imperative that you pass to the CreateFile function the FILE_FLAG_OVERLAPPED flag. That means, that in order to fix the code and be blocked in WaitForSingleObject we have to modify the CreateFile call in the following way:

9
10
11
12
handle = win32file.CreateFile (folder, FILE_LIST_DIRECTORY,
              win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
              None, win32con.OPEN_EXISTING,
              win32con.FILE_FLAG_BACKUP_SEMANTICS | win32file.FILE_FLAG_OVERLAPPED , None)

That fixes Tim-Erwin problem and should help any other person with the same issue. Sorry for not being explicit on such an important detail Tim-Erwin, please accept my apologies :) .

Read more
mandel

This is here for me to remember the next time I need to do this task:

  1. Copy the default pool definition:

    virsh pool-dumpxml default > pool.xml
  2. edit pool.xml changing the following vars:

    <pool type='dir'>
      <name>{$name}</name>
      <uuid>{$id}</uuid>
      <capacity>43544694784</capacity>
      <allocation>30412328960</allocation>
      <available>13132365824</available>
      <source>
      </source>
      <target>
        <path>{$path}</path>
        <permissions>
          <mode>0700</mode>
          <owner>-1</owner>
          <group>-1</group>
        </permissions>
      </target>
    </pool>
  3. virsh pool-create pool.xml
  4. virsh pool-refresh name

Doing the above you can add a new pool, for example one that is not in you ssd.

Read more
mandel

In the past days I have been working on implementing a python TestCase that can be used to perform integration tests to the future implementation of proxy support that will be landing in Ubuntu One. The idea of the TestCase is the following:

  • Start a proxy so that connections go throw it. The proxy has to be listening to two different ports, one in which auth is not required and a second one in which auth is required. At the moment the only supported proxy is Squid using base auth.
  • The test case should provide a way to access to the proxy details for subclasses to use.
  • The test case should integrate with the ubuntuone-dev-tools.

Initially, one of the major problems I had was to start squid in two different ports so that:

  • Port A accepts non-auth requests.
  • Port A rejects auth requests.
  • Port B accepts auth requests.

The idea is simple, if you use port A you should never auth while you must in port B, and example configuration of the ACLs and ports is the following:

auth_param basic casesensitive on
# Use a default auth using ncsa and the passed generated file.
auth_param basic program ${auth_process} ${auth_file}
#Recommended minimum configuration:
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
#
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
#
acl SSL_ports port 443		# https
acl SSL_ports port 563		# snews
acl SSL_ports port 873		# rsync
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl Safe_ports port 631		# cups
acl Safe_ports port 873		# rsync
acl Safe_ports port 901		# SWAT
acl purge method PURGE
acl CONNECT method CONNECT

# make an acl for users that have auth
acl password proxy_auth REQUIRED myportname ${auth_port_number}
acl auth_port_connected myportname ${auth_port_number}
acl nonauth_port_connected myportname ${noauth_port_number}

# Settings used for the tests:
# Allow users connected to the nonauth port
# Allow users authenticated AND connected to the auth port
http_access allow nonauth_port_connected
http_access allow password

#Recommended minimum configuration:
#
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
# Only allow purge requests from localhost
http_access allow purge localhost
http_access deny purge
# Deny requests to unknown ports
http_access deny !Safe_ports
# Deny CONNECT to other than SSL ports
http_access deny CONNECT !SSL_ports
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
#http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

#Allow ICP queries from local networks only
icp_access allow localnet
icp_access deny all

# Squid normally listens to port 3128 but we are going to listento two
# different ports, one for auth one for nonauth.
http_port ${noauth_port_number}
http_port ${auth_port_number}

#We recommend you to use at least the following line.
hierarchy_stoplist cgi-bin ?

# Default cache settings.
cache_dir ufs ${spool_temp} 100 16 256

# access log settings
access_log ${squid_temp}/access.log squid

#Default cache stroe log
cache_store_log ${squid_temp}/store.log

#Default pid file name
pid_filename ${squid_temp}/squid.pid

#Default netdb file name:
netdb_filename ${spool_temp}/logs/netdb.state

#Suggested default:
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern (Release|Packages(.gz)*)$	0	20%	2880
# example line deb packages
#refresh_pattern (\.deb|\.udeb)$   129600 100% 129600
refresh_pattern .		0	20%	4320

# Don't upgrade ShoutCast responses to HTTP
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast

# Apache mod_gzip and mod_deflate known to be broken so don't trust
# Apache to signal ETag correctly on such responses
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

extension_methods REPORT MERGE MKACTIVITY CHECKOUT

hosts_file /etc/hosts

# Leave coredumps in the first cache dir
coredump_dir ${spool_temp}

Once the above was achieved the code of the test case was quite simple for Ubuntu O, unfortunatly, it was not that issues in Ubuntu P because there we have squid3 which supports http 1.1 and keeps the proxy keeps the connection alive. The fact that the connection is kept alive means that the reactor has a selectable running because the proxy keep it there. In order to solve the issue I wrote the code so that the server could say that the connection timedout. Here is the code that does it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
# -*- coding: utf-8 -*-
#
# Copyright 2011 Canonical Ltd.
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License version 3, as published
# by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranties of
# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
# PURPOSE.  See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.
"""Test the squid test case."""
import base64
 
from twisted.application import internet, service
from twisted.internet import defer, reactor
from twisted.web import client, error, http, resource, server
 
from ubuntuone.devtools.testcases.squid import SquidTestCase
 
 
SAMPLE_RESOURCE = "<p>Hello World!</p>"
SIMPLERESOURCE = "simpleresource"
THROWERROR = "throwerror"
UNAUTHORIZED = "unauthorized"
 
# ignore common twisted lint errors
# pylint: disable=C0103, W0212
 
 
class ProxyClientFactory(client.HTTPClientFactory):
    """Factory that supports proxy."""
 
    def __init__(self, proxy_url, proxy_port, url, headers=None):
        # we set the proxy details before the init because the parent __init__
        # calls setURL
        self.proxy_url = proxy_url
        self.proxy_port = proxy_port
        self.disconnected_d = defer.Deferred()
        client.HTTPClientFactory.__init__(self, url, headers=headers)
 
    def setURL(self, url):
        self.host = self.proxy_url
        self.port = self.proxy_port
        self.url = url
        self.path = url
 
    def clientConnectionLost(self, connector, reason, reconnecting=0):
        """Connection lost."""
        self.disconnected_d.callback(self)
 
 
class ProxyWebClient(object):
    """Provide useful web methods with proxy."""
 
    def __init__(self, proxy_url=None, proxy_port=None, username=None,
            password=None):
        """Create a new instance with the proxy settings."""
        self.proxy_url = proxy_url
        self.proxy_port = proxy_port
        self.username = username
        self.password = password
        self.factory = None
        self.connectors = []
 
    def _connect(self, url, contextFactory):
        """Perform the connection."""
        scheme, _, _, _ = client._parse(url)
        # pylint: disable=E1101
        if scheme == 'https':
            from twisted.internet import ssl
            if contextFactory is None:
                contextFactory = ssl.ClientContextFactory()
            self.connectors.append(reactor.connectSSL(self.proxy_url,
                                                      self.proxy_port,
                                                      self.factory,
                                                      contextFactory))
        else:
            self.connectors.append(reactor.connectTCP(self.proxy_url,
                                                      self.proxy_port,
                                                      self.factory))
            # pylint: enable=E1101
 
    def _process_auth_error(self, failure, url, contextFactory):
        """Process an auth failure."""
        failure.trap(error.Error)
        if failure.value.status == str(http.PROXY_AUTH_REQUIRED):
            # we try to get the page using the basic auth
            auth = base64.b64encode('%s:%s' % (self.username, self.password))
            auth_header = 'Basic ' + auth.strip()
            self.factory = ProxyClientFactory(self.proxy_url, self.proxy_port,
                            url, headers={'Proxy-Authorization': auth_header})
            self._connect(url, contextFactory)
            return self.factory.deferred
        else:
            return failure
 
    def get_page(self, url, contextFactory=None, *args, **kwargs):
        """Download a webpage as a string.
 
        This method relies on the twisted.web.client.getPage but adds and extra
        step. If there is an auth error the method will perform a second try
        so that the username and password are used.
        """
        self.factory = ProxyClientFactory(self.proxy_url, self.proxy_port, url,
                                          headers={'Connection': 'close'})
        self._connect(url, contextFactory)
        self.factory.deferred.addErrback(self._process_auth_error, url,
                                    contextFactory)
        return self.factory.deferred
 
    @defer.inlineCallbacks
    def shutdown(self):
        """Clean all connectors."""
        for connector in self.connectors:
            yield connector.disconnect()
        defer.returnValue(True)
 
 
class SimpleResource(resource.Resource):
    """A simple web resource."""
 
    def render_GET(self, request):
        """Make a bit of html out of these resource's
        content."""
        return SAMPLE_RESOURCE
 
 
class SaveHTTPChannel(http.HTTPChannel):
    """A save protocol to be used in tests."""
 
    protocolInstance = None
 
    def connectionMade(self):
        """Keep track of the given protocol."""
        SaveHTTPChannel.protocolInstance = self
        http.HTTPChannel.connectionMade(self)
 
 
class SaveSite(server.Site):
    """A site that let us know when it closed."""
 
    protocol = SaveHTTPChannel
 
    def __init__(self, *args, **kwargs):
        """Create a new instance."""
        server.Site.__init__(self, *args, **kwargs)
        # we disable the timeout in the tests, we will deal with it manually.
        self.timeOut = None
 
 
class MockWebServer(object):
    """A mock webserver for testing"""
 
    def __init__(self):
        """Start up this instance."""
        root = resource.Resource()
        root.putChild(SIMPLERESOURCE, SimpleResource())
 
        root.putChild(THROWERROR, resource.NoResource())
 
        unauthorized_resource = resource.ErrorPage(resource.http.UNAUTHORIZED,
                                                "Unauthorized", "Unauthorized")
        root.putChild(UNAUTHORIZED, unauthorized_resource)
 
        self.site = SaveSite(root)
        application = service.Application('web')
        self.service_collection = service.IServiceCollection(application)
        #pylint: disable=E1101
        self.tcpserver = internet.TCPServer(0, self.site)
        self.tcpserver.setServiceParent(self.service_collection)
        self.service_collection.startService()
 
    def get_url(self):
        """Build the url for this mock server."""
        #pylint: disable=W0212
        port_num = self.tcpserver._port.getHost().port
        return "http://localhost:%d/" % port_num
 
    @defer.inlineCallbacks
    def stop(self):
        """Shut it down."""
        #pylint: disable=E1101
        # make the connection time out so that is works with squid3 when
        # the connection is kept alive.
        if self.site.protocol.protocolInstance:
            self.site.protocol.protocolInstance.timeoutConnection()
        yield self.service_collection.stopService()
 
 
class ProxyTestCase(SquidTestCase):
    """A squid test with no auth proxy."""
 
    @defer.inlineCallbacks
    def setUp(self):
        """Set the tests."""
        yield super(ProxyTestCase, self).setUp()
        self.ws = MockWebServer()
        self.proxy_client = None
        self.addCleanup(self.teardown_client_server)
        self.url = self.ws.get_url() + SIMPLERESOURCE
 
    def teardown_client_server(self):
        """Clean resources."""
        if self.proxy_client is not None:
            self.proxy_client.shutdown()
            return defer.gatherResults([self.ws.stop(),
                               self.proxy_client.shutdown(),
                               self.proxy_client.factory.disconnected_d])
        else:
            return self.ws.stop()
 
    def access_noauth_url(self, address, port):
        """Access a url throught the proxy."""
        self.proxy_client = ProxyWebClient(proxy_url=address, proxy_port=port)
        return self.proxy_client.get_page(self.url)
 
    def access_auth_url(self, address, port, username, password):
        """Access a url throught the proxy."""
        self.proxy_client = ProxyWebClient(proxy_url=address, proxy_port=port,
                                         username=username, password=password)
        return self.proxy_client.get_page(self.url)
 
    @defer.inlineCallbacks
    def test_noauth_url_access(self):
        """Test accessing to the url."""
        settings = self.get_nonauth_proxy_settings()
        # if there is an exception we fail.
        data = yield self.access_noauth_url(settings['host'],
                                            settings['port'])
        self.assertEqual(SAMPLE_RESOURCE, data)
 
    @defer.inlineCallbacks
    def test_auth_url_access(self):
        """Test accessing to the url."""
        settings = self.get_auth_proxy_settings()
        # if there is an exception we fail.
        data = yield self.access_auth_url(settings['host'],
                                          settings['port'],
                                          settings['username'],
                                          settings['password'])
        self.assertEqual(SAMPLE_RESOURCE, data)
 
    def test_auth_url_401(self):
        """Test failing accessing the url."""
        settings = self.get_auth_proxy_settings()
        # swap password for username to fail
        d = self.failUnlessFailure(self.access_auth_url(settings['host'],
                                        settings['port'], settings['password'],
                                        settings['username']), error.Error)
        return d
 
    def test_auth_url_407(self):
        """Test failing accessing the url."""
        settings = self.get_auth_proxy_settings()
        d = self.failUnlessFailure(self.access_noauth_url(settings['host'],
                                   settings['port']), error.Error)
        return d

The above code is the tests for the test case and the important bits are:

133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
class SaveHTTPChannel(http.HTTPChannel):
    """A save protocol to be used in tests."""
 
    protocolInstance = None
 
    def connectionMade(self):
        """Keep track of the given protocol."""
        SaveHTTPChannel.protocolInstance = self
        http.HTTPChannel.connectionMade(self)
 
 
class SaveSite(server.Site):
    """A site that let us know when it closed."""
 
    protocol = SaveHTTPChannel
 
    def __init__(self, *args, **kwargs):
        """Create a new instance."""
        server.Site.__init__(self, *args, **kwargs)
        # we disable the timeout in the tests, we will deal with it manually.
        self.timeOut = None

The above defines a protocol that will know the instance that it was used so that we can trigger the time out in a clean up function.

190
191
        if self.site.protocol.protocolInstance:
            self.site.protocol.protocolInstance.timeoutConnection()

This tells the server to time out.

207
208
209
210
211
212
213
214
215
    def teardown_client_server(self):
        """Clean resources."""
        if self.proxy_client is not None:
            self.proxy_client.shutdown()
            return defer.gatherResults([self.ws.stop(),
                               self.proxy_client.shutdown(),
                               self.proxy_client.factory.disconnected_d])
        else:
            return self.ws.stop()

And the clean up function. That is all, now I guess I’ll move to add proxy support or ensure that the test case works on Windows, which is certainly going to be a diff issue.

Read more
mandel

The following is some code in which I have been working (and stupidly wasting time in a small error) that allows to get a page using a methos similar to twisted.web.client.getPage through a proxy that uses base auth.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# -*- coding: utf-8 -*-
#
# Copyright 2011 Canonical Ltd.
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License version 3, as published
# by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranties of
# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
# PURPOSE.  See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program.  If not, see <http://www.gnu.org/licenses/>.
"""Test the squid test case."""
import base64
 
from twisted.internet import defer, reactor
from twisted.web import client, error, http
 
from ubuntuone.devtools.testcases.squid import SquidTestCase
 
# ignore common twisted lint errors
# pylint: disable=C0103, W0212
 
 
class ProxyClientFactory(client.HTTPClientFactory):
    """Factory that supports proxy."""
 
    def __init__(self, proxy_url, proxy_port, url, headers=None):
        self.proxy_url = proxy_url
        self.proxy_port = proxy_port
        client.HTTPClientFactory.__init__(self, url, headers=headers)
 
    def setURL(self, url):
        self.host = self.proxy_url
        self.port = self.proxy_port
        self.url = url
        self.path = url
 
 
class ProxyWebClient(object):
    """Provide useful web methods with proxy."""
 
    def __init__(self, proxy_url=None, proxy_port=None, username=None,
            password=None):
        """Create a new instance with the proxy settings."""
        self.proxy_url = proxy_url
        self.proxy_port = proxy_port
        self.username = username
        self.password = password
 
    def _process_auth_error(self, failure, url, contextFactory):
        """Process an auth failure."""
        # we try to get the page using the basic auth
        failure.trap(error.Error)
        if failure.value.status == str(http.PROXY_AUTH_REQUIRED):
            auth = base64.b64encode('%s:%s' % (self.username, self.password))
            auth_header = 'Basic ' + auth.strip()
            factory = ProxyClientFactory(self.proxy_url, self.proxy_port, url,
                    headers={'Proxy-Authorization': auth_header})
            # pylint: disable=E1101
            reactor.connectTCP(self.proxy_url, self.proxy_port, factory)
            # pylint: enable=E1101
            return factory.deferred
        else:
            return failure
 
    def get_page(self, url, contextFactory=None, *args, **kwargs):
        """Download a webpage as a string.
 
        This method relies on the twisted.web.client.getPage but adds and extra
        step. If there is an auth error the method will perform a second try
        so that the username and password are used.
        """
        scheme, _, _, _ = client._parse(url)
        factory = ProxyClientFactory(self.proxy_url, self.proxy_port, url)
        if scheme == 'https':
            from twisted.internet import ssl
            if contextFactory is None:
                contextFactory = ssl.ClientContextFactory()
            # pylint: disable=E1101
            reactor.connectSSL(self.proxy_url, self.proxy_port,
                               factory, contextFactory)
            # pylint: enable=E1101
        else:
            # pylint: disable=E1101
            reactor.connectTCP(self.proxy_url, self.proxy_port, factory)
            # pylint: enable=E1101
        factory.deferred.addErrback(self._process_auth_error, url,
                                    contextFactory)
        return factory.deferred

I hope that this helps anyone out there :)

Read more
mandel

Recently a very interesting bug has been reported agains Ubuntu One on Windows. Apparently we try to sync a number of system folders that are present on Windows 7 to be backward compatible.

The problem

The actual problem in the code is that we are using os.listdir. While lisdir on python does return system folders (at the end of the day, they are there) os.walk does not, for example, lets imaging hat we have the following scenario:

Documents
    My Pictures (System folder)
    My Videos (System folder)
    Random dir
    Random Text.txt

If we run os.listdir we would have the following:

import os
>> os.listdir('Documents')
['My Pictures', 'My Videos', 'Random dir', 'Random Text.txt']

While if we use os.walk we have:

import os
path, dirs, files = os.walk('Documents')
print dirs
>> ['Random dir']
print files
>> ['Random Text.txt']

The fix is very simple, simply filter the result from os.listdir using the following function:

import win32file
 
INVALID_FILE_ATTRIBUTES = -1
 
 
def is_system_path(path):
    """Return if the function is a system path."""
    attrs = win32file.GetFileAttributesW(path)
    if attrs == INVALID_FILE_ATTRIBUTES:
        return False
    return win32file.FILE_ATTRIBUTE_SYSTEM & attrs ==\
        win32file.FILE_ATTRIBUTE_SYSTEM

File system events

An interesting question to ask after the above is, how does ReadDirectoryChangesW work with systen directories? Well, thankfully it works correctly. What does that mean? Well, it means the following:

  • Changes in the system folders do not get notified.
  • Moves from a watch directory to a system folder is not a MOVE_TO, MOVE_FROM couple but a FILE_DELETED

The above means that if you have a system folder in a watch path you do not need to worry since the events will work correctly, which are very very good news.

Read more
mandel

Having a DirtyReactorException in your tests is a PITA and bug 885342 was that type on annoying bug. Since I use this bug not only to tell others what I’m doing but as a log for myself here it is the way to clean the resources nicely when you are testing your PB clients and servers (I mention PB because we use that, a similar approach can be used with any protocol) inspired by this way more interesting post.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
 
class SaveProtocolServerFactory(PBServerFactory):
    """A PBServerFactory that saves the latest connected client."""
 
    protocolInstance = None
 
    def clientConnectionMade(self, protocol):
        """Keep track of the given protocol."""
        self.protocolInstance = protocol
 
 
class SaveClientFactory(PBClientFactory):
    """Client Factory that knows when we disconnected."""
 
    def __init__(self, connected_d, disconnected_d):
        """Create a new instance."""
        PBClientFactory.__init__(self)
        self.connected_d = connected_d
        self.disconnected_d = disconnected_d
 
    def clientConnectionMade(self, broker):
        """Connection made."""
        PBClientFactory.clientConnectionMade(self, broker)
        self.connected_d.callback(True)
 
    def clientConnectionLost(self, connector, reason, reconnecting=0):
        """Connection lost."""
        self.disconnected_d.callback(True)
 
 
class ServerProtocol(Broker):
    """Server protocol that allows us to clean the tests."""
 
    def connectionLost(self, *a):
        self.factory.onConnectionLost.callback(self)
 
 
class ConnectedTestCase(TestCase):
    """Base test case with a client and a server."""
 
    @defer.inlineCallbacks
    def setUp(self):
        """Set up for the tests."""
        yield super(ConnectedTestCase, self).setUp()
        self.server_disconnected = defer.Deferred()
        self.client_disconnected = defer.Deferred()
        self.listener = None
        self.connector = None
        self.server_factory = None
        self.client_factory = None
 
    def setup_client_server(self, sso_root):
        """Set tests."""
        port = get_sso_pb_port()
        self.listener = self._listen_server(sso_root,
                                self.server_disconnected,
                                port)
        connected = defer.Deferred()
        self.connector = self._connect_client(connected,
                                   self.client_disconnected, port)
        self.addCleanup(self.teardown_client_server)
        return connected
 
    def _listen_server(self, sso_root, d, port):
        """Start listenting."""
        self.server_factory = SaveProtocolServerFactory(sso_root)
        self.server_factory.onConnectionLost = d
        self.server_factory.protocol = ServerProtocol
        return reactor.listenTCP(port, self.server_factory)
 
     def _connect_client(self, d1, d2, port):
        """Connect client."""
        self.client_factory = SaveClientFactory(d1, d2)
        return reactor.connectTCP(LOCALHOST, port, self.client_factory)
 
     def teardown_client_server(self):
         """Clean resources."""
         self.connector.disconnect()
         d = defer.maybeDeferred(self.listener.stopListening)
         return defer.gatherResults([d, self.client_disconnected,
                                               self.server_disconnected])

Read more
mandel

I use KVM daily for testing purposes of Ubuntu One on Windows. Recetly I created a Vista VM with 20Gb thinking that it was going to be big enough, turns out that after installation I had a single Gb left (WTF!). Not wanting to have to go through the painful installation process again I decided to find out how to re-size a KVM disk image. Here are the steps if you have to do the same:

  1. Create a new image with the extra size

    sudo qemu-img create -f raw addon.raw 30G
  2. Add the new data, my old vm image is called caranage_old.img. Do remember the order is important, otherwise the image won’t boot.

    cat carnage_old.img addon.raw >> carnage.img
  3. Create a new vm that uses the new image and resize the hardrive accordingly. For example, on Windows Vista I had to go to Mamangement Tools and resize the C: partition to use the new 30Gb.
  4. I hope it helps!

    Read more
mandel

At the moment we are working on providing support for proxy on Ubuntu One. In order to test this correctly I have been setting up a LAN in my office so that I can test as many scenarion as possible. On of those scenarios is the one in which the auth if the proxy uses Active Directory.

Because I use bind9 to set one of my boxed for the DNS I had to dig out how to configure it to work with AD. In order to do that I did the following:

  1. Edited named.conf.local to add a subdomain for the AD machine:

    zone "ad.example.com" {
            type master;
            file "/etc/bind/db.ad.example.com";
            allow-update { 192.168.1.103; };
    };
    
  2. Configured the subzone to work with AD.

    ; BIND data file for local loopback interface
    ;
    $TTL    604800
    @       IN      SOA     ad.example.com. root.ad.example.com. (
                                  2         ; Serial
                             604800         ; Refresh
                              86400         ; Retry
                            2419200         ; Expire
                             604800 )       ; Negative Cache TTL
    ;
    @       IN      NS      ad.marvel.
    @       IN      A       127.0.0.1
    @       IN      AAAA    ::1
    ;
    ; AD horrible domains
    ;
    dc1.ad.example.com.    A       192.168.1.103
    _ldap._tcp.ad.example.com.     SRV     0 0 389  dc1.ad.example.com.
    _kerberos._tcp.ad.example.com.    SRV     0 0 88   dc1.ad.example.com.
    _ldap._tcp.dc._msdcs.ad.example.com.   SRV     0 0 389  dc1.ad.example.com.
    _kerberos._tcp.dc._msdcs.ad.example.com.    SRV     0 0 88   dc1.ad.example.com.
    gc._msdcs.ad.example.com.      SRV     0 0 3268 dc1.ad.example.com.
    

    Note:Is important to remember that the computer name of the server that has the AD role is dc1, if we used a diff name we have to change the configuration accordingly.

  3. Restart the bind9 service:

    sudo /etc/init.d/bind9 restart
    
  4. Install the AD server and specify that you DO NOT want to set that server as a DNS server too.
  5. Set the AD server to use your Ubuntu with your bind9 as the DNS server.

There are lots of things missing if you wanted to use this a set up for a corporate network, but it does the trick in my LAN since I do not have AD duplication or other fancy things. Maybe is useful for you home, who knows..

Read more
mandel

Two really good developers, Alecu and Diego, have discovered a very interestning bug in the os.path.expanduser function in Python. If you have a user in your Windows machine with a name hat uses Japanese characters like “??????” you will have the following in your system:

  • The Windows Shell will show the path correctly, that is: “C:\Users\??????”
  • cmd.exe will show: “C:\Users\??????”
  • All the env variables will be wrong, which means they will be similar to the info shown in cmd.exe

The above is clearly a problem, specially when the implementation of os.path.expanduser on Winodws is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
def expanduser(path):
    """Expand ~ and ~user constructs.
 
    If user or $HOME is unknown, do nothing."""
    if path[:1] != '~':
        return path
    i, n = 1, len(path)
    while i < n and path[i] not in '/\\':
        i = i + 1
 
    if 'HOME' in os.environ:
        userhome = os.environ['HOME']
    elif 'USERPROFILE' in os.environ:
        userhome = os.environ['USERPROFILE']
    elif not 'HOMEPATH' in os.environ:
        return path
    else:
        try:
            drive = os.environ['HOMEDRIVE']
        except KeyError:
            drive = ''
        userhome = join(drive, os.environ['HOMEPATH'])
 
    if i != 1: #~user
        userhome = join(dirname(userhome), path[1:i])
 
    return userhome + path[i:]

For the time being my proposed fix for Ubuntu One is to do the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
import ctypes
from ctypes import windll, wintypes
 
class GUID(ctypes.Structure):
    _fields_ = [
         ('Data1', wintypes.DWORD),
         ('Data2', wintypes.WORD),
         ('Data3', wintypes.WORD),
         ('Data4', wintypes.BYTE * 8)
    ]
    def __init__(self, l, w1, w2, b1, b2, b3, b4, b5, b6, b7, b8):
        """Create a new GUID."""
        self.Data1 = l
        self.Data2 = w1
        self.Data3 = w2
        self.Data4[:] = (b1, b2, b3, b4, b5, b6, b7, b8)
 
    def __repr__(self):
        b1, b2, b3, b4, b5, b6, b7, b8 = self.Data4
        return 'GUID(%x-%x-%x-%x%x%x%x%x%x%x%x)' % (
                   self.Data1, self.Data2, self.Data3, b1, b2, b3, b4, b5, b6, b7, b8)
 
# constants to be used according to the version on shell32
CSIDL_PROFILE = 40
FOLDERID_Profile = GUID(0x5E6C858F, 0x0E22, 0x4760, 0x9A, 0xFE, 0xEA, 0x33, 0x17, 0xB6, 0x71, 0x73)
 
def expand_user():
    # get the function that we can find from Vista up, not the one in XP
    get_folder_path = getattr(windll.shell32, 'SHGetKnownFolderPath', None)
 
    if get_folder_path is not None:
        # ok, we can use the new function which is recomended by the msdn
        ptr = ctypes.c_wchar_p()
        get_folder_path(ctypes.byref(FOLDERID_Profile), 0, 0, ctypes.byref(ptr))
        return ptr.value
    else:
        # use the deprecated one found in XP and on for compatibility reasons
       get_folder_path = getattr(windll.shell32, 'SHGetSpecialFolderPathW', None)
       buf = ctypes.create_unicode_buffer(300)
       get_folder_path(None, buf, CSIDL_PROFILE, False)
       return buf.value

The above code ensure that we only use SHGetFolderPathW when SHGetKnownFolderPathW is not available in the system. The reasoning for that is that SHGetFolderPathW is deprecated and new applications are encourage to use SHGetKnownFolderPathW.

A much better solution is to patch ntpath.py so that is something like what I propose for Ubuntu One. Does anyone know if this is fixed in Python 3? Shall I propose a fix?

PS: For ref I got the GUI value from here.

Read more
mandel

On Ubuntu One we use BtiRock to create the packages for Windows. One of the new features I’m working on is to check if there are updates every so often so that the user gets notified. This code on Ubuntu is not needed because the Update Manger takes care of that, but when you work in an inferior OS…

Generate the auto-update.exe

In order to check for updates we use the generated auto-update.exe wizard from BitRock. Generating the wizard is very straight forward first, as with most of the BitRock stuff, we generate the XML to configure the generated .exe.

<autoUpdateProject>
    <fullName>Ubuntu One</fullName>
    <shortName>ubuntuone</shortName>
    <vendor>Canonical</vendor>
    <version>201</version>
    <singleInstanceCheck>1</singleInstanceCheck>
    <requireInstallationByRootUser>0</requireInstallationByRootUser>
    <requestedExecutionLevel>asInvoker</requestedExecutionLevel>
</autoUpdateProject>

There is just a single thing that is worth mentioning about the above XML. The requireInstallationByRootUser is true because we use the generated .exe to check if there are updates present and we do not what the user to have to be root for that, it does not make sense. Once you have the above or similar XML you can execute:

{$bitrock_installation$}\autoupdate\bin\customize.exe" build ubuntuone_autoupdate.xml windows

Which generates the .exe (the output is in ~\Documents\AutoUpdate\output).

Putting it together with Twisted

The following code provides an example of a couple of functions that can be used by the application, first to check for an update, and to perform the actual update.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import os
import sys
 
# Avoid pylint error on Linux
# pylint: disable=F0401
import win32api
# pylint: enable=F0401
 
from twisted.internet import defer
from twisted.internet.utils import getProcessValue
 
AUTOUPDATE_EXE_NAME = 'autoupdate-windows.exe'
 
def _get_update_path():
    """Return the path in which the autoupdate command is found."""
    if hasattr(sys, "frozen"):
        exec_path = os.path.abspath(sys.executable)
    else:
        exec_path = os.path.dirname(__file__)
    folder = os.path.dirname(exec_path)
    update_path = os.path.join(folder, AUTOUPDATE_EXE_NAME)
    if os.path.exists(update_path):
        return update_path
    return None
 
 
@defer.inlineCallbacks
def are_updates_present(logger):
    """Return if there are updates for Ubuntu One."""
    update_path = _get_update_path()
    logger.debug('Update path %s', update_path)
    if update_path is not None:
        # If there is an update present we will get 0 and other number
        # otherwise
        retcode = yield getProcessValue(update_path, args=('--mode',
            'unattended'), path=os.path.dirname(update_path))
        logger.debug('Return code %s', retcode)
        if retcode == 0:
            logger.debug('Returning True')
            defer.returnValue(True)
    logger.debug('Returning False')
    defer.returnValue(False)
 
 
def perform_update():
    """Spawn the autoupdate process and call the stop function."""
    update_path = _get_update_path()
    if update_path is not None:
        # lets call the updater with the commands that are required,
        win32api.ShellExecute(None, 'runas',
            update_path,
            '--unattendedmodeui none', '', 0)

With the above you should be able to easily update the installation of your frozen python app on Windows when using BitRock.

Read more
mandel

Following my last post regarding how to list all installed applications using python here is the code that one will require to remove an installed msi from a Windows machine using python.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class MsiException(Exception):
    """Raised when there are issues with the msi actions."""
 
 
def uninstall_product(uid):
    """Will remove the old beta from the users machine."""
    # we use the msi lib to be able to uninstall apps
    property_name = u'LocalPackage'
    uninstall_path = get_property_for_product(uid, property_name)
    if uninstall_path is not None:
        # lets remove the package.
        command_line = u'REMOVE=ALL'
        result = windll.msi.MsiInstallProductW(uninstall_path, command_line)
        if result != ERROR_SUCCESS:
            raise MsiException('Could not remove product %s' % uid)

The missing functions can be found in the last post about the topic.

Read more
mandel

The new Ubuntu One Windows client is very close to be released (we have already been sending the new code to our beta testers) and in order to make life easier to new user we wanted to provide a migration script that will allow the user migrate his data to the new client and uninstall the old one. In order to be able to know if the old msi is present in the system we had to use the Windows Installer SDK to query the installed applications and find if the old Ubuntu One client is present.

The following code is a small script that contains the functions to query the installed software in the system which is very similar to the script found in WiLstPrd.vbs but using python instead of VB and ctypes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
# This scripts allows to get a list of all installed products in a windows
# machine. The code uses ctypes becuase there were a number of issues when
# trying to achieve the same win win32com.client
from collections import namedtuple
from ctypes import byref, create_unicode_buffer, windll
from ctypes.wintypes import DWORD
from itertools import count
 
# defined at http://msdn.microsoft.com/en-us/library/aa370101(v=VS.85).aspx
UID_BUFFER_SIZE = 39
PROPERTY_BUFFER_SIZE = 256 
ERROR_MORE_DATA = 234
ERROR_INVALID_PARAMETER = 87
ERROR_SUCCESS = 0
ERROR_NO_MORE_ITEMS = 259 
ERROR_UNKNOWN_PRODUCT = 1605 
 
# diff propoerties of a product, not all products have all properties
PRODUCT_PROPERTIES = [u'Language',
                      u'ProductName',
                      u'PackageCode',
                      u'Transforms',
                      u'AssignmentType',
                      u'PackageName',
                      u'InstalledProductName',
                      u'VersionString',
                      u'RegCompany',
                      u'RegOwner',
                      u'ProductID',
                      u'ProductIcon',
                      u'InstallLocation',
                      u'InstallSource',
                      u'InstallDate',
                      u'Publisher',
                      u'LocalPackage',
                      u'HelpLink',
                      u'HelpTelephone',
                      u'URLInfoAbout',
                      u'URLUpdateInfo',] 
 
# class to be used for python users :)
Product = namedtuple('Product', PRODUCT_PROPERTIES)
 
 
def get_property_for_product(product, property, buf_size=PROPERTY_BUFFER_SIZE):
    """Retruns the value of a fiven property from a product."""
    property_buffer = create_unicode_buffer(buf_size)
    size = DWORD(buf_size)
    result = windll.msi.MsiGetProductInfoW(product, property, property_buffer,
                                           byref(size))
    if result == ERROR_MORE_DATA:
        return get_property_for_product(product, property,
                2 * buf_size)
    elif result == ERROR_SUCCESS:
        return property_buffer.value
    else:
        return None
 
 
def populate_product(uid):
    """Return a Product with the different present data."""
    properties = []
    for property in PRODUCT_PROPERTIES:
        properties.append(get_property_for_product(uid, property))
    return Product(*properties) 
 
 
def get_installed_products_uids():
    """Returns a list with all the different uid of the installed apps."""
    # enum will return an error code according to the result of the app
    products = []
    for i in count(0):
        uid_buffer = create_unicode_buffer(UID_BUFFER_SIZE)
        result = windll.msi.MsiEnumProductsW(i, uid_buffer)
        if result == ERROR_NO_MORE_ITEMS:
            # done interating over the collection
            break
        products.append(uid_buffer.value)
    return products
 
 
def get_installed_products():
    """Returns a collection of products that are installed in the system."""
    products = []
    for puid in  get_installed_products_uids():
        products.append(populate_product(puid))
    return products 
 
 
def is_product_installed_uid(uid):
    """Return if a product with the given id is installed.
 
    uid Most be a unicode object with the uid of the product using
    the following format {uid}
    """
    # we try to get the VersisonString for the uid, if we get an error it means
    # that the product is not installed in the system.
    buf_size = 256
    uid_buffer = create_unicode_buffer(uid)
    property = u'VersionString'
    property_buffer = create_unicode_buffer(buf_size)
    size = DWORD(buf_size)
    result = windll.msi.MsiGetProductInfoW(uid_buffer, property, property_buffer,
                                           byref(size))
    if result == ERROR_UNKNOWN_PRODUCT:
        return False
    else:
        return True

The above code will allow a python developer to check which products are installed on Windows as well as to know if a product with the given UID is indeed installed.

Read more
mandel

In a few days (well, if I find some kind person to take care of Iron) I will be attending the Ubuntu One Developer evening in which we should be able to hear Stuart will be talking about the bunch of crazy ideas he has for developers to use or infrastructure to do cool stuff. I’ll be there for two reason:

  • Hear what Stuart has been planning. I’ve got to recognized I should know a lot more of the Ubuntu One developer program but unfortunately I have been in the working in the Windows port too much and I have ignored the rests of the teams… mea culpa :( .
  • Learn a few more things of the APIs so that I can find my little Chrome extension that uses Ubuntu One (no, it is not bookmark sync, I cannot be less interested in that!).
  • See a bunch of developers and learn about there ideas and what they are doing.
  • Drinks, drinks, drinks! I’m a University of Manchester alumni and a bloody miss Manchester, I don’t care what people say, it is a great city.

If you are going to be in Manchester you should join us, the event is FREE and trust me Stuart is a great guy to go out for drinks with (I’m not bad either, but I always get in trouble :P ).

Read more
mandel

On of the features that I really like from Ubuntu One is the ability to have Read Only shares that will allow me to share files with some of my friends without them having the chance to change my files. In order to support that in a more explicit way on Windows we needed to be able to change the ACEs of an ACL from a file to stop the user from changing the files. In reality there is no need to change the ACEs since the server will ensure that the files are not changed, but as with python, is better to be explicit that to be implicit.

Our solution has the following details:

  • The file system is not using FAT.
  • We assume that the average user does not change the ACEs of a file usually.
  • If the user changes the ACEs he does not add any deny ACE.
  • We want to keep the already present ACEs.

The idea is very simple, we will add a ACE for the path that will remove the user the write rights so that we cannot edit/rename/delete a file and that he can only list the directories. The full code is the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
USER_SID = LookupAccountName("", GetUserName())[0]
 
def _add_deny_ace(path, rights):
    """Remove rights from a path for the given groups."""
    if not os.path.exists(path):
        raise WindowsError('Path %s could not be found.' % path)
 
    if rights is not None:
        security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
        dacl = security_descriptor.GetSecurityDescriptorDacl()
        # set the attributes of the group only if not null
        dacl.AddAccessDeniedAceEx(ACL_REVISION_DS,
                CONTAINER_INHERIT_ACE | OBJECT_INHERIT_ACE, rights,
                USER_SID)
        security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
        SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)
 
 
def _remove_deny_ace(path):
    """Remove the deny ace for the given groups."""
    if not os.path.exists(path):
        raise WindowsError('Path %s could not be found.' % path)
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = security_descriptor.GetSecurityDescriptorDacl()
    # if we delete an ace in the acl the index is outdated and we have
    # to ensure that we do not screw it up. We keep the number of deleted
    # items to update accordingly the index.
    num_delete = 0
    for index in range(0, dacl.GetAceCount()):
        ace = dacl.GetAce(index - num_delete)
        # check if the ace is for the user and its type is 1, that means
        # is a deny ace and we added it, lets remove it
        if USER_SID == ace[2] and ace[0][0] == 1:
            dacl.DeleteAce(index - num_delete)
            num_delete += 1
    security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
    SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)
 
 
def set_no_rights(path):
    """Set the rights for 'path' to be none.
 
    Set the groups to be empty which will remove all the rights of the file.
 
    """
    os.chmod(path, 0o000)
    rights = FILE_ALL_ACCESS
    _add_deny_ace(path, rights)
 
 
def set_file_readonly(path):
    """Change path permissions to readonly in a file."""
    # we use the win32 api because chmod just sets the readonly flag and
    # we want to have more control over the permissions
    rights = FILE_WRITE_DATA | FILE_APPEND_DATA | FILE_GENERIC_WRITE
    # the above equals more or less to 0444
    _add_deny_ace(path, rights)
 
 
def set_file_readwrite(path):
    """Change path permissions to readwrite in a file."""
    # the above equals more or less to 0774
    _remove_deny_ace(path)
    os.chmod(path, stat.S_IWRITE)
 
 
def set_dir_readonly(path):
    """Change path permissions to readonly in a dir."""
    rights = FILE_WRITE_DATA | FILE_APPEND_DATA
 
    # the above equals more or less to 0444
    _add_deny_ace(path, rights)
 
 
def set_dir_readwrite(path):
    """Change path permissions to readwrite in a dir.
 
    Helper that receives a windows path.
 
    """
    # the above equals more or less to 0774
    _remove_deny_ace(path)
    # remove the read only flag
    os.chmod(path, stat.S_IWRITE)

Adding the Deny ACE

The idea of the code is very simple, we will add a Deny ACE to the path so that the user cannot write it. The Deny ACE is different if it is a file or a directory since we want the user to be able to list the contents of a directory.

3
4
5
6
7
8
9
10
11
12
13
14
15
16
def _add_deny_ace(path, rights):
    """Remove rights from a path for the given groups."""
    if not os.path.exists(path):
        raise WindowsError('Path %s could not be found.' % path)
 
    if rights is not None:
        security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
        dacl = security_descriptor.GetSecurityDescriptorDacl()
        # set the attributes of the group only if not null
        dacl.AddAccessDeniedAceEx(ACL_REVISION_DS,
                CONTAINER_INHERIT_ACE | OBJECT_INHERIT_ACE, rights,
                USER_SID)
        security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
        SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)

Remove the Deny ACE

Very similar to the above but doing the opposite, lets remove the Deny ACES present for the current user. If you notice we store how many we removed, the reason is simple, if we remove an ACE the index is no longer valid so we have to calculate the correct one by knowing how many we removed.

19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
def _remove_deny_ace(path):
    """Remove the deny ace for the given groups."""
    if not os.path.exists(path):
        raise WindowsError('Path %s could not be found.' % path)
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = security_descriptor.GetSecurityDescriptorDacl()
    # if we delete an ace in the acl the index is outdated and we have
    # to ensure that we do not screw it up. We keep the number of deleted
    # items to update accordingly the index.
    num_delete = 0
    for index in range(0, dacl.GetAceCount()):
        ace = dacl.GetAce(index - num_delete)
        # check if the ace is for the user and its type is 1, that means
        # is a deny ace and we added it, lets remove it
        if USER_SID == ace[2] and ace[0][0] == 1:
            dacl.DeleteAce(index - num_delete)
            num_delete += 1
    security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
    SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)

Implement access

Our access implementation takes into account the Deny ACE added to ensure that we do not only look at the flags.

def access(path):
    """Return if the path is at least readable."""
    # lets consider the access on an illegal path to be a special case
    # since that will only occur in the case where the user created the path
    # for a file to be readable it has to be readable either by the user or
    # by the everyone group
    # XXX: ENOPARSE ^ (nessita)
    if not os.path.exists(path):
        return False
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = security_descriptor.GetSecurityDescriptorDacl()
    for index in range(0, dacl.GetAceCount()):
        # add the sid of the ace if it can read to test that we remove
        # the r bitmask and test if the bitmask is the same, if not, it means
        # we could read and removed it.
        ace = dacl.GetAce(index)
        if USER_SID == ace[2] and ace[0][0] == 1:
            # check wich access is denied
            if ace[1] | FILE_GENERIC_READ == ace[1] or\
               ace[1] | FILE_ALL_ACCESS == ace[1]:
                return False
    return True

Implement can_write

The following code is similar to access but checks if we have a readonly file.

def can_write(path):
    """Return if the path is at least readable."""
    # lets consider the access on an illegal path to be a special case
    # since that will only occur in the case where the user created the path
    # for a file to be readable it has to be readable either by the user or
    # by the everyone group
    # XXX: ENOPARSE ^ (nessita)
    if not os.path.exists(path):
        return False
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = security_descriptor.GetSecurityDescriptorDacl()
    for index in range(0, dacl.GetAceCount()):
        # add the sid of the ace if it can read to test that we remove
        # the r bitmask and test if the bitmask is the same, if not, it means
        # we could read and removed it.
        ace = dacl.GetAce(index)
        if USER_SID == ace[2] and ace[0][0] == 1:
            if ace[1] | FILE_GENERIC_WRITE == ace[1] or\
               ace[1] | FILE_WRITE_DATA == ace[1] or\
               ace[1] | FILE_APPEND_DATA == ace[1] or\
               ace[1] | FILE_ALL_ACCESS == ace[1]:
                # check wich access is denied
                return False
    return True

And that is about it, I hope it helps other projects :D

Read more
mandel

A little more than a year ago I started working for Canonical full time on the Windows port of Ubuntu One. One of the great things of working for Canonical is that you work at home, that is, you do not have to move to the USA or the UK to do the job you love, but don’t get confused, working at home does has its downs and here are a list of some them.

Social interaction

Most of us, geeks, most of the time we do not require as much social interaction as the ‘normal’ people. This does not mean we are less social, but due to the nature of our work we need to concentrate for long periods of time to be able to solve complicated problem which might involve thousands of lines of code, in this situation having a colleague popping in your cubicle asking if you wanna have a coffee is less than ideal. When working at home you do not have this type of social interactions which at the very begging seems to be a very convenient thing, unfortunately it is not like that.

As Aristotle said:

He who is unable to live in society, or who has no need because he is sufficient for himself, must be either a beast or a god.

The above is something certainly important to remember. No matter how much we believe that we do not need social contact, at the end of the day, it is needed because it is an intrinsic part of our being. We need human contact and no type of online interaction will be ever be able to replace a face to face interaction between two people.

I you do work at home make sure you get such interaction, and not only with your wife or girlfriend. I have learned this the hard way, I believe that this happened because during a period or time I became a monster that did not need any type of social interaction which is something I terribly regret and I have strongly tried to solve. The following are somethings I have decided to do:

  • I’m not literally working at home anymore. I’m looking for a shared office space so that I have to leave the house. As soon as I have found a place I will only be using the IPad at home, any other machine will be forbidden.
  • For a time I was going less and less to the rugby trainings, this will change as soon as the season starts.
  • As with rugby, I stopped going to the gym, this will stop.
  • I’ll force myself to go out at least once a week (although I have to admit that after this I have been going out a lot more)

Working hours

Before I started working for Canonical I worked at GDF (Electrabel to be more precise) dealing with issues between the interaction of an ASP.Net front end and a Java backend. There is no need to say that I hated the job. While I worked there I would never to an extra hour, I’d arrive at 9 and will leave at 6 without giving a damm if anyone needed me to stay an extra hour. I did this because during my free hours I wanted to work on applicatios to be used on Ubuntu and I had no passion for my everyday job.

Now this has changed, and my everyday job is my passion. This is great but it has become a small problem regarding the working hours, I do to many. I seem not to be able to stop and that is no good. I have started to be more strict with the working hours so that I do not work more than 9/10 hours (I don’t like to count reading emails and bug reports as work).

Summary

This are the problems that trouble me, which are yours? Nevertheless the oatmeal is bloody right.

Read more
mandel

Last week was probably one of the best coding sprints I have had since I started working in Canonical, I’m serious!. I had the luck to pair program with alecu on the FilesystemMonitor that we use in Ubuntu One on windows. The implementation has improved so much that I wanted to blog about it and show it as an example of how to hook the ReadDirectoryChangesW call from COM into twisted so that you can process the events using twisted which is bloody cool.

We have reduce the implementation of the Watch and WatchManager to match our needs and reduce the API provided since we do not use all the API provided by pyinotify. The Watcher implementation is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
class Watch(object):
    """Implement the same functions as pyinotify.Watch."""
 
    def __init__(self, watch_descriptor, path, mask, auto_add, processor,
        buf_size=8192):
        super(Watch, self).__init__()
        self.log = logging.getLogger('ubuntuone.SyncDaemon.platform.windows.' +
            'filesystem_notifications.Watch')
        self.log.setLevel(TRACE)
        self._processor = processor
        self._buf_size = buf_size
        self._wait_stop = CreateEvent(None, 0, 0, None)
        self._overlapped = OVERLAPPED()
        self._overlapped.hEvent = CreateEvent(None, 0, 0, None)
        self._watching = False
        self._descriptor = watch_descriptor
        self._auto_add = auto_add
        self._ignore_paths = []
        self._cookie = None
        self._source_pathname = None
        self._process_thread = None
        # remember the subdirs we have so that when we have a delete we can
        # check if it was a remove
        self._subdirs = []
        # ensure that we work with an abspath and that we can deal with
        # long paths over 260 chars.
        if not path.endswith(os.path.sep):
            path += os.path.sep
        self._path = os.path.abspath(path)
        self._mask = mask
        # this deferred is fired when the watch has started monitoring
        # a directory from a thread
        self._watch_started_deferred = defer.Deferred()
 
    @is_valid_windows_path(path_indexes=[1])
    def _path_is_dir(self, path):
        """Check if the path is a dir and update the local subdir list."""
        self.log.debug('Testing if path %r is a dir', path)
        is_dir = False
        if os.path.exists(path):
            is_dir = os.path.isdir(path)
        else:
            self.log.debug('Path "%s" was deleted subdirs are %s.',
                path, self._subdirs)
            # we removed the path, we look in the internal list
            if path in self._subdirs:
                is_dir = True
                self._subdirs.remove(path)
        if is_dir:
            self.log.debug('Adding %s to subdirs %s', path, self._subdirs)
            self._subdirs.append(path)
        return is_dir
 
    def _process_events(self, events):
        """Process the events form the queue."""
        # do not do it if we stop watching and the events are empty
        if not self._watching:
            return
 
        # we transform the events to be the same as the one in pyinotify
        # and then use the proc_fun
        for action, file_name in events:
            if any([file_name.startswith(path)
                        for path in self._ignore_paths]):
                continue
            # map the windows events to the pyinotify ones, tis is dirty but
            # makes the multiplatform better, linux was first :P
            syncdaemon_path = get_syncdaemon_valid_path(
                                        os.path.join(self._path, file_name))
            is_dir = self._path_is_dir(os.path.join(self._path, file_name))
            if is_dir:
                self._subdirs.append(file_name)
            mask = WINDOWS_ACTIONS[action]
            head, tail = os.path.split(file_name)
            if is_dir:
                mask |= IN_ISDIR
            event_raw_data = {
                'wd': self._descriptor,
                'dir': is_dir,
                'mask': mask,
                'name': tail,
                'path': '.'}
            # by the way in which the win api fires the events we know for
            # sure that no move events will be added in the wrong order, this
            # is kind of hacky, I dont like it too much
            if WINDOWS_ACTIONS[action] == IN_MOVED_FROM:
                self._cookie = str(uuid4())
                self._source_pathname = tail
                event_raw_data['cookie'] = self._cookie
            if WINDOWS_ACTIONS[action] == IN_MOVED_TO:
                event_raw_data['src_pathname'] = self._source_pathname
                event_raw_data['cookie'] = self._cookie
            event = Event(event_raw_data)
            # FIXME: event deduces the pathname wrong and we need to manually
            # set it
            event.pathname = syncdaemon_path
            # add the event only if we do not have an exclude filter or
            # the exclude filter returns False, that is, the event will not
            # be excluded
            self.log.debug('Event is %s.', event)
            self._processor(event)
 
    def _call_deferred(self, f, *args):
        """Executes the defeered call avoiding possible race conditions."""
        if not self._watch_started_deferred.called:
            f(args)
 
    def _watch(self):
        """Watch a path that is a directory."""
        # we are going to be using the ReadDirectoryChangesW whihc requires
        # a directory handle and the mask to be used.
        handle = CreateFile(
            self._path,
            FILE_LIST_DIRECTORY,
            FILE_SHARE_READ | FILE_SHARE_WRITE,
            None,
            OPEN_EXISTING,
            FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,
            None)
        self.log.debug('Watching path %s.', self._path)
        while True:
            # important information to know about the parameters:
            # param 1: the handle to the dir
            # param 2: the size to be used in the kernel to store events
            # that might be lost while the call is being performed. This
            # is complicated to fine tune since if you make lots of watcher
            # you migh used too much memory and make your OS to BSOD
            buf = AllocateReadBuffer(self._buf_size)
            try:
                ReadDirectoryChangesW(
                    handle,
                    buf,
                    self._auto_add,
                    self._mask,
                    self._overlapped,
                )
                reactor.callFromThread(self._call_deferred,
                    self._watch_started_deferred.callback, True)
            except error:
                # the handle is invalid, this may occur if we decided to
                # stop watching before we go in the loop, lets get out of it
                reactor.callFromThread(self._call_deferred,
                    self._watch_started_deferred.errback, error)
                break
            # wait for an event and ensure that we either stop or read the
            # data
            rc = WaitForMultipleObjects((self._wait_stop,
                                         self._overlapped.hEvent),
                                         0, INFINITE)
            if rc == WAIT_OBJECT_0:
                # Stop event
                break
            # if we continue, it means that we got some data, lets read it
            data = GetOverlappedResult(handle, self._overlapped, True)
            # lets ead the data and store it in the results
            events = FILE_NOTIFY_INFORMATION(buf, data)
            self.log.debug('Events from ReadDirectoryChangesW are %s', events)
            reactor.callFromThread(self._process_events, events)
 
        CloseHandle(handle)
 
    @is_valid_windows_path(path_indexes=[1])
    def ignore_path(self, path):
        """Add the path of the events to ignore."""
        if not path.endswith(os.path.sep):
            path += os.path.sep
        if path.startswith(self._path):
            path = path[len(self._path):]
            self._ignore_paths.append(path)
 
    @is_valid_windows_path(path_indexes=[1])
    def remove_ignored_path(self, path):
        """Reaccept path."""
        if not path.endswith(os.path.sep):
            path += os.path.sep
        if path.startswith(self._path):
            path = path[len(self._path):]
            if path in self._ignore_paths:
                self._ignore_paths.remove(path)
 
    def start_watching(self):
        """Tell the watch to start processing events."""
        for current_child in os.listdir(self._path):
            full_child_path = os.path.join(self._path, current_child)
            if os.path.isdir(full_child_path):
                self._subdirs.append(full_child_path)
        # start to diff threads, one to watch the path, the other to
        # process the events.
        self.log.debug('Start watching path.')
        self._watching = True
        reactor.callInThread(self._watch)
        return self._watch_started_deferred
 
    def stop_watching(self):
        """Tell the watch to stop processing events."""
        self.log.info('Stop watching %s', self._path)
        SetEvent(self._wait_stop)
        self._watching = False
        self._subdirs = []
 
    def update(self, mask, auto_add=False):
        """Update the info used by the watcher."""
        self.log.debug('update(%s, %s)', mask, auto_add)
        self._mask = mask
        self._auto_add = auto_add
 
    @property
    def path(self):
        """Return the patch watched."""
        return self._path
 
    @property
    def auto_add(self):
        return self._auto_add

The important details of this implementations are the following:

Use a deferred to notify that the watch started.

During or tests we noticed that the start watch function was slow which would mean that from the point when we start watching the directory and the point when the thread actually started we would be loosing events. The function now returns a deferred that will be fired when the ReadDirectoryChangesW has been called which ensures that no events will be lost. The interesting parts are the following:

define the deferred

31
32
33
       # this deferred is fired when the watch has started monitoring
        # a directory from a thread
        self._watch_started_deferred = defer.Deferred()

Call the deferred either when we successfully started watching:

128
129
130
131
132
133
134
135
136
137
138
            buf = AllocateReadBuffer(self._buf_size)
            try:
                ReadDirectoryChangesW(
                    handle,
                    buf,
                    self._auto_add,
                    self._mask,
                    self._overlapped,
                )
                reactor.callFromThread(self._call_deferred,
                    self._watch_started_deferred.callback, True)

Call it when we do have an error:

139
140
141
142
143
144
            except error:
                # the handle is invalid, this may occur if we decided to
                # stop watching before we go in the loop, lets get out of it
                reactor.callFromThread(self._call_deferred,
                    self._watch_started_deferred.errback, error)
                break

Threading and firing the reactor.

There is an interesting detail to take care of in this code. We have to ensure that the deferred is not called more than once, to do that you have to callFromThread a function that will fire the event only when it was not already fired like this:

103
104
105
106
    def _call_deferred(self, f, *args):
        """Executes the defeered call avoiding possible race conditions."""
        if not self._watch_started_deferred.called:
            f(args)

If you do not do the above, but the code bellow you will have a race condition in which the deferred is called more than once.

            buf = AllocateReadBuffer(self._buf_size)
            try:
                ReadDirectoryChangesW(
                    handle,
                    buf,
                    self._auto_add,
                    self._mask,
                    self._overlapped,
                )
                if not self._watch_started_deferred.called:
                    reactor.callFromThread(self._watch_started_deferred.callback, True)
            except error:
                # the handle is invalid, this may occur if we decided to
                # stop watching before we go in the loop, lets get out of it
                if not self._watch_started_deferred.called:
                    reactor.callFromThread(self._watch_started_deferred.errback, error)
                break

Execute the processing of events in the reactor main thread.

Alecu has bloody great ideas way too often, and this is one of his. The processing of the events is queued to be executed in the twisted reactor main thread which reduces the amount of threads we use and will ensure that the events are processed in the correct order.

153
154
155
156
157
158
            # if we continue, it means that we got some data, lets read it
            data = GetOverlappedResult(handle, self._overlapped, True)
            # lets ead the data and store it in the results
            events = FILE_NOTIFY_INFORMATION(buf, data)
            self.log.debug('Events from ReadDirectoryChangesW are %s', events)
            reactor.callFromThread(self._process_events, events)

Just for this the flight to Buenos Aires was well worth it!!! For anyone to see the full code feel free to look at ubuntuone.platform.windows from ubuntuone.

Read more
mandel

At Ubuntu One we required to be able to use named pipes on windows for IPC. This is a ver normal process in multi-process applications like the one we are going to provide, but in our case we had a twist, we are using twisted. As some of you may know there is not default reactor that would allow you to write a protocol in twisted and allows to use named pipes as the transport of the protocol. Well this was until very recently.

Txnamedpipes (lp:txnamedpipes) is a project that provides a ICOP based reactor that allows to use namedpipes for the transport of your protocol. At the moment we are confident that the implementation would allow you to use spred.pb or a custom protocol on twisted 10 and later on Windows 7 (we have been able to find a number of issues on Windows XP). The following is a small example of a spread.pb service and client that uses a named pipe for communication.

from txnamedpipes.reactor import install
install()
from twisted.spread import pb
from twisted.internet import reactor
 
class Echoer(pb.Root):
    def remote_echo(self, st):
        print 'echoing:', st
        return st
 
if __name__ == '__main__':
    reactor.listenPipe('\\\\.\\pipe\\test_pipe',
                               pb.PBServerFactory(Echoer()))
    reactor.run()
from txnamedpipes.reactor import install
install()
from twisted.spread import pb
from twisted.internet import reactor
from twisted.python import util
 
factory = pb.PBClientFactory()
reactor.connectPipe('\\\\.\\pipe\\test_pipe', factory)
d = factory.getRootObject()
d.addCallback(lambda object: object.callRemote("echo", 
                      "hello network"))
d.addCallback(lambda echo: 'server echoed: '+echo)
d.addErrback(lambda reason: 'error: '+str(reason.value))
d.addCallback(util.println)
d.addCallback(lambda _: reactor.stop())
reactor.run()

The code has the MIT license and we hope that other people find it useful.

Read more
mandel

Pywin32 is a very cool project that allows you to access the win api without having to go through ctypes and deal with all the crazy parameters that COM is famous for. Unfortunately sometimes it has som issues which you face only a few times in your life.

This case I found a bug where GetFileSecurity does not use the GetFileSecurityW method but the w-less version. For those who don’t have to deal with this terrible details, the W usually means that the functions knows how to deal with utf-8 strings (backward compatibility can be a problem sometimes). I have reported the bug but for those that are in a hurry here is the patch:

diff -r 7dce71d174a9 win32/src/win32security.i
--- a/win32/src/win32security.i	Sat Jun 18 10:16:06 2011 -0400
+++ b/win32/src/win32security.i	Mon Jun 20 14:15:27 2011 +0200
@@ -2108,7 +2108,7 @@
  if (!PyWinObject_AsTCHAR(obFname, &fname))
   goto done;
 
-	if (GetFileSecurity(fname, info, psd, dwSize, &dwSize)) {
+	if (GetFileSecurityW(fname, info, psd, dwSize, &dwSize)) {
   PyErr_SetString(PyExc_RuntimeError, "Can't query for SECURITY_DESCRIPTOR size info?");
   goto done;
  }
@@ -2117,7 +2117,7 @@
   PyErr_SetString(PyExc_MemoryError, "allocating SECURITY_DESCRIPTOR");
   goto done;
  }
- if (!GetFileSecurity(fname, info, psd, dwSize, &dwSize)) {
+ if (!GetFileSecurityW(fname, info, psd, dwSize, &dwSize)) {
   PyWin_SetAPIError("GetFileSecurity");
   goto done;
  }
@@ -2153,7 +2153,7 @@
  PSECURITY_DESCRIPTOR psd;
  if (!PyWinObject_AsSECURITY_DESCRIPTOR(obsd, &psd))
   goto done;
-	if (!SetFileSecurity(fname, info, psd)) {
+	if (!SetFileSecurityW(fname, info, psd)) {
   PyWin_SetAPIError("SetFileSecurity");
   goto done;
  }

Read more
mandel

At the moment some of the tests (and I cannot point out which ones) of ubuntuone-client fail when they are ran on Windows. The reason for this is due to the way in which we get the notifications out of the file system and the way the tests are written. Before I blame the OS or the tests, let me explain a number of facts about the Windows filesystem and the possible ways to interact with it.

To be able to get file system changes from the OS the Win32 API provides the following:

SHChangedNotifyRegister

This function was broken up to Vista when it was fixed, Unfortunately AFAIK we also support Windows XP which means that we cannot trust this function. On top of that taking this path means that we can have a performance issue. Because the function is build on top of Windows messages, if too many changes occur the sync daemon would start receiving roll up messages that just state that something changed and it would be up to the sync daemon to decide what really happened. Therefore we can all agree that this is a no no, right?

FindFirstChangeNotification

This is a really easy function to use which is based on ReadDirectoryChangesW (I think is a simple wrapper around it) that lets you know that something changed but gives no information about what changed. Because if is based on ReadDirectoryChangesW it suffers from the same issues.

ReadDirectoryChangesW

This is by far the most common way to get the notification changes from the system. Now, in theory there are two possible cases which can go wrong that would affect the events raised by this function:

  1. There are too many events and the buffer gets overloaded and we start loosing events. A simple way to solve this issues is to process the events in a diff thread asap so that we can keep reading the changes.
  2. We use the sync version of the function which means that we could have the following issues:
    • Blue screen of death because we used too much memory from the kernel space.
    • We cannot close the handles used to watch the changes in the directories. This makes the threads to end up blocked.

As I mentioned this is the theory and therefore makes perfect sense to choose this option as the way to get notified by the changes until… you hit a great little feature of Windows called write-behind caching. The idea of write-behind caching is the following one:

When you attempt to write a new file on your HD Windows does not directly modify the HD. Instead it makes a not of the fact that your intention is to write on disk and saves your changes in memory. Ins’t that smart?

Well, that lovely feature does come set as default AFAIK from XP onwards. Any smart person would wonder how does that interact with FindFirstChangeNotification/ReadDirectoryChangesW, well after some work here is what I have managed to find out:

The IO Manager (internal to the kernel) is queueing up disk-write requests in an internal buffer, and the actual changes are not physically committed until some condition is met which I believe is for the “write-behind caching” feature. The problem appears to be that the user-space callback via FileSystemWatcher/ReadDirectoryChanges does not occur when disk-write requests are inserted into the queue, but rather occurs when they are leaving the queue and being physically committed to disk. For what I have been able to manage through observation, the life time of a queue is based on:

  1. Whether more writes are being inserted in the q.
  2. Is another app request a read from an item in the q.

This means that when using FileSystemWatcher/ReadDirectoryChanges the events are fired only when the changes are actually committed and as for a user-space program this follows a non-deterministic process (insert spanish swearing here). a way to work around this issue is to use the FluxhFileBuffers function on the volume, which does need admin rights, yeah!

Change Journal records

Well, this allows to track the changes that have been committed in an NTFS system (that means that we do not have support to FAT). This technique allows to keep track of the changes using an update sequence number that keeps track of changes in an interesting manner. At first look, although parsing the data is hard, this solution seems to be very similar to the one used by pyinotify and therefore someone will say, hey, let just ell twisted to do a select on that file and read the changes. Well, no, is not that easy, files do not provide the functionality used for select, just sockets (http://msdn.microsoft.com/en-us/library/aa363803%28VS.85%29.aspx) /me jumps of happiness

File system filterr

Well, this is an easy one to summarize, you have to write a driver like piece of code. Means C, COM and being able to crash the entire system with a nice blue screen (although I can change the color to aubergine before we crash)

Conclusion

At this point I hope I have convinced a few to believe that ReadDirectoryChangesW is the best option to take but might be wondering why I mentioned the write-behind caching feature, well here comes my complain towards the tests. We do use the real file system notifications for testing and the trial test cases do have a timeout! Those two facts plus the lovely write-behind caching feature mean that the tests on Windows fail just because the bloody evens are not raise until the leave the q from the IO manager.

Read more
mandel

During the past few days I have been trying to track down an issue in the Ubuntu One client tests when ran on Windows that would use all the threads that the python process could have. As you can imaging finding out why there are deadlocks is quite hard, specially when I though that the code was thread safe, guess what? it wasn’t

The bug I had in the code was related to the way in which ReadDirectoryChangesW works. This functions has two different ways to be executed:

Synchronous

The ReadDirectoryChangesW can be executed in a sync mode by NOT providing a OVERLAPPED structure to perform the IO operations, for example:

def _watcherThread(self, dn, dh, changes):
        flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME
        while 1:
            try:
                print "waiting", dh
                changes = win32file.ReadDirectoryChangesW(dh,
                                                          8192,
                                                          False,
                                                          flags)
                print "got", changes
            except:
                raise
            changes.extend(changes)

The above example has the following two problems:

  • ReadDirectoryChangesW without an OVERLAPPED blocks infinitely.
  • If another thread attempts to close the handle while ReadDirectoryChangesW is waiting on it, the CloseHandle() method blocks (which has nothing to do with the GIL – it is correctly managed)

I got bitten in the ass by the second item which broke my tests in two different ways since it let thread block and a Handle used so that the rest of the tests could not remove the tmp directories that were under used by the block threads.

Asynchronous

In other to be able to use the async version of the function we just have to use an OVERLAPPED structure, this way the IO operations will no block and we will also be able to close the handle from a diff thread.

def _watcherThreadOverlapped(self, dn, dh, changes):
        flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME
        buf = win32file.AllocateReadBuffer(8192)
        overlapped = pywintypes.OVERLAPPED()
        overlapped.hEvent = win32event.CreateEvent(None, 0, 0, None)
        while 1:
            win32file.ReadDirectoryChangesW(dh,
                                            buf,
                                            False, #sub-tree
                                            flags,
                                            overlapped)
            # Wait for our event, or for 5 seconds.
            rc = win32event.WaitForSingleObject(overlapped.hEvent, 5000)
            if rc == win32event.WAIT_OBJECT_0:
                # got some data!  Must use GetOverlappedResult to find out
                # how much is valid!  0 generally means the handle has
                # been closed.  Blocking is OK here, as the event has
                # already been set.
                nbytes = win32file.GetOverlappedResult(dh, overlapped, True)
                if nbytes:
                    bits = win32file.FILE_NOTIFY_INFORMATION(buf, nbytes)
                    changes.extend(bits)
                else:
                    # This is "normal" exit - our 'tearDown' closes the
                    # handle.
                    # print "looks like dir handle was closed!"
                    return
            else:
                print "ERROR: Watcher thread timed-out!"
                return # kill the thread!

Using the ReadDirectoryW function in this way does solve all the other issues that are found on the sync version and the only extra overhead added is that you need to understand how to deal with COM events which is not that hard after you have worked with it for a little.

I leave this here for people that might find the same issue and for me to remember how much my ass hurt.

References

Read more