Canonical Voices

Posts tagged with 'win32'

mandel

During the past few days I have been trying to track down an issue in the Ubuntu One client tests when ran on Windows that would use all the threads that the python process could have. As you can imaging finding out why there are deadlocks is quite hard, specially when I though that the code was thread safe, guess what? it wasn’t

The bug I had in the code was related to the way in which ReadDirectoryChangesW works. This functions has two different ways to be executed:

Synchronous

The ReadDirectoryChangesW can be executed in a sync mode by NOT providing a OVERLAPPED structure to perform the IO operations, for example:

def _watcherThread(self, dn, dh, changes):
        flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME
        while 1:
            try:
                print "waiting", dh
                changes = win32file.ReadDirectoryChangesW(dh,
                                                          8192,
                                                          False,
                                                          flags)
                print "got", changes
            except:
                raise
            changes.extend(changes)

The above example has the following two problems:

  • ReadDirectoryChangesW without an OVERLAPPED blocks infinitely.
  • If another thread attempts to close the handle while ReadDirectoryChangesW is waiting on it, the CloseHandle() method blocks (which has nothing to do with the GIL – it is correctly managed)

I got bitten in the ass by the second item which broke my tests in two different ways since it let thread block and a Handle used so that the rest of the tests could not remove the tmp directories that were under used by the block threads.

Asynchronous

In other to be able to use the async version of the function we just have to use an OVERLAPPED structure, this way the IO operations will no block and we will also be able to close the handle from a diff thread.

def _watcherThreadOverlapped(self, dn, dh, changes):
        flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME
        buf = win32file.AllocateReadBuffer(8192)
        overlapped = pywintypes.OVERLAPPED()
        overlapped.hEvent = win32event.CreateEvent(None, 0, 0, None)
        while 1:
            win32file.ReadDirectoryChangesW(dh,
                                            buf,
                                            False, #sub-tree
                                            flags,
                                            overlapped)
            # Wait for our event, or for 5 seconds.
            rc = win32event.WaitForSingleObject(overlapped.hEvent, 5000)
            if rc == win32event.WAIT_OBJECT_0:
                # got some data!  Must use GetOverlappedResult to find out
                # how much is valid!  0 generally means the handle has
                # been closed.  Blocking is OK here, as the event has
                # already been set.
                nbytes = win32file.GetOverlappedResult(dh, overlapped, True)
                if nbytes:
                    bits = win32file.FILE_NOTIFY_INFORMATION(buf, nbytes)
                    changes.extend(bits)
                else:
                    # This is "normal" exit - our 'tearDown' closes the
                    # handle.
                    # print "looks like dir handle was closed!"
                    return
            else:
                print "ERROR: Watcher thread timed-out!"
                return # kill the thread!

Using the ReadDirectoryW function in this way does solve all the other issues that are found on the sync version and the only extra overhead added is that you need to understand how to deal with COM events which is not that hard after you have worked with it for a little.

I leave this here for people that might find the same issue and for me to remember how much my ass hurt.

References

Read more
Martin Pool

A new plugin bzr-wincrypt stores passwords encrypted in the Windows CryptoAPI keyring.

Read more
mandel

While working on making the Ubuntu One code more multiplatform I founded myself having to write some code that would set the attributes of a file on Windows. Ideally os.chmod would do the trick, but of course this is windows, and it is not fully supported. According to the python documentation:

Note: Although Windows supports chmod(), you can only set the file’s read-only flag with it (via the stat.S_IWRITE and stat.S_IREAD constants or a corresponding integer value). All other bits are ignored.

Grrrreat… To solve this issue I have written a small function that will allow to set the attributes of a file by using the win32api and win32security modules. This solves partially the issues since 0444 and others cannot be perfectly map to the Windows world. In my code I have made the assumption that using the groups ‘Everyone’, ‘Administrators’ and the user name would be close enough for our use cases.

Here is the code in case anyone has to go through this:

from win32api import MoveFileEx, GetUserName
 
from win32file import (
    MOVEFILE_COPY_ALLOWED,
    MOVEFILE_REPLACE_EXISTING,
    MOVEFILE_WRITE_THROUGH
)
from win32security import (
    LookupAccountName,
    GetFileSecurity,
    SetFileSecurity,
    ACL,
    DACL_SECURITY_INFORMATION,
    ACL_REVISION
)
from ntsecuritycon import (
    FILE_ALL_ACCESS,
    FILE_GENERIC_EXECUTE,
    FILE_GENERIC_READ,
    FILE_GENERIC_WRITE,
    FILE_LIST_DIRECTORY
)
 
EVERYONE_GROUP = 'Everyone'
ADMINISTRATORS_GROUP = 'Administrators'
 
def _get_group_sid(group_name):
    """Return the SID for a group with the given name."""
    return LookupAccountName('', group_name)[0]
 
 
def _set_file_attributes(path, groups):
    """Set file attributes using the wind32api."""
    security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
    dacl = ACL()
    for group_name in groups:
        # set the attributes of the group only if not null
        if groups[group_name]:
            group_sid = _get_group_sid(group_name)
            dacl.AddAccessAllowedAce(ACL_REVISION, groups[group_name],
                group_sid)
    # the dacl has all the info of the dff groups passed in the parameters
    security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
    SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)
 
def set_file_readonly(path):
    """Change path permissions to readonly in a file."""
    # we use the win32 api because chmod just sets the readonly flag and
    # we want to have imore control over the permissions
    groups = {}
    groups[EVERYONE_GROUP] = FILE_GENERIC_READ
    groups[ADMINISTRATORS_GROUP] = FILE_GENERIC_READ
    groups[GetUserName()] = FILE_GENERIC_READ
    # the above equals more or less to 0444
    _set_file_attributes(path, groups)

For those who might want to remove the read access from a group, you just have to not pass the group in the groups parameter which would remove the group from the security descriptor.

Read more
Martin Pool

A nice mail from Martitza this morning:

Hi.  Some of you may recall that I’ve provided measurements of some slow operations in both bzr itself and bzr-explorer based on bzrlib 2.1 and earlier.  I don’t recall doing any testing on bzr 2.2.0 specifically, but now that I’m on bzr 2.2.1 bzr (and bzr-explorer) are noticeably faster and even more pleasant to use now.   So far it seems that each of the add/commit/status operations on sets of 10K+ files (500MB+ branches) takes no more than two or three minutes each.  This is twice as fast as some of my previous measurements on the same or smaller branches.  Also, memory consumption is much reduced, which I suspect helps a lot.  I even have one of these large repos hosted on an old 512MB Pentium 3 machine with no problem today.  This is wonderful!  I hesitate to name the people I know made this possible, because I do not want to leave out the many more people whose contributions I don’t know.  Please accept my thanks to the whole team.


Read more
Martin Pool

TortoiseBzr (integration into the Microsoft Windows shell) is now being internationalized, with Japanese and Spanish translations now almost complete.

If you speak any language other than English, you can help with translation.


Read more