Canonical Voices

Posts tagged with 'kernel'

Colin Ian King

Before I started some analysis on benchmarking various popular file systems on Linux I was recommended to read "Systems Performance: Enterprise  and the Cloud" by Brendan Gregg.

In today's modern server and cloud based systems the multi-layered complexity can make it hard to pin point performance issues and bottlenecks. This book is packed full useful analysis techniques covering tracing, kernel internals, tools and benchmarking.

Critical to getting a well balanced and tuned system are all the different components, and the book has chapters covering CPU optimisation (cores, threading, caching and internconnects),  memory optimisation (virtual memory, paging, swapping, allocators, busses),  file system I/O, storage, networking (protcols, sockets, physical connections) and typical issues facing cloud computing.

The book is full of very useful examples and practical instructions on how to drill down and discover performance issues in a system and also includes some real-world case studies too.

It has helped me become even more focused on how to analyse performance issues and consider how to do deep system instrumentation to be able to understand where any why performance regressions occur.

All-in-all, a most systematic and well written book that I'd recommend to anyone running large complex servers and cloud computing environments.






Read more
Colin Ian King

even more stress in stress-ng

Over the past few weeks in spare moments I've been adding more stress methods to stress-ng  ready for Ubuntu 15.04 Vivid Vervet.   My intention is to produce a rich set of stress methods that can stress and exercise many facets of a system to force out bugs, catch thermal over-runs and generally torture a kernel in a controlled repeatable manner.

I've also re-structured the tool in several ways to enhance the features and make it easier to maintain.  The cpu stress method has been re-worked to include nearly 40 different ways to stress a processor, covering:

  • Bit manipulation: bitops, crc16, hamming
  • Integer operations: int8, int16, int32, int64, rand
  • Floating point:  long double, double,  float, ln2, hyperbolic, trig
  • Recursion: ackermann, hanoi
  • Computation: correlate, euler, explog, fibonacci, gcd, gray, idct, matrixprod, nsqrt, omega, phi, prime, psi, rgb, sieve, sqrt, zeta
  • Hashing: jenkin, pjw
  • Control flow: jmp, loop
..the intention was to have a wide enough eclectic mix of CPU exercising tests that cover a wide range of typical operations found in computationally intense software.   Use the new --cpu-method option to select the specific CPU stressor, or --cpu-method all to exercise all of them sequentially.

I've also added more generic system stress methods too:
  • bigheap - re-allocs to force OOM killing
  • rename - rename files rapidly
  • utime - update file modification times to create lots of dirty file metadata
  • fstat - rapid fstat'ing of large quantities of files
  • qsort - sorting of large quantities of random data
  • msg - System V message sending/receiving
  • nice - rapid re-nicing processes
  • sigfpe - catch rapid division by zero errors using SIGFPE
  • rdrand - rapid reading of Intel random number generator using the rdrand instruction (Ivybridge and later CPUs only)
Other new options:
  • metrics-brief - this dumps out only the bogo-op metrics that are relevant for just the tests that were run.
  • verify - this will sanity check the stress results per iteration to ensure memory operations and CPU computations are working as expected. Hopefully this will catch any errors on a hot machine that has errors in the hardware. 
  • sequential - this will run all the stress methods one by one (for a default of 60 seconds each) rather than all in parallel.   Use this with the --timeout option to run all the stress methods sequentially each for a specified amount of time. 
  • Specifying 0 instances of any stress method will run an instance of the stress method on all online CPUs. 
The tool also builds and runs on Debian kFreeBSD and GNU HURD kernels although some stress methods or stress options are not included due to lack of support on these other kernels.
The stress-ng man page gives far more explanation of each stress method and more detailed examples of how to use the tool.

For more details, visit here or read the manual.

Read more
Jussi Pakkanen

If you read discussions on the Internet about memory allocation (and who doesn’t, really), one surprising tidbit that always comes up is that in Linux, malloc never returns null because the kernel does a thing called memory overcommit. This is easy to verify with a simple test application.

#include<stdio.h>
#include<malloc.h>

int main(int argc, char **argv) {
  while(1) {
    char *x = malloc(1);
    if(!x) {
      printf("Malloc returned null.\n");
      return 0;
    }
    *x = 0;
  }
  return 1;
}

This app tries to malloc memory one byte at a time and writes to it. It keeps doing this until either malloc returns null or the process is killed by the OOM killer. When run, the latter happens. Thus we have now proved conclusively that malloc never returns null.

Or have we?

Let’s change the code a bit.

#include<stdio.h>
#include<malloc.h>

int main(int argc, char **argv) {
  long size=1;
  while(1) {
    char *x = malloc(size*1024);
    if(!x) {
      printf("Malloc returned null.\n");
      printf("Tried to alloc: %ldk.\n", size);
      return 0;
    }
    *x = 0;
    free(x);
    size++;
  }
  return 1;
}

In this application we try to allocate a block of ever increasing size. If the allocation is successful, we release the block before trying to allocate a bigger one. This program does receive a null pointer from malloc.

When run on a machine with 16 GB of memory, the program will fail once the allocation grows to roughly 14 GB. I don’t know the exact reason for this, but it may be that the kernel reserves some part of the address space for itself and trying to allocate a chunk bigger than all remaining memory fails.

Summarizing: malloc under Linux can either return null or not and the non-null pointer you get back is either valid or invalid and there is no way to tell which one it is.

Happy coding.

Read more

The Ubuntu Developer Summit was held in Copenhagen last week, to discuss plans for the next six-month cycle of Ubuntu. This was the most productive UDS that I've been to — maybe it was the shorter four-day schedule, or the overlap with Linaro Connect, but it sure felt like a whirlwind of activity.

I thought I'd share some details about some of the sessions that cover areas I'm working on at the moment. In no particular order:

Improving cross-compilation

Blueprint: foundations-r-improve-cross-compilation

This plan is a part of a mutli-cycle effort to improve cross-compilation support in Ubuntu. Progress is generally going well — the consensus from the session was that the components are fairly close to complete, but we still need some work to pull those parts together into something usable.

So, this cycle we'll be working on getting that done. While we have a few bugfixes and infrastructure updates to do, one significant part of this cycle’s work will be to document the “best-practices” for cross builds in Ubuntu, on wiki.ubuntu.com. This process will be heavily based on existing pages on the Linaro wiki. Because most of the support for cross-building is already done, the actual process for cross-building should be fairly straightforward, but needs to be defined somewhere.

I'll post an update when we have a working draft on the Ubuntu wiki, stay tuned for details.

Rapid archive bringup for new hardware

Blueprint: foundations-r-rapid-archive-bringup

I'd really like for there to be a way to get an Ubuntu archive built “from scratch”, to enable custom toolchain/libc/other system components to be built and tested. This is typically useful when bringing up new hardware, or testing rebuilds with new compiler settings. Because we may be dealing with new hardware, doing this bootstrap through cross-compilation is something we'd like too.

Eventually, it would be great to have something as straightforward as the OpenEmbedded or OpenWRT build process to construct a repository with a core set of Ubuntu packages (say, minbase), for previously-unsupported hardware.

The archive bootstrap process isn't done often, and can require a large amount of manual intervention. At present, there's only a couple of folks who know how to get it working. The plan here is to document the bootstrap process in this cycle, so that others can replicate the process, and possibly improve the bits that are a little too janky for general consumption.

ARM64 / ARMv8 / aarch64 support

Blueprint: foundations-r-aarch64

This session is an update for progress on the support for ARMv8 processors in Ubuntu. While no general-purpose hardware exists at the moment, we want to have all the pieces ready for when we start seeing initial implementations. Because we don't have hardware yet, this work has to be done in a cross-build environment; another reason to keep on with the foundations-r-improve-cross-compilation plan!

So far, toolchain progress is going well, with initial cross toolchains available for quantal.

Although kernel support isn’t urgent at the moment, we’ll be building an initial kernel-headers package for aarch64. There's also a plan to get a page listing the aarch64-cross build status of core packages, so we'll know what is blocked for 64-bit ARM enablement.

We’ve also got a bunch of workitems for volunteers to fix cross-build issues as they arise. If you're interested, add a workitem in the blueprint, and keep an eye on it for updates.

Secure boot support in Ubuntu

Blueprint: foundations-r-secure-boot

This session covered progress of secure boot support as at the 12.10 Quantal Quetzal release, items that are planned for 13.04, and backports for 12.04.2.

As for 12.10, we’ve got the significant components of secure boot support into the release — the signed boot chain. The one part that hasn't hit 12.10 yet is the certificate management & update infrastructure, but that is planned to reach 12.10 by way of a not-too-distant-future update.

The foundations team also mentioned that they were starting the 12.04.2 backport right after UDS, which will bring secure boot support to our current “Long Term Support” (LTS) release. Since the LTS release is often preferred Ubuntu preinstall situations, this may be used as a base for hardware enablement on secure boot machines. Combined with the certificate management tools (described at sbkeysync & maintaining uefi key databases), and the requirement for “custom mode” in general-purpose hardware, this will allow for user-defined trust configuration in an LTS release.

As for 13.04, we're planning to update the shim package to a more recent version, which will have Matthew Garrett's work on the Machine Owner Key plan from SuSE.

We're also planning to figure out support for signed kernel modules, for users who wish to verify all kernel-level code. Of course, this will mean some changes to things like DKMS, which run custom module builds outside of the normal Ubuntu packages.

Netboot with secure boot is still in progress, and will require some fixes to GRUB2.

And finally, the sbsigntools codebase could do with some new testcases, particularly for the PE/COFF parsing code. If you're interested in contributing, please contact me at jeremy.kerr@canonical.com.

Read more

Over the last couple of weeks I've been working on a set of secure-boot tools. Originally, this project was intended as a quick implementation of an EFI-image-signing utility, but it has since grown a little. I've now added code to help maintain the UEFI signature databases from within a running OS.

A new utility, sbkeysync, reads the current EFI signature databases from firmware, and reads a set of keys from a standard location in the filesystem - for example, /etc/secureboot/keys. It then updates the firmware key databases with any keys that are not already present.

A filesystem keystore will look something like this:

  • /etc/secureboot/keys/PK/<pk-file>
  • /etc/secureboot/keys/KEK/<kek-file-1>
  • /etc/secureboot/keys/KEK/<kek-file-2>
  • /etc/secureboot/keys/KEK/…
  • /etc/secureboot/keys/db/<db-file-1>
  • /etc/secureboot/keys/db/<db-file-2>
  • /etc/secureboot/keys/db/…
  • /etc/secureboot/keys/dbx/<dbx-file-1>
  • /etc/secureboot/keys/dbx/<dbx-file-2>
  • /etc/secureboot/keys/dbx/…

These files need to be in a certain format: signed EFI_SIGNATURE_LIST data. There's two other utilities in the sbtools tree to help to create the key files: sbsiglist and sbvarsign. The following example shows how you'd use these tools to do a basic secure boot key configuration.

An example key setup

If you're interested in trying sbkeysync, the following guide should get you set up. To start, you'll need:

  • A build of the secure boot tools (git repository information below);
  • A kernel with the efivars filesystem. Either build your own kernel with efivars-1f087c6.patch (from Matthew Garrett, with some minor changes), or use linux-image-3.5.0-13-generic_3.5.0-13.14~efivars1_amd64.deb, which should work on Ubuntu 12.04 or 12.10; and
  • A machine with firmware that implements UEFI secure boot, configured to be in setup mode (ie, no PK installed).

Be warned that you're playing with three different layers of development code here: the secure boot tools are new, the efivars implementation hasn't had a lot of review yet, and firmware secure boot implementations are still fairly recent too. I'd recommend against doing this testing on a production machine.

generating keys

We'll generate a test key, and a self-signed certificate:

[jk@pecola ~]$ openssl genrsa -out test-key.rsa 2048
[jk@pecola ~]$ openssl req -new -x509 -sha256 \
        -subj '/CN=test-key' -key test-key.rsa -out test-cert.pem
[jk@pecola ~]$ openssl x509 -in test-cert.pem -inform PEM \
        -out test-cert.der -outform DER

We'll also need a GUID to represent the "key owner". Just generate one with uuidgen.

[jk@pecola ~]$ guid=$(uuidgen)
[jk@pecola ~]$ echo $guid

generating key updates

In order to install this key into the firmware signature databases, we need to create an EFI_SIGNATURE_LIST container for the key, and provide an EFI_VARIABLE_AUTHENTICATION_2 descriptor. The update data will be self-signed, to keep things simple.

First, we create the EFI_SIGNATURE_LIST containing the certificate:

[jk@pecola ~]$ sbsiglist --owner $guid --type x509 --output test-cert.der.siglist test-cert.der

Next, we create a signed update for the EFI signature databases. The signed update consists of the certificate, prefixed with an EFI_VARIABLE_AUTHENTICATION_2 descriptor. The authentication descriptor signs the key data, plus the variable name and attributes. Becuase the variable name is included, we need to generate a separate signed update for each variable (PK, KEK and db):

[jk@pecola ~]$ for n in PK KEK db
> do
>   sbvarsign --key test-key.rsa --cert test-cert.pem \
>     --output test-cert.der.siglist.$n.signed \
>     $n test-cert.der.siglist
> done

creating a keystore

Next up, we'll put our keys into standard locations for sbkeysync to find:

[jk@pecola ~]$ sudo mkdir -p /etc/secureboot/keys/{PK,KEK,db,dbx}
[jk@pecola ~]$ sudo cp *.PK.signed /etc/secureboot/keys/PK/
[jk@pecola ~]$ sudo cp *.KEK.signed /etc/secureboot/keys/KEK/
[jk@pecola ~]$ sudo cp *.db.signed /etc/secureboot/keys/db/

If you'd rather use a different location for the keystore, just use the --keystore and/or --no-default-keystores arguments to the sbkeysync commands that follow.

using sbkeysync

We can now use sbkeysync to synchronise the firmware key databases with the keystore we just created. Do a dry-run first to make sure all is OK:

[jk@pecola ~]$ sbkeysync --verbose --pk --dry-run
Filesystem keystore:
  /etc/secureboot/keys/db/test-cert.der.siglist.db.signed [2116 bytes]
  /etc/secureboot/keys/KEK/test-cert.der.siglist.KEK.signed [2116 bytes]
  /etc/secureboot/keys/PK/test-cert.der.siglist.PK.signed [2116 bytes]
firmware keys:
  PK:
  KEK:
  db:
  dbx:
filesystem keys:
  PK:
    /CN=test-key
     from /etc/secureboot/keys/PK/test-cert.der.siglist.PK.signed
  KEK:
    /CN=test-key
     from /etc/secureboot/keys/KEK/test-cert.der.siglist.KEK.signed
  db:
    /CN=test-key
     from /etc/secureboot/keys/db/test-cert.der.siglist.db.signed
  dbx:
New keys in filesystem:
 /etc/secureboot/keys/db/test-cert.der.siglist.db.signed
 /etc/secureboot/keys/KEK/test-cert.der.siglist.KEK.signed
 /etc/secureboot/keys/PK/test-cert.der.siglist.PK.signed

The output will list the keys were found in the EFI key databases, keys found in the filesystem keystore, and which keys should be inserted to the EFI key databases to bring them in sync with the keystore.

If all looks good, we can remove the --dry-run argument to actually update the firmware key databases. However, be careful here - once a PK is enrolled, the machine is no longer in setup mode, and secure boot is enforced. At the very least, ensure that your firmware setup screens have a facility for returning your machine to setup mode, and/or removing the PK.

Note that some firmware implementations may require a reboot for the changes to take effect in the EFI variables. Before you do this though, you'll probably want to sign your bootloader, so that you can actually boot something!

signing a bootloader

Now that secure boot is enabled, it'd be nice if we actually had something to boot. Although it isn't recommended for production systems, we'll just sign the GRUB2 binary that's already there:

[jk@pecola ~]$ sbsign --key test-key.rsa --cert test-cert.pem \
        --output grubx64.efi /boot/efi/efi/ubuntu/grubx64.efi
[jk@pecola ~]$ sudo cp /boot/efi/efi/ubuntu/grubx64.efi{,.bak}
[jk@pecola ~]$ sudo cp grubx64.efi /boot/efi/efi/ubuntu/

reverting to setup mode

Theoretically, since we have the private-key component of PK, we can revert the machine from user-mode to setup mode. This requires writing an empty signed update to the PK variable:

[jk@pecola ~]$ : > empty
[jk@pecola ~]$ sbvarsign --key test-key.rsa --cert test-cert.pem \
        --attrs NON_VOLATILE,BOOTSERVICE_ACCESS,RUNTIME_ACCESS \
        --include-attrs --output empty.PK.signed PK empty
[jk@pecola ~]$ sudo dd bs=4k if=empty.PK.signed \
        of=/sys/firmware/efi/vars/PK-8be4df61-93ca-11d2-aa0d-00e098032b8c

However, I have not been able to reset the PK on all firmware implementations so far; there may be bugs in the signing tools (or firmware) that prevent the update from being properly verified. Because of this, I strongly suggest checking for the facility to clear the PK through your firmware setup screens before attempting to set the PK.

secure boot tools resources

If you'd like to check out the code, the following links may be useful:

Read more
Colin Ian King

Testing eCryptfs

Over the past several months I've been occasionally back-porting a bunch of eCryptfs patches onto older Ubuntu releases.  Each back-ported fix needs to be properly sanity checked and so I've been writing test cases for each one and adding them to the eCryptfs test suite.

To get hold of the test suite, check it out using bzr:

 bzr checkout lp:ecryptfs  
and install the dependencies so one can build the test suite:
 sudo apt-get install debhelper autotools-dev autoconf automake \
intltool libtool libgcrypt11-dev libglib2.0-dev libkeyutils-dev \
libnss3-dev libpam0g-dev pkg-config python-dev swig acl \
ecryptfs-utils
If you want to test eCrytpfs with xfs and btrfs as the lower file system onto which eCryptfs is mounted, then one needs to also install the tools for these:
 sudo apt-get install xfsprogs btrfs-tools  
And then build the test programs:
 cd ecryptfs  
autoreconf -ivf
intltoolize -c -f
./configure --enable-tests --disable-pywrap
make
To run the tests, one needs to create lower and upper mount points. The tests allow one to create ext2, ext3, ext4, xfs or btrfs loop-back mounted file systems on the lower mount point, and then eCryptfs is mounted on the upper mount point on top.   To create these, use something like:
 sudo mkdir /lower /upper  
The loop-back file system image needs to be placed somewhere too, I generally place mine in a directory /tmp/image, so this needs creating too:
 mkdir /tmp/image  
There are two categories of tests, "safe" and "destructive".  Safe tests should run in such a ways as to not lock up the machine.  Destructive tests try hard to force bugs that can cause kernel oopses or panics. One specifies the test category with the -c option.  Now to run the tests, use:
 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper  
The -K option tells the test suite to run the kernel specific tests. These are the ones I am generally interested in since I'm testing kernel patches.

The -b option specifies the size in 1K blocks of the loop-back mounted /lower file system size.  I generally use 1000000 blocks as a minimum.

The -D option specifies the path where the temporary loop-back mounted image is kept and the -l and -u options specified the paths of the lower and upper mount points.

By default the tests will use an ext4 lower filesystem. One can also run specify which file systems to run the tests on using the -f option, this can be a comma separated list of one or more file systems, for example:
 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper \
-f ext2,ext3,ext4,xfs
And also, instead of running a bunch of tests, one can just a particular test using the -t option:
 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper \
-f ext2,ext3,ext4,xfs -t lp-926292.sh
..which tests the fix for LaunchPad bug 926292
 
We also run these tests regularly on new kernel images to ensure we don't introduce and regressions.   As it stands, I'm currently adding in tests for each bug fix that we back-port and for most new bugs that require a thorough test. I hope to expand the breadth of the tests to ensure we get better general test coverage.

And finally, thanks to Tyler Hicks for writing the test framework and for his valuable help in describing how to construct a bunch of these tests.

Read more
Colin Ian King

A new Ubuntu portal http://odm.ubuntu.com is a jump-start page containing links to pages and documents useful for Original Design Manufactures (ODMs), Original Equipment Manufacturers (OEMs) and Independent BIOS vendors.

Some of the highlights include:

  • A BIOS/UEFI requirements document that containing recommendations to ensure firmware is compatible with the Linux kernel.
  • Getting started links describing how to download, install, configure and debug Ubuntu.
  • Links to certified hardware, debugging tools, SystemTap guides, packaging guides, kernel building notes.
  • Debugging tips, covering: hotkeys, suspend/resume, sound, X and wireless and an A5 sized Ubuntu Debugging booklet.
  • Link to fwts-live, the Firmware Test Suite live image.
 ..so lots of useful technical resources to call upon.

Kudos to Chris Van Hoof for organizing this useful portal.

Read more
~apw

The Internet has been alive with doom saying since the IPv4 global address pool was parcelled out.  Now I do not subscribe to the view that the Internet is going to end imminently, but I do feel that if the technical people out there do not start playing with IPv6 soon then what hope is there for the masses?

In the UK getting native IPv6 is not a trivial task, only one ISP I can find seems to offer it and of course it is not the one I am with.  So what options do I have?  Well there are a number of different types of IPv4 tunnelling techniques such as 6to4 but these seem to require the ability to handle the transition on your NAT router, not an option here.  The other is a proper 6in4 tunnel to a tunnel broker but this needs an end-point.

As I have a local server that makes a sensible anchor for such a tunnel.  Talking round with those in the know I settled on getting a tunnel from Hurricane Electric (HE), a company which gives out tunnels to individuals for free and seems to have local presence for their tunnel hosts.  HE even supply you with tools to cope with your endpoint having a dynamic address, handy.  So with an HE tunnel configuration in hand I set about making my backup server into my IPv6 gateway.

First I had to ensure that protocol 41 (the tunnelling protocol) was being forwarded to the appropriate host.  This is a little tricky as this required me to talk to the configurator for my wireless router.  With that passed on to my server I was able to start configuring the tunnel.

Following the instructions on my HE tunnel broker page, a simple cut-n-paste into /etc/network/interfaces added the new tunnel network device, a quick ifup and my server started using IPv6.  Interestingly my apt-cacher-ng immediately switched backhaul of its incoming IPv4 requests to IPv6 no configuration needed.

Enabling IPv6 for the rest of the network was surprisingly easy.  I had to install and configure radv with my assigned prefix.  It also passed out information on the HE DNS servers, prioritising IPv6 in DNS lookup results.  No changes were required for any of the client systems; well other than enabling firewalls.  Win.

Overall IPv6 is still not simple as it is hard to obtain native IPv6 support, but if you can get it onto your network the client side is working very well indeed.

Read more
Andy Whitcroft

We have uploaded a new Precise linux-ti-omap4 kernel. The most notable changes
are as follows:

* rebased on master branch 3.2.0-19.31

The full changelog can be seen at:

https://launchpad.net/ubuntu/+source/linux-ti-omap4/3.2.0-1410.13

 

Read more
Andy Whitcroft

We have uploaded a new Precise linux-ti-omap4 kernel. The most notable changes
are as follows:

* rebased on master branch 3.2.0-18.29
* a number of CVE fixes

The full changelog can be seen at:

https://launchpad.net/ubuntu/+source/linux-ti-omap4/3.2.0-1408.11

Read more
~apw

After 39 2.6.x releases Linus Torvalds has chosen to revisit the upstream kernel version.  The plan is to release what would have been 2.6.40 instead as version 3.0:

"I decided to just bite the bullet, and call the next version 3.0. It
will get released close enough to the 20-year mark, which is excuse
enough for me, although honestly, the real reason is just that I can
no longe rcomfortably count as high as 40."
When 3.0-rc1 was released the Kernel Team had to decide what version to use for it in Ubuntu.  We typically upload every -rcN release within a couple of days of its release so the pressure was on.  We could simply call it 3.0.0 knowing that all the current scripting would cope, or as 3.0 better matching its official name knowing this would not be plain sailing.  This was not a decision we could delay as in Debian versioning 3.0 < 3.0.0 so we were likely to be committed for Oneiric if we uploaded using 3.0.0.  It is also not clear from upstream discussion what version number the final release will carry, as 3.0 clearly will cause breakage on older userspace.

After much discussion we decided we bite the bullet and upload a 3.0 kernel.  At least we get a chance  to identify problematic applications, while still keeping our options open to move to a 3.0.0 kernel for release should that be prudent.  As expected this was not smooth sailing, not least for the kernel packaging which needed much love to even correctly build this version.  Plus we had to hack the meta packages to allow that to be reversioned later too.

Once successfully uploaded the problem applications started to crawl out of the woodwork:
  • depmod -- the depmod incantion to create the module dependancies identifies the kernel version in its command line but was assuming that a version contained three digits, this lead it to miss the version entirely and rebuild the wrong dependancies;
  • libc6 -- both the runtime and the installation control scripts manipulate the kernel version number, in both cases assuming the version was three digits, enormous fun getting the pending updates installed;
  • ps/top -- when starting the kernel version was checked, and miss decoded triggering a rather nasty sounding version warning whenever they are started;
  • nfs-utils -- when attempting to read and identify the kernel version the nfs-utils would trigger a SIGSEGV and die, triggering boot failures on machines with NFS roots; and
  • lm-sensors-3 -- this package is only compatible with 2.6.5 and above, failed version detection lead to this test failing and sensors being unconfigured.
Those are the ones we have found so far, I am sure there will be more.  If you do find one please file a bug against the failing package but tag it kernel-3.0 then we can find them.

Read more
~apw

During the early part of the Maverick cycle we once again revisited out Union Mount solution.  At that time VFS union-mounts was the hit of the day, set to finally to produce something which might get into the kernel.  Since then the complexity of changing every filesystem to support whiteouts, its invasiveness, and its affects on Posix semantics have lead to it falling by the wayside.  In its place has sprung overlayfs.

overlayfs is a small patch set which is a hybrid of the VFS union-mount approach and that of aufs/unionfs in that it also provides a filesystem.  This greatly reduces the complexity of the patch set, reducing its invasiveness and thus increasing its chances of ever being merged.  So much so simpler is it that your author is actually able to understand and debug it.  Win.

We have been tickling overlayfs for most of the Natty cycle, but with Natty in the can I have had had some time to catch up with its development and help out a little, both with testing and bug fixing.  Culminating today in my being able to inject a kernel containing overlayfs support into an Ubuntu LiveCD and boot it, then update it to the latest Natty, all without error.

overlayfs may shortly be in a mergable state, nirvana for all union mount lovers.  Only time and testing will tell.

Read more
Jeremy Foshee

Hi Folks,
It is that time again. Time for our regularly scheduled Bug Day [0]. Like last time we will be focusing on the bugs in the new state. We had a great amount of work done last time, and I’d like for us to keep that momentum going. There are a number of these bugs that have been improperly set to New when they should be either Triaged or Confirmed. My goal is for us to get as many of them properly set as possible. Additionally, we’d like to perform our basic triage on the ones that are, in fact, new. This includes a brief review of the information that has been provided and a request for what is missing so that we can get as many as possible into the triaged state for further review by the team.
I’d like to take a moment and thank those people on the team for working on the bugs with patches that we have. Through their efforts the number is the lowest it has been since I started here. There has also been quite a bit of effort put into continual review of the regression-proposed bugs. We now have them in a good place. All of this is due to the extra effort the team has put in, in addition to the massive amounts of work they normally do to achieve this. Thanks guys for your efforts. :-)

Parties interested in beginning triage or if you have questions about what to do, or if you just find yourself stuck on a bug and are not certain how to continue, please reach out to me in the #ubuntu-kernel channel on FreeNode. I’m always there, and I am always willing to help. :-)

[0] https://wiki.ubuntu.com/Kernel/BugTriage/BugDay

Read more
Jeremy Foshee

As many of you have experienced this week, I have begun a (somewhat) aggressive review of old bugs for the Kernel package. In some cases bugs are being marked Invalid or Won’t Fix, and I am sure some of you will take issue with that. I’d like to explain some of the rationale behind these decisions and, hopefully, allay the majority of your fears concerning these issues.

First, I am not attempting to lessen the difficulty of any of these reported issues. I am marking the majority of the bugs Invalid due to the overwhelming responses of reporters having ‘similar’ issues. This has resulted in a watering down of the issue as reported by the Original Reporter. I do not wish to convey the notion that those of you encountering those issues are not important, quite the opposite. I am simply wishing to close an issue that is not really able to be solved and asking those of you experiencing issues of a similar nature in unique bugs of your own. This empowers me as a triager to drill down to individual causes of your issue and get that information in front of the team as opposed to trying to understand a variety of, possibly unrelated, issues from a multitude of reporters.

In some cases, you will see the Invalid status being applied to bugs that are very old. In these cases my preference is for affected users to, again, submit me a new bug with all of the relevant information. This gives me all of the benefits I have outlined above in addition to giving us brand new evidence of an issue in the current development release and will give the team a chance to take a fresh look at a longstanding issue.

There are also those bugs that have been Fix Released at some point in the past due to updates to a kernel. Well meaning reporters who experience a seemingly similar issue have, in the past, reopened these under an incorrect status and they have been lost in the shuffle of thousands of other issues. Let me say right now that it is not our policy to reopen Fixed bugs and I consider it a major no-no. :-) To those of you I fuss at for doing this, I apologize in advance.

Some of you will have seen bugs that are set to Won’t Fix. These are issues that are (once again, in most cases) being defined as hardware faults, BIOS issues or unrelated software configuration problems and cannot be fixed by changes in the kernel. In these cases, they are being marked Won’t Fix not our of a desire to keep from actually fixing an issue but in an effort to allow those invested in a product to take the opportunity to solve the problem in the correct location.

I speak from an experience around the team when I say that we would love to solve all of the issues presented to us. The reality is that we are able to take time on very few reported issues directly. The vast majority of fixes come through upstream commits or via new drivers provided by third parties.

To those of you who get annoyed when it seems that all I ever say on a bug is, “Is this still a problem in ” all I can do is shrug. Given the avenues open to us whereby a possible fix could come, I can only ask in that way. Please understand, I’d prefer there were some easier way. :-) Thanks for your patience as the team and I work to make this one of the easiest to use distributions while supporting the most hardware possible.

Finally, I’d like to say that, with such a broad install base and with such a large number of legacy devices available in the wild, there will be some cases where our answer to whether we will be able to support your specific device may end up being “No”. Please forgive me in those instances when I must be the bearer of bad news.

Thank you to all of you who help to triage bugs, update wiki pages and attend learning events put out by the Community team. With all of your help we can continue to improve our chosen operating system and make it more useful to those who would be free.

~JFo

Read more
Jeremy Foshee

Hi everyone!

It is time once again to hold a Kernel Bug Day. I will start blogging these events now instead of only sending out the usual e-mail. I hope that some of you will join in.

This Tuesday, December 7th, the Kernel team will conduct a Bug Day focused on bugs tagged ‘regression-update’. These bugs are an indication that some update to the kernel caused a failure. We’d like for these bugs to be brought current by requesting testing on one of the other versions to include mainline and current development. Your help in requesting that information and in completing the general triage of these bugs would be greatly appreciated. :-)

Information concerning the Bug Day itself can be located on the wiki at https://wiki.ubuntu.com/Kernel/BugTriage/BugDay. As noted there and in the e-mail I sent out earlier, should you desire a head start on the triage day, that contribution too is most appreciated.

I will be available in the #ubuntu-kernel channel of the FreeNode IRC server should you have any questions.

Happy Triaging!

~JFo

Read more
Jeremy Foshee

On Saturday September 11th the Kernel Team will take the first in what I hope to be a series of steps toward educating ourselves and our community in the triage of the thousands of bugs that pass our way daily.

My goal for this event is to begin the process of training those interested in helping with kernel bugs in the way we process our bug tickets. This first event is meant to help us both educate and document. The information on the first ever Triage Summit is located on the wiki here.

As with everything we do, your feedback is appreciated. Please don’t hesitate to send us e-mail to the team list at kernel-team@lists.ubuntu.com or even on the wiki page itself. Your feedback will go a long way toward our plans for future events like this.

The schedule for Saturday is as follows:

1) at 1400 UTC we will hold a General session centered around providing information as to where to locate us, who our upstreams are and where to find them as well as the new subsystem breakdown of bugs to help us gather related issues more easily and get them in front of the right people.

2) at 1500 UTC there will be a session on Graphics and all of its related bits and bytes. There will be a focus on basic, effective triage for this subset as well as locations to find troubleshooting information and further reading on Graphics issues

3) at 1600 UTC there will be a session on audio bugs and the related subsystem parts involved there. This session will also have a basic troubleshooting and location portion so as to guide those interested in triaging audio bugs with further information.

4) at 1700 UTC we will have something of a lightning round in which we briefly discuss the USB/Firewire and Bluetooth stacks as well as helpful debugging and triage steps for this subset of bugs

As stated before, your feedback on these sessions as well as the documentation that will be provided will be most helpful to us. You are also welcome to approach us for other avenues of support. We are always looking for more help on a variety of kernel related areas such as documentation, bugs and spelling/grammar applications :-) so please don’t hesitate to ask how you can help.

I look forward to seeing a great many of you at the Summit on Saturday!

~JFo

Read more
Brad Figg

Meeting Minutes

IRC Log of the meeting.

Agenda

2010-06-04 Meeting Agenda

Outstanding actions from last meeting

None this week.

Lucid Release Status: Bugs

Beta 2 Milestoned Bugs Release Targeted Bugs
linux 6 29
linux-fsl-imx51 0 0
linux-ec2 0 0
linux-mvl-dove 0 0

Blueprints: kernel-lucid-bug-handling

All work items done or postponed, only remaining tasks are for release.

Blueprints: kerne-lucid-kernel-config-review

Over-active beep in VT is under investigation. We expect to write up the configuration report in time for Beta-2.

Blueprints: kernel-lucid-kms

We have ongoing issues with Lid detection, currently we have LVDS lid detection reverted and we are looking at the fallout from that change.

Blueprints: kernel-lucid-suspend-resume

apport — allow us to detect frequency of failure:POSTPONED all other work items done

Blueprints: kernel-lucid-apparmor-development

Still looking for the root cause of Bug #552225, Bug #544764, Bug #549428, Bug #458299

  • Launchpad bug 552225 in apparmor “system bogs down when apparmor is running” [Undecided,New] https://launchpad.net/bugs/552225
  • Launchpad bug 544764 in apparmor “unkillable apparmor_parser” [Undecided,New] https://launchpad.net/bugs/544764
  • Launchpad bug 549428 in apparmor “Triggers permanent high i/o load after upgrade” [Undecided,Incomplete] https://launchpad.net/bugs/549428
  • Launchpad bug 458299 in linux “apparmor_parser: page allocation failure. order:5″ [High,Fix released] https://launchpad.net/bugs/458299

Not positive, but very likely one root cause. It is always replacement related, under high memory pressure/fragmented memory where fast path allocation fails and we fall back to vmalloc for the dfa. It seems to be very much related to the use of swap as I have reports that turning off swap makes it go away and in the end may be a bug in the mm and not apparmor. Also verified that the policy compiler is building isomorphs on multi-policy load and not suffering from a bug, and looked at ways to fix this for M so that we can use multi-profile load and have automated policy verification.

We haven’t been using multi-policy load because we have known that it is generating alternate dfas, and haven’t been able to verify them. I found a few more problems with what I was trying to kick out last week so it got delayed into this week, I have been working through, the checks to send out this morning I expect a few more audit changes will be required yet, and the path generation is some what dependent upon the __d_path patch discussion that I am also rekicking out patches for

Other Release Tasks: Lucid Audio Support

More arsenal work. Wrapping up the survey this week.

Other Release Tasks: Lucid Better Power Mgt

Nothing new this week.

Other Release Tasks: EC2 Lucid Kernel Status

We have Bug #532553, which is just a config oversight as far I can tell

  • Launchpad bug 532553 in linux-ec2 “linux-image-2.6.32-302-ec2 is missing iptables module xt_recent” [Low,Confirmed] https://launchpad.net/bugs/532553

The only question being whether we want to turn it on this late. I (apw) would think anything netfilter related is handy to ahve in that environment, and we have a window to respin for release for somethign which only adds a module which is very low risk. Okay, I’ll send a patch out this morning

Status: Lucid

Beta-2 kernels are in an built. We are not expecting to change the kernel before beta-2. We are expecting some change still before release. Otherwise the worst issues are graphics related to my mind, and mostly getting resolved. Doubt they will all get resolved before release however. We are expecting to be getting a stable update too.

Security & Bugfix Kernels

Hardy 2.6.24-27.68 (security) *
  2.6.24-27.69 (proposed) [ 6] 1/ 3 verifications done (+1)
Intrepid 2.6.27-17.46 (security)
Jaunty 2.6.28-18.60 (security)
Karmic 2.6.31-19.58 (security) *
 
&lbsp; 2.6.31-19.59 (proposed) [ 6] 2/19 verifications done (+2)
LBM 2.6.31-20.22 (updates) *
&lbsp; 2.6.31-20.23 (proposed) [ 6] 0/ 2 verifications done
mvl-dove 2.6.31-211.26 (security) *
&lbsp; 2.6.31-213.27 (proposed) [ 6]
fsl-imx51 2.6.31-108.25 (security) *
&lbsp; 2.6.31-110.26 (proposed) [ 6] 0/ 1 verifications done
1/ 1 failed! #507416
ec2 2.6.31-304.13 (security) *
&lbsp; 2.6.31-110.26 (proposed) [ 6]

Nothing new this week.

Incoming Bugs: Regressions

Current regression stats (broken down by release):

regression-potential (up 130)

  • 223 lucid bugs

regression-update (down 1)

  • 11 karmic bugs
  • 5 jaunty bugs
  • 2 intrepid bugs
  • 1 hardy bug

regression-release (down 1)

  • 52 karmic bugs
  • 21 jaunty bugs
  • 11 intrepid bugs
  • 4 hardy bugs

regression-proposed (no change)

  • 1 karmic bug

Incoming Bugs: Bug day report

The next Kernel Team ‘regression-’ bug day is Wednesday, April 7. Thanks for working on these last week.

Open Discussion or Questions

Just FYI — I (kamalm) am working on the “volume keys never release” issue — seems to be a common problem for various laptop models (bug 550979, bug 420473, bug 374884).
The problem can be fixed in ‘udev’ by adding bits to /lib/udev/… scripts — I plan to produce a patched ‘udev’ package for those bug submitters to test.

  • Launchpad bug 550979 in linux “Volume increase & decrease by function buttons never release on Dell Studio 1558″ [Medium,Confirmed] https://launchpad.net/bugs/550979
  • Launchpad bug 420473 in linux “Coolbox QBook 270-02: volume keys produce more than one key event” [Medium,Incomplete] https://launchpad.net/bugs/420473
  • Launchpad bug 374884 in linux “Keyboard quirk is required for Mitac 8050QDA Fn Volume keys to function.” [Undecided,New] https://launchpad.net/bugs/374884

Read more
Brad Figg

Meeting Minutes

IRC Log of the meeting.

Agenda

2010-23-03 Meeting Agenda

Outstanding actions from last meeting

  • JFo to send out regression bug day announcements on monday
    e-mail went out last week for the schedule to all bug lists that normal bug days go to

  • JFo to do a wiki page on regression bug days
    In progress, carried forward to next week.
    Wiki page for the Bug Day is located here: https://wiki.ubuntu.com/KernelTeam/KernelBugDay.

Lucid Release Status: Bugs

Beta 2 Milestoned Bugs Release Targeted Bugs
linux 13 32
linux-fsl-imx51 0 0
linux-ec2 1 1
linux-mvl-dove 1 2

Blueprints: kernel-lucid-bug-handling

Only outstanding item for beta 2 still open: [leannogasawara] apport — sub-system directed reporting [amber]:INPROGRESS all other beta 2 tasks are complete

Blueprints: kerne-lucid-kernel-config-review

PATA and SATA pull out is done and some debian installer fallout sorted out (thanks smb), and the review of the builtin sub-systems complete.

Blueprints: kernel-lucid-kms

I’ve just released a test kernel which addresses the flickering problem with some Intel graphics devices. There are five bugs open against this: http://bit.ly/aiKuuL. Otherwise we are continuing to debug issues as they appear.

Blueprints: kernel-lucid-suspend-resume

Nothing new to report.

Blueprints: kernel-lucid-apparmor-development

Looking at Bug #458299, Bug #529288, Bug #544764, Bug #549428, which all seem to be inter-related or even possibly symptoms of the same bug

  • Launchpad bug 458299 in linux “apparmor_parser: page allocation failure. order:5″ [High,Confirmed] https://launchpad.net/bugs/458299
  • Launchpad bug 549428 in apparmor “Triggers permanent high i/o load after upgrade” [Undecided,New] https://launchpad.net/bugs/549428
  • apparmor “unkillable apparmor_parser” [Undecided,New] https://launchpad.net/bugs/544764
  • linux “”Kernel Oops” – unable to handle kernel paging request at ffff880323279bf2 RIP is at aa_dfa_match_len+0xd9/0xf0″ [Low,Incomplete] https://launchpad.net/bugs/529288

Blueprints: kernel-lucid-boot-performance

Looks like the remaining ureadahead work will get postponed to Lucid+1 as the userspace bits are not likely to make Lucid. That likely will close off the blueprint.

Other Release Tasks: Lucid Audio Support

Did a bunch of arsenal work, blasted several hundred alsa-driver bugs with automated comments.

Other Release Tasks: Lucid Better Power Mgt

Pushed a new set of packages out and a CFT (call for testing). No news since.

Other Release Tasks: EC2 Lucid Kernel Status

Did lots of testing for Bug #540378, Bug #527208, haven’t been able to trip either so we are looking good for EC2.

  • Launchpad bug 540378 in linux-ec2 “BUG: soft lockup – CPU#1 stuck for 66s! [swapper:0]” [Medium,New] https://launchpad.net/bugs/540378
  • Launchpad bug 527208 in linux-ec2 “ec2 instance fails boot, no console output on c1.xlarge” [High,Confirmed] https://launchpad.net/bugs/527208

Status: Lucid

Lucid is at stable 2.6.32.10+drm33.1. We are prepping to the Beta-2 freeze on Thursday, and expect to upload kernels for that early Wednesday. Anything we need in should be up for review _today_.

Security & Bugfix Kernels

Hardy 2.6.24-27.68 (security) *
  2.6.24-27.69 (proposed) [ 6] 1/ 3 verifications done (+1)
Intrepid 2.6.27-17.46 (security)
Jaunty 2.6.28-18.60 (security)
Karmic 2.6.31-19.58 (security) *
 
&lbsp; 2.6.31-19.59 (proposed) [ 6] 2/19 verifications done (+2)
LBM 2.6.31-20.22 (updates) *
&lbsp; 2.6.31-20.23 (proposed) [ 6] 0/ 2 verifications done
mvl-dove 2.6.31-211.26 (security) *
&lbsp; 2.6.31-213.27 (proposed) [ 6]
fsl-imx51 2.6.31-108.25 (security) *
&lbsp; 2.6.31-110.26 (proposed) [ 6] 0/ 1 verifications done
1/ 1 failed! #507416
ec2 2.6.31-304.13 (security) *
&lbsp; 2.6.31-110.26 (proposed) [ 6]

Karmic ec2 and mvl-dove have no open bug reports but were rebased to the version of master in proposed. Asked for quick re-test.
Karmic fsl-imx51 needs to set the status right. An update for it has been uploaded today (consisting of previous and new changelog).

Incoming Bugs: Regressions

Current regression stats (broken down by release):

regression-potential (up 130)

  • 166 lucid bugs

regression-update (up 1)

  • 12 karmic bugs
  • 5 jaunty bugs
  • 2 intrepid bugs
  • 1 hardy bug

regression-release (down 1)

  • 53 karmic bugs
  • 21 jaunty bugs
  • 11 intrepid bugs
  • 4 hardy bugs

regression-proposed (no change)

  • 1 karmic bug

Incoming Bugs: Bug day report

Today is another Kernel Team ‘regression-’ bug day. Thanks for working on these last week. The next Kernel Regression Bug Day will be on Thursday of this week.

Open Discussion or Questions

  1. Nothing new this week.

Read more
Brad Figg

Meeting Minutes

IRC Log of the meeting.

Agenda

2010-23-03 Meeting Agenda

Outstanding actions from last meeting

  1. JFo to send out regression bug day announcements on monday
    Carried forward to next week.
  2. JFo to do a wiki page on regression bug days
    In progress, carried forward to next week.

Lucid Release Status: Bugs

Beta 1 Milestoned Bugs Release Targeted Bugs
linux 13 30
linux-fsl-imx51 0 0
linux-ec2 1 1
linux-mvl-dove 1 2

Blueprints: kernel-lucid-bug-handling

Nothing new to report.

Blueprints: kerne-lucid-kernel-config-review

Nothing new to report.

Blueprints: kernel-lucid-kms

Testing of the beta has been mostly good so far, there are a couple of “blank screen on install” bugs to look at.

Blueprints: kernel-lucid-suspend-resume

Nothing new to report.

Blueprints: kernel-lucid-apparmor-development

Another lkml push going out today, will also issue pull requests for the bug fixes it includes.
Since beta 1 release have picked up another couple bugs to look at with security team
Bug #544819, Bug #544789, Bug #544764 as well as #529288 which I haven’t been able to
reproduce and currently looks as if it is caused by memory corruption.

  • apparmor “apparmor does not work with midori” [Undecided,New] https://launchpad.net/bugs/544819
  • apparmor “apparmor-logprof not working” [Undecided,New] https://launchpad.net/bugs/544789
  • apparmor “unkillable apparmor_parser” [Undecided,New] https://launchpad.net/bugs/544764
  • linux “”Kernel Oops” – unable to handle kernel paging request at ffff880323279bf2 RIP is at aa_dfa_match_len+0xd9/0xf0″ [Low,Incomplete] https://launchpad.net/bugs/529288

Blueprints: kernel-lucid-boot-performance

Nothing new to report.

Other Release Tasks: Lucid Audio Support

Still working on the bug survey. I’ve moved to working on modifications to arsenal
scripts to do more of the work. I’ve done a bit of refactoring of Bryce’s process-new-bugs.py
script and sent it to him. I’ll take his feedback and appy the changes to our scripts
as well. I’m trying to make it so we don’t fork the scripts quite so much.

Other Release Tasks: Lucid Better Power Mgt

Sent some patches upstream to pm-utils and am waiting on feedback. Will push out a new set of packages when I hear back.

Other Release Tasks: EC2 Lucid Kernel Status

Testing fix for Bug #527208, which also seems to fix Bug #540378

  • linux-ec2 “ec2 instance fails boot, no console output on c1.xlarge” [High,Confirmed] https://launchpad.net/bugs/527208
  • linux-ec2 “BUG: soft lockup – CPU#1 stuck for 66s! [swapper:0]” [Medium,New] https://launchpad.net/bugs/540378

Other than those its looking good.

Status: Lucid

Ok, apw uploaded a kernel including 2.6.32.10 and a newer igb driver. Uploaded meta today so it will probably get offered soonish.
Will gather updates till Friday and then do another upload.

Security & Bugfix Kernels

Dapper 2.6.15-55.83 (security)
Hardy 2.6.24-27.68 (security) *
Intrepid 2.6.27-17.46 (security)
Jaunty 2.6.28-18.60 (security)
Karmic 2.6.31-19.58 (security) *
 
LBM 2.6.31-20.22 (updates) *
mvl-dove 2.6.31-211.26 (security) *
fsl-imx51 2.6.31-108.25 (security) *
ec2 2.6.31-304.13 (security) *

Packages marked with ‘*’ have been uploaded to proposed. Planned acceptance into proposed is tomorrow.

Incoming Bugs: Regressions

Current regression stats (broken down by release):

regression-potential (up 40)

  • 128 lucid bugs

regression-update (up 1)

  • 12 karmic bugs
  • 5 jaunty bugs
  • 2 intrepid bugs
  • 1 hardy bug

regression-release (down 1)

  • 53 karmic bugs
  • 22 jaunty bugs
  • 11 intrepid bugs
  • 4 hardy bugs

regression-proposed (no change)

  • 1 karmic bug

Incoming Bugs: Bug day report

Open Discussion or Questions

  1. A warm welcome from all to Kamal Mostafa, the latest kernel team member.

Read more
Brad Figg

Meeting Minutes

IRC Log of the meeting.

Agenda

2010-09-03 Meeting Agenda

Outstanding actions from last meeting

  1. None

Lucid Release Status: Bugs

Beta 1 Milestoned Bugs Release Targeted Bugs
linux 1 28
linux-fsl-imx51 2 2
linux-ec2 0 1
linux-mvl-dove 1 2

Blueprints: kernel-lucid-bug-handling

Almost done with the analysis of the X debugging pages. JFo will update the team on the current state of the Kernel Team pages and an initial draft of recommended changes to the structure and layout via the e-mail list. Will have the arsenal scripts running (not in dry run mode) by the end of this week.

  • I’m almost ready to present my findings on the X wiki debug pages and how I’d like to restructure the Kernel Team pages. I hope to send something on this to the list this week

  • I am reworking the way the arsenal scripts gather and process bugs so that I can extend the ‘reach’ of the script deeper into the backlog. I’ll probably do a rather intense round of bug processing during the lull after release freeze.

  • I sent out an e-mail to the c-k-t list (meant to send to the k-t list) regarding a wiki page describing the rationale for automated bug processing. Please provide some feedback. I’d like to get this URL integrated into the arsenal scripts soon.

Blueprints: kerne-lucid-kernel-config-review

Discussion thread started on the remaining sub-systems and PATA/SATA driver status. Should have patches for immediatly after Beta-1.

Blueprints: kernel-lucid-kms

We are seeing a lot of issues with i945 and older graphics cards, which includes all the atom netbooks. This is looking to be new occurance of the i915.powersave problems, we will disable powersaving for these ‘older’ cards by default.

Blueprints: kernel-lucid-suspend-resume

Looks like we are getting some useful information as a result of the patch to measure suspend resume times. I am currently looking at the suspend/resume bugs filed recently and trying to get a sense of any common elements we can find.

Blueprints: kernel-lucid-apparmor-development

The latest upstream push is ready go out.

Blueprints: kernel-lucid-boot-performance

The ureadahead patches are with Foundations for testing. Again we should have results in time for the first upload after Beta-1.

Other Release Tasks: Lucid Audio Support

Nothing significantly new this week.

I’m several days into my als-driver bug survey and have looked at almost 200 bugs. I’m basicly going through them one by one and adding tags onto them. If they have not had any action for more than a couple months I’m requesting a test of a dev image and the mainline kernel. I’m pretty amazed at the number of bugs that have not seen any attention since their initial submission.

The biggest buckets are “no sound at all” and “internal mic not working”. There are a few that look like upgrade isses, people running out and buying the latest HW expecting it to “just work”, etc.

I also see that there are a bunch of bugs that are “past expiration”. I asked around about this and was told that the bot that ran around marking these “invalid” had to be turned off because it wasn’t smart enought and was invalidating bugs it shouldn’t. I was told that a new status is being created “Expired” specifically for these bugs and that the bot would be smartened up. This work is supposed to happen this month.

Other Release Tasks: Lucid Better Power Mgt

The release team has pretty much veto’d putting the pmutils functionality in as it didn’t hit b-1. The rest has been postponed due to workload.

Other Release Tasks: EC2 Lucid Kernel Status

Working on Launchpad bug 527208 in linux-ec2 “ec2 instance fails boot, no console output on c1.xlarge” [High,Confirmed] https://launchpad.net/bugs/527208

Have confirmed that it isn’t exhibiting under similar circumstances in rackspace cloud. I haven’t made anymore progress on testing pv-ops.

Status: Lucid

Lucid remains at stable v2.6.32.9, though v2.6.32.10 is now available and will be applied once Beta-1 is over. We have commited patches to expose the DRM backports version such that when v2.6.33.1 DRM patches we can tell that that has occured in bug reports. We have also changed the default settings for the CDROM trays to allow direct removal of disks, userspace will handle this shortly.

All of the main kernels (linux, linux-fsl-imx51, linux-mvl-dove, linux-qcm-msm, and linux-ec2) remain unchanged from last week. linux-ti-omap was just accepted into the archive. Anything which requires a kernel change will have to wait until after beta-1 and will have to pass the abbreviated SRU process (2 acks required).

Security & Bugfix Kernels

Dapper 2.6.15-55.82 (security)
Hardy 2.6.24-27.65 (security)
 
  2.6.24-27.67 (updates)
Intrepid 2.6.27-17.45 (security)
Jaunty 2.6.28-18.59 (security)
Karmic 2.6.31-19.57 (security)
 
LBM 2.6.31-20.22 (updates)
mvl-dove 2.6.31-211.22 (security)
fsl-imx51 2.6.31-108.21 (security)
  2.6.31-108.23 (proposed)[12] 0/ 1 verifications done (+0)
ec2 2.6.31-304.11 (updates)

Security update is expected somewhen today (after some final tests)

Incoming Bugs: Regressions

Current regression stats (broken down by release):

regression-potential (up 30)

  • 88 lucid bugs

regression-update (no change)

  • 11 karmic bugs
  • 5 jaunty bugs
  • 2 intrepid bugs
  • 1 hardy bug

regression-release (no change)

  • 54 karmic bugs
  • 22 jaunty bugs
  • 11 intrepid bugs
  • 4 hardy bugs

regression-proposed (no change)

  • 1 karmic bug

Incoming Bugs: Bug day report

Today is another Kernel Team ‘regression-’ bug day. Thanks for working on these last week. I’ll be looking at the schedule for the rest of this release and proposing more bug days to work out regressions to Pete for inclusion in the schedule. I anticipate that, as we move closer to the release freeze, we will probably do more than one bug day a week, but I want to see what the timeframe looks like before I propose that.

I didn’t get to schedule a community bug day today as I had planned. I will get things ready to hold another bug day against bugs with patches attached for next week.

Open Discussion or Questions

  1. Nothing new this week.

Read more