Archive for the ‘PulseAudio’ Category
All roads lead to Rome, but PulseAudio is not far behind! In fact, how the PulseAudio client library determines how to try to connect to the PulseAudio server has no less than 13 different steps. Here they are, in priority order:
1) As an application developer, you can specify a server string in your call to pa_context_connect. If you do that, that’s the server string used, nothing else.
2) If the PULSE_SERVER environment variable is set, that’s the server string used, and nothing else.
3) Next, it goes to X to check if there is an x11 property named PULSE_SERVER. If there is, that’s the server string, nothing else. (There is also a PulseAudio module called module-x11-publish that sets this property. It is loaded by the start-pulseaudio-x11 script.)
4) It also checks client.conf, if such a file is found, for the default-server key. If that’s present, that’s the server string.
So, if none of the four methods above gives any result, several items will be merged and tried in order.
First up is trying to connect to a user-level PulseAudio, which means finding the right path where the UNIX socket exists. That in turn has several steps, in priority order:
5) If the PULSE_RUNTIME_PATH environment variable is set, that’s the path.
6) Otherwise, if the XDG_RUNTIME_DIR environment variable is set, the path is the “pulse” subdirectory below the directory specified in XDG_RUNTIME_DIR.
7) If not, and the “.pulse” directory exists in the current user’s home directory, that’s the path. (This is for historical reasons – a few years ago PulseAudio switched from “.pulse” to using XDG compliant directories, but ignoring “.pulse” would throw away some settings on upgrade.)
8) Failing that, if XDG_CONFIG_HOME environment variable is set, the path is the “pulse” subdirectory to the directory specified in XDG_CONFIG_HOME.
9) Still no path? Then fall back to using the “.config/pulse” subdirectory below the current user’s home directory.
Okay, so maybe we can connect to the UNIX socket inside that user-level PulseAudio path. But if it does not work, there are still a few more things to try:
10) Using a path of a system-level PulseAudio server. This directory is /var/run/pulse on Ubuntu (and probably most other distributions), or /usr/local/var/run/pulse in case you compiled PulseAudio from source yourself.
11) By checking client.conf for the key “auto-connect-localhost”. If so, also try connecting to tcp4:127.0.0.1…
12) …and tcp6:[::1], too. Of course we cannot leave IPv6-only systems behind.
13) As the last straw of hope, the library checks client.conf for the key “auto-connect-display”. If it’s set, it checks the DISPLAY environment variable, and if it finds a hostname (i e, something before the “:”), then that host will be tried too.
To summarise, first the client library checks for a server string in step 1-4, if there is none, it makes a server string – out of one item from steps 5-9, and then up to four more items from steps 10-13.
And that’s all. If you ever want to customize how you connect to a PulseAudio server, you have a smorgasbord of options to choose from!
2.1 surround sound is (by a very unscientific measure) the third most popular surround speaker setup, after 5.1 and 7.1. Yet, ALSA and PulseAudio has since a long time back supported more unusual setups such as 4.0, 4.1 but not 2.1. It took until 2015 to get all pieces in the stack ready for 2.1 as well.
So what made adding 2.1 surround more difficult than other setups? Well, first and foremost, because ALSA used to have a fixed mapping of channels. The first six channels were decided to be:
1. Front Left
2. Front Right
3. Rear Left
4. Rear Right
5. Front Center
6. LFE / Subwoofer
Thus, a four channel stream would default to the first four, which would then be a 4.0 stream, and a three channel stream would default to the first three. The only way to send a 2.1 channel stream would then be to send a six channel stream with three channels being silence.
This was not good enough, because some cards, including laptops with internal subwoofers, would only support streaming four channels maximum.
(To add further confusion, it seemed some cards wanted the subwoofer signal on the third channel of four, and others wanted the same signal on the fourth channel of four instead.)
ALSA channel map API
The first part of the solution was a new alsa-lib API for channel mapping, allowing drivers to advertise what channel maps they support, and alsa-lib to expose this information to programs (see snd_pcm_query_chmaps, snd_pcm_get_chmap and snd_pcm_set_chmap).
The second step was for the alsa-lib route plugin to make use of this information. With that, alsa-lib could itself determine whether the hardware was 5.1 or 2.1, and change the number of channels automatically.
PulseAudio bass / treble filter
With the alsa-lib additions, just adding another channel map was easy.
However, there was another problem to deal with. When listening to stereo material, we would like the low frequencies, and only those, to be played back from the subwoofer. These frequencies should also be removed from the other channels. In some cases, the hardware would have a built-in filter to do this for us, so then it was just a matter of setting enable-lfe-remixing in daemon.conf. In other cases, this needed to be done in software.
Therefore, we’ve integrated a crossover filter into PulseAudio. You can configure it by setting lfe-crossover-freq in daemon.conf.
If you have a laptop with an internal subwoofer, chances are that it – with all these changes to the stack – still does not work. Because the HDA standard (which is what your laptop very likely uses for analog audio), does not have much of a channel mapping standard either! So vendors might decide to do things differently, which means that every single hardware model might need a patch in the kernel.
If you don’t have an internal subwoofer, but a separate external one, you might be able to use hdajackretask to reconfigure your headphone jack to an “Internal Speaker (LFE)” instead. But the downside of that, is that you then can’t use the jack as a headphone jack…
Do I have it?
In Ubuntu, it’s been working since the 15.04 release (vivid). If you’re not running Ubuntu, you need alsa-lib 1.0.28, PulseAudio 7, and a kernel from, say, mid 2014 or later.
Takashi Iwai wrote the channel mapping API, and also provided help and fixes for the alsa-lib route plugin work.
The crossover filter code was imported from CRAS (but after refactoring and cleanup, there was not much left of that code).
Hui Wang helped me write and test the PulseAudio implementation.
PulseAudio upstream developers, especially Alexander Patrakov, did a thorough review of the PulseAudio patch set.
This is a technical post about PulseAudio internals and the upcoming protocol improvements in the upcoming PulseAudio 6.0 release.
PulseAudio memory copies and buffering
PulseAudio is said to have a “zero-copy” architecture. So let’s look at what copies and buffers are involved in a typical playback scenario.
When PulseAudio server and client runs as the same user, PulseAudio enables shared memory (SHM) for audio data. (In other cases, SHM is disabled for security reasons.) Applications can use pa_stream_begin_write to get a pointer directly into the SHM buffer. When using pa_stream_write or through the ALSA plugin, there will be one memory copy into the SHM.
Server resampling and remapping
On the server side, the server might need to convert the stream into a format that fits the hardware (and potential other streams that might be running simultaneously). This step is skipped if deemed unnecessary.
First, the samples are converted to either signed 16 bit or float 32 bit (mainly depending on resampler requirements).
In case resampling is necessary, we make use of external resampler libraries for this, the default being speex.
Second, if remapping is necessary, e g if the input is mono and the output is stereo, that is performed as well. Finally, the samples are converted to a format that the hardware supports.
So, in worst case, there might be up to four different buffers involved here (first: after converting to “work format”, second: after resampling, third: after remapping, fourth: after converting to hardware supported format), and in best case, this step is entirely skipped.
Mixing and hardware output
PulseAudio’s built in mixer multiplies each channel of each stream with a volume factor and writes the result to the hardware. In case the hardware supports mmap (memory mapping), we write the mix result directly into the DMA buffers.
The best we can do is one copy in total, from the SHM buffer directly into the DMA hardware buffer. I hope this clears up any confusion about what PulseAudio’s advertised “zero copy” capabilities means in practice.
However, memory copies is not the only thing you want to avoid to get good performance, which brings us to the next point:
Protocol improvements in 6.0
PulseAudio does pretty well CPU wise for high latency loads (e g music playback), but a bit worse for low latency loads (e g VOIP, gaming). Or to put it another way, PulseAudio has a low per sample cost, but there is still some optimisation that can be done per packet.
For every playback packet, there are three messages sent: from server to client saying “I need more data”, from client to server saying “here’s some data, I put it in SHM, at this address”, and then a third from server to client saying “thanks, I have no more use for this SHM data, please reclaim the memory”. The third message is not sent until the audio has actually been played back.
For every message, it means syscalls to write, read, and poll a unix socket. This overhead turned out to be significant enough to try to improve.
So instead of putting just the audio data into SHM, as of 6.0 we also put the messages into two SHM ringbuffers, one in each direction. For signalling we use eventfds. (There is also an optimisation layer on top of the eventfd that tries to avoid writing to the eventfd in case no one is currently waiting.) This is not so much for saving memory copies but to save syscalls.
From my own unscientific benchmarks (i e, running “top”), this saves us ~10% – 25% of CPU power in low latency use cases, half of that being on the client side.
Headsets come in many sorts and shapes. And laptops come with different sorts of headset jacks – there is the classic variant of one 3.5 mm headphone jack and one 3.5 mm mic jack, and the newer (common on smartphones) 3.5 mm headset jack which can do both. USB and Bluetooth headsets are also quite common, but that’s outside the scope for this article, which is about different types of 3.5 mm (1/8 inch) jacks and how we support them in Ubuntu 14.04.
You’d think this would be simple to support, and for the classic (and still common) version of having one headphone jack and one mic jack that’s mostly true, but newer hardware come in several variants.
If we talk about the typical TRRS headset – for the headset itself there are two competing standards, CTIA and OMTP. CTIA is the more common variant, at least in the US and Europe, but it means that we have laptop jacks supporting only one of the variants, or both by autodetecting which sort has been plugged in.
Speaking of autodetection, hardware differs there as well. Some computers can autodetect whether a headphone or a headset has been plugged in, whereas others can not. Some computers also have a “mic in” mode, so they would have only one jack, but you can manually retask it to be a microphone input.
Finally, a few netbooks have one 3.5 mm TRS jack where you can plug in either a headphone or a mic but not a headset.
So, how would you know which sort of headset jack(s) you have on your device? Well, I found the most reliable source is to actually look at the small icon present next to the jack. Does it look like a headphone (without mic), headset (with mic) or a microphone? If there are two icons separated by a slash “/”, it means “either or”.
For the jacks where the hardware cannot autodetect what has been plugged in, the user needs to do this manually. In Ubuntu 14.04, we now have a dialog:
In previous versions of Ubuntu, you would have to go to the sound settings dialog and make sure the correct input and output were selected. So still solvable, just a few more clicks. (The dialog might also be present in some Ubuntu preinstalls running Ubuntu 12.04.)
So in userspace, we should be all set. Now let’s talk about kernels and individual devices.
Quite common with Dell machines manufactured in the last year or so, is the version where the hardware can’t distinguish between headphones and headsets. These machines need to be quirked in the kernel, which means that for every new model, somebody has to insert a row in a table inside the kernel. Without that quirk, the jack will work, but with headphones only.
So if your Dell machine is one of these and not currently supporting headset microphones in Ubuntu 14.04, here’s what you can do:
- Check which codec you have: We currently can enable this for ALC255, ALC283, ALC292 and ALC668. “grep -r Realtek /proc/asound/card*” would be the quickest way to figure this out.
- Try it for yourself: edit /etc/modprobe.d/alsa-base.conf and add the line “options snd-hda-intel model=dell-headset-multi”. (A few instead need “options snd-hda-intel model=dell-headset-dock”, but it’s not that common.) Reboot your computer and test.
- Regardless of whether you manage to resolve this or not, feel free to file a bug using the “ubuntu-bug audio” command. Please remove the workaround from the previous step (and reboot) before filing the bug. This might help others with the same hardware, as well as helping us upstreaming your fix to future kernels in case the workaround was successful. Please keep separate machines in separate bugs as it helps us track when a specific hardware is fixed.
Notes for people not running Ubuntu
- Kernel support for most newer devices appeared in 3.10. Additional quirks have been added to even newer kernels, but most of them are with CC to stable, so will hopefully appear in 3.10 as well.
- PulseAudio support is present in 4.0 and newer.
- The “what did you plug in”-dialog is a part of unity-settings-daemon. The code is free software and available here.
Up until now, we’ve been using Android’s AudioFlinger for playing back and recording audio. Starting with tomorrow’s image, that is no longer true. Instead we’re talking directly from PulseAudio to ALSA, or the Android audio HAL when necessary.
In short, here’s how PulseAudio now works:
- For normal playback and recording, PulseAudio talks directly to alsa-lib, just as on the desktop.
- For detecting whether a headphone/headset is plugged in or not, PulseAudio now has code for reading that from the Android kernel, through the “switch” interface.
- For normal mixer setup, we use ALSA UCM mixer files.
- For setting up voice calls, we talk to the Android Audio HAL through a PulseAudio module.
This provides somewhat of a compromise between features and porting effort: By using the ALSA library whenever we can, we can access PulseAudio’s timer scheduling and dynamic latency features. Having the straightest path possible for playing back music should help efficiency (and in extension, battery life). At least in theory – we haven’t actually done measurements.
Using the Audio HAL for everything mixer related would have been optimal, but it turns out that the audio HAL is too smart: it refuses to set up the mixer, unless PCM data is also sent to it, which is what we wanted to avoid. So then we had to set up the mixer manually too. However, we still could not avoid using the Audio HAL altogether: when starting and stopping voice calls, the Audio HAL talks to the modem and other components in the kernel to route the voice call between the modem and the sound card. Hence we ended up with this compromise approach.
At the time of this writing, this is working best on Nexus 4. The Galaxy Nexus works for the most part, except for bug 1217072. I intend to add Nexus 7 support shortly. If anyone wants to help testing Nexus 10, let me know.
For porters: if you need to do the same
Unfortunately, this means some additional work for porters, because you need to write UCM mixer files. What’s worse, UCM is lacking good documentation. For that reason, I hesitated somewhat before deciding to actually use UCM at all, but it’s the closest we have to a standard for setting up mixers on embedded devices right now.
But to give you a two-minute crash course in UCM and how it’s used in Ubuntu Touch – start by having a look in /usr/share/alsa/ucm/apq8064-tabla-snd-card/ directory. You’ll need to create a similar directory for your device. You’ll find the right directory name if you look in /proc/asound/cards.
Second, look at apq8064-tabla-snd-card.conf. Rename and copy into your own UCM directory. If you’re making a tablet image (that can’t make voice calls), you can remove the VoiceCall part (and the corresponding file).
Third, look at the HiFi file. This is where all fun happens. Notice the device names, which are hardcoded into telepathy-ofono and need to match: “Speaker”, “Earpiece” and “Headphone” for playback, plus “Handset” and “Headset” for recording.
Fourth, if you need voice calls, also look at the VoiceCall file. Btw, the verb names “HiFi” and “VoiceCall” also need to match.) This is largely empty, because the mixer setup is handled by the Audio HAL, but there is a twist here that took a while to get right: For PulseAudio’s UCM to work, it needs to open a PCM device. However, at the time where UCM tests this, the voice call is not yet set up. So, you might need to set up the mixer just a little, so that the PCM can open. (On desktops, PCM can always open, regardless of mixer state. This is not always true on embedded devices, that are using ASoC.) It’s a bonus if you can find a PCM that actually plays back audio, because then you can get notification sounds while on the phone.
And this concludes the two minute crash course – happy porting!
(Side note: Sorry if the permalink – or comment pages etc – to this blog leads you to a blank page. I’ve reported the bug to the relevant team in Canonical, but at the time of this posting, they are looking into it but have not yet fixed it.)
The first PulseAudio conference is approaching quickly. This is a shoutout for people who might be interested, but missed the mailinglist announcement.
The conference will be Friday 2nd November 2012, and colocated with Ubuntu Developer Summit and Linaro Connect, in Bella Center, Copenhagen.
There is no attendance fee, but you’ll need a UDS or LC registration (which are also free) to be able to get into the conference area. (Note: LC might be the safer bet here as UDS ends on Thursday)
There had been a few topics brought up on the mailinglist, but we welcome anyone who would like to contribute to constructive discussions and help to shape the future of PulseAudio!
If you would like to attend, how about you write an email to the pulseaudio-discuss mailinglist with the topic(s) you want to bring up, and maybe a small presentation with your background and interests? Of course, if you mostly want to listen in rather than bring your own topics, that’s okay too!
The audio stack in Linux/Ubuntu evolves over time. What used to be good advice is not necessarily good advice anymore. (That also means, that if you happen to read this blog post in 2019 or something, don’t trust it!)
Here are some things that people try, and sometimes they even fix the problem, but are often bad in one way or the other. Or at least, they have side effects one needs to be aware of. So – while there are valid exceptions, as a rule of thumb, don’t do the following:
5. Don’t add your user to the “audio” group
A user has access to the audio card if that person is either logged in – both VT and GUI login counts, but not SSH logins, or if that user is in the “audio” group. However, on the level of access we’re talking about here, only one user has access at a time. So the typical problem scenario goes like:
- User Homer has an audio issue, and tries to fix it by adding himself to the audio group. This doesn’t help to resolve the problem.
- Homer discovers his audio is muted, and unmutes it. Happy to have his audio issue resolved, he forgets he’s still in the audio group, or doesn’t realise it leads to problems.
- User Marge comes and wants to borrow the computer. Homer does a fast-user-switching so Marge can log in.
- Because Homer is in the audio group, he has still access to the audio device. If some software, e g PulseAudio, has the audio device opened, it blocks access to other software trying to use it.
- Now Marge has an audio issue!
I’ve written a longer article about the audio group here. In short, there are some usages for it, including that it is also the standard group name for assigning realtime priorities when used together with JACK. But don’t leave a user in the audio group unless you have a good reason.
4. Don’t try different “model” strings
A common way to try to get HDA Intel soundcards to work is to edit /etc/modprobe.d/alsa-base.conf and add the following line:
options snd-hda-intel model=[something]
…where [something] are values you find in some file. Contrary to official documentation, this is in most cases obsolete. In particular, avoid model=generic – that is almost guaranteed to give you trouble. In many cases, when trying different models, you will find that you might fix one thing but break another.
In fact, there is only one model to try, and that is model=auto. If your machine happen to be one of those quirked to use an older model parser, changing to model=auto can improve the situation.
It still happens that BIOS/UEFI assigns the wrong values to pin nodes, which causes an output or input not to work correctly. If so, I recommend trying to tweak this with hda-jack-retask.
In some cases, trying different modules can actually be okay – sometimes, these models point to lightweight fixups instead of the earlier, more heavyweight code that was used in previous kernels. (In this context, I have to mention that Takashi Iwai has done a fantastic job of converting the older models to the newer auto-parser.)
3. Don’t upgrade ALSA drivers by following random blog posts
I’ve seen far too many people reporting bugs on Launchpad where they’ve been following some random blog post that tells you how to upgrade ALSA, and are having audio issues as a result. These guides are of varying quality and often come without good uninstall instructions, so you have no way to revert in case the upgrade did not solve your problem, or broke something else.
First, something not everybody is aware of: 95% of ALSA code is in the kernel, and follows the kernel’s release cycle. That means that even if “/proc/asound/version” says something that was released a year or two ago, don’t panic. It’s the kernel release that tells you how new your sound drivers are, so if you have a new kernel, and you see an ALSA release coming out, you are unlikely to gain from an upgrade.
In some case you do have an old kernel, and newer sound drivers can be worth a try. The Ubuntu Audio Developer’s team provides daily snapshot drivers for HDA Intel cards. Guide is available here and it also comes with proper uninstall instructions.
In the past we have also provided drivers for other cards, but due to the maintenance required to keep this up-to-date, in combination with that the vast majority of people’s bugs concern HDA Intel anyway, this support has been discontinued.
2. Don’t purge PulseAudio
First, PulseAudio itself isn’t perfect, some of the bindings to PulseAudio aren’t perfect, and some of the drivers are not perfect in the way PulseAudio wants to use it either. So there might be valid reasons to temporarily move it out of your way, even if it would be better to actually fix the problem and submit a bug fix patch (if you’re capable of doing so).
But don’t try uninstalling the PulseAudio package, as it has far too many dependencies.
If you just need direct access to your sound card, you can run the “pasuspender” command. You can either run “pasuspender” (in a terminal) to make PulseAudio stay away for the duration of the application. Or if you think that’s simpler, just run “pasuspender bash” (in a terminal), start your application through the menu/dash/whatever you prefer, and when you’re done, write “exit” in the terminal.
If you need to stop the PulseAudio process completely, execute these commands:
echo autospawn=no > ~/.pulse/client.conf
If you need PulseAudio back again, remove ~/.pulse/client.conf, then try to start an application that uses PulseAudio, and it should start automatically.
Unexpected side effects:
- The Gnome sound settings, the sound indicator and the volume up/down keys relies on PulseAudio, so they won’t work when PulseAudio is off.
- PulseAudio mixes audio, so that means that only one application at a time can output audio if PulseAudio is disabled (and you aren’t using some other sound server).
- Several applications have PulseAudio backends. Some of them will need reconfiguration to use ALSA directly, some will just automatically redirect themselves, and some won’t work at all.
- Bluetooth audio might not work without PulseAudio.
1. Don’t replace ALSA with OSS
OSS was the standard used before ALSA came along. These days, ALSA is much better, both when it comes to hardware support, and when it comes to how much software that supports outputting sound to either sound system. OSS is also entirely unsupported, at least by Ubuntu. In addition, I’m not sure exactly how to get back to a working system after you’ve tried OSS…!
If you know your problem is in ALSA, either drivers or userspace, try to track down and/or fix the bug, and talk to us about it. If you’re running Ubuntu, file a bug against the alsa-driver package. You can also contact the alsa-devel mailinglist. While we won’t guarantee responses due to the high volume of bugs/traffic, we are often able to help out.
Note 1. HDA Intel cards are the built-in audio inputs and outputs on your motherboard (at least if you bought your computer after ~2006 or so). HDMI and DisplayPort audio are also HDA Intel cards, but they are covered in more detail here.
Note 2. I have had some problems with spammers posting spam comments to my blog post. I don’t want to spend too much time just reading spam and marking it as such, so I might close for comments in a relatively short period. Sorry for the inconvenience.
- Missing speakers or internal mic in sound settings?
- Seeing a S/PDIF device show up for your USB device?
- Having no or extremely low sound from your internal microphone?
If so, I might need your help to be able to fix it for you!
In Ubuntu 12.04 – and very likely, other recent distribution releases as well – there are at least these three audio bugs that need manual quirking for every machine. This means that you submit your hardware info in the meta-bug, I will look in that info for some ID numbers and include them in a list of devices for which a specific workaround has to be applied. We can’t apply the workaround for every device, as that could potentially cause problems for other devices.
So if you’re suffering from one of the bugs I’m describing below, I could use your help to make sure they are fixed for future releases of Ubuntu and ALSA/PulseAudio.
Missing speakers or internal mic in sound settings
If you’re suffering from one of the following problems:
1) In the sound settings dialog, when you plug your headphones into the designated 3.5 mm jack, there is a “Headphones” device correctly shown, but when you unplug it, there is no “Speaker” device shown, even though you have internal speakers on your computer.
2) In the sound settings dialog, when you plug your external microphone into the designated 3.5 mm jack, there is a “Microphone” (or similar) device correctly shown, but when you unplug it, there is no “Internal Mic” device shown, even though you have an internal mic on your computer.
Then you might be suffering from bug 946232. Please look in that bug for your device in that list, and if it is not there, try the workaround and if it’s working, add your alsa-info as suggested in the bug. Thanks!
S/PDIF device showing up for your USB device
If you have a USB headset, or another USB device that does not have any S/PDIF output, but yet sometimes there is an extra device called “Digital Output (S/PDIF)” for that device in Sound Preferences. If so, you’re likely suffering from bug 1002952. At the time of this writing there are also a few devices that are on their way to becoming fixed, if you have one of these, please help out with testing the proposed repository. There is more information in bug 1002952 about that.
Inverted Internal Mic
In this case, you have an internal mic showing up, but it seems not to work: it’s either completely silent, or you can possibly pick a very small sound, with much background noise, even though you have set gain to maximum.
There is something you could try. Install the pavucontrol application, start it and go to the “Input Devices” tab. Unlock the channels (click the keylock icon), then mute the right channel while keeping the left channel at the volume you want.
If the internal mic is now working correctly, you have an inverted internal mic, so that your right channel cancels out the left one. (Why the hardware is constructed that way is beyond me.)
This seems to be most common on Acer Aspire laptops, but I’ve seen them on other laptops as well, and fixing these are more of a long term project as they are non-trivial kernel patches. So far I’ve created a patch for Thinkpad U300s. Anyway, I would like to track the remaining ones in bug 1002978, so please add your system there according to the instructions.
Last but not least – thanks in advance for helping Ubuntu, as well as the greater Linux ecosystem (I’ll make sure that the patches pushed into Ubuntu gets upstream as well)!
Ok, for those of you who just want it up and working, I’m including a quickstart section before we dive into the details:
1) If you have an ATI/AMD or NVidia card, you need proprietary drivers.
2) You need to activate your secondary screen. For Intel, this is done in the regular “Screens” dialog, and on NVidia this is done in the nvidia-settings dialog. (I haven’t tested fglrx.)
3) You need to select the HDMI/DisplayPort output in the sound settings dialog, which is quickest reachable from the sound indicator.
Can’t we switch audio output automatically?
Choosing whether to automatically switch to HDMI/DisplayPort – essentially, switching sound to use the HDMI/DisplayPort whenever that screen is activated – is not trivial. It is not obvious to me whether the user wants to do that, or not. And in fact, in Ubuntu 11.10, we did switch, but only for some cards. And we did not switch back when the screen was deactivated. After a discussion where different opinions were voiced, I reached the conclusion that given the current pieces of infrastructure in place, the best option would be to disable automatic HDMI/DisplayPort switching for Ubuntu 12.04.
The problem of four devices
As mentioned in an earlier post, much HDMI/DisplayPort hardware have phantom outputs, and there is no way we know what outputs are real until something is plugged in. With the new sound settings UI in Ubuntu 12.04, we finally have a good user experience in this scenario: Only the outputs that are actually plugged in and possible to activate will be shown.
Sound settings in Ubuntu 12.04
Most of the code to activate HDMI/DisplayPort audio is in the video driver, rather than the audio driver. Therefore, if this is not working, it is more likely that the problem is within the video driver.
It is also notable that the open source driver for ATI/AMD (called radeon), has experimental support for HDMI/DisplayPort audio, at least for some cards. It is disabled by default, but you can activate it by adding radeon.audio=1 as a kernel boot parameter.
PulseAudio 2.0 is soon to be released (hopefully). PulseAudio 2.0 and Ubuntu 12.04 have the same feature set when it comes to HDMI/DisplayPort audio support.
The new sound settings UI in Ubuntu 12.04 has not yet been upstreamed.
As a part of the Ubuntu Hardware Summit, I held a presentation on the topic “audio debugging techniques”, focused on HDA Intel cards. I also wrote down some notes for some of those slides. I share the slides and the notes with the hope that you will find the information useful if you run into troubles with your audio hardware.
Audio stack overview
The audio stack can seem a bit complex, but first look at the line all the way from the applications to the hardware. This is the optimal audio path. If the audio path is different, complexity will increase and you might run into undesired behaviour, such as one application blocking another from playing audio. There are valid exceptions though – we have a separate sound server for professional, low-latency audio. But that’s outside the scope of this presentation.
Let’s start from the top. On the top we have different kinds of audio applications, which talk to PulseAudio. GStreamer is a library to help media playback, it can for example decode ogg and mp3 files. PulseAudio mixes these audio streams and send them down to the kernel. The ALSA library and the ALSA kernel core do not do much here but send the audio pointers through. The HDA controller driver is responsible for talking directly to the hardware, and so it sets up all necessary DMA streams between the HDA controller and memory. The HDA controller driver also talks to the HDA codec driver, which is different for every codec vendor.
As some of you probably know, between the HDA controller – which is a part of the southbridge in most computers – and the HDA codec, a special HDA bus is used. This means that the only way we can talk to the codec is through the controller.
Controlling audio volume goes the same path. When you use your volume control application, it controls PulseAudio’s volume. PulseAudio in turn modifies the volume controls being exposed by the kernel, and the kernel in turn talks to the hardware to set volume control registers on the codec. There are two levels of abstraction here: first, the kernel might choose not to expose all of the hardware’s volume controls, and second, PulseAudio exposes only one big volume control which is the sum of some of the volume controls the kernel exposes. So there is filtering on two levels.
Audio stack overview – codec
Let us have a look at the HDA codec chip and how its internals are represented to the driver. The codec is constructed as a graph, and on this slide one of the more simple HDA codec graphs is shown (just because it would fit the screen). A while ago upstream made a small program to extract this graph from the codec and make a picture of it. Thanks to Keng-Yü, who works for Canonical in Taipei, this tool is available as a package in Ubuntu 11.10. Just install the “codecgraph” package.
In this graph we have nodes correspondings to DACs, ADCs, mixers, and pins. In this example we can see what pins are connected to which DACs by following the solid line. The dotted line shows a connection that is possible but not currently active.
As the Linux codec driver code grows more intelligent, we depend more and more on this information to be accurate. This way we do not hard code as much in the driver, so we can adapt to future codecs without having to rewrite much code.
The information coming from the codec is usually correct. One problem we have from time to time is though, is that sometimes chip vendors add features which they choose not to document in this graph (and not in any other way either). There is a mechanism called “processing coefficients” in the specification, where the vendor can add its own functionality without telling anyone. When that happens, and it is required to use these undocumented “processing coefficients” to enable all inputs and outputs, we usually run into difficult problems that require vendor support to resolve.
Also, in some cases the graph cannot describe the functionality needed, e g if some hardware is depending on special pins on the codec. We need to know about this when it happens, so we can support it in the driver. So if you are a hardware designer, my message is: Try to use the standard way of doing things as much as possible. Do this and it will work out of the box on Linux, and likely other operating systems as well. If you do anything special, you’re causing headache for driver writers, possibly causing a slower time to market.
An example of this would be how you control external amplifiers: you can use the EAPD pins, which is the standard way, and you can use GPIO pins, ACPI, or anything else, that will be more problematic and require special driver support.
Pin configuration default
We also depend on information from the writers of BIOS/UEFI, i e the computer’s firmware. As a hardware designer, you have the freedom to choose which pins of the codec that go to what physical jack. You might decide that you want a digital out, or you decide that this machine should not have that functionality, and then you leave that pin unconnected.
Then the firmware engineer needs to know this, and program this into the codec when the computer boots. This is done by setting the “Pin Configuration Default” register. This register tells us not only the device type (headphone, mic, etc), but also the location (Internal, External, Docking Station), the color, and the channel mapping (to use for surround functionality).
Several years ago, we did not read this register much, but these days, we depend on that for all new computers for setting up the codec correctly. So what do we do if this register is wrong? Well, if we work with hardware pre-release, there might be a chance we can feed this information back to the firmware writers so they can correct the problem. If the hardware is already released, we have to create a “quirk”. This means that the driver overrides the firmware’s pin value(s) and instead uses its own value.
Because this value is so important, I’ve written an application where you can try out different combinations of this register.
One of the most common problems with getting audio up and running on Linux is to make sure the mixer is correct. Typical symptoms of this would be that some outputs are working where others are not, or that there is something wrong with the volume control.
Here are some initial checks of these problems. We do this at the two levels of mixer abstraction. First, let’s have a look at the PulseAudio volume control. You can do that in Gnome’s volume control application.
Also, PulseAudio controls the volume of mixers at the ALSA level. You can see how this works by starting the alsamixer program. In this program, you can also see additional sliders, which you can also use to verify that they are in the correct to enable sound. You start alsamixer from a terminal (in Ubuntu the quickest way to launch a terminal is the Ctrl-Alt-T shortcut).
Mixer control names
So let’s look at these two abstraction levels in more detail and how you can inspect what is actually going on. First, let’s look at the codec level. If you are familiar with the codec’s nodes and how they are connected, e g by running “codecgraph”, you can also find out which ALSA level controls that are connected to which nodes on the codec. This is done by inspecting the “codec proc” file. Every codec in the system has this file, and its name is made up of the sound card name, and the codec’s address on the HDA bus. In this file, you can also see a lot of other information about the codec.
So next, we will also take a look at PulseAudio’s abstraction of these controls. This is done by looking at the files in /usr/share/pulseaudio/alsa-mixer. In this case, if we look at /usr/share/pulseaudio/alsa-mixer/paths/analog-output-headphones.conf, you can e g find the sections [Element Master] and [Element Headphones]. That means that the ALSA-level controls “Master” and “Headphones” are being merged in PulseAudio’s volume control when the “Headphones” port has been selected.
So these two places are the keys to understanding what is going on when you have mixer problems.
So up next is when you have problems with the streaming. That is usually shown as the audio is breaking up, crackling or glitching. Unfortunately these problems are typically quite hard to resolve.
Sometimes this can be a bug in PulseAudio, or in the driver. But more often the problem is on either the application side or the hardware side.
If an application is not submitting data to PulseAudio in time, the PulseAudio has no audio to play back, so therefore playback breaks up. Once some more data has reached PulseAudio, it starts playback again, and so playback is started and stopped repeatedly.
The other problem could be with bad position reports from the hardware. PulseAudio depends on being able to ask the hardware for its current position at all times, and this should be sample accurate. You can test this by trying to run PulseAudio with timer scheduling disabled, in this case PulseAudio will rely more on DMA interrupts and less on position reports. However, this will also make PulseAudio draw more power than necessary from the machine, so please avoid this if you can.
When I try to debug these problems I usually start with making a PulseAudio verbose log. It often takes some knowledge and experience to be able to analyze this log though.
Over the last six months or so, one of the things I’ve been working with is trying to get better jack detection handling, throughout the audio stack.
“Jack sensing” in this context means what to do when something has been plugged in, or unplugged.
When this happens, an interrupt (IRQ) is triggered and control is passed to the HDA codec driver. The driver takes the first action itself. Now, this is an area, unfortunately, when things differ a lot between different drivers, mostly between different vendors, but also between different chips of the same vendor, or even between configurations of the same chip.
But as a general rule, and for the most common vendors – that means Realtek, IDT and Conexant – these rules are the ones that are followed:
- For headphones – when you plug them in, the Internal Speakers are muted. Remember, this is still all at the kernel level.
- For what we’re doing with Line Outs – it’s not completely standardised everywhere yet, but it seems upstream is leaning on having Headphones mute Line Outs and having Line Outs mute Internal Speakers by default. Some drivers also have a special control where the automute behaviour can be changed.
- For Microphones – the only rule here is that if we have only one internal microphone and one external microphone, the external microphone takes over when you plug it in, and the internal microphone regains control when you unplug. Should there be any other inputs, e g two external mic jacks, or a line in jack, no autoswitching is done at the kernel level.
After this has been done, a signal is sent to userspace. Hopefully – this also varies between vendors. We’ll get back to that. What’s new in Ubuntu 11.10, is that this signal is being picked up by PulseAudio. This is important, because it enables PulseAudio, to switch port for volume control. So this means, when you press your media keys (or use the sound menu) to control your volume, you control your headphone’s volume when you have headphones plugged in, and your speakers’ volume when your headphones are unplugged.
So this not working properly, is one of the more common problems. I have written a small tool that helps you to debug whether this issue is in hardware or software. This tool is called “hda-jack-sense-test”. This program sends the “get pin sense” command to each codec and outputs the results. I actually had use for it earlier this week, and confirmed that it was a hardware issue: although the headphones were unplugged, the “get pin sense” command returned that the headphones were being plugged in and unplugged all the time.
If you can confirm that things are working at this level, you can also look in “Sound settings” to see if the port (this is known as a “connector”) is automatically switched whenever headphones – or microphone – is plugged in. If it is not, the most common cause is that kernel driver does not notify userspace correctly about that change.
One of the most common problem with HDMI these days are with newer chips supporting more than one output. These outputs could be HDMI, DisplayPort or DVI (with audio supported through a DVI to HDMI adapter). NVidia has supported four outputs for quite some time and Intel has supported three. But usually, not all of these are actually connected on the board.
Now, the problem is: How do we know what pin to output to? And the answer is, that there is no good way to figure that out until something is actually plugged in.
If you remember me talking about the pin config default earlier, you would say that maybe the graphics chip could mark the pins not connected to anything. If this was done, it would be a great start (and if they are, we make use of it to hide the outputs that are marked as not connected), but unfortunately, more often than not, these pins are set up as all pins connected and present. So if you write firmware for internal or external graphics cards, please do set up these pins.
So if we don’t know, what do we do? Well, here’s also work in progress at the userspace level. First, PulseAudio has to probe how many ports there are. Then we can use the new jack detection feature, to determine what has actually been plugged in. I’m currently working on redesigning the sound settings dialog so that the ports that are not plugged in will be actually hidden from the dialog, and I hope this will land in Ubuntu 12.04 which will be released in April next year.
And a final note, just so you don’t forget it: For NVidia and ATI, they both require proprietary video drivers to enable HDMI and DisplayPort audio. The ATI driver used to have support for some of the cards in its open source driver, but this feature was recently removed because they had some problems with it.
Intel has no proprietary drivers at all, so there it works with the standard open source driver.