Some notes and questions on different backends

[milasudril]

On GNU/Linux, there is typically a sound daemon running that has exclusive access to the sound device. This means that it is not guaranteed that the operating system level API (ALSA) will work from user applications. Instead the application must communicate with the sound daemon. There are two sound daemons to choose from: JACKD and PulseAudio. Jackd targets pro-audio usecases and features no extra overhead. It does only mixing and no sample rate conversions. PulseAudio is more geared towards "consumer" audio applications, where low-latency is less important. Since only one sound daemon can be running at a time, the library must support both JACKD and PulseAudio.

I have some experience with JACKD. It uses a callback based API, and buffers are non-interleaved (channels is the major index). Jackd applications registers a client with some name, and then named input and output ports can be added to the client. Then ports can be connected either through an external applilcation such as Catia, or by using the JACK port connection API directly. Jackd also has more callbacks than the audio callback, such as change of sample rate, change of buffer size, port connection events, etc.

• Will port manipulation and extra (optional?) callbacks fit within libstdaudio?
• If not, is it possible to write backend specific extensions?

I have no experience with PulseAudio, so I do not know how that works, but I guess there are some difference in features compared to JACKD. This raises another question:

• Should libstdaudio expose the union of all API:s in use, or should it focus on the common subset?

If it exposes the union, the API becomes bloated. If using the common subset, it will be useless for those that need to take advantage of the specific backend. Maybe there should be two sets of API:s with an independent set of possible backends. This also solves the pull vs push problem. The old MME API is excellent for streaming audio: Simply push data with waveOutAddBuffer and you are ready to go. However, it is not a good pro-audio API as it converts sample formats, and uses interleaved samples. Jackd on the other hand is the PITA for streaming but excellent for pro-audio: No sample format conversions removes all extra latency and potential loss in quality.

Splitting the API in two has more benefits:

You discussed play("foo.wav");. Well, this is not C++. The high-level API in C++ way would be like:

// Playback
std::copy(std::begin(audiobuf), std::end(audiobuf), std::audio::audio_output_iterator{device});

// Record
std::copy(std::audio::begin_record{device}, std::audio::stop_record{device}, std::back_inserter(v));

This is the stream-based approach used by media applications. A pro-audio app will instead use the callback API presented at ACCU, but for it to be useful within pro-audio, there has to be a standardized API for managing the backend (adding/removing ports, add callbacks to port events, sample rate changes, or buffer size changes). This really calls for different abstractions for different usecases.

You can set default ALSA device to be PulseAudio and you can set PulseAudio to use JACK. This way ALSA programs will still output sound when JACK is running assuming PulseAudio goes somewhere in JACK.