The main concept for audio processing are filters and sinks.

Filter is an object that either generates audio samples (a source filter) or an object that takes audio samples and modifies them in some way.

Sink is a special kind of filter that consumes the samples (and plays them out, stores to a file, …).

The processing them takes place in a form of a filter chain, which usually consists of a source filter, arbitrary number of other filters and a sink at the end.

Simplified interface for creating a filter chain is provided by the class iimavlib::filter_chain.

Consider this example:

using namespace iimavlib;
auto sink = filter_chain<WaveSource>("file.wav")
        .add<SimpleEchoFilter>(0.2)
        .add<PlatformSink>(device_id)
        .sink();

This creates a filter chain that takes a file (file.wav), adds an echo (with delay 0,2s) and then adds a sink to play the file through an platform specific audio device (with id device_id).

file.wav → echo → output

The reason for the trailing .sink() is to ensure we really have a sink. It return a pointer to (actually a shared_ptr<> to) the last object in the chain, provided it's a sink. If the last object is not a sink (and thus the chain is not complete), it returns an empty pointer.

It's also possible to create parts of the chain separately. Following code should yield the same result as the previous one:

using namespace iimavlib;
auto chain = filter_chain<WaveSource>("file.wav");
chain.add<SimpleEchoFilter>(0.2)
chain.add<PlatformSink>(device_id)
auto sink = chain.sink();

To create a new filter, you need to inherit from iimavlib::AudioFilter class and implement it's do_process method.

Let's walk-through a simple example from the library - SineMultiplyFilter. This filter takes samples and multiplies them with a sine function with specified frequency.

SineMultiplyFilter.h

#ifndef SINEMULTIPLY_H_
#define SINEMULTIPLY_H_
 
#include "iimavlib/AudioFilter.h"
namespace iimavlib {
class SineMultiply: public AudioFilter {
public:
	SineMultiply(const pAudioFilter& child, double frequency);
	virtual ~SineMultiply();
private:
	virtual error_type_t do_process(audio_buffer_t& buffer);
	double frequency_;
	double time_;
};
}
#endif /* SINEMULTIPLY_H_ */

So what does it say here?

class SineMultiply: public AudioFilter {

SineMultiply(const pAudioFilter& child, double frequency);

private:
    virtual error_type_t do_process(audio_buffer_t& buffer);

    double frequency_;
    double time_;

The important points are the constructor that HAS TO take first parameter of const pAudioFilter& in order to be usable in the filter_chain helper class and then it can take any other arguments.

Other important part is the method do_process that has to be implemented to do the real work.

Let's now look at the implementation:

SineMultiply::SineMultiply(const pAudioFilter& child, double frequency)
:AudioFilter(child), frequency_(frequency), time_(0.0) {}

Nothing really interesting here, just initialization of the parent AudioFilter and the internal state.

Now for the real work:

error_type_t SineMultiply::do_process(audio_buffer_t& buffer)

The method should take single parameter - a ref to audio_buffer object containing the samples and information about the sampling format. The return code should be one of following:

• error_type_t::ok - Filtering finished successfully.
• error_type_t::unsupported - The sampling format is not supported
• error_type_t::failed - Failed to apply the filter. This should be considered fatal and the application should consider the filtering chain broken.

const audio_params_t& params = buffer.params;
const double step = 1.0/convert_rate_to_int(params.rate);

This help us to write cleaner code (very important thing) and also it should help compiler to optimize the method (it can assume nothing is going to change). params stores current audio parameters (sampling rate). step is time representing one interval in the sampling frequency.

for (auto& sample: buffer.data) {
    sample = sample * std::sin(time_*frequency_*pi2);
    time_=time_+ step;
}

pi2 is an constant meaning 2*PI.

error_type_t SineMultiply::do_process(audio_buffer_t& buffer)
{
    const audio_params_t& params = buffer.params;
    const double step = 1.0/convert_rate_to_int(params.rate);
    for (auto& sample: buffer.data) {
        sample = sample * std::sin(time_*frequency_*pi2);
        time_=time_+ step;
    }
    return error_type_t::ok;
}

The pi2 constant is defined beforehand like this:

namespace {
    const double pi2 = 8 * std::atan(1.0);
}

Implementing an source filter is the same as implementing an usual filter, with two exceptions.

1. - There's no child for the source filter. This means that constructor does not have the pAudioFilter parameter and the parent AudioFilter is initialized with an empty child pointer. An example for a sine generator:

SineGenerator(double frequency):AudioFilter(pAudioFilter()),
    frequency_(frequency),time_(0.0) {}

2. - The method do_process does not modify the data, but creates them. That means that if we replace the data modification from SineMultiplyFilter by generation, we have sine generator now:

    sample = max_val * std::sin(time_*frequency_*pi2);

A sink filter is a bit different than other filters. It has to inherit from iimavlib::AudioSink instead of the the AudioFilter and has to implement method called do_run. It can also implement method do_process and then it can be usable as a normal filter as well. Aside from inheriting from the AudioSink class, the implementation is the same as for ordinary audio filter. The only difference is the method do_run, that requests the samples from the chain.

It has to:

• Prepare audio_buffer to pass down through the chain.
• Periodically ask the sink for new data.
• Consume the data (== use the data to whatever it wants)

An simple example can be taken from WaveSink class (implementing storing the file to a .wav file). The class behaves as an ordinary filter, so it implements the login in it's do_process method. The do_run method is used only when it's used as a sink (i.e. at the end of filtering chain).

error_type_t WaveSink::do_run()
{
    const size_t buffer_size=512;
    audio_buffer_t buffer;
    buffer.params = get_params();
    buffer.data.resize(buffer_size);
 
    std::fill(buffer.data.begin(),buffer.data.end(),0);
 
    while (still_running()) {
        buffer.data.resize(buffer_size);
        buffer.valid_samples = buffer_size;
        if (process(buffer)!=error_type_t::ok) {
            stop();
            break;
        }
    }
    return error_type_t::ok;
}

So, what does it say? Let's walk-through it.

    audio_buffer_t buffer;
    buffer.params = get_params();

Here, we define a buffer and set the sampling parameters. Method get_params() is a part of the API and (unless overriden by the filter) it returns the format used in a child. This may sound weird, but imagine reading from a file or capturing sound from an audio card. The source filter in this case knows the format used better than sink, so we stick with that.

    buffer.data.resize(buffer_size, 0);

Here we allocate the data for out buffer and initialize them to 0.

    while (still_running()) {

The method still_running() returns true while the filter should run and starts returning false, when it should quit.

        buffer.valid_samples = buffer_size;

        if (process(buffer)!=error_type_t::ok) {
            stop();
            break;
        }

And .. that's all. The whole processing is done in the process method. process ensures that the data will be filled by the source filter and modified by all the other filters and finally passes them to the do_process method of the current sink. So the call to process really means something like:

• source_filter::do_process() → generates data
• other_filters::do_process() → modifies data
• sink::do_process() → Finally, the data will arrive to the sink's do_process method.

If the sink doesn't need to work as an ordinary filter, it doesn't need to implement the do_process method at all (AudioSink has some default pass-through implementation) and can process the data in the main loop.

API for AudioFilter class:

API for AudioSink class: