IPA Writer’s Guide¶

The IPA interface and protocol¶

The IPA interface defines the interface between the pipeline handler and the IPA. Specifically, it defines the functions that the IPA exposes that the pipeline handler can call, and the signals that the pipeline handler can connect to, in order to receive data from the IPA asynchronously. In addition, it contains any custom data structures that the pipeline handler and IPA may pass to each other.

It is possible to use the same IPA interface with multiple pipeline handlers on different hardware platforms. Generally in such cases, these platforms would have a common hardware ISP pipeline. For instance, the rkisp1 pipeline handler supports both the RK3399 and the i.MX8MP as they integrate the same ISP. However, the i.MX8MP has a more complex camera pipeline, which may call for a dedicated pipeline handler in the future. As the ISP is the same as for RK3399, the same IPA interface could be used for both pipeline handlers. The build files provide a mapping from pipeline handler to the IPA interface name as detailed in Compiling the IPA interface.

The IPA protocol refers to the agreement between the pipeline handler and the IPA regarding the expected response(s) from the IPA for given calls to the IPA. This protocol doesn’t need to be declared anywhere in code, but it shall be documented, as there may be multiple IPA implementations for one pipeline handler.

As part of the design of libcamera, IPAs may be isolated in a separate process, or run in the same process but a different thread from libcamera. The pipeline handler and IPA shall not have to change their operation based on whether the IPA is isolated or not, but the possibility of isolation needs to be kept in mind. Therefore all data that is passed between them must be serializable, so they must be defined separately in the mojo Interface Definition Language (IDL), and a code generator will generate headers and serializers corresponding to the definitions. Every interface is defined in a mojom file and includes:

• the functions that the pipeline handler can call from the IPA
• signals in the pipeline handler that the IPA can emit
• any data structures that are to be passed between the pipeline handler and the IPA

All IPA modules of a given pipeline handler use the same IPA interface. The IPA interface definition is thus written by the pipeline handler author, based on how they design the interactions between the pipeline handler and the IPA.

The entire IPA interface, including the functions, signals, and any custom structs shall be defined in a file named {interface_name}.mojom under include/libcamera/ipa/.

Namespacing¶

To avoid name collisions between data types defined by different IPA interfaces and data types defined by libcamera, each IPA interface must be defined in its own namespace.

The namespace is specific with mojo’s module directive. It must be the first non-comment line in the mojo data definition file. For example, the Raspberry Pi IPA interface uses:

module ipa.rpi;

This will become the ipa::rpi namespace in C++ code.

Data containers¶

Since the data passed between the pipeline handler and the IPA must support serialization, any custom data containers must be defined with the mojo IDL.

The following list of libcamera objects are supported in the interface definition, and may be used as function parameter types or struct field types:

• libcamera.ControlInfoMap
• libcamera.ControlList
• libcamera.FileDescriptor
• libcamera.IPABuffer
• libcamera.IPACameraSensorInfo
• libcamera.IPASettings
• libcamera.IPAStream
• libcamera.Point
• libcamera.Rectangle
• libcamera.Size
• libcamera.SizeRange

To use them, core.mojom must be included in the mojo data definition file:

import "include/libcamera/ipa/core.mojom";

Other custom structs may be defined and used as well. There is no requirement that they must be defined before usage. enums and structs are supported.

The following is an example of a definition of an enum, for the purpose of being used as flags:

enum ConfigParameters {
        ConfigLsTable = 0x01,
        ConfigStaggeredWrite = 0x02,
        ConfigSensor = 0x04,
        ConfigDropFrames = 0x08,
};

The following is an example of a definition of a struct:

struct ConfigInput {
        uint32 op;
        uint32 transform;
        libcamera.FileDescriptor lsTableHandle;
        int32 lsTableHandleStatic = -1;
        map<uint32, libcamera.IPAStream> streamConfig;
        array<libcamera.IPABuffer> buffers;
};

This example has some special things about it. First of all, it uses the FileDescriptor data type. This type must be used to ensure that the file descriptor that it contains is translated properly across the IPC boundary (when the IPA is in an isolated process).

This does mean that if the file descriptor should be sent without being translated (for example, for the IPA to tell the pipeline handler which fd that the pipeline handler holds to act on), then it must be in a regular int32 type.

This example also illustrates that struct fields may have default values, as is assigned to lsTableHandleStatic. This is the value that the field will take when the struct is constructed with the default constructor.

Arrays and maps are supported as well. They are translated to C++ vectors and maps, respectively. The members of the arrays and maps are embedded, and cannot be const.

Note that nullable fields, static-length arrays, handles, and unions, which are supported by mojo, are not supported by our code generator.

The Main IPA interface¶

The IPA interface is split in two parts, the Main IPA interface, which describes the functions that the pipeline handler can call from the IPA, and the Event IPA interface, which describes the signals received by the pipeline handler that the IPA can emit. Both must be defined. This section focuses on the Main IPA interface.

The main interface must be named as IPA{interface_name}Interface.

The functions that the pipeline handler can call from the IPA may be synchronous or asynchronous. Synchronous functions do not return until the IPA returns from the function, while asynchronous functions return immediately without waiting for the IPA to return.

At a minimum, the following three functions must be present (and implemented):

• init();
• start();
• stop();

All three of these functions are synchronous. The parameters for start() and init() may be customized.

init() initializes the IPA interface. It shall be called before any other function of the IPAInterface.

stop() informs the IPA module that the camera is stopped. The IPA module shall release resources prepared in start().

A configure() function is recommended. Any ControlInfoMap instances that will be used by the IPA must be sent to the IPA from the pipeline handler, at configure time, for example.

All input parameters will become const references, except for arithmetic types, which will be passed by value. Output parameters will become pointers, unless the first output parameter is an int32, or there is only one primitive output parameter, in which case it will become a regular return value.

const is not allowed inside of arrays and maps. mojo arrays will become C++ std::vector<>.

By default, all functions defined in the main interface are synchronous. This means that in the case of IPC (i.e. isolated IPA), the function call will not return until the return value or output parameters are ready. To specify an asynchronous function, the [async] attribute can be used. Asynchronous functions must not have any return value or output parameters, since in the case of IPC the call needs to return immediately.

It is also possible that the IPA will not be run in isolation. In this case, the IPA thread will not exist until start() is called. This means that in the case of no isolation, asynchronous calls cannot be made before start(). Since the IPA interface must be the same regardless of isolation, the same restriction applies to the case of isolation, and any function that will be called before start() must be synchronous.

In addition, any call made after start() and before stop() must be asynchronous. The motivation for this is to avoid damaging real-time performance of the pipeline handler. If the pipeline handler wants some data from the IPA, the IPA should return the data asynchronously via an event (see “The Event IPA interface”).

The following is an example of a main interface definition:

interface IPARPiInterface {
        init(libcamera.IPASettings settings, string sensorName)
                => (int32 ret, bool metadataSupport);
        start() => (int32 ret);
        stop();

        configure(libcamera.IPACameraSensorInfo sensorInfo,
                  map<uint32, libcamera.IPAStream> streamConfig,
                  map<uint32, libcamera.ControlInfoMap> entityControls,
                  ConfigInput ipaConfig)
                => (int32 ret, ConfigOutput results);

        mapBuffers(array<IPABuffer> buffers);
        unmapBuffers(array<uint32> ids);

        [async] signalStatReady(uint32 bufferId);
        [async] signalQueueRequest(libcamera.ControlList controls);
        [async] signalIspPrepare(ISPConfig data);
};

The first three functions are the required functions. Functions do not need to have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case of asynchronous functions, as explained before, they must not have return values.

The Event IPA interface¶

The event IPA interface describes the signals received by the pipeline handler that the IPA can emit. It must be defined. If there are no event functions, then it may be empty. These emissions are meant to notify the pipeline handler of some event, such as request data is ready, and must not be used to drive the camera pipeline from the IPA.

The event interface must be named as IPA{interface_name}EventInterface.

Functions defined in the event interface are implicitly asynchronous. Thus they cannot return any value. Specifying the [async] tag is not necessary.

Functions defined in the event interface will become signals in the IPA interface. The IPA can emit signals, while the pipeline handler can connect slots to them.

The following is an example of an event interface definition:

interface IPARPiEventInterface {
        statsMetadataComplete(uint32 bufferId,
                              libcamera.ControlList controls);
        runIsp(uint32 bufferId);
        embeddedComplete(uint32 bufferId);
        setIsp(libcamera.ControlList controls);
        setStaggered(libcamera.ControlList controls);
};

Compiling the IPA interface¶

After the IPA interface is defined in include/libcamera/ipa/{interface_name}.mojom, an entry for it must be added in meson so that it can be compiled. The filename must be added to the pipeline_ipa_mojom_mapping variable in include/libcamera/ipa/meson.build. This variable maps the pipeline handler name to its IPA interface file.

For example, adding the raspberrypi.mojom file to meson:

pipeline_ipa_mojom_mapping = [
    'rpi/vc4': 'raspberrypi.mojom',
]

This will cause the mojo data definition file to be compiled. Specifically, it generates five files:

• a header describing the custom data structures, and the complete IPA interface (at {$build_dir}/include/libcamera/ipa/{interface}_ipa_interface.h)
• a serializer implementing de/serialization for the custom data structures (at {$build_dir}/include/libcamera/ipa/{interface}_ipa_serializer.h)
• a proxy header describing a specialized IPA proxy (at {$build_dir}/include/libcamera/ipa/{interface}_ipa_proxy.h)
• a proxy source implementing the IPA proxy (at {$build_dir}/src/libcamera/proxy/{interface}_ipa_proxy.cpp)
• a proxy worker source implementing the other end of the IPA proxy (at {$build_dir}/src/libcamera/proxy/worker/{interface}_ipa_proxy_worker.cpp)

The IPA proxy serves as the layer between the pipeline handler and the IPA, and handles threading vs isolation transparently. The pipeline handler and the IPA only require the interface header and the proxy header. The serializer is only used internally by the proxy.

Using the custom data structures¶

To use the custom data structures that are defined in the mojo data definition file, the following header must be included:

#include <libcamera/ipa/{interface_name}_ipa_interface.h>

The POD types of the structs simply become their C++ counterparts, eg. uint32 in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo array becomes C++ std::vector. All members of maps and vectors are embedded, and are not pointers. The members cannot be const.

The names of all the fields of structs can be used in C++ in exactly the same way as they are defined in the data definition file. For example, the following struct as defined in the mojo file:

struct SensorConfig {
     uint32 gainDelay = 1;
     uint32 exposureDelay;
     uint32 sensorMetadata;
};

Will become this in C++:

struct SensorConfig {
     uint32_t gainDelay;
     uint32_t exposureDelay;
     uint32_t sensorMetadata;
};

The generated structs will also have two constructors, a constructor that fills all fields with the default values, and a second constructor that takes a value for every field. The default value constructor will fill in the fields with the specified default value if it exists. In the above example, gainDelay_ will be initialized to 1. If no default value is specified, then it will be filled in as zero (or -1 for a FileDescriptor type).

All fields and constructors/destructors in these generated structs are public.

Using the IPA interface (pipeline handler)¶

The following headers are necessary to use an IPA in the pipeline handler (with raspberrypi as an example):

#include <libcamera/ipa/raspberrypi_ipa_interface.h>
#include <libcamera/ipa/raspberrypi_ipa_proxy.h>

The first header includes definitions of the custom data structures, and the definition of the complete IPA interface (including both the Main and the Event IPA interfaces). The name of the header file comes from the name of the mojom file, which in this case was raspberrypi.mojom.

The second header includes the definition of the specialized IPA proxy. It exposes the complete IPA interface. We will see how to use it in this section.

In the pipeline handler, we first need to construct a specialized IPA proxy. From the point of view of the pipeline hander, this is the object that is the IPA.

To do so, we invoke the IPAManager:

std::unique_ptr<ipa::rpi::IPAProxyRPi> ipa_ =
        IPAManager::createIPA<ipa::rpi::IPAProxyRPi>(pipe_, 1, 1);

The ipa::rpi namespace comes from the namespace that we defined in the mojo data definition file, in the “Namespacing” section. The name of the proxy, IPAProxyRPi, comes from the name given to the main IPA interface, IPARPiInterface, in the “The Main IPA interface” section.

The return value of IPAManager::createIPA shall be error-checked, to confirm that the returned pointer is not a nullptr.

After this, before initializing the IPA, slots should be connected to all of the IPA’s signals, as defined in the Event IPA interface:

ipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete);
ipa_->runIsp.connect(this, &RPiCameraData::runIsp);
ipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete);
ipa_->setIsp.connect(this, &RPiCameraData::setIsp);
ipa_->setStaggered.connect(this, &RPiCameraData::setStaggered);

The slot functions have a function signature based on the function definition in the Event IPA interface. All plain old data (POD) types are as-is (with their C++ versions, eg. uint32 -> uint32_t), and all structs are const references.

For example, for the following entry in the Event IPA interface:

statsMetadataComplete(uint32 bufferId, ControlList controls);

A function with the following function signature shall be connected to the signal:

void statsMetadataComplete(uint32_t bufferId, const ControlList &controls);

After connecting the slots to the signals, the IPA should be initialized (using the main interface definition example from earlier):

IPASettings settings{};
bool metadataSupport;
int ret = ipa_->init(settings, "sensor name", &metadataSupport);

At this point, any IPA functions that were defined in the Main IPA interface can be called as if they were regular member functions, for example (based on the main interface definition example from earlier):

ipa_->start();
int ret = ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result);
ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast<unsigned int>(index));

Remember that any functions designated as asynchronous must not be called before start().

Notice that for both init() and configure(), the first output parameter is a direct return, since it is an int32, while the other output parameter is a pointer-based output parameter.

Using the IPA interface (IPA Module)¶

The following header is necessary to implement an IPA Module (with raspberrypi as an example):

#include <libcamera/ipa/raspberrypi_ipa_interface.h>

This header includes definitions of the custom data structures, and the definition of the complete IPA interface (including both the Main and the Event IPA interfaces). The name of the header file comes from the name of the mojom file, which in this case was raspberrypi.mojom.

The IPA module must implement the IPA interface class that is defined in the header. In the case of our example, that is ipa::rpi::IPARPiInterface. The ipa::rpi namespace comes from the namespace that we defined in the mojo data definition file, in the “Namespacing” section. The name of the interface is the same as the name given to the Main IPA interface.

The function signature rules are the same as for the slots in the pipeline handler side; PODs are passed by value, and structs are passed by const reference. For the Main IPA interface, output values are also allowed (only for synchronous calls), so there may be output parameters as well. If the first output parameter is a POD it will be returned by value, otherwise it will be returned by an output parameter pointer. The second and any other output parameters will also be returned by output parameter pointers.

For example, for the following function specification in the Main IPA interface definition:

configure(libcamera.IPACameraSensorInfo sensorInfo,
          uint32 exampleNumber,
          map<uint32, libcamera.IPAStream> streamConfig,
          map<uint32, libcamera.ControlInfoMap> entityControls,
          ConfigInput ipaConfig)
=> (int32 ret, ConfigOutput results);

We will need to implement a function with the following function signature:

int configure(const IPACameraSensorInfo &sensorInfo,
              uint32_t exampleNumber,
              const std::map<unsigned int, IPAStream> &streamConfig,
              const std::map<unsigned int, ControlInfoMap> &entityControls,
              const ipa::rpi::ConfigInput &data,
              ipa::rpi::ConfigOutput *response);

The return value is int, because the first output parameter is int32. The rest of the output parameters (in this case, only response) become output parameter pointers. The non-POD input parameters become const references, and the POD input parameter is passed by value.

At any time after start() and before stop() (though usually only in response to an IPA call), the IPA may send data to the pipeline handler by emitting signals. These signals are defined in the C++ IPA interface class (which is in the generated and included header).

For example, for the following function defined in the Event IPA interface:

statsMetadataComplete(uint32 bufferId, libcamera.ControlList controls);

We can emit a signal like so:

statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_);