The libcamera camera sensor model

libcamera defines an abstract camera sensor model in order to provide a description of each of the processing steps that result in image data being sent on the media bus and that form the image stream delivered to applications.

Applications should use the abstract camera sensor model defined here to precisely control the operations of the camera sensor.

The libcamera camera sensor model targets image sensors producing frames in RAW format, delivered through a MIPI CSI-2 compliant bus implementation.

The abstract sensor model maps libcamera components to the characteristics and operations of an image sensor, and serves as a reference to model the libcamera CameraSensor class and SensorConfiguration classes and operations.

In order to control the configuration of the camera sensor through the SensorConfiguration class, applications should understand this model and map it to the combination of image sensor and kernel driver in use.

The camera sensor model defined here is based on the MIPI CCS specification, particularly on Section 8.2 - Image readout of Chapter 8 - Video Timings.

Glossary

Pixel array
The full grid of pixels, active and inactive ones
Pixel array active area
The portion(s) of the pixel array that contains valid and readable pixels; corresponds to the libcamera properties::PixelArrayActiveAreas
Analog crop rectangle
The portion of the pixel array active area which is read out and passed to further processing stages
Subsampling
Pixel processing techniques that reduce the image size by binning or by skipping adjacent pixels
Digital crop
Crop of the sub-sampled image data before scaling
Frame output
The frame (image) as output on the media bus by the camera sensor

Camera sensor model

The abstract sensor model is described in the following diagram.

_images/sensor_model.svg

  1. The sensor reads pixels from the pixel array. The pixels being read out are selected by the analog crop rectangle.

  2. The pixels can be subsampled to reduce the image size without affecting the field of view. Two subsampling techniques can be used:

    • Binning: combines adjacent pixels of the same colour by averaging or summing their values, in the analog domain and/or the digital domain.

      _images/binning.svg

    • Skipping: skips the read out of a number of adjacent pixels.

      _images/skipping.svg

  3. The output of the optional sub-sampling stage is then cropped after the conversion of the analogue pixel values in the digital domain.

  4. The resulting output frame is sent on the media bus by the sensor.

Camera Sensor configuration parameters

The libcamera camera sensor model defines parameters that allow users to control:

  1. The image format bit depth

  2. The size and position of the Analog crop rectangle

  3. The subsampling factors used to downscale the pixel array readout data to a smaller frame size without reducing the image field of view. Two configuration parameters are made available to control the downscaling factor:

    • binning

      A vertical and horizontal binning factor can be specified, the image will be downscaled in its vertical and horizontal sizes by the specified factor.

      Definition: The horizontal and vertical binning factors
      horizontal_binning = xBin;
vertical_binning = yBin;
    • skipping

      Skipping reduces the image resolution by skipping the read-out of a number of adjacent pixels. The skipping factor is specified by the ‘increment’ number (number of pixels to ‘skip’) in the vertical and horizontal directions and for even and odd rows and columns.

      Definition: The horizontal and vertical skipping factors
      horizontal_skipping = (xOddInc + xEvenInc) / 2;
vertical_skipping = (yOddInc + yEvenInc) / 2;

    Different sensors perform the binning and skipping stages in different orders. For the sake of computing the final output image size the order of execution is not relevant. The overall down-scaling factor is obtained by combining the binning and skipping factors.

    Definition: The total scaling factor (binning + sub-sampling)
    total_horizontal_downscale = horizontal_binning + horizontal_skipping;
total_vertical_downscale = vertical_binning + vertical_skipping;

  4. The output size is used to specify any additional cropping on the sub-sampled frame.

  5. The total line length and frame height (visibile pixels + blankings) as sent on the MIPI CSI-2 bus.

  6. The pixel transmission rate on the MIPI CSI-2 bus.

The above parameters are combined to obtain the following high-level configurations:

  • frame output size

    Obtained by applying a crop to the physical pixel array size in the analog domain, followed by optional binning and sub-sampling (in any order), followed by an optional crop step in the output digital domain.

  • frame rate

    The combination of the total frame size, the image format bit depth and the pixel rate of the data sent on the MIPI CSI-2 bus allows to compute the image stream frame rate. The equation is the well known:

    frame_duration = total_frame_size / pixel_rate;
frame_rate = 1 / frame_duration;

    where the pixel_rate parameter is the result of the sensor’s configuration of the MIPI CSI-2 bus (the following formula applies to MIPI CSI-2 when used on MIPI D-PHY physical protocol layer only)

    pixel_rate = csi_2_link_freq * 2 * nr_of_lanes / bits_per_sample;