The IPU3 IPA implementation. More...

Inheritance diagram for libcamera::ipa::ipu3::IPAIPU3:

Collaboration diagram for libcamera::ipa::ipu3::IPAIPU3:

Public Member Functions
int	init (const IPASettings &settings, const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls) override
	Initialize the IPA module and its controls. More...

int	start () override
	Perform any processing required before the first frame.

void	stop () override
	Ensure that all processing has completed.

int	configure (const IPAConfigInfo &configInfo, ControlInfoMap *ipaControls) override
	Configure the IPU3 IPA. More...

void	mapBuffers (const std::vector< IPABuffer > &buffers) override
	Map the parameters and stats buffers allocated in the pipeline handler. More...

void	unmapBuffers (const std::vector< unsigned int > &ids) override
	Unmap the parameters and stats buffers. More...

void	queueRequest (const uint32_t frame, const ControlList &controls) override
	Queue a request and process the control list from the application. More...

void	fillParamsBuffer (const uint32_t frame, const uint32_t bufferId) override
	Fill and return a buffer with ISP processing parameters for a frame. More...

void	processStatsBuffer (const uint32_t frame, const int64_t frameTimestamp, const uint32_t bufferId, const ControlList &sensorControls) override
	Process the statistics generated by the ImgU. More...

Public Member Functions inherited from libcamera::ipa::Module< _Context, _FrameContext, _Config, _Params, _Stats >
const std::list< std::unique_ptr< Algorithm< Module > > > &	algorithms () const
	Retrieve the list of instantiated algorithms. More...

int	createAlgorithms (Context &context, const YamlObject &algorithms)
	Create algorithms from YAML configuration data. More...

Protected Member Functions
std::string	logPrefix () const override
	Retrieve a string to be prefixed to the log message. More...

Protected Member Functions inherited from libcamera::Loggable
LogMessage	_log (const LogCategory category, LogSeverity severity, const char fileName=__builtin_FILE(), unsigned int line=__builtin_LINE()) const
	Create a temporary LogMessage object to log a message. More...

Additional Inherited Members
Public Types inherited from libcamera::ipa::Module< _Context, _FrameContext, _Config, _Params, _Stats >
using	Context = _Context
	The type of the shared IPA context.

using	FrameContext = _FrameContext
	The type of the frame context.

using	Config = _Config
	The type of the IPA configuration data.

using	Params = _Params
	The type of the ISP specific parameters.

using	Stats = _Stats
	The type of the IPA statistics and ISP results.

Static Public Member Functions inherited from libcamera::ipa::Module< _Context, _FrameContext, _Config, _Params, _Stats >
static void	registerAlgorithm (AlgorithmFactoryBase< Module > *factory)
	Add an algorithm factory class to the list of available algorithms. More...

Detailed Description

The IPU3 IPA implementation.

The IPU3 Pipeline defines an IPU3-specific interface for communication between the PipelineHandler and the IPA module.

We extend the IPAIPU3Interface to implement our algorithms and handle calls from the IPU3 PipelineHandler to satisfy requests from the application.

At initialisation time, a CameraSensorHelper is instantiated to support camera-specific calculations, while the default controls are computed, and the algorithms are instantiated from the tuning data file.

The IPU3 ImgU operates with a grid layout to divide the overall frame into rectangular cells of pixels. When the IPA is configured, we determine the best grid for the statistics based on the pipeline handler Bayer Down Scaler output size.

Two main events are then handled to operate the IPU3 ImgU by populating its parameter buffer, and adapting the settings of the sensor attached to the IPU3 CIO2 through sensor-specific V4L2 controls.

In fillParamsBuffer(), we populate the ImgU parameter buffer with settings to configure the device in preparation for handling the frame queued in the Request.

When the frame has completed processing, the ImgU will generate a statistics buffer which is given to the IPA with processStatsBuffer(). In this we run the algorithms to parse the statistics and cache any results for the next fillParamsBuffer() call.

The individual algorithms are split into modular components that are called iteratively to allow them to process statistics from the ImgU in the order defined in the tuning data file.

The current implementation supports five core algorithms:

Auto focus (AF)
Automatic gain and exposure control (AGC)
Automatic white balance (AWB)
Black level correction (BLC)
Tone mapping (Gamma)

AWB is implemented using a Greyworld algorithm, and calculates the red and blue gains to apply to generate a neutral grey frame overall.

AGC is handled by calculating a histogram of the green channel to estimate an analogue gain and shutter time which will provide a well exposed frame. A low-pass IIR filter is used to smooth the changes to the sensor to reduce perceivable steps.

The tone mapping algorithm provides a gamma correction table to improve the contrast of the scene.

The black level compensation algorithm subtracts a hardcoded black level from all pixels.

The IPU3 ImgU has further processing blocks to support image quality improvements through bayer and temporal noise reductions, however those are not supported in the current implementation, and will use default settings as provided by the kernel driver.

Demosaicing is operating with the default parameters and could be further optimised to provide improved sharpening coefficients, checker artifact removal, and false color correction.

Additional image enhancements can be made by providing lens and sensor-specific tuning to adapt for Black Level compensation (BLC), Lens shading correction (SHD) and Color correction (CCM).

Member Function Documentation

◆ configure()

int libcamera::ipa::ipu3::IPAIPU3::configure	(	const IPAConfigInfo &	configInfo,
		ControlInfoMap *	ipaControls
	)

override

Configure the IPU3 IPA.

Parameters

[in]	configInfo	The IPA configuration data, received from the pipeline handler
[in]	ipaControls	The IPA controls to update

Calculate the best grid for the statistics based on the pipeline handler BDS output, and parse the minimum and maximum exposure and analogue gain control values.

Todo:: Document what the BDS is, ideally in a block diagram of the ImgU.

All algorithm modules are called to allow them to prepare the IPASessionConfiguration structure for the IPAContext.

◆ fillParamsBuffer()

void libcamera::ipa::ipu3::IPAIPU3::fillParamsBuffer	(	const uint32_t	frame,
		const uint32_t	bufferId
	)

override

Fill and return a buffer with ISP processing parameters for a frame.

Parameters

[in]	frame	The frame number
[in]	bufferId	ID of the parameter buffer to fill

Algorithms are expected to fill the IPU3 parameter buffer for the next frame given their most recent processing of the ImgU statistics.

◆ init()

int libcamera::ipa::ipu3::IPAIPU3::init	(	const IPASettings &	settings,
		const IPACameraSensorInfo &	sensorInfo,
		const ControlInfoMap &	sensorControls,
		ControlInfoMap *	ipaControls
	)

override

Initialize the IPA module and its controls.

This function receives the camera sensor information from the pipeline handler, computes the limits of the controls it handles and returns them in the ipaControls output parameter.

◆ logPrefix()

std::string libcamera::ipa::ipu3::IPAIPU3::logPrefix ( ) const

overrideprotectedvirtual

Retrieve a string to be prefixed to the log message.

This function allows classes inheriting from the Loggable class to extend the logger with an object-specific prefix output right before the log message contents.

Returns: A string to be prefixed to the log message

Implements libcamera::Loggable.

◆ mapBuffers()

void libcamera::ipa::ipu3::IPAIPU3::mapBuffers ( const std::vector< IPABuffer > & buffers )

override

Map the parameters and stats buffers allocated in the pipeline handler.

Parameters

[in] buffers The buffers to map

◆ processStatsBuffer()

void libcamera::ipa::ipu3::IPAIPU3::processStatsBuffer	(	const uint32_t	frame,
		const int64_t	frameTimestamp,
		const uint32_t	bufferId,
		const ControlList &	sensorControls
	)

override

Process the statistics generated by the ImgU.

Parameters

[in]	frame	The frame number
[in]	frameTimestamp	Timestamp of the frame
[in]	bufferId	ID of the statistics buffer
[in]	sensorControls	Sensor controls

Parse the most recently processed image statistics from the ImgU. The statistics are passed to each algorithm module to run their calculations and update their state accordingly.

◆ queueRequest()

void libcamera::ipa::ipu3::IPAIPU3::queueRequest	(	const uint32_t	frame,
		const ControlList &	controls
	)

override

Queue a request and process the control list from the application.

Parameters

[in]	frame	The number of the frame which will be processed next
[in]	controls	The controls for the frame

Parse the request to handle any IPA-managed controls that were set from the application such as manual sensor settings.

◆ unmapBuffers()

void libcamera::ipa::ipu3::IPAIPU3::unmapBuffers ( const std::vector< unsigned int > & ids )

override

Unmap the parameters and stats buffers.

Parameters

[in] ids The IDs of the buffers to unmap

The documentation for this class was generated from the following file:

src/ipa/ipu3/ipu3.cpp

Public Member Functions

Protected Member Functions

Additional Inherited Members

Detailed Description

Member Function Documentation

◆ configure()

◆ fillParamsBuffer()

◆ init()

◆ logPrefix()

◆ mapBuffers()

◆ processStatsBuffer()

◆ queueRequest()

◆ unmapBuffers()