GSoC 2021: Final report for multistream support in the GStreamer element¶

Project Details¶

• Name: Vedant Paranjape
• Mentors: Paul Elder, Nicolas Dufresne
• Organization: libcamera
• Code: https://git.libcamera.org/libcamera/libcamera.git/
• Project Page: https://summerofcode.withgoogle.com/projects/#6332672818806784
• Progress log: https://ve0x10.in/gsoc2021/
• Midway Blog: https://ve0x10.in/blog/2021/07/19/gsoc-libcamera.html

Brief overview¶

The GStreamer libcamera element allows libcamera to be used in GStreamer pipelines. libcamera supports simultaneous streaming, for example having one lower quality stream for preview, while another higher quality stream is used for recording. This project extends support for the GStreamer libcamera element to support this multistreaming.

Usage¶

Multistream is supported on IPU3, RockChip RK3399, Raspberry Pi 3 and 4 devices camera pipeline. This will only work on those devices.

1. Install libcamera and GStreamer using this guide: https://libcamera.org/getting-started.html

2. If the GStreamer element isn’t installed system wide, set the environment variable specifying the location:

   export GST_PLUGIN_PATH=$BUILDPATH/build/src/gstreamer
   
   # here $BUILDPATH points to the build directory.
   

3. Install gst-plugins-bad as follows, it is required for autovideosink

   sudo apt install gstreamer1.0-plugins-bad
   

4. Run the following command on the terminal

   gst-launch-1.0 libcamerasrc name=src src.src ! queue ! videoconvert ! autovideosink src.src_0 ! queue ! videoconvert ! autovideosink
   

If it runs correctly, you should see such two windows with camera output as follows:

Examples¶

This functionality can be done programmatically, the example codes are here

1. Single Stream
2. Multi Stream
3. Using Device Provider

Work Summary¶

All the patches submitted by me that merged can be seen here

1. gstreamer: Add error checking in gst_libcamera_src_task_enter()
2. libcamera: Add OV5647 sensor properties
3. gstreamer: Added virtual functions needed to support request pads
4. libcamera: Fix the V4L2 pixel format for formats::XBGR8888
5. libcamera: Add pixel format BGRX8888
6. gstreamer: Store group_id in GstLibcameraSrcState
7. gstreamer: Update format specifier in Request Pad template
8. test: gstreamer: Add test for gstreamer single stream

Notes¶

• Use the following command to verify working of multistream on Raspberry Pi 4B+ using Raspberry Pi Cam v1

  gst-launch-1.0 libcamerasrc camera-name="/base/soc/i2c0mux/i2c@1/ov5647@36" name=src src.src ! queue ! videoconvert ! autovideosink src.src_0 ! queue ! videoconvert ! autovideosink`
  

• Use the following command to compile gst code

  gcc basic-tutorial-3.c -o basic-tutorial-3 `pkg-config --cflags --libs gstreamer-1.0`
  

• Use the following to view camera using libcamerasrc (replace “Camera 1” with the actual name of the camera)

  gst-launch-1.0 libcamerasrc camera-name="Camera 1" ! videoconvert ! autovideosink
  

• Use the following to export GStreamer libcamera element, execute this in libcamera root path

  export GST_PLUGIN_PATH=$(pwd)/build/src/gstreamer
  

Conclusion¶

All goals that were originally proposed have been completed. I will continue to support this project in the future, take care of any bugs, issues and will continue implementing additional features whenever required.

It has been a really great experience for me. I have learned so much in these 10 weeks, which will surely help me achieve my future goals in life. I would like to thank libcamera and Google for giving me such an amazing opportunity. Furthermore, I would also like to thank my mentors Paul and Nicolas for helping me out and supporting me throughout the project, also the members of libcamera community, namely Laurent, Kieran, Umang and Jacopo.

On an ending note, if my words or actions hurt anyone, I am extremely sorry !