View source for V4L2 mode for Raspberry Pi

[[V4L2 mode for Raspberry Pi/zh|查看中文]]

[[V4L2 mode for Raspberry Pi|<big>How to use VEYE and CS series camera module on Raspberry Pi(V4L2 mode)</big>]]

=== Introduction ===
For the use of VEYE series and CS series camera modules on the raspberry Pi platform, we provide veye_raspicam software, an application layer open source software similar to the raspicam. This series of software, without driver support, has good compatibility with different versions of piOS.

However, we think that the V4L2 driver mode also has a wide range of applications. The two modes cannot be used at the same time, which is embodied in whether the driver is installed or not. It will be described in detail later.

This article describes how to call the camera modules of VEYE series and CS series through V4L2 on the raspberry Pi platform.

=== Hardware Setup ===
[[VEYE-MIPI-290/327 for Raspberry Pi#Hardware Setup|VEYE series camera module setup on rpi]]

[[CS-MIPI-X for Raspberry Pi#Hardware Setup|CS series camera module setup on rpi]]

=== RaspberryPi System Setup and Configuration ===
[[CS-MIPI-X for Raspberry Pi#RaspberryPi System Setup and Configuration|RaspberryPi System Setup and Configuration]]

=== Driver Installation ===

==== Download the driver ====
<code>git clone <nowiki>https://github.com/veyeimaging/raspberrypi_v4l2.git</nowiki></code>

==== Install the driver ====
<code>cd raspberrypi_v4l2/release/</code>

<code>chmod +x *</code>

<code>sudo ./install_driver.sh [camera module]</code>

camera module：could be veye327,csimx307,cssc132,veyecam2m.

'''Note: veyecam2m is a new version driver that can replace veye327 and supports all modules of the VEYE series with 200W resolution.'''

'''please use veyecam2m instead of veye327 which will be discarded.'''

Note: For VEYE-MIPI-IMX327S or VEYE-MIPI-IMX462, please make sure '''hdver''' must >= 0x5.

http://wiki.veye.cc/index.php/VEYE-MIPI-IMX327S_version_log

==== Uninstall the driver ====
If you need to change to using the camera by veye_raspicam software, or if you want to change to a driver of another camera module model, you must first uninstall the current driver.

<code>sudo ./uninstall_driver.sh [camera module]</code>

camera module：could be veye327,csimx307,cssc132,veyecam2m.

=== Check and Test the Camera ===
Check the camera probe status

* VEYE-MIPI-327

<code>dmesg | grep veye</code>

You can see:

<code>veye327 camera probed</code>

* CS-MIPI-IMX307

<code>dmesg | grep 307</code>

You can see:

<code>camera id is cs-mipi-imx307</code>

* CS-MIPI-SC132

<code>dmesg | grep 132</code>

You can see:

<code>camera id is cs-mipi-sc132</code>

==== List the video device ====
<code>ls /dev/video0</code>

The device node exsit.

==== List the available video modes ====
<code>v4l2-ctl --list-formats-ext</code>

=== Preview using VLC ===
1. Open VLC with command line <code>vlc</code> , or click the icon to launch.

2. Hit the <code>▶</code>(Play) button to call the open media window.

3. In <code>Capture Device</code> >> <code>Device Selection</code> >> <code>Video device name</code>, select the camera video node.

4. Hit <code>Advanced Options...</code> button

5. In Advanced Options window >> <code>Video input chroma format</code>, type <code>UYUV</code>.

6. Type in the <code>width</code> and <code>height</code>, for example, 1920 and 1080.

7. Hit <code>OK</code> to save the settings and see the video feed.
[[File:Play camera using VLC on RPI.png|alt=Play camera using VLC on RPI|center|thumb|1000x1000px|Play camera using VLC on RPI]]
<br />

=== Gstreamer usage samples ===
<code>export DISPLAY=:0</code>

==== install gstreamer ====
<code>sudo apt-get install gstreamer1.0-tools</code>

<code>sudo apt-get install libx264-dev libjpeg-dev</code>

<code>sudo apt-get install libgstreamer1.0-devlibgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-ugly gstreamer1.0-tools gstreamer1.0-gl gstreamer1.0-gtk3</code>

==== Run gstreamer videotest command ====
<code>gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink</code>

==== Snap a picture (VEYE-MIPI-327,CS-MIPI-IMX307 @1080p mode) ====
<code>gst-launch-1.0 v4l2src num-buffers=1 device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=1920,height=1080' ! jpegenc ! filesink location=test_image.jpg</code>

==== Frame rate test (VEYE-MIPI-327,CS-MIPI-IMX307 @1080p mode) ====
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! videoconvert ! fpsdisplaysink video-sink=fakesink -v</code>

==== Preview (VEYE-MIPI-327,CS-MIPI-IMX307 @1080p mode) ====
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! videoconvert ! autovideosink sync=false -v</code>

==== Preview and Scale with timestamp on (VEYE-MIPI-327,CS-MIPI-IMX307 @1080p mode) ====
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1"  ! videoconvert ! videoscale ! clockoverlay time-format="%D %H:%M:%S" ! video/x-raw, width=640, height=360 ! autovideosink</code>

==== Preview (CS-MIPI-IMX307,CS-MIPI-SC132 @720p mode) ====
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1280, height=(int)720,framerate=(fraction)60/1" ! videoconvert ! autovideosink sync=false -v</code>

==== Preview (CS-MIPI-IMX307,CS-MIPI-SC132 @vga mode) ====
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)640, height=(int)480" ! videoconvert ! autovideosink sync=false -v</code>

==== Preview (CS-MIPI-SC132 @1280*1080@45 mode) ====
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1280, height=(int)1080,framerate=(fraction)45/1" ! videoconvert ! autovideosink sync=false -v</code>

==== Streaming to file，save as mkv file(VEYE-MIPI-X,CS-MIPI-IMX307 @1080p mode) ====
<code>gst-launch-1.0 -e v4l2src device=/dev/video0  num-buffers=300  ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! v4l2h264enc extra-controls="controls, h264_profile=4, video_bitrate=6200000" ! 'video/x-h264, profile=high, level=(string)4' ! h264parse ! matroskamux ! filesink location=output.mkv</code>

==== Streaming to file，save as mp4 file(VEYE-MIPI-X,CS-MIPI-IMX307 @1080p mode) ====
<code>gst-launch-1.0 -e v4l2src device=/dev/video0  num-buffers=300  ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! v4l2h264enc extra-controls="controls, h264_profile=4, video_bitrate=6200000" ! 'video/x-h264, profile=high, level=(string)4' ! h264parse ! mp4mux ! filesink location=video.mp4</code>
==== TCP streaming====
'''''raspberrypi（sender）'''''

<code>gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=-1 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! v4l2h264enc extra-controls="controls, h264_profile=4, video_bitrate=4000000" ! 'video/x-h264, profile=high, level=(string)4' ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink  host=x.x.x.x port=5000</code>

The bandwidth of the stream is 4Mbps, continuous transmission, and the listening port is 5000.

'''''Client'''''

<code>gst-launch-1.0 -v tcpclientsrc host=x.x.x.x port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! autovideosink sync=false</code>

Where x.x.x.x is the IP address of the Raspberry Pi (Server).

It is recommended to use powershell if client is a windows system. '''''gstreamer''''' windows version download [https://gstreamer.freedesktop.org/documentation/installing/on-windows.html?gi-language=c here].

==== Stream to OpenCV ====
<code>gst-launch-1.0 v4l2src device=/dev/video0  ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! videoscale ! "video/x-raw,width=640,height=480" ! videoconvert ! "video/x-raw, format=(string)BGR" ! appsink</code>
===V4l2-ctl usage examples===
====install v4l2-utils====
<code>sudo apt-get install v4l-utils</code>
====v4l2-ctl test====

*Snap one UYVY raw data picture (1080p mode)

<code>v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --stream-mmap --stream-count=1 --stream-to=uyvy-1920x1080.yuv</code>

* Snap 10 UYVY raw data pictures (1080p mode)

<code>v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --stream-mmap --stream-count=10 --stream-to=uyvy-1920x1080_stream.yuv</code>

=== yavta usage examples ===

==== download yavta ====
<code>git clone <nowiki>https://git.ideasonboard.org/yavta.git</nowiki></code> 

<code>cd yavta;make</code>

==== yavta test ====
<code>./yavta -c8 -Fuyvy_1920X1080.yuv --skip 0 -f UYVY -s 1920x1080 /dev/video0</code>

PS, UYVY file can be played by [https://yuv-player-deluxe.software.informer.com/2.6/ YUV Displayer Deluxe].

=== Video Control Toolkits Manual ===
Because of the high flexibility of our camera parameters, we do not use V4L2 parameters to control, but use scripts to configure parameters.

<nowiki>https://github.com/veyeimaging/raspberrypi/tree/master/i2c_cmd</nowiki>

[[VEYE-MIPI-290/327 for Raspberry Pi#Video Control Toolkits Manual|VEYE Series Video Control Toolkits Manual]]

[[CS-MIPI-X for Raspberry Pi#Video Control Toolkits Manual|CS Series Video Control Toolkits Manual]]

=== Notes for CM4 ===
Cm4 supports the use of two cameras at the same time. Following above steps will only be able to use CAM1. To use two cameras, follow the following steps:

==== Upgrade dt-blob.bin ====
<code>wget <nowiki>https://www.raspberrypi.org/documentation/hardware/computemodule/dt-blob-dualcam.bin</nowiki></code>

<code>sudo cp dt-blob-dualcam.bin /boot/dt-blob.bin</code>

==== Upgrade dual camera version dtbo file ====
Use csimx307,5.10.17-v7l+ kernel as an example：

<code>sudo cp raspberrypi_v4l2/release/driver_bin/5.10.17-v7l+/csimx307-dual-cm4.dtbo /boot/overlays/csimx307.dtbo</code>

<code>sudo reboot</code>

==== Device node description ====
The CM4 module uses two I2C channels to communicate with the two cameras respectively.
{| class="wikitable"
! description
!i2c bus num
!video node
|-
|CAM0
|0
|video0
|-
|CAM1
|10
|video2
|}
Note: If there is only one camera, video node is always video0.

==== Gstreamer usage samples ====
<code>export DISPLAY=:0</code>

*CAM0 preview(VEYE-MIPI-327,CS-MIPI-IMX307 @1080p mode)

<code>gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! videoconvert ! fpsdisplaysink videosink=autovideosink sync=false text-overlay=false -v</code>

*CAM1 preview(VEYE-MIPI-327,CS-MIPI-IMX307 @1080p mode)

<code>gst-launch-1.0 v4l2src device=/dev/video2 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080,framerate=(fraction)30/1" ! videoconvert ! fpsdisplaysink videosink=autovideosink sync=false text-overlay=false -v</code>
=== Source code cross compilation ===
The main resources in this section are the [https://www.raspberrypi.org/documentation/linux/kernel/building.md official piOS building method]. We use the method of cross-compilation, and the 5.4.72 version of 32bitOS is taken as an example.

The following operations are done on ubuntu PC.

==== Build cross-compilation environment on Ubuntu 64-bitoperating system) ====
<code>sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev</code>

* 32-bit piOS target version

<code>sudo apt install crossbuild-essential-armhf</code>

* 64-bit piOS target version

<code>sudo apt install crossbuild-essential-arm64</code>

==== Download the standard version of the piOS source code ====

===== Confirm the piOS version of your raspberry Pi =====

* <code>Release version</code>

<code>$ uname -a</code>

<code>Linux raspberrypi 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux</code>

* <code>Code tag</code>

<code>$ cp /usr/share/doc/raspberrypi-bootloader/changelog.Debian.gz ./</code>

<code>gunzip changelog.Debian.gz</code>

Check the top lines and you will know the tag.
===== There are two ways to get the source code of the version you want: =====

# using git to clone the corresponding branch and checkout the corresponding tag.

<code>git clone --branch rpi-5.4.y <nowiki>https://github.com/raspberrypi/linux</nowiki></code>

<code>git checkout raspberrypi-kernel_1.20201022-1</code>

''PS: Please replace the above two commands with your own corresponding versions.''

2. Manually download the version code of the corresponding tag directly from the link below.

https://github.com/raspberrypi/linux/tags

==== Patch our  code to kernel ====

*Driver source code

camera driver path is ： linux/drivers/media/i2c，copy our camera module drivers to this path.

*Modify the the Makefile and Kconfig files

Modify the Config and Makefile in the same path，add the corresponding camera driver.

*dts file

dts file path is ：linux/arch/arm/boot/dts/overlays，copy our [camera]-overlay.dts files to this path.

*Modify dts Makefile

Modify the Makefile in the same path，add the corresponding dts compilation option.

==== Building ====

==== Preparing ====

*For Raspberry Pi 1, Zero and Zero W, and Raspberry Pi Compute Module 1 default (32-bit only) build configuration

<code>KERNEL=kernel</code>

<code>make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig</code>

*For Raspberry Pi 2, 3, 3+ and Zero 2 W, and Raspberry Pi Compute Modules 3 and 3+ default 32-bit build configuration

<code>KERNEL=kernel7</code>

<code>make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig</code>

*For Raspberry Pi 4 and 400, and Raspberry Pi Compute Module 4 default 32-bit build configuration

<code>KERNEL=kernel7l</code>

<code>make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2711_defconfig</code>

* For Raspberry Pi 3, 3+, 4, 400 and Zero 2 W, and Raspberry Pi Compute Modules 3, 3+ and 4 default 64-bit build configuration

<code>KERNEL=kernel8</code>

<code>make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig</code>
==== Add compilation options ====

* 32-bit version

<code>make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig</code>

* 64-bit version

<code>make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig</code>

Add the compilation options by the corresponding camera module, for 5.4 version kernel the path is driver-- > multimedia-- > i2C.

For 5.10 version kernel,the path is Device Drivers --> Multimedia Support --> Media ancillary drivers --> Camera sensor devices.

===== Build =====

* <code>32-bit version</code>

<code>make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j4</code>

* <code>64-bit version</code>

<code>make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs -j4</code>

=== Frequently Questions ===

* disagrees about version of symbol module_layout

After the system has been booted, <code>dmesg</code> can see this error reported when the driver of camera is loaded. It is caused by the mismatch between the original piOS version on the board and the compiled camera driver version.

It is recommended to refer to this [https://www.raspberrypi.com/documentation/computers/linux_kernel.html#cross-compiling-the-kernel link] to recompile from the code and install Image, dtb, modules as a whole.

=== References ===
<nowiki>https://www.raspberrypi.org/documentation/linux/kernel/building.md</nowiki>

=== Document History ===

* 20220302

Add v4l2-ctl and yavta examples.

* 20220301

Support 64-bit piOS

* 20220225

Add the application samples of gstreamer.

*20211125

Add VLC preview section.<br />