Mv series camera appnotes 4 jetson
1 Overview
The MV series cameras are cameras introduced for AI applications in the industrial field. It uses the MIPI CSI-2 interface, which is especially suitable for embedded computing platforms.
It features rich data formats and trigger characteristics, extremely low latency, extremely high bandwidth and reliable stability.
This article describes how to use the MV series camera on the NVIDIA Jetson platform.
1.1 Camera Module List
| Series | Model | Status |
|---|---|---|
| MV series | MV-MIPI-IMX178M | Done |
| MV series | MV-MIPI-SC130M | Done |
1.2 Jetson Board List
| Jetson Board | Status |
|---|---|
| Nano A02 | Done |
| Nano B01 | Done |
| Nano 2GB | Done |
| TX2 NX | Done |
| XAVIER NX | Done |
| TX2 Devkit | Done |
| AGX Xavier | Done |
| AGX Orin | Coming soon |
1.3 Supported L4T versions
- Jetpack4.6.1,L4T r32.7.1
1.3.1 How to check the current L4T version
On Jetson board
cat /etc/nv_tegra_release
If it shows:
# R32 (release), REVISION: 7.1......
It means L4t Version is 32.7.1, and the Jetpack version is 4.6.1.
2 Hardware Setup
MV series cameras require an adapter board to access the Jetson platform. The following table shows the support status.
| Jetson Board | adapter board | camera number | Power |
|---|---|---|---|
| Nano A02 | ADP-MV1 | 1 | 5V DC（Required） |
| Nano B01 | ADP-MV1 | 1 | 5V DC（Required） |
| Nano 2GB | ADP-MV1 | 2 | 5V DC（Required） |
| TX2 NX | ADP-MV1 | 2 | 5V DC（Required） |
| XAVIER NX | ADP-MV1 | 2 | 5V DC（Required） |
| TX2 Devkit | ADP-N4 | 6 | 5V or 12V DC (Required) |
| AGX Xavier | ADP-N4 | 6 | 5V or 12V DC (Required) |
| Orin series | ADP-N4 | 6 | 5V or 12V DC (Required) |
2.1 Connection of MV-MIPI-CAM and ADP-MV1
The two are connected using 0.5 mm pitch*30P FFC cable with opposite direction. The cable must be inserted with the silver contacts facing outside.
2.2 Connection using ADP-MV1
2.2.1 Power supply
The ADP-MV1 requires a separate 5V power supply and can be powered directly from the Jetson board using a Dupont cable.
2.3 Connection with Nano and NX using ADP-MV1
The Nano Series and NX Series are connected in this way.
2.4 Connection of MV-MIPI-CAM and ADP-N4
The two are connected using 0.5 mm pitch*30P FFC cable with same direction. The cable must be inserted with the silver contacts facing outside.
2.5 Connection using ADP-N4
2.5.1 Power supply
The ADP-N4 requires power supply. It supports two types of power supply, any one of them will work.
- 5V DC is connected to J19-9, which can be powered directly from the Jetson motherboard using a DuPont cable.
- 12V DC is connected to J11, which needs to be powered by an external adapter.
2.5.2 Connection with TX2 Devkitand AGX Series using ADP-MV1
3 Upgrade Jetson system
This section describes how to upgrade the Jetson system to support MV camera module. For OS update method, please refer to Update Jetson OS.
Specially, for MV series camera, besides adding the camera driver in Linux system, we also add a kernel patch - veye_mv_l4t_32.7.1.patch.
This patch has two features：
- Added support for both Y10 and Y12 data formats for Mono cameras.
- Added support for trigger mode.
3.1 Description of raw data image format
On TX2 and XAVIER, 10 bit depth and 12 bit depth raw data is stored in memory in a format that is not compliant with the V4L2 standard. We have extended the definition in the linux kernel to support this case.
TY10, TY12, XY10 and XY12 are the new types we have added.
3.1.1 Nano
| Depth | Bit order | FourCC | Enumerator |
|---|---|---|---|
| 8 | B7 B6 B5 B4 B3 B2 B1 B0 | GREY | V4L2_PIX_FMT_GREY |
| 10 | 0 0 0 0 0 0 B9 B8 B7 B6 B5 B4 B3 B2 B1 B0 | 'Y10 ' | V4L2_PIX_FMT_Y10 |
| 12 | 0 0 0 0 B11 B10 B9 B8 B7 B6 B5 B4 B3 B2 B1 B0 | 'Y12 ' | V4L2_PIX_FMT_Y12 |
3.1.2 TX2
| Depth | Bit order (X is undefined) | FourCC | Enumerator |
|---|---|---|---|
| 8 | B7 B6 B5 B4 B3 B2 B1 B0 | GREY | V4L2_PIX_FMT_GREY |
| 10 | 0 0 B9 B8 B7 B6 B5 B4 B3 B2 B1 B0 X X X X | TY10 | V4L2_PIX_FMT_TX2_Y10 |
| 12 | 0 0 B11 B10 B9 B8 B7 B6 B5 B4 B3 B2 B1 B0 X X | TY12 | V4L2_PIX_FMT_TX2_Y12 |
3.1.3 XAVIER
| Depth | Bit order (X is undefined) | FourCC | Enumerator |
|---|---|---|---|
| 8 | B7 B6 B5 B4 B3 B2 B1 B0 | GREY | V4L2_PIX_FMT_GREY |
| 10 | 0 B9 B8 B7 B6 B5 B4 B3 B2 B1 B0 X X X X X | XY10 | V4L2_PIX_FMT_XAVIER_Y10 |
| 12 | 0 B11 B10 B9 B8 B7 B6 B5 B4 B3 B2 B1 B0 X X X | XY12 | V4L2_PIX_FMT_XAVIER_Y12 |
Also, we provide tool software in order to convert these special data formats to standard formats: https://github.com/veyeimaging/pixel_layer_convert.
We recommend using vooya as the player.
3.1.4 Support for trigger mode
The default driver for Jetson systems only supports video streaming mode. In its VI driver, the data reception function has a timeout mechanism. We have added a settable vi_time_out_disable option to dynamically turn this timeout mechanism on and off.
Refer to the following application example for specific applications.
4 Check system status
After system update, reboot the Jetson board.
During the Jetson system boot process, it detects the presence of cameras on all i2c buses and generates the /dev/videoX device node if it exists.
Execute the following command on the Jetson board to check if the camera is properly connected.
dmesg | grep mvcam
You can see the camera model and the camera version number probed.
A prompt as below indicates that the MV-MIPI-IMX178M camera is detected on the i2c-10 bus.
mvcam 10-003b: camera is: MV-MIPI-IMX178M
mvcam 10-003b: firmware version: 0x1080103
4.1 /dev/videoX node
The camera module is mapped as /dev/videoX device node in the Jetson system.
During the OS boot process, the cameras are detected in the order of the i2c bus. The X value is incremented from 0 according to the logical order of detection.
For instance, if only one camera is connected, X is 0 regardless of the location to which the hardware is connected. If 5 cameras are connected, X is [0-4] according to i2c bus from smallest to largest.
The v4l2-ctl command uses -d /dev/videoX to access different cameras.
In gstreamer, v4l2src can access different cameras by specifying device=/dev/videoX.
5 v4l2-ctl Application Examples
5.1 Install v4l2-utils
sudo apt-get install v4l-utils
5.2 Configure parameters using v4l2-ctl
5.2.1 List the data formats supported by the camera
v4l2-ctl --list-formats-ext
The following is an example in the XAVIER system：
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'UYVY'
Name : UYVY 4:2:2
Size: Discrete 3088x2064
Interval: Discrete 0.045s (22.000 fps)
Index : 1
Type : Video Capture
Pixel Format: 'NV16'
Name : Y/CbCr 4:2:2
Size: Discrete 3088x2064
Interval: Discrete 0.045s (22.000 fps)
Index : 2
Type : Video Capture
Pixel Format: 'GREY'
Name : 8-bit Greyscale
Size: Discrete 3088x2064
Interval: Discrete 0.045s (22.000 fps)
Index : 3
Type : Video Capture
Pixel Format: 'XY10'
Name : XAVIER 10-bit/16-bit Greyscale
Size: Discrete 3088x2064
Interval: Discrete 0.045s (22.000 fps)
Index : 4
Type : Video Capture
Pixel Format: 'XY12'
Name : XAVIER 12-bit/16-bit Greyscale
Size: Discrete 3088x2064
Interval: Discrete 0.045s (22.000 fps)
It should be noted that the UYVY and NV16 modes provided by the MV series cameras are for debugging purposes only and the maximum width cannot exceed 2880.
5.2.2 List the configurable parameters of the camera implemented in the driver
v4l2-ctl -L
User Controls
horizontal_flip 0x00980914 (bool) : default=0 value=0
vertical_flip 0x00980915 (bool) : default=0 value=0
trigger_mode 0x00981901 (int) : min=0 max=2 step=1 default=0 value=0 flags=volatile, execute-on-write
trigger_src 0x00981902 (int) : min=0 max=1 step=1 default=1 value=1 flags=volatile, execute-on-write
soft_trgone 0x00981903 (button) : flags=write-only, execute-on-write
frame_rate 0x00981904 (int) : min=1 max=22 step=1 default=22 value=22 flags=volatile, execute-on-write
roi_x 0x00981905 (int) : min=0 max=2712 step=8 default=0 value=0
roi_y 0x00981906 (int) : min=0 max=1744 step=4 default=0 value=0
Camera Controls
low_latency_mode 0x009a206d (bool) : default=0 value=0
preferred_stride 0x009a206e (int) : min=0 max=65535 step=1 default=0 value=0
vi_time_out_disable 0x009a2078 (bool) : default=0 value=0
Parameters can be set and get using the following methods.
v4l2-ctl --set-ctrl [ctrl_type]=[val]
v4l2-ctl --get-ctrl [ctrl_type]
All the above functions can be implemented using mv_mipi_i2c.sh.
None of the above parameters can be modified in the state when the image acquisition is started.
Each of them is described below:
5.2.3 horizontal and vertical flip
- horizontal flip
v4l2-ctl --set-ctrl horizontal_flip=1
- vertical flip
v4l2-ctl --set-ctrl vertical_flip=1
5.2.4 Trigger Mode
v4l2-ctl --set-ctrl trigger_mode=[0-2]
0：Video streaming mode
1：Normal trigger mode.
2：High-speed continuous trigger mode.
5.2.5 Trigger Source
v4l2-ctl --set-ctrl trigger_src=[0-1]
0: Software trigger mode.
1: Hardware trigger mode.
5.2.6 Software trigger command
v4l2-ctl --set-ctrl soft_trgone=1
5.2.7 Set frame rate
v4l2-ctl --set-ctrl frame_rate=[1-max]
The maximum frame rate is automatically updated as the resolution changed.
5.2.8 Set ROI and pixel format
For example, for MV-MIPI-IMX178M:
v4l2-ctl --set-ctrl roi_x=0
v4l2-ctl --set-ctrl roi_y=0
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY
The maximum frame rate will be adjusted automatically after setting ROI. Please note that the camera ROI parameters need to comply with the requirements in the camera manual.
5.2.9 preferred_stride
The preferred_stride refers to the memory size required for one line of image data. Image data for the Jetson platform is stored unpacked, i.e., one pixel occupies two bytes in the 10-bit depth and 12-bit depth data formats.
- In the normal case, the buffer needs to be 64 bytes aligned.
preferred_stride= ALIGN_UP（width*bytes_per_pixel,64）
For instance, if the width is 3088:
8bit depth，preferred_stride=3136;
10bit depth，preferred_stride=6208;
12bit depth，preferred_stride=6208;
- If VIC buffer is used, 256 bytes alignment is required.
preferred_stride= ALIGN_UP（width*bytes_per_pixel,256）
5.3 Video Streaming mode
5.3.1 Set ROI and format
Take MV-MIPI-IMX178M, 3088*2064 as an example.
v4l2-ctl --set-ctrl roi_x=0
v4l2-ctl --set-ctrl roi_y=0
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY
5.3.2 Frame rate statistics
In streaming mode, the following commands can be used for frame rate statistics.
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY --stream-mmap --stream-count=-1 --stream-to=/dev/null
5.3.3 Save image to file
- raw8
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=3136
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY --stream-mmap --stream-count=1 --stream-to=y8-3136x2064.raw
- raw10
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=6208
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=XY10 --stream-mmap --stream-count=1 --stream-to=y10-3104x2064.raw
- raw12
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=6208
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=XY12 --stream-mmap --stream-count=1 --stream-to=y12-3104x2064.raw
For the image format, please refer to the section above: Description of raw data image format.
5.3.4 Preview
The camera can be previewed in real time using the following command:
v4l2-ctl -d /dev/video0 --set-fmt-video=width=2816,height=2064,pixelformat=UYVY
gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)2816, height=(int)2064, framerate=(fraction)22/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false
The maximum width supported by UYVY mode is 2880.
5.4 Trigger mode
5.4.1 Prepare
Take MV-MIPI-IMX178M, 3088*2064 as an example.
v4l2-ctl --set-ctrl roi_x=0
v4l2-ctl --set-ctrl roi_y=0
v4l2-ctl --set-ctrl low_latency_mode=1
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=3136
5.4.2 Software trigger mode
5.4.2.1 Set mode
v4l2-ctl --set-ctrl trigger_mode=1
v4l2-ctl --set-ctrl trigger_src=0
v4l2-ctl --set-ctrl vi_time_out_disable=1
5.4.2.2 Start acquisition
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY --stream-mmap --stream-count=-1 --stream-to=y8-3104x2064.raw
5.4.2.3 Perform soft trigger operation
In other shell terminals, you can execute the following command multiple times for multiple triggers.
v4l2-ctl --set-ctrl soft_trgone=1
5.4.2.4 Stop triggering and capturing
Since the kernel driver is waiting for new images, you need to cancel vi_time_out_disable before exiting the acquisition operation.
v4l2-ctl --set-ctrl vi_time_out_disable=0
In the terminal of the acquisition command, press Ctrl+C to exit the acquisition operation.
5.4.3 Hardware trigger mode
To use jetson-gpio for trigger operation, please install and configure jetson-gpio first.
The following is an example of using jetson GPIO40 (Board number) as the trigger source and rising edge triggering.
You can use the mv_mipi_i2c.sh script to set other trigger parameters.
5.4.3.1 Hardware Connection
5.4.3.2 Set mode
v4l2-ctl --set-ctrl trigger_mode=1
v4l2-ctl --set-ctrl trigger_src=1
v4l2-ctl --set-ctrl vi_time_out_disable=1
5.4.3.3 Start acquisition
v4l2-ctl --set-fmt-video=width=3088,height=2064,pixelformat=GREY --stream-mmap --stream-count=-1 --stream-to=y8-3104x2064.raw
5.4.3.4 Perform hardware trigger operation
python gpio_trigger_jetson.py
Note: script link.
5.4.3.5 Stop triggering and capturing
Since the kernel driver is waiting for new images, you need to cancel vi_time_out_disable before exiting the acquisition operation.
v4l2-ctl --set-ctrl vi_time_out_disable=0
In the terminal of the acquisition command, press Ctrl+C to exit the acquisition operation.
6 Application demo
6.1 v4l2grab_mvcam_jetson
v4l2grab_mvcam is a sample developed in C language, supporting stream mode and trigger mode.
Please refer to：https://github.com/veyeimaging/nvidia_jetson_veye_bsp/tree/master/mv_tools_jetson/examples/v4l2grab_mvcam
6.2 yavta(Video Streaming mode only)
6.2.1 install yavta
git clone https://github.com/veyeimaging/yavta.git
cd yavta;make
6.2.2 Set ROI and format
Take MV-MIPI-IMX178M, 3088*2064 as an example.
v4l2-ctl --set-ctrl roi_x=0
v4l2-ctl --set-ctrl roi_y=0
6.2.3 Save image to file
- raw8
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=3136
./yavta -c1 -Fy8-3136x2064.raw --skip 0 -f Y8 -s 3088x2064 /dev/video0
- raw10
Take XAVIER as example.
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=6208
./yavta -c1 -Fy10-3104x2064.raw --skip 0 -f XY10 -s 3088x2064 /dev/video0
- raw12
Take XAVIER as example.
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=6208
./yavta -c1 -Fy12-3104x2064.raw --skip 0 -f XY12 -s 3088x2064 /dev/video0
For the image format, please refer to the section above: Description of raw data image format.
7 i2c script for parameter configuration
We provide shell scripts to configure the parameters.
8 References
9 Document History
- 2022-05-20
Release 1st version.