VEYE CS Camera for Jetson TX2
How to use VEYE and CS series camera module on NVIDIA Jetson Nano,TX2,Xavier and Orin
1 Overview
This guide shows how to use VEYE and CS series camera modules on Jetson Nano,Jetson Nano 2GB,TX2,TX2 NX,AGX Xavier and Xavier NX. Jetson SDK Version is
- Jetpack4.2.2,L4T r32.2.1
- Jetpack4.3,L4T r32.3.1
- Jetpack4.4,L4T r32.4.3
- Jetpack4.4.1,L4T r32.4.4
- Jetpack4.5,L4T r32.5
- Jetpack4.5.1,L4T r32.5.1
- Jetpack4.6,L4T r32.6.1
- Jetpack4.6.1, L4T r32.7.1
We provide two ways to do so: Prebuilt Binaries or Source Code. Yes, It's Open Source!
VEYE and CS series camera modules are camera module with ISP functions build in. It output UYVY data using MIPI-CSI2. We provide V4L2 interface for video streaming apps , and Video Control Toolkits (which is Shell Script) to control the camera module directly, which is called DRA(Directly Register Access).
1.1 Camera module list
| Series | Model | Status |
|---|---|---|
| VEYE series | VEYE-MIPI-IMX327S | Done |
| VEYE series | VEYE-MIPI-IMX385 | Done |
| VEYE series | VEYE-MIPI-IMX462 | Done |
| VEYE series | VEYE-MIPI-IMX335 | Done |
| CS series | CS-MIPI-IMX307 | Done |
| CS series | CS-MIPI-SC132 | Done |
| TX2-XAVIER-nCAM Series | CS-TX2-XAVIER-nCAM | Done |
1.2 How to check the current L4T version
On Jetson board
cat /etc/nv_tegra_release
If it shows:
# R32 (release), REVISION: 4.3......
It means L4t Version is 32.4.3
2 Hardware Setup
We have designed a 6cam interposer board for TX2 devkit,Xavier and Orin, It support up to 6 cameras at the same time.
In particular, the VEYE-MIPI-IMX335 must use a 12V DC terminal to provide auxiliary power due to its high power consumption. Refer to J11 of ADP-N1.
2.1 NVIDIA TX2 Developer Kit
It is connected as shown in figure:
Be sure to screw the ADP adapter board to the bottom board.
2.2 NVIDIA AGX Xavier(the same with Orin)
Be sure to screw the ADP adapter board to the bottom board.
2.3 NVIDIA Nano ，TX2 NX and Xavier NX Devkit
The Xavier NX interface is compatible with raspberry pi and Jetson Nano. The connection mode is shown in figure:
3 Upgrade Jetson Nano, TX2,Xavier and Orin system
3.1 Overview
This section describes how to upgrade the Jetson system to support our camera module.
- Jetpack4.x versions
To support our camera module, we need to update the two parts of the L4T (Linux for Tegra) of the Jetson system, Image and DTB.
In the Image, we added the camera driver, while the DTB indicates the camera model used. In general, you only need to use the Image and DTB, that we have prebuilt, and you don't need to build from source when it is not necessary.
- Jetpack5.x versions
The driver is no longer compiled into the Image, but loaded into the system as a separate module. The dtb file is still compiled together as a whole to facilitate replacement.
The dtb can also be loaded dynamically as an overlay.
3.2 BSP package introduction
3.2.1 Path
https://github.com/veyeimaging/nvidia_jetson_veye_bsp
3.2.2 bsp package includes：
- prebuilt dtb for different boards
- driver source code
- dts source code
- i2c toolkits
In release packages
- prebuilt linux kernel：Image
- prebuilt camera drivers
3.3 DTB file corresponds to Jetson boards
3.3.1 Nano A02
tegra210-p3448-0000-p3449-0000-a02.dtb
3.3.2 Nano B01
tegra210-p3448-0000-p3449-0000-b00.dtb
3.3.3 Nano 2GB
tegra210-p3448-0003-p3542-0000.dtb
3.3.4 TX2 Devkit
tegra186-quill-p3310-1000-c03-00-base.dtb
3.3.5 TX2 NX(p3509-0000 carrier board)
use the same carrier board with XAVIER NX Devkit
tegra186-p3636-0001-p3509-0000-a01.dtb
3.3.6 AGX XAVIER
tegra194-p2888-0001-p2822-0000.dtb
3.3.7 XAVIER NX Devkit
tegra194-p3668-all-p3509-0000.dtb
3.3.8 AGX Orin
tegra234-p3701-0000-p3737-0000.dtb
3.4 DTB file name and camera module correspondence
Under the dtb directory of each motherboard, the directory corresponding to the camera model is stored, and you can usually know the corresponding camera model directly by the directory name.
Specially:
1, VEYE-MIPI-CAM2M is the new version of VEYE series dtb, which supports all VEYE 2M series products, including VEYE-MIPI-327E, VEYE-MIPI-IMX327S, VEYE-MIPI-IMX462, VEYE-MIPI-IMX385.
2, The ones with fpdlink as suffix correspond to fpdlink coaxial model module.
3.5 Upgrade directly on the jetson board
Applicable models: Jetson Nano, Xavier and Orin.
Note: AGX Xavie Jetpack version >= 4.4 , support this upgrade mode. Earlier versions have not been verified and are not recommended.
3.5.1 Check the current L4T version
First check the current L4T version and try to replace it with the same version.
cat /etc/nv_tegra_release
If it shows:
# R32 (release), REVISION: 4.3......
It means L4t Verion is 32.4.3
3.5.2 Download the bsp package to the Jetson board
On Jetson Board:
wget https://github.com/veyeimaging/nvidia_jetson_veye_bsp/releases/latest/download/nvidia_jetson_veye_bsp.tgz
tar -xzvf nvidia_jetson_veye_bsp.tgz
3.5.3 Upgrade Image(For Jetpack4.x)
Backup old version Image
cp /boot/Image /boot/Image.backup
cp /boot/Image.sig /boot/Image.sig.backup
Under the bsp package directory, find the corresponding version of the Image package, decompress it, and execute:
sudo cp <path to your Image dir>/Image /boot/Image -f
For Xavier NX Devkit
Need to use the signed file(For example，file inside Image_l4t_r32.*.*_veyecam_signed_4_xavier),copy Image and Image.sig to /boot/ dir on the Jetson Xavier board:
sudo cp <path to your Image dir>/Image /boot/Image -f
sudo cp <path to your Image dir>/Image.sig /boot/Image.sig -f
Effective after restart.
3.5.4 install camera modules(For Jetpack5.x)
Check linux version:
uname -r
Find the corresponding directory in nvidia_jetson_veye_bsp/ko :
sudo cp nvidia_jetson_veye_bsp/ko/$(uname -r)/* /lib/modules/$(uname -r)/kernel/drivers/media/i2c/ -f
sudo depmod
3.5.5 Upgrade dtb
Under the bsp package directory, find the dtb file corresponding to your camera module model and L4T version number.
sudo mkdir /boot/veyecam/
sudo cp <path to your dtb dir>/<DTB file name> /boot/veyecam/ -f
Backup extlinux.conf file first
cp /boot/extlinux/extlinux.conf / boot/extlinux/extlinux.conf .back
Then, edit the /boot/extlinux/extlinux.conf file and add the following line.
FDT /boot/veyecam/<DTB file name>
Note: <DTB file name> should be replaced with the corresponding dtb name of the mainboard, ensure that the file specified in the FDT line actually exists correctly.
Effective after restart.
3.6 Upgrade Jetson Board using Host PC
This section provides the second method of upgrading the system.
This method has the same goal as the method in the previous section, and one of the two can be chosen.
Specifically, for TX2 systems, only the methods in this section can be taken to upgrade.
3.6.1 Burn the standard Image
Note: if your Jetson board has correctly burned the standard version Image, you can skip this step.
You can burn the standard Image using SDK Manager or following the steps below.
- Set PATH
Assume SDK install directory is <TOPDIR>.
export TOP_DIR=<absolute path to top dir accroding to your jetpack version and board name;for example /home/xumm/nvidia/nvidia_sdk/JetPack_4.5_Linux_JETSON_XAVIER_NX_DEVKIT/>
export L4T_DIR=$TOP_DIR/Linux_for_Tegra
- Build default image, flashing to Jetson Board, prepare L4T environment
cd $L4T_DIR
sudo ./apply_binaries.sh
- Nano
sudo ./flash.sh jetson-nano-devkit mmcblk0p1
sudo ./flash.sh jetson-nano-2gb-devkit mmcblk0p1
- TX2
sudo ./flash.sh jetson-tx2 mmcblk0p1
- TX2 NX(p3509-0000 carrier board)
sudo ./flash.sh jetson-xavier-nx-devkit-tx2-nx mmcblk0p1
- AGX Xavier
sudo ./flash.sh jetson-xavier mmcblk0p1
- Xavier NX Devkit
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1
3.6.2 Download our bsp package for Jetson
cd $L4T_DIR
wget https://github.com/veyeimaging/nvidia_jetson_veye_bsp/releases/latest/download/nvidia_jetson_veye_bsp.tgz
tar -xzvf nvidia_jetson_veye_bsp.tgz
export RELEASE_PACK_DIR=$L4T_DIR/nvidia_jetson_veye_bsp
3.6.3 Upgrade Image and DTB
prebuilt Image on github:https://github.com/veyeimaging/nvidia_jetson_veye_bsp/tree/master/kernel_image
- Installing the Kernel
Whether you use a prebuild kernel or you build it yourself
- Nano and TX2
copy Image to /boot/ dir on the Jetson Nano or TX2 board:
sudo cp <path to your Image dir>/Image /boot/Image -f
- AGX Xavier and Xavier NX Devkit
Need to use the signed file(For example，file inside Image_l4t_r32.*.*_veyecam_signed_4_xavier),copy Image and Image.sig to /boot/ dir on the Jetson Xavier board:
sudo cp <path to your Image dir>/Image /boot/Image -f
sudo cp <path to your Image dir>/Image.sig /boot/Image -f
on the Host PC:
cd $L4T_DIR
sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1
- Xavier NX
on the Host PC：
cd $L4T_DIR
sudo ./flash.sh -k kernel jetson-xavier-nx-devkit mmcblk0p1
- Flashing dtb file
Enter the recovery mode，and on the Host PC:
cp <path to your dtb dir>/dtbfilename $L4T_DIR/kernel/dtb/ -f
cd $L4T_DIR
- Nano
sudo ./flash.sh -r -k DTB jetson-nano-qspi-sd mmcblk0p1
sudo ./flash.sh -r -k DTB jetson-nano-devkit mmcblk0p1
sudo ./flash.sh -r -k DTB jetson-nano-2gb-devkit mmcblk0p1
- TX2
sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1
- TX2 NX(p3509-0000 carrier board)
sudo ./flash.sh -r -k kernel-dtb jetson-xavier-nx-devkit-tx2-nx mmcblk0p1
- AGX Xavier
sudo ./flash.sh -r -k kernel-dtb jetson-xavier mmcblk0p1
- Xavier NX
sudo ./flash.sh -r -k kernel-dtb jetson-xavier-nx-devkit mmcblk0p1
Reset Jetson Board after flashing.
3.7 Are Image and DTB upgrade successful ?
3.7.1 For Image
ls /sys/bus/i2c/drivers/
There should have veye327 veyecam csx307 cssc132 directory.
3.7.2 For DTB
DTB varies from platform and from camera module.
- Nano A02/ Nano 2G
ls /proc/device-tree/host1x/i2c@546c0000/
- Nano B01
ls /proc/device-tree/cam_i2cmux/i2c@*
- TX2 Devkit
ls /proc/device-tree/i2c@3180000/tca9548@70/i2c@*
or
ls /proc/device-tree/i2c@3180000/tca9548@77/i2c@*
- AGX Xavier/AGX Orin
ls /proc/device-tree/i2c@3180000/tca9548@70/i2c@*
or
ls /proc/device-tree/i2c@3180000/tca9548@77/i2c@*
- Xavier NX and TX2 NX(p3509-0000 carrier board)
ls /proc/device-tree/cam_i2cmux/i2c@*
You should be able to find the camera model you used.
3.8 Protection against being overwritten by upgrades
After making sure Image and dtb are successfully updated, please execute the following command to prevent kernel and dtb from being overwritten by update.
sudo apt-mark hold nvidia-l4t-kernel nvidia-l4t-kernel-dtbs
3.9 Build the drivers from source
Please refer to: Nvidia Jetson platform VEYE and CS series camera driver source code compilation guide.
Could be used for Flashing or Upgrading.
4 Applications and Test
4.1 Check system status
Run the following command to confirm whether the camera is probed.
- VEYE-MIPI-IMX327S/VEYE-MIPI-IMX462/VEYE-MIPI-IMX385(using VEYE-MIPI-CAM2M dtb)
dmesg | grep veye
The output message appears as shown below：
camera id is veyecam
sensor is IMX327/IMX462/IMX385
subdev veyecam [i2c_bus]-003b bound
- For CS-MIPI-IMX307
dmesg | grep csx307
The output message appears as shown below.：
Detected CS307 sensor
subdev csx307 [i2c_bus]-003b bound
- For CS-MIPI-SC132
dmesg | grep cssc132
The output message appears as shown below.：
subdev cssc132 [i2c_bus]-003b bound
- For VEYE-MIPI-IMX335
dmesg | grep imx335
The output message appears as shown below.：
camera id is VEYE-MIPI-IMX335
subdev veye_imx335 [i2c_bus]-003b bound
- Run the following command to check the presence of video node.
ls /dev/video*
The output message appears as shown below.
videoX
- For VEYE-MIPI-327(using VEYE-MIPI-327 dtb)
dmesg | grep veye327
The output message appears as shown below：
Detected VEYE327 sensor
subdev veye327 [i2c_bus]-003b bound
The [i2c_bus] in the driver prompt message indicates the i2c bus corresponding to this camera.
4.2 Video Stream Toolkits Manual
The camera appears as /dev/videoX device node,where X is between [0-5].
During the Linux boot process, the cameras are detected in the order of i2c bus from smallest to largest. For example, if only one camera is connected, X is 0 regardless of the hardware connection to the location. If 5 cameras are connected, X is reflected as [0-4] according to i2c bus from smallest to largest.
In gstreamer command ，v4l2src has a param device=/dev/videoX to identify which camera to use.
4.2.1 Gstreamer Usage
- Preview FHD
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvoverlaysink sync=false
- Preview FHD(using xvimagesink sink)
export DISPLAY=:0
gst-launch-1.0 -e v4l2src io-mode=4 device=/dev/video0 do-timestamp=true ! 'video/x-raw, width=1920, height=1080, framerate=30/1, format=UYVY' ! xvimagesink sync=false
- Preview 720p@60 (CS-MIPI-IMX307)
gst-launch-1.0 nvv4l2camerasrc ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)1280, height=(int)720, framerate=(fraction)60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false
- Preview 1280*1080@45 (CS-MIPI-SC132)
gst-launch-1.0 nvv4l2camerasrc ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)1280, height=(int)1080, framerate=(fraction)45/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false
- Preview 640*480p@130 (CS-MIPI-IMX307)
gst-launch-1.0 nvv4l2camerasrc ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)130/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false
- Preview 2 cameras 1080p HD
WIDTH=960
HEIGHT=540
CAPS="video/x-raw(memory:NVMM),format=(string)UYVY, width=1920, height=1080"
gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=$WIDTH sink_0::height=$HEIGHT sink_1::xpos=$WIDTH sink_1::ypos=0 sink_1::width=$WIDTH sink_1::height=$HEIGHT ! nvoverlaysink nvv4l2camerasrc device=/dev/video0 ! $CAPS ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420"! comp. nvv4l2camerasrc device=/dev/video1 ! $CAPS ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420"! comp.
- Gstreamer cmd line for OpenCV
I think maybe OpenCV do not support I420 data format input，so you should convert it to BGR format.
gst-launch-1.0 nvv4l2camerasrc ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1920, height=(int)1080 ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)I420 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink
- Record FHD in H.264 format to a video file
gst-launch-1.0 nvv4l2camerasrc num-buffers=300 ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvv4l2h264enc control-rate=1 bitrate=10000000 ! h264parse ! qtmux ! filesink location=filename.mp4 -e
- Playback of saved video file
gst-launch-1.0 filesrc location=filename.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nvoverlaysink -e
- Capturing FHD still image
gst-launch-1.0 v4l2src num-buffers=1 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvjpegenc ! filesink location=jpgname.jpg
4.2.2 VEYE-MIPI-IMX335 gstreamer example
Since VEYE-MIPI-IMX335 supports only a few specific resolution modes, its driver we use use_sensor_mode_id mode.
| ./veye5_mipi_i2c.sh video mode | v4l2-ctl sensor_mode | video format |
|---|---|---|
| 1 | 0 | 2592x1944@20fps |
| 2 | 1 | 2592x1944@12.5fps |
| 3 | 2 | 2560x1440@25fps |
| 4 | 3 | 2560x1440@30fps |
- prepare
sudo apt-get install v4l-utils
Refer to the following section and download the veye5_mipi_i2c.sh tool.
- 2592x1944@20fps mode preview, framerate statistics
v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=0
./veye5_mipi_i2c.sh -w -f videomode -p1 1
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=5376
Preview
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! "video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false
Framerate statistics
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! "video/x-raw(memory:NVMM),format=(string)UYVY,width=(int)2592,height=(int)1944" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! fpsdisplaysink video-sink=fakesink -v
- 2560x1440@30fps mode video encode and capture
v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=3
./veye5_mipi_i2c.sh -w -f videomode -p1 4
v4l2-ctl -d /dev/video0 --set-ctrl preferred_stride=5120
Video encode
gst-launch-1.0 nvv4l2camerasrc num-buffers=300 ! "video/x-raw(memory:NVMM),format=(string)UYVY, width=(int)2560, height=(int)1440" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvv4l2h264enc control-rate=0 bitrate=10000000 ! h264parse ! qtmux ! filesink location=filename.mp4 -e
Video Replay
gst-launch-1.0 filesrc location=filename.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nvoverlaysink -e
Capture
gst-launch-1.0 v4l2src num-buffers=1 ! "video/x-raw,format=(string)UYVY, width=(int)2560, height=(int)1440" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvjpegenc ! filesink location=jpgname.jpg
4.2.3 gstreamer examples on Jetpack5.x
On Jetpack5.x：
- nvoverlaysink is obsolete, nv3dsink can be used instead.
- There are still bugs in nvv4l2camerasrc, use v4l2src for now.
- Preview 1080p HD
gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY,width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nv3dsink -e
- Video Encode1080p HD
gst-launch-1.0 v4l2src num-buffers=300 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvv4l2h264enc control-rate=1 bitrate=10000000 ! h264parse ! qtmux ! filesink location=filename.mp4 -e
- Capture
gst-launch-1.0 v4l2src num-buffers=1 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvjpegenc ! filesink location=jpgname.jpg
4.3 Video Control Toolkits Manual
- VEYE-MIPI-CAM2M Series（VEYE-MIPI-327、VEYE-MIPI-IMX327S、VEYE-MIPI-IMX462、VEYE-MIPI-IMX385）
Video Control Toolkits Manual ：VEYE-MIPI-327 I2C
- CS-MIPI-IMX307和CS-MIPI-SC132
Video Control Toolkits Manual ：CS-MIPI-X I2C
- VEYE-MIPI-IMX335
Video Control Toolkits Manual ：VEYE-MIPI-IMX335 I2C
5 How to port the driver to a third party board
6 Document History
- 20220629
Add support for Jetpack 5.0.1DP and tune through AGX-Orin.
- 20220110
Add VEYE-MIPI-IMX385 support.
- 20220105
Use nvv4l2h264enc instead of omxh264enc in gstreamer command，because omxh264enc has been deprecated.
- 20211025
Use nvcamerasrc instead of v4l2src in gstreamer command , and the data goes directly into DMA memory.
7 References
Jetson start up： https://developer.nvidia.com/embedded/learn/getting-started-jetson
Jetson Download Center: https://developer.nvidia.com/embedded/downloads
Xavier NX: https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit
TX2 development kit： https://developer.nvidia.com/embedded/jetson-tx2-developer-kit
SDK Manager: https://docs.nvidia.com/sdk-manager/index.html
L4T Doc: https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3242/index.htm
TX2 user guide: link