VEYE CS Camera for Jetson TX2

How to use VEYE and CS series camera module on NVIDIA Jetson TX2,AGX Xavier and Xavier NX

1 Overview

This guide shows how to use VEYE and CS series camera modules on Jetson TX2,AGX Xavier and Xavier NX. Jetson SDK Version is(Jetpack4.2.2,L4T r32.2.1)and(Jetpack4.3,L4Tr32.3.1) and (Jetpack4.4,L4T,r32.4.2). We provide two ways to do so: Prebuilt Binaries or Source Code. Yes, It's Open Source!

VEYE and CS series camera modules are STARVIS camera module with ISP functions build in. It output UYVY data using MIPI-CSI2. We provide V4L2 interface for video streaming apps , and Video Control Toolkits (which is Shell Script) to control the camera module directly, which is called DRA(Directly Register Access).

2 Hardware Setup

We have designed a 6cam interposer board for TX2 devkit and Xavier, It support up to 6 cameras at the same time.

6cam interposer board connected with 6 cameras

2.1 NVIDIA TX2 Developer Kit

It is connected as shown in figure:

TX2 Devkit 6 cameras connection

2.2 NVIDIA TX2 AGX Xavier

Connect VEYE cameras to AGX Xavier (the green interposer board is temporary，the official version is black. )

2.3 NVIDIA Xavier NX Devkit

The Xavier NX interface is compatible with raspberry pi and Jetson Nano. The connection mode is shown in figure:

Xavier NX connection with VEYE-MIPI-327

Xavier NX connection with CS-MIPI-IMX307

3 Buruing the prebuilt Image to upgrade Jetson Xavier NX Devkit

You can use the image prebuild by us to upgrade jetson nano system, which will support our camera.

Using whole prebuilt image

download ：

link：https://pan.baidu.com/s/1sWrcfEOivjuQ7T7P5bVcng

If Baidu cloud disk download is too slow, you can contact our company customer service xumm@csoneplus.com to obtain.

How to burn

refer to Official document

4 Upgrade Jetson TX2, AGX Xavier and Xavier NX system

4.1 Setting Up the Host PC Environment

Cross-compiling Toolchain

Please refer to this link to install toolchain on your Host PC.

L4T source code and Rootfs

You could use SDK Manager or directly download to get source code.

Sync code as this if you use SDK Manager:

- Jetpack4.2.2

./source_sync.sh -t tegra-l4t-r32.2.1

- Jetpack4.3

./source_sync.sh -t tegra-l4t-r32.3.1

- Jetpack4.4

./source_sync.sh -t tegra-l4t-r32.4.2

Setting Up the Environment

Assume SDK install directory is <TOPDIR>，source code is in $L4T_DIR/sources directory.

- TX2@Jetpack4.2.2

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.2.2_Linux_GA_P3310/

- TX2@Jetpack4.3

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.3_Linux_P3310/

- AGX XavierJetpack4.2.2

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.2.2_Linux_GA_P2888/

- AGX XavierJetpack4.3

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.3_Linux_P2888/

- Xavier NX Jetpack4.4

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT

- Common part

export L4T_DIR=$TOP_DIR/Linux_for_Tegra

export LOCALVERSION=-tegra

export LDK_ROOTFS_DIR=$TOP_DIR/Linux_for_Tegra/rootfs

export ARCH=arm64

export CROSS_COMPILE=aarch64-linux-gnu-

export CROSS32CC=arm-linux-gnueabihf-gcc

export TEGRA_KERNEL_OUT=$L4T_DIR/sources/kernel/out_kernel

export KERNEL_PATH=$L4T_DIR/sources/kernel/out_kernel

export NVIDIA_PATH=$L4T_DIR/sources/kernel/nvidia

export TX2_DTS_PATH=$L4T_DIR/sources/hardware/nvidia/platform/t18x

export XAVIER_DTS_PATH=$L4T_DIR/sources/hardware/nvidia/platform/t19x

cd $L4T_DIR/sources/kernel/kernel-4.9/

Build default image, flashing to Jetson Board, prepare L4T environment

cd $L4T_DIR

sudo ./apply_binaries.sh

- TX2

sudo ./flash.sh jetson-tx2 mmcblk0p1

- AGX Xavier

sudo ./flash.sh jetson-xavier mmcblk0p1

- Xavier NX Devkit

sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

4.2 Download out bsp package for Jetson

cd $L4T_DIR

git clone https://github.com/veyeimaging/nvidia_jetson_veye_bsp.git

export RELEASE_PACK_DIR=$L4T_DIR/nvidia_jetson_veye_bsp

4.2.1 bsp package includes：

prebuild linux kernel：Image
prebuild dtb for different boards
driver source code
dts source code
i2c toolkits

4.3 Using prebuild Image and DTB

Installing the Kernel

Whether you use a prebuild kernel or you build it yourself

- TX2

copy Image to /boot/ dir on the Jetson TX2 board:

sudo cp <path to your Image dir>/Image /boot/Image -f

- AGX Xavier

on the Host PC:

cd $L4T_DIR

sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1

- Xavier NX

on the Host PC：

cd $L4T_DIR

sudo ./flash.sh -k kernel jetson-xavier-nx-devkit mmcblk0p1

Flashing dtb file

Enter the recovery mode，and on the Host PC:

cp <path to your dtb dir>/dtbfilename $L4T_DIR/kernel/dtb/ -f

cd $L4T_DIR

- TX2

sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1

- AGX Xavier

sudo ./flash.sh -r -k kernel-dtb jetson-xavier mmcblk0p1

- AGX Xavier

sudo ./flash.sh -r -k kernel-dtb jetson-xavier-nx-devkit mmcblk0p1

Reset Jetson Board after flashing.

4.4 Building from Source

4.4.1 Build kernel

patch code

cp $RELEASE_PACK_DIR/drivers_source/cs_imx307\ veye327/* $NVIDIA_PATH/drivers/media/i2c/

- r32.2.1

cp $RELEASE_PACK_DIR/drivers_source/kernel_csimx307veye327_config_32.2.1 $L4T_DIR/sources/kernel/kernel-4.9/arch/arm64/configs/tegra_csimx307veye327_defconfig

- r32.3.1

cp $RELEASE_PACK_DIR/drivers_source/kernel_csimx307veye327_config_32.3.1 $L4T_DIR/sources/kernel/kernel-4.9/arch/arm64/configs/tegra_csimx307veye327_defconfig

- r32.4.2

cp $RELEASE_PACK_DIR/drivers_source/kernel_csimx307veye327_config_32.4.2 $L4T_DIR/sources/kernel/kernel-4.9/arch/arm64/configs/tegra_csimx307veye327_defconfig

build

cd $L4T_DIR/sources/kernel/kernel-4.9/

make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_csimx307veye327_defconfig

make ARCH=arm64 O=$TEGRA_KERNEL_OUT Image -j4

Completed Image file is $TEGRA_KERNEL_OUT/arch/arm64/boot/Image，could be used for Flashing or Upgrading.

Install Image for Flashing:

sudo cp $TEGRA_KERNEL_OUT/arch/arm64/boot/Image $L4T_DIR/kernel/ -f

4.4.2 Build DTS

patch code

- Jetson TX2@Jetpack4.2.2

cp $RELEASE_PACK_DIR/TX2/JetPack_4.2.2_Linux_GA_P3310/dts\ dtb/common/t18x/* -r $TX2_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/TX2/JetPack_4.2.2_Linux_GA_P3310/dts\ dtb/CS-MIPI-IMX307/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/TX2/JetPack_4.2.2_Linux_GA_P3310/dts\ dtb/VEYE-MIPI-327/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

- TX2@Jetpack4.3

cp $RELEASE_PACK_DIR/TX2/JetPack_4.3_Linux_P3310/dts\ dtb/common/t18x/* -r $TX2_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/TX2/JetPack_4.3_Linux_P3310/dts\ dtb/CS-MIPI-IMX307/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/TX2/JetPack_4.3_Linux_P3310/dts\ dtb/VEYE-MIPI-327/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

- AGX Xavier@Jetpack4.2.2

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.2.2_Linux_GA_P2888/dts\ dtb/common/t19x/* -r $XAVIER_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.2.2_Linux_GA_P2888/dts\ dtb/CS-MIPI-IMX307/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.2.2_Linux_GA_P2888/dts\ dtb/VEYE-MIPI-327/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

- AGX Xavier@Jetpack4.3

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.3_Linux_P2888/dts\ dtb/common/t19x/* -r $XAVIER_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.3_Linux_P2888/dts\ dtb/CS-MIPI-IMX307/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.3_Linux_P2888/dts\ dtb/VEYE-MIPI-327/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

- Xavier NX@Jetpack4.4

cp $RELEASE_PACK_DIR/Xavier-NX/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT/dts\ dtb/common/t19x/* -r $XAVIER_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/Xavier-NX/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT/dts\ dtb/CS-MIPI-IMX307/tegra194-p3509-0000-a00.dtsi $TX2_DTS_PATH/jakku/kernel-dts/common/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/Xavier-NX/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT/dts\ dtb/VEYE-MIPI-327/tegra194-p3509-0000-a00.dtsi $TX2_DTS_PATH/jakku/kernel-dts/common/

build

cd $L4T_DIR/sources/kernel/kernel-4.9/

make ARCH=arm64 O=$TEGRA_KERNEL_OUT dtbs

- TX2

cp $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/tegra186-quill-p3310-1000-c03-00-base.dtb $L4T_DIR/kernel/dtb/

- AGX XAVIER

cp $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/tegra194-p2888-0001-p2822-0000.dtb

- XAVIER NX

cp $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb $L4T_DIR/kernel/dtb/

Could be used for Flashing or Upgrading.

5 Applications and Test

5.1 Check system status

Run the following command to confirm whether the camera is probed.

For VEYE-MIPI-327

dmesg | grep veye327

The output message appears as shown below.：

Detected VEYE327 sensor

subdev veye327 3X-003b bound

此外，检查/dev/videoX设备节点，应当是存在的。

For CS-MIPI-IMX307

dmesg | grep csx307

The output message appears as shown below.：

Detected CS307 sensor

subdev csimx307 3X-003b bound

Run the following command to check the presence of video node.

ls /dev/video*

The output message appears as shown below.

videoX

5.2 Video Stream Toolkits Manual

Our camera appears as /dev/videoX device node,where X is between [0-5].

in gstreamer command ，v4l2src has a param device=/dev/videoX to identify which camera to use.

5.2.1 Gstreamer Usage

Preview FHD(HW accelerated)

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false

Preview 720p@60 (CS-MIPI-IMX307,HW accelerated)

gst-launch-1.0 v4l2src ! "video/x-raw,format=(string)UYVY, width=(int)1280, height=(int)720, framerate=(fraction)60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false

Preview 640*480p@130 (CS-MIPI-IMX307,HW accelerated)

gst-launch-1.0 v4l2src ! "video/x-raw,format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)130/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false

Preview 2 cameras 1080p HD(HW accelerated)

WIDTH=960

HEIGHT=540

CAPS="video/x-raw,format=(string)UYVY, width=1920, height=1080"

gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=$WIDTH sink_0::height=$HEIGHT sink_1::xpos=$WIDTH sink_1::ypos=0 sink_1::width=$WIDTH sink_1::height=$HEIGHT ! nvoverlaysink v4l2src device=/dev/video0 ! $CAPS ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420"! comp. v4l2src device=/dev/video1 ! $CAPS ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420"! comp.

Record FHD in H.264 format to a video file（HW accelerated）

gst-launch-1.0 v4l2src num-buffers=300 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! omxh264enc qp-range=20,20:20,20:-1,-1 ! matroskamux ! queue ! filesink location=videoname.mkv

Playback of saved video file (HW accelerated）

gst-launch-1.0 filesrc location=videoname.mkv ! matroskademux ! h264parse ! omxh264dec ! nvoverlaysink

Capturing FHD still image

gst-launch-1.0 v4l2src num-buffers=1 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! jpegenc ! filesink location=jpgname.jpg

5.3 Video Control Toolkits Manual

i2c bus description：

ADP-N1 has 6 MIPI CSI-2 interface [A-F]，the corresponding i2c address is[30-35].

It is also the same as the 3X part of dmesg message:

subdev veye327 3X-003b bound

subdev csimx307 3X-003b bound

using -b option to identify which bus you want to use.

VEYE-MIPI-327

Video Control Toolkits Manual ：VEYE-MIPI-327 I2C

CS-MIPI-IMX307

Video Control Toolkits Manual ：CS-MIPI-IMX307 I2C

6 Realtimes RTSO-9001

6.1 Hardware Setup

CS-MIPI-IMX307 and RTSO-9001（TX2）

Matching with Realtimes RTSO-9001, you need to use RTST-RPI adapter board.

Note: the power supply capacity of the MIPI CSI-2 interface of the current version of RTSO-9001 is not enough to support the power requirements of our camera, which requires an additional 5V power supply to the camera. It is not shown in the picture above.

6.2 Software bsp

The corresponding directory for Realtimes RTSO-9001, is：

nvidia_jetson_veye_bsp\TX2\JetPack_4.2.2_Linux_RTSO-9001

Subject to the agreement with Ruitai, we do not release the source code
Our code have been submitted to Realtimes, and customers are advised to contact Realtimes for a complete version of the supported image.