VEYE CS Camera for Jetson TX2

查看中文

How to use VEYE and CS series camera module on NVIDIA Jetson TX2 and AGX Xavier

1 Overview

This guide shows how to use VEYE and CS series camera modules on Jetson TX2 and AGX Xavier. Jetson SDK Version is(Jetpack4.2.2,L4T r32.2.1)and(Jetpack4.3,L4Tr32.3.1). We provide two ways to do so: Prebuilt Binaries or Source Code. Yes, It's Open Source!

VEYE and CS series camera modules are STARVIS camera module with ISP functions build in. It output UYVY data using MIPI-CSI2. We provide V4L2 interface for video streaming apps , and Video Control Toolkits (which is Shell Script) to control the camera module directly, which is called DRA(Directly Register Access).

2 Hardware Setup

We have designed a 6cam interposer board for TX2 devkit and Xavier, It support up to 6 cameras at the same time.

6cam interposer board connected with 6 cameras

2.1 NVIDIA TX2 Developer Kit

It is connected as shown in figure:

TX2 Devkit 6 cameras connection

2.2 NVIDIA TX2 AGX Xavier

Connect VEYE cameras to AGX Xavier (the green interposer board is temporary，the official version is black. )

3 Upgrade Jetson TX2 and AGX Xavier system

3.1 Setting Up the Host PC Environment

• Cross-compiling Toolchain

Please refer to this link to install toolchain on your Host PC.

• L4T source code and Rootfs

You could use SDK Manager or directly download to get source code.

Sync code as this if you use SDK Manager:

- Jetpack4.2.2

./source_sync.sh -t tegra-l4t-r32.2.1

- Jetpack4.3

./source_sync.sh -t tegra-l4t-r32.3.1

• Setting Up the Environment

Assume SDK install directory is <TOPDIR>，source code is in $L4T_DIR/sources directory.

- TX2@Jetpack4.2.2

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.2.2_Linux_GA_P3310/

- TX2@Jetpack4.3

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.3_Linux_P3310/

- AGX XavierJetpack4.2.2

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.2.2_Linux_GA_P2888/

- AGX XavierJetpack4.3

export TOP_DIR=/home/xumm/nvidia/nvidia_sdk/JetPack_4.3_Linux_P2888/

- Common part

export L4T_DIR=$TOP_DIR/Linux_for_Tegra

export LOCALVERSION=-tegra

export LDK_ROOTFS_DIR=$TOP_DIR/Linux_for_Tegra/rootfs

export ARCH=arm64

export CROSS_COMPILE=aarch64-linux-gnu-

export CROSS32CC=arm-linux-gnueabihf-gcc

export TEGRA_KERNEL_OUT=$L4T_DIR/sources/kernel/out_kernel

export KERNEL_PATH=$L4T_DIR/sources/kernel/out_kernel

export NVIDIA_PATH=$L4T_DIR/sources/kernel/nvidia

export TX2_DTS_PATH=$L4T_DIR/sources/hardware/nvidia/platform/t18x

export XAVIER_DTS_PATH=$L4T_DIR/sources/hardware/nvidia/platform/t19x

cd $L4T_DIR/sources/kernel/kernel-4.9/

• Build default image, flashing to Jetson Board, prepare L4T environment

cd $L4T_DIR

sudo ./apply_binaries.sh  

- TX2

sudo ./flash.sh jetson-tx2 mmcblk0p1

- AGX Xavier

sudo ./flash.sh jetson-xavier mmcblk0p1

3.2 Download out bsp package for Jetson

cd $L4T_DIR

git clone https://github.com/veyeimaging/nvidia_jetson_veye_bsp.git

export RELEASE_PACK_DIR=$L4T_DIR/nvidia_jetson_veye_bsp

3.2.1 bsp package includes：

• prebuild linux kernel：Image
• prebuild dtb for different boards
• driver source code
• dts source code
• i2c toolkits

3.3 Using prebuild Image and DTB

• Installing the Kernel

Whether you use a prebuild kernel or you build it yourself

- TX2

copy Image to /boot/ dir on the Jetson TX2 board:

sudo cp <path to your Image dir>/Image /boot/Image -f

- AGX Xavier

on the Host PC:

cd $L4T_DIR

sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1

• Flashing dtb file

Enter the recovery mode，and on the Host PC:

cp <path to your dtb dir>/dtbfilename $L4T_DIR/kernel/dtb/ -f

cd $L4T_DIR

- TX2

sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1

- AGX Xavier

sudo ./flash.sh -r -k kernel-dtb jetson-xavier mmcblk0p1

Reset Jetson Board after flashing.

3.4 Building from Source

3.4.1 Build kernel

• patch code

cp $RELEASE_PACK_DIR/drivers_source/cs_imx307\ veye327/* $NVIDIA_PATH/drivers/media/i2c/

- r32.2.1

cp $RELEASE_PACK_DIR/drivers_source/kernel_csimx307veye327_config_32.2.1 $L4T_DIR/sources/kernel/kernel-4.9/arch/arm64/configs/tegra_csimx307veye327_defconfig

- r32.3.1

cp $RELEASE_PACK_DIR/drivers_source/kernel_csimx307veye327_config_32.3.1 $L4T_DIR/sources/kernel/kernel-4.9/arch/arm64/configs/tegra_csimx307veye327_defconfig

• build

cd $L4T_DIR/sources/kernel/kernel-4.9/

make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_csimx307veye327_defconfig

make ARCH=arm64 O=$TEGRA_KERNEL_OUT Image -j4

Completed Image file is $TEGRA_KERNEL_OUT/arch/arm64/boot/Image，could be used for Flashing or Upgrading.

Install Image for Flashing:

sudo cp $TEGRA_KERNEL_OUT/arch/arm64/boot/Image $L4T_DIR/kernel/ -f

3.4.2 Build DTS

• patch code

- Jetson TX2@Jetpack4.2.2

cp $RELEASE_PACK_DIR/TX2/JetPack_4.2.2_Linux_GA_P3310/dts\ dtb/common/t18x/* -r $TX2_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/TX2/JetPack_4.2.2_Linux_GA_P3310/dts\ dtb/CS-MIPI-IMX307/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/TX2/JetPack_4.2.2_Linux_GA_P3310/dts\ dtb/VEYE-MIPI-327/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

- TX2@Jetpack4.3

cp $RELEASE_PACK_DIR/TX2/JetPack_4.3_Linux_P3310/dts\ dtb/common/t18x/* -r $TX2_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/TX2/JetPack_4.3_Linux_P3310/dts\ dtb/CS-MIPI-IMX307/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/TX2/JetPack_4.3_Linux_P3310/dts\ dtb/VEYE-MIPI-327/tegra186-quill-p3310-1000-a00-00-base.dts $TX2_DTS_PATH/quill/kernel-dts/

- AGX Xavier@Jetpack4.2.2

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.2.2_Linux_GA_P2888/dts\ dtb/common/t19x/* -r $XAVIER_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.2.2_Linux_GA_P2888/dts\ dtb/CS-MIPI-IMX307/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.2.2_Linux_GA_P2888/dts\ dtb/VEYE-MIPI-327/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

- AGX Xavier@Jetpack4.3

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.3_Linux_P2888/dts\ dtb/common/t19x/* -r $XAVIER_DTS_PATH/

For CS-MIPI-IMX307

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.3_Linux_P2888/dts\ dtb/CS-MIPI-IMX307/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

For VEYE-MIPI-327

cp $RELEASE_PACK_DIR/AGX-Xaviar/JetPack_4.3_Linux_P2888/dts\ dtb/VEYE-MIPI-327/tegra194-p2888-0001-p2822-0000.dts $TX2_DTS_PATH/galen/kernel-dts/

• build

cd $L4T_DIR/sources/kernel/kernel-4.9/

make ARCH=arm64 O=$TEGRA_KERNEL_OUT dtbs

- TX2

cp $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/tegra186-quill-p3310-1000-c03-00-base.dtb $L4T_DIR/kernel/dtb/

- XAVIER

cp $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/tegra194-p2888-0001-p2822-0000.dtb

Could be used for Flashing or Upgrading.

4 Applications and Test

4.1 Check system status

Run the following command to confirm whether the camera is probed.

• For VEYE-MIPI-327

dmesg | grep veye327  

The output message appears as shown below.：

Detected VEYE327 sensor

subdev veye327 3X-003b bound

此外，检查/dev/videoX设备节点，应当是存在的。

• For CS-MIPI-IMX307

dmesg | grep csimx307  

The output message appears as shown below.：

Detected CS307 sensor

subdev csimx307 3X-003b bound

• Run the following command to check the presence of video node.

ls /dev/video*

The output message appears as shown below.

videoX  

4.2 Video Stream Toolkits Manual

Our camera appears as /dev/videoX device node,where X is between [0-5].

in gstreamer command ，v4l2src has a param device=/dev/videoX to identify which camera to use.

4.2.1 Gstreamer Usage

• Preview FHD(HW accelerated)

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false

• Preview 720p@60 (CS-MIPI-IMX307,HW accelerated)

gst-launch-1.0 v4l2src ! "video/x-raw,format=(string)UYVY, width=(int)1280, height=(int)720, framerate=(fraction)60/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false

• Preview 640*480p@130 (CS-MIPI-IMX307,HW accelerated)

gst-launch-1.0 v4l2src ! "video/x-raw,format=(string)UYVY, width=(int)640, height=(int)480, framerate=(fraction)130/1" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! nvoverlaysink sync=false

• Preview 2 cameras 1080p HD(HW accelerated)

WIDTH=960

HEIGHT=540

CAPS="video/x-raw,format=(string)UYVY, width=1920, height=1080"

gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=$WIDTH sink_0::height=$HEIGHT sink_1::xpos=$WIDTH sink_1::ypos=0 sink_1::width=$WIDTH sink_1::height=$HEIGHT ! nvoverlaysink v4l2src device=/dev/video0 ! $CAPS ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420"! comp. v4l2src device=/dev/video1 ! $CAPS ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420"! comp.

• Record FHD in H.264 format to a video file（HW accelerated）

gst-launch-1.0 v4l2src num-buffers=300 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! nvvidconv ! "video/x-raw(memory:NVMM),format=(string)I420" ! omxh264enc qp-range=20,20:20,20:-1,-1 ! matroskamux ! queue ! filesink location=videoname.mkv

• Playback of saved video file (HW accelerated）

gst-launch-1.0 filesrc location=videoname.mkv ! matroskademux ! h264parse ! omxh264dec ! nvoverlaysink

• Capturing FHD still image  

gst-launch-1.0 v4l2src num-buffers=1 ! "video/x-raw,format=(string)UYVY, width=(int)1920, height=(int)1080" ! jpegenc ! filesink location=jpgname.jpg

4.3 Video Control Toolkits Manual

• i2c bus description：

ADP-N1 has 6 MIPI CSI-2 interface [A-F]，the corresponding i2c address is[30-35].

It is also the same as the 3X part of dmesg message:

subdev veye327 3X-003b bound

subdev csimx307 3X-003b bound

using -b option to identify which bus you want to use.

• VEYE-MIPI-327

Video Control Toolkits Manual ：VEYE-MIPI-327 I2C

• CS-MIPI-IMX307

Video Control Toolkits Manual ：CS-MIPI-IMX307 I2C

5 Realtimes RTSO-9001

5.1 Hardware Setup

CS-MIPI-IMX307 and RTSO-9001（TX2）

Matching with Realtimes RTSO-9001, you need to use RTST-RPI adapter board.

Note: the power supply capacity of the MIPI CSI-2 interface of the current version of RTSO-9001 is not enough to support the power requirements of our camera, which requires an additional 5V power supply to the camera. It is not shown in the picture above.

5.2 Software bsp

• The corresponding directory for Realtimes RTSO-9001, is：

nvidia_jetson_veye_bsp\TX2\JetPack_4.2.2_Linux_RTSO-9001

• Subject to the agreement with Ruitai, we do not release the source code
• Our code have been submitted to Realtimes, and customers are advised to contact Realtimes for a complete version of the supported image.

6 References

Jetson start up： https://developer.nvidia.com/embedded/learn/getting-started-jetson

TX2 development kit： https://developer.nvidia.com/embedded/jetson-tx2-developer-kit

SDK Manager: https://docs.nvidia.com/sdk-manager/index.html

TX2 user guide: link

Realtimes website：http://www.realtimes.cn/