Difference between revisions of "MV Camera on Firfly Boards"

From wiki_veye
Jump to navigation Jump to search
Line 1: Line 1:
 
[[MV Camera on Firfly Boards/zh|查看中文]]
 
[[MV Camera on Firfly Boards/zh|查看中文]]
 +
 +
=== Overview ===
 +
The MV series and RAW series cameras are cameras designed for AI applications in the industrial field. They use the MIPI CSI-2 interface and are particularly suitable for use with embedded computing platforms. They have rich data formats and triggering features, extremely low latency, high bandwidth, and reliable stability.
 +
 +
This article takes Firefly's ROC-RK3588S-PC motherboard as an example to introduce how to connect MV and RAW series cameras to the RK3588S/RK3588 system.
 +
 +
We provide drivers for the Linux operating system (using Ubuntu as an example).
 +
 +
==== Camera Module List ====
 +
{| class="wikitable"
 +
!Series
 +
!Model
 +
!Status
 +
|-
 +
|MV series
 +
|MV-MIPI-IMX178M
 +
|Done
 +
|-
 +
|MV series
 +
|MV-MIPI-SC130M
 +
|Done
 +
|-
 +
|MV series
 +
|MV-MIPI-IMX296M
 +
|Done
 +
|-
 +
|MV series
 +
|MV-MIPI-IMX287M
 +
|Done
 +
|-
 +
|MV series
 +
|MV-MIPI-IMX265M
 +
|Done
 +
|-
 +
|MV series
 +
|MV-MIPI-IMX264M
 +
|Done
 +
|-
 +
|RAW series
 +
|RAW-MIPI-SC132M
 +
|Done
 +
|}
 +
 +
=== Hardware Setup ===
 +
The MV series and RAW series cameras require an [[ADP-MV2 Adapter Board Data Sheet/zh|ADP-MV2]] adapter board to connect to the ROC-RK35xx-PC motherboard.
 +
 +
==== Connection of MV-MIPI-CAM and ADP-MV2 ====
 +
The two are connected using 0.5 mm pitch*30P FFC cable with opposite direction. The cable must be inserted with the silver contacts facing outside.
 +
{| class="wikitable"
 +
!TOP
 +
!BOTTOM
 +
|-
 +
|[[File:ADP-MV2-MV-MIPI 01.jpg|alt=|center|thumb|600x600px|ADP-MV2 to MV-MIPI-X|link=http://wiki.veye.cc/index.php/File:ADP-MV2-MV-MIPI_01.jpg]]
 +
|[[File:ADP-MV2-MV-MIPI 02.jpg|alt=|center|thumb|600x600px|ADP-MV2 to MV-MIPI-X|link=http://wiki.veye.cc/index.php/File:ADP-MV2-MV-MIPI_02.jpg]]
 +
|}
 +
 +
==== Connection of RAW-MIPI-CAM and ADP-MV2 ====
 +
The two are connected using 1.0 mm pitch*15P FFC cable with opposite direction. The cable must be inserted with the silver contacts facing outside.
 +
{| class="wikitable"
 +
!TOP
 +
!BOTTOM
 +
|-
 +
|[[File:ADP-MV2-RAW-MIPI 02.jpg|alt=|center|thumb|600x600px|ADP-MV2 to RAW-MIPI-X|link=http://wiki.veye.cc/index.php/File:ADP-MV2-RAW-MIPI_02.jpg]]
 +
|[[File:ADP-MV2-RAW-MIPI 01.jpg|alt=|center|thumb|600x600px|ADP-MV2 to RAW-MIPI-X|link=http://wiki.veye.cc/index.php/File:ADP-MV2-RAW-MIPI_01.jpg]]
 +
|}
 +
 +
==== Connection with Main board using ADP-MV2 ====
 +
[[File:RK-ADP-MV2-MV-MIPI 01.jpg|center|thumb|800x800px|RK to ADP-MV2 and MV cam|link=http://wiki.veye.cc/index.php/File:RK-ADP-MV2-MV-MIPI_01.jpg]]
 +
===Introduction to github repositories===
 +
https://github.com/veyeimaging/rk35xx_firefly
 +
 +
includes：
 +
 +
*driver source code
 +
*i2c toolkits
 +
*application demo
 +
 +
In addition, a compiled linux kernel installation package and Android image is provided in the [https://github.com/veyeimaging/rk356x_firefly/releases releases].
 +
 +
=== Upgrade Firefly Ubuntu system ===
 +
For the ROC-RK3588S-PC, we have provided an image of the release system.
 +
 +
Download the latest rk358x_firefly_ubuntu.tar.gz from https://github.com/veyeimaging/rk35xx_firefly/releases/ .
 +
 +
Refer to the [https://wiki.t-firefly.com/en/ROC-RK3588S-PC/upgrade_bootmode.html Firefly documentation] to burn in a standard system.
 +
===Check system status===
 +
 +
==== Whether the camera is correctly recognized ====
 +
After system update, reboot the main board.
 +
 +
Execute the following command on the main board to check if the camera is properly connected.
 +
 +
<code>dmesg | grep mvcam</code>
 +
 +
You can see the camera model and the camera version number probed.
 +
 +
A prompt as below indicates that the MV-MIPI-IMX296M camera is detected on the i2c-7 bus.
 +
 +
<code>mvcam 7-003b: camera is：MV-MIPI-IMX296M</code>
 +
 +
<code>mvcam 7-003b: firmware version: 0x1290133</code>
 +
 +
* Check the video0 device node:
 +
 +
<code>ls /dev/video0</code>
 +
 +
You should see:
 +
 +
<code>video0</code>
 +
 +
After successfully identifying the camera, the camera will be recognized as /dev/video0.
 +
 +
Use <code>media-ctl</code> command to view the current topology structure
 +
 +
==== Using media-ctl to view topology ====
 +
Using the media-ctl command can clearly display the current topography structure.
 +
 +
<code>media-ctl -p -d /dev/media0</code>
 +
 +
===== Link relationship =====
 +
mv camera->rockchip-csi2-dphy0->rockchip-mipi-csi2->stream_cif_mipi_id0 - - ->DDR(/dev/video0)
 +
 +
The application can obtain images through the /dev/video0 node.
 +
 +
===== mv camera entity information =====
 +
Taking the MV-MIPI-IMX296M as an example:
 +
 +
<code>- entity 63: m00_b_mvcam 7-003b (1 pad, 1 link)</code>
 +
 +
<code>             type V4L2 subdev subtype Sensor flags 0</code>
 +
 +
<code>             device node name /dev/v4l-subdev2</code>
 +
 +
<code>        pad0: Source</code>
 +
 +
<code>                [fmt:Y8_1X8/1456x1088@100/6000 field:none]</code>
 +
 +
<code>                -> "rockchip-csi2-dphy0":0 [ENABLED]</code>
 +
 +
You can see that:
 +
 +
* The complete name of this entity is: <code>m00_b_mvcam 7-003b</code>.
 +
* It is a V4L2 subdev (Sub-Device) Sensor.
 +
* Its corresponding node is <code>/dev/v4l-subdev2</code>, which can be opened and configured by applications (such as <code>v4l2-ctl</code>).
 +
* Its output format is <code>[fmt:Y8_1X8/1456x1088@100/6000 field:none]</code>, where <code>Y8_1X8</code> is a shorthand for a mbus-code, which will be listed in the next section of this article.
 +
* The current resolution is <code>1456x1088</code>.
 +
* The current frame interval is <code>100/6000</code>, which means the frame rate is 60.
 +
* The data format output by the camera can be modified using the media-ctl command.
 +
 +
===== mbus-code list =====
 +
MV series and RAW series cameras have different data format capabilities, which can be found in the data manual for each camera model.
 +
{| class="wikitable"
 +
!Format on datasheet
 +
!mbus-code for media-ctl
 +
!FourCC pixelformat for v4l2-ctl
 +
|-
 +
|RAW8
 +
|Y8_1X8
 +
|GREY
 +
|-
 +
|RAW10
 +
|Y10_1X10
 +
|'Y10 '
 +
|-
 +
|RAW12
 +
|Y12_1X12
 +
|'Y12 '
 +
|-
 +
|UYVY
 +
|UYVY8_2X8
 +
|UYVY
 +
|}
 +
 +
=== Raw data format ===
 +
The VICAP module of RK3588 supports two data saving formats, Compact and Noncompact RAW. You can modify the mode using the RKCIF_CMD_SET_CSI_MEMORY_MODE ioctl command of RKCIF. By default, the output is in Compact RAW format.[[File:Compact raw and noncompact raw of rk3588 vicap.png|center|thumb|800x800px|Compact raw and noncompact raw of rk3588 VICAP|link=http://wiki.veye.cc/index.php/File:Compact_raw_and_noncompact_raw_of_rk3588_vicap.png]]
 +
 +
==== Noncompact RAW ====
 +
For pixel data with 10-bit depth or 12-bit depth, two bytes are always used to store one pixel. This storage method is convenient for software processing, but it has the disadvantage of occupying a large amount of space.
 +
 +
Depending on whether the effective data is stored in the high bits or low bits, it can be further divided into two types: high align and low align.
 +
 +
=====Noncompact RAW(high align)=====
 +
Data is saved to the high bits, and the unused low bits are filled with 0. This is one of the data formats supported by RK VICAP.
 +
 +
===== Noncompact RAW(low align) =====
 +
In Noncompact RAW (low align) format, data is saved to the low bits, and the unused high bits are filled with 0. The V4L2 standard 'Y10' (10-bit Greyscale) and 'Y12' (12-bit Greyscale) formats are both stored in this way.
 +
 +
The pixel_layer_convert conversion tool mentioned later in the article also converts Compact RAW to this storage format for easy display using image players.
 +
 +
==== Compact RAW ====
 +
As shown above，there is no bit padding between pixels in this storage format.
 +
 +
==== Line stride ====
 +
To facilitate fast operations on images, the system usually provides row-aligned buffer sizes for each line of data. RK3588 uses 256-byte alignment for this purpose.
 +
 +
line_stride = ALIGN_UP（image_width*bits_per_pixel/8,256）
 +
 +
For example, when the image width is 1456:
 +
 +
8bit depth，line_stride=1536
 +
 +
10bit depth，preferred_stride=2048
 +
 +
12bit depth，preferred_stride=2304
 +
====Format convert tool====
 +
We have written a small tool: [https://github.com/veyeimaging/pixel_layer_convert pixel_layer_convert], which can easily convert Compact images to Noncompact (low align) images.
 +
 +
For example, the following command can convert a Compact RAW10 image with a width of 1456 to Noncompact RAW10 format:
 +
 +
<code>./pixel_layer_convert -I R10C -i y10-1456x1088_0001.raw -o y10-1456x1088_0001_new.raw -w 1456</code>
 +
====Raw data image player====
 +
We recommend using [https://www.offminor.de/ vooya] as the player, which supports GREY, and unpacked image formats.
 +
 +
Also, y8 file can be used with this player: [https://yuv-player-deluxe.software.informer.com/2.6/ YUV Displayer Deluxe].

Revision as of 20:14, 12 April 2023

查看中文

1 Overview

The MV series and RAW series cameras are cameras designed for AI applications in the industrial field. They use the MIPI CSI-2 interface and are particularly suitable for use with embedded computing platforms. They have rich data formats and triggering features, extremely low latency, high bandwidth, and reliable stability.

This article takes Firefly's ROC-RK3588S-PC motherboard as an example to introduce how to connect MV and RAW series cameras to the RK3588S/RK3588 system.

We provide drivers for the Linux operating system (using Ubuntu as an example).

1.1 Camera Module List

Series Model Status
MV series MV-MIPI-IMX178M Done
MV series MV-MIPI-SC130M Done
MV series MV-MIPI-IMX296M Done
MV series MV-MIPI-IMX287M Done
MV series MV-MIPI-IMX265M Done
MV series MV-MIPI-IMX264M Done
RAW series RAW-MIPI-SC132M Done

2 Hardware Setup

The MV series and RAW series cameras require an ADP-MV2 adapter board to connect to the ROC-RK35xx-PC motherboard.

2.1 Connection of MV-MIPI-CAM and ADP-MV2

The two are connected using 0.5 mm pitch*30P FFC cable with opposite direction. The cable must be inserted with the silver contacts facing outside.

TOP BOTTOM
ADP-MV2 to MV-MIPI-X
ADP-MV2 to MV-MIPI-X

2.2 Connection of RAW-MIPI-CAM and ADP-MV2

The two are connected using 1.0 mm pitch*15P FFC cable with opposite direction. The cable must be inserted with the silver contacts facing outside.

TOP BOTTOM
ADP-MV2 to RAW-MIPI-X
ADP-MV2 to RAW-MIPI-X

2.3 Connection with Main board using ADP-MV2

RK to ADP-MV2 and MV cam

3 Introduction to github repositories

https://github.com/veyeimaging/rk35xx_firefly

includes：

  • driver source code
  • i2c toolkits
  • application demo

In addition, a compiled linux kernel installation package and Android image is provided in the releases.

4 Upgrade Firefly Ubuntu system

For the ROC-RK3588S-PC, we have provided an image of the release system.

Download the latest rk358x_firefly_ubuntu.tar.gz from https://github.com/veyeimaging/rk35xx_firefly/releases/ .

Refer to the Firefly documentation to burn in a standard system.

5 Check system status

5.1 Whether the camera is correctly recognized

After system update, reboot the main board.

Execute the following command on the main board to check if the camera is properly connected.

dmesg | grep mvcam

You can see the camera model and the camera version number probed.

A prompt as below indicates that the MV-MIPI-IMX296M camera is detected on the i2c-7 bus.

mvcam 7-003b: camera is：MV-MIPI-IMX296M

mvcam 7-003b: firmware version: 0x1290133

  • Check the video0 device node:

ls /dev/video0

You should see:

video0

After successfully identifying the camera, the camera will be recognized as /dev/video0.

Use media-ctl command to view the current topology structure

5.2 Using media-ctl to view topology

Using the media-ctl command can clearly display the current topography structure.

media-ctl -p -d /dev/media0

5.2.1 Link relationship

mv camera->rockchip-csi2-dphy0->rockchip-mipi-csi2->stream_cif_mipi_id0 - - ->DDR(/dev/video0)

The application can obtain images through the /dev/video0 node.

5.2.2 mv camera entity information

Taking the MV-MIPI-IMX296M as an example:

- entity 63: m00_b_mvcam 7-003b (1 pad, 1 link)

             type V4L2 subdev subtype Sensor flags 0

             device node name /dev/v4l-subdev2

        pad0: Source

                [fmt:Y8_1X8/1456x1088@100/6000 field:none]

                -> "rockchip-csi2-dphy0":0 [ENABLED]

You can see that:

  • The complete name of this entity is: m00_b_mvcam 7-003b.
  • It is a V4L2 subdev (Sub-Device) Sensor.
  • Its corresponding node is /dev/v4l-subdev2, which can be opened and configured by applications (such as v4l2-ctl).
  • Its output format is [fmt:Y8_1X8/1456x1088@100/6000 field:none], where Y8_1X8 is a shorthand for a mbus-code, which will be listed in the next section of this article.
  • The current resolution is 1456x1088.
  • The current frame interval is 100/6000, which means the frame rate is 60.
  • The data format output by the camera can be modified using the media-ctl command.
5.2.3 mbus-code list

MV series and RAW series cameras have different data format capabilities, which can be found in the data manual for each camera model.

Format on datasheet mbus-code for media-ctl FourCC pixelformat for v4l2-ctl
RAW8 Y8_1X8 GREY
RAW10 Y10_1X10 'Y10 '
RAW12 Y12_1X12 'Y12 '
UYVY UYVY8_2X8 UYVY

6 Raw data format

The VICAP module of RK3588 supports two data saving formats, Compact and Noncompact RAW. You can modify the mode using the RKCIF_CMD_SET_CSI_MEMORY_MODE ioctl command of RKCIF. By default, the output is in Compact RAW format.

Compact raw and noncompact raw of rk3588 VICAP

6.1 Noncompact RAW

For pixel data with 10-bit depth or 12-bit depth, two bytes are always used to store one pixel. This storage method is convenient for software processing, but it has the disadvantage of occupying a large amount of space.

Depending on whether the effective data is stored in the high bits or low bits, it can be further divided into two types: high align and low align.

6.1.1 Noncompact RAW(high align)

Data is saved to the high bits, and the unused low bits are filled with 0. This is one of the data formats supported by RK VICAP.

6.1.2 Noncompact RAW(low align)

In Noncompact RAW (low align) format, data is saved to the low bits, and the unused high bits are filled with 0. The V4L2 standard 'Y10' (10-bit Greyscale) and 'Y12' (12-bit Greyscale) formats are both stored in this way.

The pixel_layer_convert conversion tool mentioned later in the article also converts Compact RAW to this storage format for easy display using image players.

6.2 Compact RAW

As shown above，there is no bit padding between pixels in this storage format.

6.3 Line stride

To facilitate fast operations on images, the system usually provides row-aligned buffer sizes for each line of data. RK3588 uses 256-byte alignment for this purpose.

line_stride = ALIGN_UP（image_width*bits_per_pixel/8,256）

For example, when the image width is 1456:

8bit depth，line_stride=1536

10bit depth，preferred_stride=2048

12bit depth，preferred_stride=2304

6.4 Format convert tool

We have written a small tool: pixel_layer_convert, which can easily convert Compact images to Noncompact (low align) images.

For example, the following command can convert a Compact RAW10 image with a width of 1456 to Noncompact RAW10 format:

./pixel_layer_convert -I R10C -i y10-1456x1088_0001.raw -o y10-1456x1088_0001_new.raw -w 1456

6.5 Raw data image player

We recommend using vooya as the player, which supports GREY, and unpacked image formats.

Also, y8 file can be used with this player: YUV Displayer Deluxe.

Retrieved from "http://wiki.veye.cc/index.php?title=MV_Camera_on_Firfly_Boards&oldid=6582"