Introduction
============
-This file documents Intel IPU3 (3rd generation Image Processing Unit) Imaging
-Unit driver located under drivers/media/pci/intel/ipu3.
+This file documents the Intel IPU3 (3rd generation Image Processing Unit)
+Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
+as under drivers/staging/media/ipu3 (ImgU).
The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
-platforms (U/Y processor lines) is made up of two parts namely Imaging Unit
-(ImgU) and CIO2 device (MIPI CSI2 receiver).
+platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
+(ImgU) and the CIO2 device (MIPI CSI2 receiver).
-The CIO2 device receives the raw bayer data from the sensors and outputs the
-frames in a format that is specific to IPU3 (for consumption by IPU3 ImgU).
-CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2* and is
-enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
+The CIO2 device receives the raw Bayer data from the sensors and outputs the
+frames in a format that is specific to the IPU3 (for consumption by the IPU3
+ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
+and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
The Imaging Unit (ImgU) is responsible for processing images captured
-through IPU3 CIO2 device. The ImgU driver sources can be found under
-drivers/media/pci/intel/ipu3 directory. The driver is enabled through the
+by the IPU3 CIO2 device. The ImgU driver sources can be found under
+drivers/staging/media/ipu3 directory. The driver is enabled through the
CONFIG_VIDEO_IPU3_IMGU config option.
-The two driver modules are named ipu3-csi2 and ipu3-imgu, respectively.
+The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
-The driver has been tested on Kaby Lake platforms (U/Y processor lines).
+The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
-The driver implements V4L2, Media controller and V4L2 sub-device interfaces.
-Camera sensors that have CSI-2 bus, which are connected to the IPU3 CIO2
-device are supported. Support for lens and flash drivers depends on the
-above sensors.
+Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
+interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
+MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
-ImgU device nodes
-=================
+CIO2
+====
-The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
-subdev interface to the user space.
+The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
+interface to the user space. There is a video node for each CSI-2 receiver,
+with a single media controller interface for the entire device.
-Each V4L2 subdev represents a pipe, which can support a maximum of 2
-streams. A private ioctl can be used to configure the mode (video or still)
-of the pipe.
+The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
+receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
+to userspace as a V4L2 sub-device node and has two pads:
-This helps to support advanced camera features like Continuous View Finder
-(CVF) and Snapshot During Video(SDV).
+.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
-CIO2 device
-===========
+.. flat-table::
-The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
-interface to the user space. There is a video node for each CSI-2 receiver,
-with a single media controller interface for the entire device.
+ * - pad
+ - direction
+ - purpose
-Media controller
-----------------
+ * - 0
+ - sink
+ - MIPI CSI-2 input, connected to the sensor subdev
-The media device interface allows to configure the ImgU links, which defines
-the behavior of the IPU3 firmware.
+ * - 1
+ - source
+ - Raw video capture, connected to the V4L2 video interface
-Device operation
-----------------
+The V4L2 video interfaces model the DMA engines. They are exposed to userspace
+as V4L2 video device nodes.
-With IPU3, once the input video node ("ipu3-imgu 0/1":0,
-in <entity>:<pad-number> format) is queued with buffer (in packed raw bayer
-format), IPU3 ISP starts processing the buffer and produces the video output
-in YUV format and statistics output on respective output nodes. The driver
-is expected to have buffers ready for all of parameter, output and
-statistics nodes, when input video node is queued with buffer.
+Capturing frames in raw Bayer format
+------------------------------------
-At a minimum, all of input, main output, 3A statistics and viewfinder
-video nodes should be enabled for IPU3 to start image processing.
+CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
+from the raw sensors connected to the CSI2 ports. The captured frames are used
+as input to the ImgU driver.
-Each ImgU V4L2 subdev has the following set of video nodes.
+Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
+yavta [#f2]_ due to the following unique requirements and / or features specific
+to IPU3.
-input, output and viewfinder video nodes
-----------------------------------------
+-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
+raw Bayer format that is specific to IPU3.
-The frames (in packed raw bayer format specific to IPU3) received by the
-input video node is processed by the IPU3 Imaging Unit and is output to 2
-video nodes, with each targeting different purpose (main output and viewfinder
-output).
+-- Multiple video nodes have to be operated simultaneously.
-Details on raw bayer format specific to IPU3 can be found as below.
-Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
+2592x1944 image capture.
-The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
+Using the media contorller APIs, the ov5670 sensor is configured to send
+frames in packed raw Bayer format to IPU3 CSI2 receiver.
-Only the multi-planar API is supported. More details can be found at
-:ref:`planar-apis`.
+# This example assumes /dev/media0 as the CIO2 media device
+export MDEV=/dev/media0
-parameters video node
----------------------
+# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
-The parameter video node receives the ISP algorithm parameters that are used
-to configure how the ISP algorithms process the image.
+export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
-Details on raw bayer format specific to IPU3 can be found as below.
-Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
+# Establish the link for the media devices using media-ctl [#f3]_
+media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
-3A statistics video node
-------------------------
+# Set the format for the media devices
+media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
-3A statistics video node is used by the ImgU driver to output the 3A (auto
-focus, auto exposure and auto white balance) statistics for the frames that
-are being processed by the ISP to user space applications. User space
-applications can use this statistics data to arrive at desired algorithm
-parameters for ISP.
+media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
-CIO2 device nodes
-=================
+media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
-CIO2 is represented as a single V4L2 sub-device with a video node for each
-CSI-2 receiver. The video node represents the DMA engine.
+Once the media pipeline is configured, desired sensor specific settings
+(such as exposure and gain settings) can be set, using the yavta tool.
-Configuring the Intel IPU3
-==========================
+e.g
-The Intel IPU3 ImgU driver supports V4L2 interface. Using V4L2 ioctl calls,
-the ISP can be configured and enabled.
+yavta -w 0x009e0903 444 $SDEV
-The IPU3 ImgU pipelines can be configured using media controller APIs,
-defined at :ref:`media_controller`.
+yavta -w 0x009e0913 1024 $SDEV
-Capturing frames in raw bayer format
-------------------------------------
+yavta -w 0x009e0911 2046 $SDEV
-IPU3 MIPI CSI2 receiver is used to capture frames (in packed raw bayer
-format) from the raw sensors connected to the CSI2 ports. The captured
-frames are used as input to the ImgU driver.
+Once the desired sensor settings are set, frame captures can be done as below.
-Image processing using IPU3 ImgU requires tools such as v4l2n [#f1]_,
-raw2pnm [#f1]_, and yavta [#f2]_ due to the following unique requirements
-and / or features specific to IPU3.
+e.g
--- The IPU3 CSI2 receiver outputs the captured frames from the sensor in
-packed raw bayer format that is specific to IPU3
+yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
+ -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
--- Multiple video nodes have to be operated simultaneously
+With the above command, 10 frames are captured at 2592x1944 resolution, with
+sGRBG10 format and output as IPU3_SGRBG10 format.
-Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
-2592x1944 image capture.
+The captured frames are available as /tmp/frame-#.bin files.
-Using the media contorller APIs, the ov5670 sensor is configured to send
-frames in packed raw bayer format to IPU3 CSI2 receiver.
+ImgU
+====
-# This example assumes /dev/media0 as the ImgU media device
+The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
+subdev interface to the user space.
-export MDEV=/dev/media0
+Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
+This helps to support advanced camera features like Continuous View Finder (CVF)
+and Snapshot During Video(SDV).
-# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
+The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
+exposed to userspace as a V4L2 sub-device node.
-export SDEV="ov5670 10-0036"
+Each pipe has two sink pads and three source pads for the following purpose:
-# Establish the link for the media devices using media-ctl [#f3]_
-media-ctl -d $MDEV -l "ov5670 ":0 -> "ipu3-csi2 0":0[1]
+.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
-media-ctl -d $MDEV -l "ipu3-csi2 0":1 -> "ipu3-cio2 0":0[1]
+.. flat-table::
-# Set the format for the media devices
-media-ctl -d $MDEV -V "ov5670 ":0 [fmt:SGRBG10/2592x1944]
+ * - pad
+ - direction
+ - purpose
-media-ctl -d $MDEV -V "ipu3-csi2 0":0 [fmt:SGRBG10/2592x1944]
+ * - 0
+ - sink
+ - Input raw video stream
-media-ctl -d $MDEV -V "ipu3-csi2 0":1 [fmt:SGRBG10/2592x1944]
+ * - 1
+ - sink
+ - Processing parameters
-Once the media pipeline is configured, desired sensor specific settings
-(such as exposure and gain settings) can be set, using the yavta tool.
+ * - 2
+ - source
+ - Output processed video stream
-e.g
+ * - 3
+ - source
+ - Output viewfinder video stream
-yavta -w 0x009e0903 444 $(media-ctl -d $MDEV -e "$SDEV")
+ * - 4
+ - source
+ - 3A statistics
-yavta -w 0x009e0913 1024 $(media-ctl -d $MDEV -e "$SDEV")
+Each pad is connected to a corresponding V4L2 video interface, exposed to
+userspace as a V4L2 video device node.
+
+Device operation
+----------------
-yavta -w 0x009e0911 2046 $(media-ctl -d $MDEV -e "$SDEV")
+With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
+<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
+format), ImgU starts processing the buffer and produces the video output in YUV
+format and statistics output on respective output nodes. The driver is expected
+to have buffers ready for all of parameter, output and statistics nodes, when
+input video node is queued with buffer.
-Once the desired sensor settings are set, frame captures can be done as below.
+At a minimum, all of input, main output, 3A statistics and viewfinder
+video nodes should be enabled for IPU3 to start image processing.
-e.g
+Each ImgU V4L2 subdev has the following set of video nodes.
-yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin
--f IPU3_GRBG10 media-ctl -d $MDEV -e ipu3-cio2 0
+input, output and viewfinder video nodes
+----------------------------------------
-With the above command, 10 frames are captured at 2592x1944 resolution, with
-sGRBG10 format and output as IPU3_GRBG10 format.
+The frames (in packed raw Bayer format specific to the IPU3) received by the
+input video node is processed by the IPU3 Imaging Unit and are output to 2 video
+nodes, with each targeting a different purpose (main output and viewfinder
+output).
-The captured frames are available as /tmp/frame-#.bin files.
+Details onand the Bayer format specific to the IPU3 can be found in
+:ref:`v4l2-pix-fmt-ipu3-sbggr10`.
-Processing the image in raw bayer format
+The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
+
+Only the multi-planar API is supported. More details can be found at
+:ref:`planar-apis`.
+
+Parameters video node
+---------------------
+
+The parameters video node receives the ImgU algorithm parameters that are used
+to configure how the ImgU algorithms process the image.
+
+Details on processing parameters specific to the IPU3 can be found in
+:ref:`v4l2-meta-fmt-params`.
+
+3A statistics video node
+------------------------
+
+3A statistics video node is used by the ImgU driver to output the 3A (auto
+focus, auto exposure and auto white balance) statistics for the frames that are
+being processed by the ImgU to user space applications. User space applications
+can use this statistics data to compute the desired algorithm parameters for
+the ImgU.
+
+Configuring the Intel IPU3
+==========================
+
+The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
+:ref:`media_controller`.
+
+Firmware binary selection
+-------------------------
+
+The firmware binary is selected using the V4L2_CID_INTEL_IPU3_MODE, currently
+defined in drivers/staging/media/ipu3/include/intel-ipu3.h . "VIDEO" and "STILL"
+modes are available.
+
+Processing the image in raw Bayer format
----------------------------------------
Configuring ImgU V4L2 subdev for image processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The ImgU V4L2 subdevs have to be configured with media controller APIs to
-have all the video nodes setup correctly.
+The ImgU V4L2 subdevs have to be configured with media controller APIs to have
+all the video nodes setup correctly.
Let us take "ipu3-imgu 0" subdev as an example.
media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
-media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "output":0[1]
+media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
-media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "viewfinder":0[1]
+media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
-media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "3a stat":0[1]
+media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
-Also the pipe mode of the corresponding V4L2 subdev should be set as
-desired (e.g 0 for video mode or 1 for still mode) through the
-control id 0x009819a1 as below.
-
-e.g
+Also the pipe mode of the corresponding V4L2 subdev should be set as desired
+(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
+below.
-v4l2n -d /dev/v4l-subdev7 --ctrl=0x009819A1=1
+yavta -w "0x009819A1 1" /dev/v4l-subdev7
-RAW bayer frames go through the following ISP pipeline HW blocks to
-have the processed image output to the DDR memory.
+RAW Bayer frames go through the following ImgU pipeline HW blocks to have the
+processed image output to the DDR memory.
-RAW bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
+RAW Bayer frame -> Input Feeder -> Bayer Down Scaling (BDS) -> Geometric
Distortion Correction (GDC) -> DDR
-The ImgU V4L2 subdev has to be configured with the supported resolutions
-in all the above HW blocks, for a given input resolution.
+The ImgU V4L2 subdev has to be configured with the supported resolutions in all
+the above HW blocks, for a given input resolution.
-For a given supported resolution for an input frame, the Input Feeder,
-Bayer Down Scaling and GDC blocks should be configured with the supported
-resolutions. This information can be obtained by looking at the following
-IPU3 ISP configuration table.
+For a given supported resolution for an input frame, the Input Feeder, Bayer
+Down Scaling and GDC blocks should be configured with the supported resolutions.
+This information can be obtained by looking at the following IPU3 ImgU
+configuration table.
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
-Under baseboard-poppy/media-libs/arc-camera3-hal-configs-poppy/files/gcss
+Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
directory, graph_settings_ov5670.xml can be used as an example.
-The following steps prepare the ImgU ISP pipeline for the image processing.
+The following steps prepare the ImgU pipeline for the image processing.
1. The ImgU V4L2 subdev data format should be set by using the
VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
respectively.
-Once this is done, the received raw bayer frames can be input to the ImgU
-V4L2 subdev as below, using the open source application v4l2n.
+Once this is done, the received raw Bayer frames can be input to the ImgU
+V4L2 subdev as below, using the open source application v4l2n [#f1]_.
For an image captured with 2592x1944 [#f4]_ resolution, with desired output
resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
-v4l2n command can be used. This helps process the raw bayer frames and
-produces the desired results for the main output image and the viewfinder
-output, in NV12 format.
+v4l2n command can be used. This helps process the raw Bayer frames and produces
+the desired results for the main output image and the viewfinder output, in NV12
+format.
v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069
where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
input, output, viewfinder and 3A statistics video nodes respectively.
-Converting the raw bayer image into YUV domain
+Converting the raw Bayer image into YUV domain
----------------------------------------------
The processed images after the above step, can be converted to YUV domain
Main output frames
~~~~~~~~~~~~~~~~~~
-raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.pnm
+raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
where 2560x1920 is output resolution, NV12 is the video format, followed
by input frame and output PNM file.
Viewfinder output frames
~~~~~~~~~~~~~~~~~~~~~~~~
-raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.pnm
+raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
where 2560x1920 is output resolution, NV12 is the video format, followed
by input frame and output PNM file.
References
==========
-include/uapi/linux/intel-ipu3.h
+.. [#f5] include/uapi/linux/intel-ipu3.h
.. [#f1] https://github.com/intel/nvt
projects / mirror_ubuntu-disco-kernel.git / commitdiff