]>
Commit | Line | Data |
---|---|---|
5377d91f MH |
1 | .. -*- coding: utf-8; mode: rst -*- |
2 | ||
3 | .. _subdev: | |
4 | ||
5 | ******************** | |
6 | Sub-device Interface | |
7 | ******************** | |
8 | ||
9 | The complex nature of V4L2 devices, where hardware is often made of | |
10 | several integrated circuits that need to interact with each other in a | |
11 | controlled way, leads to complex V4L2 drivers. The drivers usually | |
12 | reflect the hardware model in software, and model the different hardware | |
13 | components as software blocks called sub-devices. | |
14 | ||
15 | V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver | |
16 | implements the media device API, they will automatically inherit from | |
17 | media entities. Applications will be able to enumerate the sub-devices | |
18 | and discover the hardware topology using the media entities, pads and | |
19 | links enumeration API. | |
20 | ||
21 | In addition to make sub-devices discoverable, drivers can also choose to | |
22 | make them directly configurable by applications. When both the | |
23 | sub-device driver and the V4L2 device driver support this, sub-devices | |
24 | will feature a character device node on which ioctls can be called to | |
25 | ||
26 | - query, read and write sub-devices controls | |
27 | ||
28 | - subscribe and unsubscribe to events and retrieve them | |
29 | ||
30 | - negotiate image formats on individual pads | |
31 | ||
32 | Sub-device character device nodes, conventionally named | |
33 | ``/dev/v4l-subdev*``, use major number 81. | |
34 | ||
35 | ||
36 | Controls | |
37 | ======== | |
38 | ||
39 | Most V4L2 controls are implemented by sub-device hardware. Drivers | |
40 | usually merge all controls and expose them through video device nodes. | |
41 | Applications can control all sub-devices through a single interface. | |
42 | ||
43 | Complex devices sometimes implement the same control in different pieces | |
44 | of hardware. This situation is common in embedded platforms, where both | |
45 | sensors and image processing hardware implement identical functions, | |
46 | such as contrast adjustment, white balance or faulty pixels correction. | |
47 | As the V4L2 controls API doesn't support several identical controls in a | |
48 | single device, all but one of the identical controls are hidden. | |
49 | ||
50 | Applications can access those hidden controls through the sub-device | |
51 | node with the V4L2 control API described in :ref:`control`. The ioctls | |
52 | behave identically as when issued on V4L2 device nodes, with the | |
53 | exception that they deal only with controls implemented in the | |
54 | sub-device. | |
55 | ||
56 | Depending on the driver, those controls might also be exposed through | |
57 | one (or several) V4L2 device nodes. | |
58 | ||
59 | ||
60 | Events | |
61 | ====== | |
62 | ||
63 | V4L2 sub-devices can notify applications of events as described in | |
64 | :ref:`event`. The API behaves identically as when used on V4L2 device | |
65 | nodes, with the exception that it only deals with events generated by | |
66 | the sub-device. Depending on the driver, those events might also be | |
67 | reported on one (or several) V4L2 device nodes. | |
68 | ||
69 | ||
70 | .. _pad-level-formats: | |
71 | ||
72 | Pad-level Formats | |
73 | ================= | |
74 | ||
75 | **Warning** | |
76 | ||
77 | Pad-level formats are only applicable to very complex device that | |
78 | need to expose low-level format configuration to user space. Generic | |
79 | V4L2 applications do *not* need to use the API described in this | |
80 | section. | |
81 | ||
82 | **Note** | |
83 | ||
84 | For the purpose of this section, the term *format* means the | |
85 | combination of media bus data format, frame width and frame height. | |
86 | ||
87 | Image formats are typically negotiated on video capture and output | |
88 | devices using the format and | |
89 | :ref:`selection <vidioc-subdev-g-selection>` ioctls. The driver is | |
90 | responsible for configuring every block in the video pipeline according | |
91 | to the requested format at the pipeline input and/or output. | |
92 | ||
93 | For complex devices, such as often found in embedded systems, identical | |
94 | image sizes at the output of a pipeline can be achieved using different | |
95 | hardware configurations. One such example is shown on | |
96 | :ref:`pipeline-scaling`, where image scaling can be performed on both | |
97 | the video sensor and the host image processing hardware. | |
98 | ||
99 | ||
100 | .. _pipeline-scaling: | |
101 | ||
102 | .. figure:: dev-subdev_files/pipeline.* | |
103 | :alt: pipeline.pdf / pipeline.png | |
104 | :align: center | |
105 | ||
106 | Image Format Negotiation on Pipelines | |
107 | ||
108 | High quality and high speed pipeline configuration | |
109 | ||
110 | ||
111 | ||
112 | The sensor scaler is usually of less quality than the host scaler, but | |
113 | scaling on the sensor is required to achieve higher frame rates. | |
114 | Depending on the use case (quality vs. speed), the pipeline must be | |
115 | configured differently. Applications need to configure the formats at | |
116 | every point in the pipeline explicitly. | |
117 | ||
118 | Drivers that implement the :ref:`media API <media-controller-intro>` | |
119 | can expose pad-level image format configuration to applications. When | |
120 | they do, applications can use the | |
121 | :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` and | |
122 | :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` ioctls. to | |
123 | negotiate formats on a per-pad basis. | |
124 | ||
125 | Applications are responsible for configuring coherent parameters on the | |
126 | whole pipeline and making sure that connected pads have compatible | |
127 | formats. The pipeline is checked for formats mismatch at | |
128 | :ref:`VIDIOC_STREAMON <vidioc-streamon>` time, and an EPIPE error | |
129 | code is then returned if the configuration is invalid. | |
130 | ||
131 | Pad-level image format configuration support can be tested by calling | |
132 | the :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` ioctl on pad | |
133 | 0. If the driver returns an EINVAL error code pad-level format | |
134 | configuration is not supported by the sub-device. | |
135 | ||
136 | ||
137 | Format Negotiation | |
138 | ------------------ | |
139 | ||
140 | Acceptable formats on pads can (and usually do) depend on a number of | |
141 | external parameters, such as formats on other pads, active links, or | |
142 | even controls. Finding a combination of formats on all pads in a video | |
143 | pipeline, acceptable to both application and driver, can't rely on | |
144 | formats enumeration only. A format negotiation mechanism is required. | |
145 | ||
146 | Central to the format negotiation mechanism are the get/set format | |
147 | operations. When called with the ``which`` argument set to | |
148 | ``V4L2_SUBDEV_FORMAT_TRY``, the | |
149 | :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` and | |
150 | :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` ioctls operate on | |
151 | a set of formats parameters that are not connected to the hardware | |
152 | configuration. Modifying those 'try' formats leaves the device state | |
153 | untouched (this applies to both the software state stored in the driver | |
154 | and the hardware state stored in the device itself). | |
155 | ||
156 | While not kept as part of the device state, try formats are stored in | |
157 | the sub-device file handles. A | |
158 | :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` call will return | |
159 | the last try format set *on the same sub-device file handle*. Several | |
160 | applications querying the same sub-device at the same time will thus not | |
161 | interact with each other. | |
162 | ||
163 | To find out whether a particular format is supported by the device, | |
164 | applications use the | |
165 | :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` ioctl. Drivers | |
166 | verify and, if needed, change the requested ``format`` based on device | |
167 | requirements and return the possibly modified value. Applications can | |
168 | then choose to try a different format or accept the returned value and | |
169 | continue. | |
170 | ||
171 | Formats returned by the driver during a negotiation iteration are | |
172 | guaranteed to be supported by the device. In particular, drivers | |
173 | guarantee that a returned format will not be further changed if passed | |
174 | to an :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` call as-is | |
175 | (as long as external parameters, such as formats on other pads or links' | |
176 | configuration are not changed). | |
177 | ||
178 | Drivers automatically propagate formats inside sub-devices. When a try | |
179 | or active format is set on a pad, corresponding formats on other pads of | |
180 | the same sub-device can be modified by the driver. Drivers are free to | |
181 | modify formats as required by the device. However, they should comply | |
182 | with the following rules when possible: | |
183 | ||
184 | - Formats should be propagated from sink pads to source pads. Modifying | |
185 | a format on a source pad should not modify the format on any sink | |
186 | pad. | |
187 | ||
188 | - Sub-devices that scale frames using variable scaling factors should | |
189 | reset the scale factors to default values when sink pads formats are | |
190 | modified. If the 1:1 scaling ratio is supported, this means that | |
191 | source pads formats should be reset to the sink pads formats. | |
192 | ||
193 | Formats are not propagated across links, as that would involve | |
194 | propagating them from one sub-device file handle to another. | |
195 | Applications must then take care to configure both ends of every link | |
196 | explicitly with compatible formats. Identical formats on the two ends of | |
197 | a link are guaranteed to be compatible. Drivers are free to accept | |
198 | different formats matching device requirements as being compatible. | |
199 | ||
200 | :ref:`sample-pipeline-config` shows a sample configuration sequence | |
201 | for the pipeline described in :ref:`pipeline-scaling` (table columns | |
202 | list entity names and pad numbers). | |
203 | ||
204 | ||
205 | .. _sample-pipeline-config: | |
206 | ||
207 | .. flat-table:: Sample Pipeline Configuration | |
208 | :header-rows: 1 | |
209 | :stub-columns: 0 | |
210 | ||
211 | ||
212 | - .. row 1 | |
213 | ||
214 | - | |
215 | - Sensor/0 format | |
216 | ||
217 | - Frontend/0 format | |
218 | ||
219 | - Frontend/1 format | |
220 | ||
221 | - Scaler/0 format | |
222 | ||
223 | - Scaler/0 compose selection rectangle | |
224 | ||
225 | - Scaler/1 format | |
226 | ||
227 | - .. row 2 | |
228 | ||
229 | - Initial state | |
230 | ||
231 | - 2048x1536/SGRBG8_1X8 | |
232 | ||
233 | - (default) | |
234 | ||
235 | - (default) | |
236 | ||
237 | - (default) | |
238 | ||
239 | - (default) | |
240 | ||
241 | - (default) | |
242 | ||
243 | - .. row 3 | |
244 | ||
245 | - Configure frontend sink format | |
246 | ||
247 | - 2048x1536/SGRBG8_1X8 | |
248 | ||
249 | - *2048x1536/SGRBG8_1X8* | |
250 | ||
251 | - *2046x1534/SGRBG8_1X8* | |
252 | ||
253 | - (default) | |
254 | ||
255 | - (default) | |
256 | ||
257 | - (default) | |
258 | ||
259 | - .. row 4 | |
260 | ||
261 | - Configure scaler sink format | |
262 | ||
263 | - 2048x1536/SGRBG8_1X8 | |
264 | ||
265 | - 2048x1536/SGRBG8_1X8 | |
266 | ||
267 | - 2046x1534/SGRBG8_1X8 | |
268 | ||
269 | - *2046x1534/SGRBG8_1X8* | |
270 | ||
271 | - *0,0/2046x1534* | |
272 | ||
273 | - *2046x1534/SGRBG8_1X8* | |
274 | ||
275 | - .. row 5 | |
276 | ||
277 | - Configure scaler sink compose selection | |
278 | ||
279 | - 2048x1536/SGRBG8_1X8 | |
280 | ||
281 | - 2048x1536/SGRBG8_1X8 | |
282 | ||
283 | - 2046x1534/SGRBG8_1X8 | |
284 | ||
285 | - 2046x1534/SGRBG8_1X8 | |
286 | ||
287 | - *0,0/1280x960* | |
288 | ||
289 | - *1280x960/SGRBG8_1X8* | |
290 | ||
291 | ||
292 | ||
293 | 1. Initial state. The sensor source pad format is set to its native 3MP | |
294 | size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the | |
295 | host frontend and scaler sink and source pads have the default | |
296 | values, as well as the compose rectangle on the scaler's sink pad. | |
297 | ||
298 | 2. The application configures the frontend sink pad format's size to | |
299 | 2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The | |
300 | driver propagates the format to the frontend source pad. | |
301 | ||
302 | 3. The application configures the scaler sink pad format's size to | |
303 | 2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to | |
304 | match the frontend source size and media bus code. The media bus code | |
305 | on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver | |
306 | propagates the size to the compose selection rectangle on the | |
307 | scaler's sink pad, and the format to the scaler source pad. | |
308 | ||
309 | 4. The application configures the size of the compose selection | |
310 | rectangle of the scaler's sink pad 1280x960. The driver propagates | |
311 | the size to the scaler's source pad format. | |
312 | ||
313 | When satisfied with the try results, applications can set the active | |
314 | formats by setting the ``which`` argument to | |
315 | ``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try | |
316 | formats by drivers. To avoid modifying the hardware state during format | |
317 | negotiation, applications should negotiate try formats first and then | |
318 | modify the active settings using the try formats returned during the | |
319 | last negotiation iteration. This guarantees that the active format will | |
320 | be applied as-is by the driver without being modified. | |
321 | ||
322 | ||
323 | .. _v4l2-subdev-selections: | |
324 | ||
325 | Selections: cropping, scaling and composition | |
326 | --------------------------------------------- | |
327 | ||
328 | Many sub-devices support cropping frames on their input or output pads | |
329 | (or possible even on both). Cropping is used to select the area of | |
330 | interest in an image, typically on an image sensor or a video decoder. | |
331 | It can also be used as part of digital zoom implementations to select | |
332 | the area of the image that will be scaled up. | |
333 | ||
334 | Crop settings are defined by a crop rectangle and represented in a | |
335 | struct :ref:`v4l2_rect <v4l2-rect>` by the coordinates of the top | |
336 | left corner and the rectangle size. Both the coordinates and sizes are | |
337 | expressed in pixels. | |
338 | ||
339 | As for pad formats, drivers store try and active rectangles for the | |
340 | selection targets :ref:`v4l2-selections-common`. | |
341 | ||
342 | On sink pads, cropping is applied relative to the current pad format. | |
343 | The pad format represents the image size as received by the sub-device | |
344 | from the previous block in the pipeline, and the crop rectangle | |
345 | represents the sub-image that will be transmitted further inside the | |
346 | sub-device for processing. | |
347 | ||
348 | The scaling operation changes the size of the image by scaling it to new | |
349 | dimensions. The scaling ratio isn't specified explicitly, but is implied | |
350 | from the original and scaled image sizes. Both sizes are represented by | |
351 | struct :ref:`v4l2_rect <v4l2-rect>`. | |
352 | ||
353 | Scaling support is optional. When supported by a subdev, the crop | |
354 | rectangle on the subdev's sink pad is scaled to the size configured | |
355 | using the | |
356 | :ref:`VIDIOC_SUBDEV_S_SELECTION <vidioc-subdev-g-selection>` IOCTL | |
357 | using ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the | |
358 | subdev supports scaling but not composing, the top and left values are | |
359 | not used and must always be set to zero. | |
360 | ||
361 | On source pads, cropping is similar to sink pads, with the exception | |
362 | that the source size from which the cropping is performed, is the | |
363 | COMPOSE rectangle on the sink pad. In both sink and source pads, the | |
364 | crop rectangle must be entirely contained inside the source image size | |
365 | for the crop operation. | |
366 | ||
367 | The drivers should always use the closest possible rectangle the user | |
368 | requests on all selection targets, unless specifically told otherwise. | |
369 | ``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round | |
370 | the image size either up or down. :ref:`v4l2-selection-flags` | |
371 | ||
372 | ||
373 | Types of selection targets | |
374 | -------------------------- | |
375 | ||
376 | ||
377 | Actual targets | |
378 | ^^^^^^^^^^^^^^ | |
379 | ||
380 | Actual targets (without a postfix) reflect the actual hardware | |
381 | configuration at any point of time. There is a BOUNDS target | |
382 | corresponding to every actual target. | |
383 | ||
384 | ||
385 | BOUNDS targets | |
386 | ^^^^^^^^^^^^^^ | |
387 | ||
388 | BOUNDS targets is the smallest rectangle that contains all valid actual | |
389 | rectangles. It may not be possible to set the actual rectangle as large | |
390 | as the BOUNDS rectangle, however. This may be because e.g. a sensor's | |
391 | pixel array is not rectangular but cross-shaped or round. The maximum | |
392 | size may also be smaller than the BOUNDS rectangle. | |
393 | ||
394 | ||
395 | Order of configuration and format propagation | |
396 | --------------------------------------------- | |
397 | ||
398 | Inside subdevs, the order of image processing steps will always be from | |
399 | the sink pad towards the source pad. This is also reflected in the order | |
400 | in which the configuration must be performed by the user: the changes | |
401 | made will be propagated to any subsequent stages. If this behaviour is | |
402 | not desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This | |
403 | flag causes no propagation of the changes are allowed in any | |
404 | circumstances. This may also cause the accessed rectangle to be adjusted | |
405 | by the driver, depending on the properties of the underlying hardware. | |
406 | ||
407 | The coordinates to a step always refer to the actual size of the | |
408 | previous step. The exception to this rule is the source compose | |
409 | rectangle, which refers to the sink compose bounds rectangle --- if it | |
410 | is supported by the hardware. | |
411 | ||
412 | 1. Sink pad format. The user configures the sink pad format. This format | |
413 | defines the parameters of the image the entity receives through the | |
414 | pad for further processing. | |
415 | ||
416 | 2. Sink pad actual crop selection. The sink pad crop defines the crop | |
417 | performed to the sink pad format. | |
418 | ||
419 | 3. Sink pad actual compose selection. The size of the sink pad compose | |
420 | rectangle defines the scaling ratio compared to the size of the sink | |
421 | pad crop rectangle. The location of the compose rectangle specifies | |
422 | the location of the actual sink compose rectangle in the sink compose | |
423 | bounds rectangle. | |
424 | ||
425 | 4. Source pad actual crop selection. Crop on the source pad defines crop | |
426 | performed to the image in the sink compose bounds rectangle. | |
427 | ||
428 | 5. Source pad format. The source pad format defines the output pixel | |
429 | format of the subdev, as well as the other parameters with the | |
430 | exception of the image width and height. Width and height are defined | |
431 | by the size of the source pad actual crop selection. | |
432 | ||
433 | Accessing any of the above rectangles not supported by the subdev will | |
434 | return ``EINVAL``. Any rectangle referring to a previous unsupported | |
435 | rectangle coordinates will instead refer to the previous supported | |
436 | rectangle. For example, if sink crop is not supported, the compose | |
437 | selection will refer to the sink pad format dimensions instead. | |
438 | ||
439 | ||
440 | .. _subdev-image-processing-crop: | |
441 | ||
442 | .. figure:: dev-subdev_files/subdev-image-processing-crop.* | |
443 | :alt: subdev-image-processing-crop.svg | |
444 | :align: center | |
445 | ||
446 | Image processing in subdevs: simple crop example | |
447 | ||
448 | In the above example, the subdev supports cropping on its sink pad. To | |
449 | configure it, the user sets the media bus format on the subdev's sink | |
450 | pad. Now the actual crop rectangle can be set on the sink pad --- the | |
451 | location and size of this rectangle reflect the location and size of a | |
452 | rectangle to be cropped from the sink format. The size of the sink crop | |
453 | rectangle will also be the size of the format of the subdev's source | |
454 | pad. | |
455 | ||
456 | ||
457 | .. _subdev-image-processing-scaling-multi-source: | |
458 | ||
459 | .. figure:: dev-subdev_files/subdev-image-processing-scaling-multi-source.* | |
460 | :alt: subdev-image-processing-scaling-multi-source.svg | |
461 | :align: center | |
462 | ||
463 | Image processing in subdevs: scaling with multiple sources | |
464 | ||
465 | In this example, the subdev is capable of first cropping, then scaling | |
466 | and finally cropping for two source pads individually from the resulting | |
467 | scaled image. The location of the scaled image in the cropped image is | |
468 | ignored in sink compose target. Both of the locations of the source crop | |
469 | rectangles refer to the sink scaling rectangle, independently cropping | |
470 | an area at location specified by the source crop rectangle from it. | |
471 | ||
472 | ||
473 | .. _subdev-image-processing-full: | |
474 | ||
475 | .. figure:: dev-subdev_files/subdev-image-processing-full.* | |
476 | :alt: subdev-image-processing-full.svg | |
477 | :align: center | |
478 | ||
479 | Image processing in subdevs: scaling and composition with multiple sinks and sources | |
480 | ||
481 | The subdev driver supports two sink pads and two source pads. The images | |
482 | from both of the sink pads are individually cropped, then scaled and | |
483 | further composed on the composition bounds rectangle. From that, two | |
484 | independent streams are cropped and sent out of the subdev from the | |
485 | source pads. | |
486 | ||
487 | ||
488 | .. toctree:: | |
489 | :maxdepth: 1 | |
490 | ||
491 | subdev-formats | |
492 | ||
493 | ||
494 | ||
495 | ||
496 | .. ------------------------------------------------------------------------------ | |
497 | .. This file was automatically converted from DocBook-XML with the dbxml | |
498 | .. library (https://github.com/return42/sphkerneldoc). The origin XML comes | |
499 | .. from the linux kernel, refer to: | |
500 | .. | |
501 | .. * https://github.com/torvalds/linux/tree/master/Documentation/DocBook | |
502 | .. ------------------------------------------------------------------------------ |