zio: try to execute TYPE_NULL ZIOs on the current task

Many TYPE_NULL ZIOs are used to provide a sync point for child ZIOs, and
do not do any actual work themselves. However, they are still dispatched
to a dedicated, single-thread taskq, which leads to their execution
being entirely task switch and dequeue overhead for no actual reason.

This commit changes it so that when selecting a parent ZIO to execute,
if the parent is TYPE_NULL and has no done function (that is, no
additional work), it is executed on the same thread. This reduces task
switches and frees up CPU cores for other work.

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
Closes #16134

diff --git a/module/zfs/zio.c b/module/zfs/zio.c
index ce967a7cdc68ecf7decbfad493429ee32bee2f16..0e7993d87e8715c6ae72d1515661841b4a174c3d 100644 (file)
--- a/module/zfs/zio.c
+++ b/module/zfs/zio.c
@@ -803,9 +803,10 @@ zio_notify_parent(zio_t *pio, zio_t *zio, enum zio_wait_type wait,
 
                /*
                 * If we can tell the caller to execute this parent next, do
-                * so. We only do this if the parent's zio type matches the
-                * child's type. Otherwise dispatch the parent zio in its
-                * own taskq.
+                * so. We do this if the parent's zio type matches the child's
+                * type, or if it's a zio_null() with no done callback, and so
+                * has no actual work to do. Otherwise dispatch the parent zio
+                * in its own taskq.
                 *
                 * Having the caller execute the parent when possible reduces
                 * locking on the zio taskq's, reduces context switch
@@ -825,7 +826,8 @@ zio_notify_parent(zio_t *pio, zio_t *zio, enum zio_wait_type wait,
                 * of writes for spa_sync(), and the chain of ZIL blocks.
                 */
                if (next_to_executep != NULL && *next_to_executep == NULL &&
-                   pio->io_type == zio->io_type) {
+                   (pio->io_type == zio->io_type ||
+                   (pio->io_type == ZIO_TYPE_NULL && !pio->io_done))) {
                        *next_to_executep = pio;
                } else {
                        zio_taskq_dispatch(pio, type, B_FALSE);