io_uring: fix imbalanced sqo_mm accounting

We do the initial accounting of locked_vm and pinned_vm before we have
setup ctx->sqo_mm, which means we can end up having not accounted the
memory at setup time, but still decrement it when we exit. This causes
an imbalance in the accounting.

Setup ctx->sqo_mm earlier in io_uring_create(), before we do the first
accounting of mm->{locked,pinned}_vm. This also unifies the state
grabbing for the ctx, and eliminates a failure case in
io_sq_offload_start().

Fixes: f74441e6311a ("io_uring: account locked memory before potential error case")
Reported-by: Robert M. Muncrief <rmuncrief@humanavance.com>
Reported-by: Niklas Schnelle <schnelle@linux.ibm.com>
Tested-by: Niklas Schnelle <schnelle@linux.ibm.com>
Tested-by: Robert M. Muncrief <rmuncrief@humanavance.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/fs/io_uring.c b/fs/io_uring.c
index e030b33fa53e12574ecb391d534741f615f3e352..384df86dfc6921fa5c7eae49af43b85c49aeb130 100644 (file)
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -7447,9 +7447,6 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx,
 {
        int ret;
 
-       mmgrab(current->mm);
-       ctx->sqo_mm = current->mm;
-
        if (ctx->flags & IORING_SETUP_SQPOLL) {
                ret = -EPERM;
                if (!capable(CAP_SYS_ADMIN))
@@ -7494,10 +7491,6 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx,
        return 0;
 err:
        io_finish_async(ctx);
-       if (ctx->sqo_mm) {
-               mmdrop(ctx->sqo_mm);
-               ctx->sqo_mm = NULL;
-       }
        return ret;
 }
 
@@ -8547,6 +8540,9 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p,
        ctx->user = user;
        ctx->creds = get_current_cred();
 
+       mmgrab(current->mm);
+       ctx->sqo_mm = current->mm;
+
        /*
         * Account memory _before_ installing the file descriptor. Once
         * the descriptor is installed, it can get closed at any time. Also