block, bfq: set next_rq to waker_bfqq->next_rq in waker injection

Since commit c5089591c3ba ("block, bfq: detect wakers and
unconditionally inject their I/O"), when the in-service bfq_queue, say
Q, is temporarily empty, BFQ checks whether there are I/O requests to
inject (also) from the waker bfq_queue for Q. To this goal, the value
pointed by bfqq->waker_bfqq->next_rq must be controlled. However, the
current implementation mistakenly looks at bfqq->next_rq, which
instead points to the next request of the currently served queue.

This mistake evidently causes losses of throughput in scenarios with
waker bfq_queues.

This commit corrects this mistake.

Fixes: c5089591c3ba ("block, bfq: detect wakers and unconditionally inject their I/O")
Signed-off-by: Jia Cheng Hu <jia.jiachenghu@gmail.com>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Paolo Valente <paolo.valente@linaro.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index eb2ca32d5b635abb179c7a74d04e69641b0e7b35..fdc5e163b2fedac1f3f3d18261501b577862978d 100644 (file)
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -4499,7 +4499,7 @@ check_queue:
                        bfqq = bfqq->bic->bfqq[0];
                else if (bfq_bfqq_has_waker(bfqq) &&
                           bfq_bfqq_busy(bfqq->waker_bfqq) &&
-                          bfqq->next_rq &&
+                          bfqq->waker_bfqq->next_rq &&
                           bfq_serv_to_charge(bfqq->waker_bfqq->next_rq,
                                              bfqq->waker_bfqq) <=
                           bfq_bfqq_budget_left(bfqq->waker_bfqq)