rcu: Accelerate callbacks for CPU initiating a grace period

Because grace-period initialization is carried out by a separate
kthread, it might happen on a different CPU than the one that
had the callback needing a grace period -- which is where the
callback acceleration needs to happen.

Fortunately, rcu_start_gp() holds the root rcu_node structure's
->lock, which prevents a new grace period from starting.  This
allows this function to safely determine that a grace period has
not yet started, which in turn allows it to fully accelerate any
callbacks that it has pending.  This commit adds this acceleration.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 74df86bd9204aef5ec9d14ca8b4b0094777bef87..93d6871bf7f960b9ed43b37a6049d2acfb12513d 100644 (file)
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1404,15 +1404,37 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
            !cpu_needs_another_gp(rsp, rdp)) {
                /*
                 * Either we have not yet spawned the grace-period
-                * task or this CPU does not need another grace period.
+                * task, this CPU does not need another grace period,
+                * or a grace period is already in progress.
                 * Either way, don't start a new grace period.
                 */
                raw_spin_unlock_irqrestore(&rnp->lock, flags);
                return;
        }
 
+       /*
+        * Because there is no grace period in progress right now,
+        * any callbacks we have up to this point will be satisfied
+        * by the next grace period.  So promote all callbacks to be
+        * handled after the end of the next grace period.  If the
+        * CPU is not yet aware of the end of the previous grace period,
+        * we need to allow for the callback advancement that will
+        * occur when it does become aware.  Deadlock prevents us from
+        * making it aware at this point: We cannot acquire a leaf
+        * rcu_node ->lock while holding the root rcu_node ->lock.
+        */
+       rdp->nxttail[RCU_NEXT_READY_TAIL] = rdp->nxttail[RCU_NEXT_TAIL];
+       if (rdp->completed == rsp->completed)
+               rdp->nxttail[RCU_WAIT_TAIL] = rdp->nxttail[RCU_NEXT_TAIL];
+
        rsp->gp_flags = RCU_GP_FLAG_INIT;
-       raw_spin_unlock_irqrestore(&rnp->lock, flags);
+       raw_spin_unlock(&rnp->lock); /* Interrupts remain disabled. */
+
+       /* Ensure that CPU is aware of completion of last grace period. */
+       rcu_process_gp_end(rsp, rdp);
+       local_irq_restore(flags);
+
+       /* Wake up rcu_gp_kthread() to start the grace period. */
        wake_up(&rsp->gp_wq);
 }