main-loop: drop the BQL if the I/O appears to be spinning

The char-flow refactoring introduced a busy-wait that depended on
an action from the VCPU thread.  However, the VCPU thread could
never take that action because the busy-wait starved the VCPU thread
of the BQL because it never dropped the mutex while running select.

Paolo doesn't want to drop this optimization for fear that we will
stop detecting these busy waits.  I'm afraid to keep this optimization
even with the busy-wait fixed because I think a similar problem can
occur just with heavy I/O thread load manifesting itself as VCPU pauses.

As a compromise, introduce an artificial timeout after a thousand
iterations but print a rate limited warning when this happens.  This
let's us still detect when this condition occurs without it being
a fatal error.

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Message-id: 1365169560-11012-1-git-send-email-aliguori@us.ibm.com

diff --git a/main-loop.c b/main-loop.c
index eb80ff369f9ef9e327a2170201086702ba9b4db2..f46aece8b86b7fc27698521aa319afb1bf69f228 100644 (file)
--- a/main-loop.c
+++ b/main-loop.c
@@ -188,14 +188,39 @@ static void glib_pollfds_poll(void)
     }
 }
 
+#define MAX_MAIN_LOOP_SPIN (1000)
+
 static int os_host_main_loop_wait(uint32_t timeout)
 {
     int ret;
+    static int spin_counter;
 
     glib_pollfds_fill(&timeout);
 
+    /* If the I/O thread is very busy or we are incorrectly busy waiting in
+     * the I/O thread, this can lead to starvation of the BQL such that the
+     * VCPU threads never run.  To make sure we can detect the later case,
+     * print a message to the screen.  If we run into this condition, create
+     * a fake timeout in order to give the VCPU threads a chance to run.
+     */
+    if (spin_counter > MAX_MAIN_LOOP_SPIN) {
+        static bool notified;
+
+        if (!notified) {
+            fprintf(stderr,
+                    "main-loop: WARNING: I/O thread spun for %d iterations\n",
+                    MAX_MAIN_LOOP_SPIN);
+            notified = true;
+        }
+
+        timeout = 1;
+    }
+
     if (timeout > 0) {
+        spin_counter = 0;
         qemu_mutex_unlock_iothread();
+    } else {
+        spin_counter++;
     }
 
     ret = g_poll((GPollFD *)gpollfds->data, gpollfds->len, timeout);