When flushing, discard only memb_join messages

Patch solves problem when 1 ring out of 2 went up/down quite often.

The simplest setup to reproduce bug is following:
- 2 VMs, connected by 2 network interfaces
- OS: Linux
- On one of the VMs, a test program sending some CPG messages (see the
  script "test_corosync.sh" joined to this mail for example)

Here are the Corosync logs we get when we do this setup:

Jun 06 16:23:40 corosync [TOTEM ] A processor joined or left the
membership and a new membership was formed.
Jun 06 16:23:40 corosync [CPG   ] chosen downlist: sender r(0)
ip(192.168.56.104) r(1) ip(192.168.57.104) ; members(old:1 left:0)
Jun 06 16:23:40 corosync [MAIN  ] Completed service synchronization,
ready to provide service.
Jun 06 16:24:37 corosync [TOTEM ] Marking ringid 1 interface
192.168.57.105 FAULTY
Jun 06 16:24:38 corosync [TOTEM ] Automatically recovered ring 1
Jun 06 16:25:33 corosync [TOTEM ] Marking ringid 1 interface
192.168.57.105 FAULTY
Jun 06 16:25:34 corosync [TOTEM ] Automatically recovered ring 1
Jun 06 16:26:35 corosync [TOTEM ] Marking ringid 1 interface
192.168.57.105 FAULTY
Jun 06 16:26:36 corosync [TOTEM ] Automatically recovered ring 1
(...)

The second ring goes down about every 2 minutes and automatically back
up right after.

We spent some times looking for the commit that introduced this bug, and
it appears it's due the following one:
Corosync 1.3.3 -> 1.3.4: e27a58d93d0d3795beb550f87b660c9c04f11386
Corosync 1.4.1 -> 1.4.2: be608c050247e5f9c8266b8a0f9803cc0a3dc881
Commit message: Ignore memb_join messages during flush operations

I had a look at this commit, and it seems to me it's dropping too many
packets:
Because of this commit, while totemrrp_recv_flush() is called, Corosync
drops memb_join packets, but also ORF tokens. In the end, it seems that
sometimes, we drop so many of them that Corosync marks the ring as
faulty.

To fix that, only memb_join messages are dropped now.

Signed-off-by: Jerome FLESCH <jerome.flesch@netasq.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

diff --git a/exec/totemudp.c b/exec/totemudp.c
index e40300a574258f2832b4c6ebb545a872ebf089c1..ac1d81fd92c15756c435d9cffd2e070387370718 100644 (file)
--- a/exec/totemudp.c
+++ b/exec/totemudp.c
@@ -87,7 +87,7 @@
 #define BIND_STATE_REGULAR     1
 #define BIND_STATE_LOOPBACK    2
 
-#define MESSAGE_TYPE_MCAST     1
+#define MESSAGE_TYPE_MEMB_JOIN 3
 
 struct totemudp_socket {
        int mcast_recv;
@@ -454,7 +454,7 @@ static int net_deliver_fn (
         * messages should be dropped)
         */
        message_type = (char *)iovec->iov_base;
-       if (instance->flushing == 1 && *message_type != MESSAGE_TYPE_MCAST) {
+       if (instance->flushing == 1 && *message_type == MESSAGE_TYPE_MEMB_JOIN) {
                iovec->iov_len = FRAME_SIZE_MAX;
                return (0);
        }