votequorum: improve wait_for_all reset handling when merging parititions
this change allow a node to rejoin a cluster that has already seen
wait_for_all and reset the flag if the partition that the node is joining
is quorate.
Also propagate current wait_for_all_status and quorate info via nodeinfo.
Reviewed-by: Steven Dake <sdake@redhat.com> Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
votequorum: fix regression introduced by 05b4e99a6e (dispatch notifications only once)
Effectively there are 2 kind of quorum notifications that cannot be merged into 1.
config_change: when the quorum membership changes, triggered by totem/membership changes
(node join/leave)
quorum_status_changes: same membership node becomes quorate or not.
A quorum status change does not necessarely match a membership change, hence it needs
to be dispatched separately.
An example is a cluster that is not quorate, user changes the amount of votes in a node to
regain quorum. No membership changes happen at this point, but votequorum will
effectively broadcast the votes: XX change and recalculate quorum.
In turn we need to notify quorum users.
Reviewed-by: Steven Dake <sdake@redhat.com> Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
first pass to bring votequorum at corosync codying style.
fix whitespaces, add missing {}, fix comments, be consistent with
ENTER/LEAVE usage, be consistent with some functions variable names
and some more cosmetic changes
Reviewed-by: Steven Dake <sdake@redhat.com> Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
votequorum: add last_man_standing support (default: off)
this flag (0|1) can be configured via quorum.last_man_standing and when
enabled, it allows expected_votes to be dynamically recalculated.
Assuming an 8 nodes cluster, every node votes 1 (mandatory requirement for
this feature).
In the first event, 3 nodes are lost.
The remaining partition of 5 is barely quorate.
After a configurable timeout (quorum.last_man_standing_window, default 10sec)
the quorate partition is allow to recalculate expected_votes based on
the remaining nodes.
This operation will bring expected_votes to 5 and quorum to 3.
Repeating the above loop, in the next event, 2 more nodes are allowed to
die. etc. etc.
Reviewed-by: Steven Dake <sdake@redhat.com> Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
quorum: change API to return quorum type at initialization time
corosync internal theory of operation is that without a quorum provider
the cluster is always quorate. This is fine for membership free clusters
but it does pose a problem for applications that need membership and
"real" quorum.
this change add quorum_type to quorum_initialize call to return QUORUM_FREE
or QUORUM_SET. Applications can then make their own decisions to error out
or continue operating.
The only other way to know if a quorum provider is enabled/configured is
to poke at confdb/objdb, but adds an unnecessary burden to applications
that really don't need to use an entire library for a boolean value.
Reviewed-by: Steven Dake <sdake@redhat.com> Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Steven Dake [Fri, 6 Jan 2012 06:32:15 +0000 (23:32 -0700)]
Move logsys.c into corosync binary instead of a shared object
Our preferred shared logging system is exported via the libqb library. As
a result, the corosync project no longer needs to export logsys.so and the
code can be directly included in the binary. The header file can also be
removed.
Signed-off-by: Steven Dake <sdake@redhat.com> Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Angus Salkeld [Wed, 14 Dec 2011 23:43:00 +0000 (10:43 +1100)]
Fix cpgbench (large message sizes)
To allow async cpg messages of 1M we need to:
1) increase the totem queue size 4 times
2) align the critical level to one large message free
There are a number of reasons for doing this:
We can't let cpg_mcast_joined() fail because the user will not see it
and will assume is has succeded.
The reason we are getting good performance is by providing a negative
feedback loop from the totem q to the IPC/poll system. This relies
on 4 q states low/med/high/crit. With messages of size 1M you
now have a q of size one and now go from level low to crit instantly
then back to low as messages are put on and taken off. I don't think
this is the best behaviour. By having a q size of 4 allows the system
to utilize the q better and give us time to respond to changes in
the q level.
To effectively achieve flow control with a q of size 1 would require
all the clients to request the space on the q like is done in
totempg_groups_joined_reserve() but probably in shared memory
This would take quite a bit of re-work.
Angus Salkeld [Wed, 14 Dec 2011 01:03:18 +0000 (12:03 +1100)]
Be more flexible (correct) with flowcontrol.
Many functions do not require flowcontrol and are two-way
so they can get failures from corosync.
Only cpg_mcast_joined() _really_ needs the current level
of flowcontrol.
Signed-off-by: Angus Salkeld <asalkeld@redhat.com> Reviewed-by: Steven Dake <sdake@redhat.com>
Steven Dake [Tue, 29 Nov 2011 14:41:53 +0000 (07:41 -0700)]
From: Yunkai Zhang:
Today, I have observed one of the reason that corosync running into
FAILED TO RECEIVE state.
There was five nodes(A,B,C,D,E) in my testing, and I limited the UDP
transmission rate of C nodes by iptables command:
iptables -A INPUT -i eth0 -p udp -m limit --limit 10000/s
--limit-burst 1 -j ACCEPT
iptables -A INPUT -i eth0 -p udp -j DROP
After one hour later, C node had been missing some MCAST messages,
it's state described as following:
==state of C node==
my_aru:0x805
my_high_seq_received:0xC2C
my_aru_count:7
=>receved MCAST message with seq:806 from B nodes
=>enter *message_handler_mcast*
=>add this message to regular_sort_queue
...
=>enter *update_aru* function
=> range = (my_high_seq_received - my_aru)
= (0xC2C - 0x805)
= 1063
=> if range>1024, do nothing and and return directly.
==END==
According this logic, after (my_high_req_received-my_aru)>1024, my_aru
will not be updated though corosync can receive MCAST messages
retransmitted by other nodes.
But at that timte, my_aru_count was only 7. So the corosync at C node
would keep in this status until my_aru_count increased to
fail_to_recv_const(the default value is 2500). This was a long time
for corosync, but we wasted it.
To solve this issue, maybe we can enlarge the range condition in
update_aru function? Or we just ingnore the checking of range value,
it seems no harmfull, because we have been using fail_to_recv_const to
control the things.
Signed-off-by: Steven Dake <sdake@redhat.com> Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Yunkai Zhang [Mon, 28 Nov 2011 10:32:36 +0000 (18:32 +0800)]
Correct nodeid of token when we retransmit it
Although incorrect nodeid will not affect program's logic, but it will
make us confused when we add some logs to record the transmission path of
token in debug mode.
Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com> Reviewed-by: Steven Dake <sdake@redhat.com>
Yunkai Zhang [Sat, 26 Nov 2011 09:42:54 +0000 (17:42 +0800)]
Fixed bug when corosync receive JoinMSG in OPERATIONAL state
Accordig the totem protocal, nodes should enter GATHER state when it
receive JoinMSG in OPERATIONAL state. If we discard it in OPERATIONAL
state, the nodes sending this JoinMSG could not receive the response
untill other nodes reach token lost timeout.
This bug will cause nodes having entered GATHER state spend more time to
rejoin the ring, and then it will make nodes reach token expired timeout
more easily.
Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com> Reviewed-by: Steven Dake <sdake@redhat.com>
projects / mirror_corosync.git / log