quorum: don't leak memory on error

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

testvotequorum: fix test loop to break if votequorum goes away

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

quorumtool: fix return code

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

testquorum: check for quorum_dispatch return code

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

quorumtools: check for quorum_dispatch return code

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

specfile: ship new man pages

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

man: add *quorum_track* devel man pages

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

quorum: drop dead code

spotted while writing man pages. There are no users for this struct

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

man: add quorum_overview.8 man page

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

man: hook quorum and votequorum devel man pages with genman script

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

man: rename all devel man pages to .3.in

tidy up man/Makefile.am a bit in the process

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

man: add build infrastructure to generate devel man pages

this is useful to include ipc_common errors into all man pages

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

move cs_strerror() to common_lib

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Treat ENOBUFS as TRY_AGAIN

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

move hdb_error_to_cs to common_lib

Note the previous inconsistent implementation.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Add a common library that can be shared between libs and corosync

We have always had this problem and worked around it by coping code
or using inline functions. Both not good IMO.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Remove cs_config.h from global header install

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove include/engine/quorum and integrate it into exec/engine.h

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove swab.h from global headers

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove list.h from global header install

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove mar_gen.h from global header install since it is not needed

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Add CS_DISPATCH_ONE_NONBLOCKING dispatch type

Add missing option for dispatch, which fills gap in combination of
block/nonblock and one/all dispatch types. New type doesn't mask
CS_ERR_TRY_AGAIN, and it means "no message was processed".

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

corotype: drop deprecated CPG_ defines

the only user of those obsoleted defines is dlm master (already ported)
to use CS_ and cmirror (that needs full porting to new corosync either way).

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

corotypes: drop deprecated QUORUM_ defines

neither corosync or any of the dependencies use it.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

corotypes: drop deprecated EVS_ defines

none of our current dependencies use it.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Free state variable allocated in wd_resource_state_is_ok

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove leaked resource error from wd_resource_state_is_ok

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove use after free and free of uninit value in mainconfig error path

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove use after free in corosync_main_config_set in error path

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove dead code in sam test agent

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

votequorum: fix quorum_ringid setting before any delivery occours

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

Make sure ipc functions return CS_ERR_TRY_AGAIN and not CS_ERR_TIMEOUT

This is because most applications that use corosync do not test
for TIMEOUT but only for TRY_AGAIN.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-and-Tested-by: Fabio M. Di Nitto <fdinitto@redhat.com>

Remove deprecated function qb_util_set_log_function()

Use the standard qb_log api.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

LOG: add libqb as a "subsys"

So we can see libqb internal logs

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

cmap: Check RO flag in adjust int function

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CMAP man pages

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CPG: Send CPG_REASON_PROCDOWN when really needed

This patch fixes the issue that in some cases where cpg_finalize()
was called just after cpg_leave() was called, CPG_REASON_PROCDOWN
might also be sent while CPG_REASON_LEAVE had already been sent.
This behavior is not aligned with what the man page has described:
"CPG_REASON_PROCDOWN - the process left a group without calling
cpg_leave()."
And it will confuse CPG's clients in that one process left results
in two different reasons being sent.

The root cause of this issue is cpg_leave() will return after
adding the LEAVE message to the sending queue, but the cpg's group
name has not been cleared yet. Just at that time, cpg_finalize()
is being called, then it determines if there is the calling of
cpg_leave() happened only by the checking of cpg's group name, so
this method is not sufficient.

Signed-off-by: Jiaju Zhang <jjzhang@suse.de>
Reviewed-by: Steven Dake <sdake@redhat.com>

quorumtool: fix return codes for show_status and monitor

correct return codes should be:
 1 if node is quorate
 0 if node is not quorate
-1 if there is any error gather info on the node

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

quorumtools: fix nodes display on status

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: fix expected votes manual override from quorumtools

votequorum internal quorum/expected_vote check was slightly too
conservative and was not done correctly when leave_remove feature
is enabled.

this fix allows admins to effectively override expected_votes
and drive ev_barrier as expected.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

Better checks of integer values in coroparse

Instead of atoi, strtol is used. This allows detection of typical
problems like empty value of key and incorrectly entered numbers.

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

votequorum: add runtime internal data to icmap runtime.votequorum.*

specifically ev_barrier, two_node, lowest_node_id and wait_for_all_status
are values that change internally at runtime and keeping track
of those can make debugging rather easy, specially when LOG_DEBUG is not
set.

Also track our node id.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-By: Christine Caulfield <ccaulfie@redhat.com>

Wait for corosync-notifyd exit in init script

Without wait for real exit of corosync-notifyd it can happen, that new
corosync-notifyd is killed. To prevent such condition, stop now wait for
process to die, before exit of stop function.

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Show correct error when open of logfile failed

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Store error str if can't open logfile

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

IPC: reference count the connection whilst flushing the outq

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

build: fix make dist and make rpm

do some cleanup around to include all files that need to be shipped
and honor conditional builds properly

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Angus Salkeld <asalkeld@redhat.com>

IPC: allow for failures in the connection_created callback

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

votequorum: add leave_remove option

this also cleanup NODESTATE for good. JOINING was never used

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: honor onwire node flags change

internal flags were not propagated correctly in the node status

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

quorum: fix load/unload priority for quorum services

all main services are loaded at priority 1.
vfs_quorum and votequorum did not specify a priority and
automatically defaulting to 0, that has a special meaning
of being loaded last and unloaded last.

this is not correct behavior and limits what votequorum
can do at shutdown, for example notify other nodes that
it is leaving (something that cannot be gathered by
totem membership change callback).

fix vsf_quorum to load at priority 1 as the other
default services and bump votequorum to 2 (needs to
unload before everything else currently known).

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

service: fix service unload regression introduced by lcrso dropping

service exec_exit_fn was not honored because the loop was looking
into the wrong icmap key

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

votequorum: fix possible string overflow (-1) in qdevice_register

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

votequorum: drop unnecessary flags

code inspection shows that those internal flags are never used

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

CTS: cleanup the cpg test agent

improve the diagnostic log messages

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: make the systemd logic more reliable

rely on positive logic as there can be multiple
failure reasons.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>

Honor exec_init_fn call

exec_init_fn now either returns NULL (success) or a string which indicates
the error that occured during service engine initialization.  If an error
occurs, corosync will exit.  This patch adds ykd and makes other suggestions
from Fabio Di Nitto.

Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Fabio Di Nitto <fdinitto@redhat.com>

Return an exit code of 1 if an interface is faulty in corosync-cfgtool

Signed-off-by: Oren Nechustan <theoren28@hotmail.com>
Signed-off-by: Steven Dake <sdake@redhat.com>
Reviewed-by: Fabio Di Nitto <fdinitto@redhat.com>

cmap: add iterator finalize

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

cmap: add -D option to getopt

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

votequorum: rename qdisk to qdevice

a quorum device is not necessarely a disk and this also aligns
various names to be generic

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-By: Christine Caulfield <ccaulfie@redhat.com>

quorum: drop quorum.quorate config option

it's unused / unnecessary

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-By: Christine Caulfield <ccaulfie@redhat.com>

CTS: add test VoteQuorumWaitForAll

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: remove test service config

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

augeas: update the lense (rm amf & add update quorum options)

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: be consistent with the cpg group name

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: make the status command more accurate

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: remove SamTestQuorum as there is not test_quorum anymore

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: ignore blackbox shm

(only whilst running as it is still visable)

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: delete resourses recursively

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: init votequorum by default

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

CTS: account for change in sam resource path.

This was:
process_name:pid
now:
pid

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: handle socket exceptions better

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: fix shell script variable name

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

update TODO list

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: fix expected_votes propagation

it is not correct to randomly accept expected_votes from any node in
the cluster. We can only allow expected_votes from quorate nodes.

A quorate cluster is "always" right and have the correct expected_votes.

One of the different bug triggers:

quorum {
  expected_votes: 8
  auto_tie_breaker: 1
  last_man_standing: 1
}

start all 8 nodes.
clean shut down 2 nodes.
wait for lms to kick in.
kill 3 nodes with highest nodeid
(we want to retain a quorate partition of 3 nodes)
start one node again -> cluster will be unquorate

This happens because the node rebooting/rejoining with
non current cluster status will propagate an expected_votes of 8,
while in reality the cluster is down to expected_votes: 3.

4 nodes are still < 5 (quorum for 8 nodes/votes).

In order to avoid this condition, we need to exchange expected_votes
information among nodes but we cannot randomly trust everybody.

1) Allow expected_votes to be changed cluster-wide only if the
   information is coming from a quorate node.
2) Fix node->expected_votes based on quorate status
3) allow a joining node to decrease quorum and expected_votes
   if the node is not yet quorate, but it's joining a quorate
   cluster

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: fix auto_tie_breaker design and simplify code a lot

auto_tie_breaker requires to know the lowest node id in the currently
quorate partition and not of the whole cluster.

this allow us to determine the lowest node id as soon as we are quorate
and remove the complexity to read it from WFA or nodelist. Add
the same time it adds the flexibility for dynamic nodeids in a cluster.

drop requirement on WFA if nodelist is not specified

update man page

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: drop NODESTATE_LEAVING

this is another leftover from cman compatibility layer

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

update TODO list

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: add documentation and man pages

fix a few typos on the way and separate config / library bits

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: change quorum.expected_votes override behavior

as agreed on the mailing list, quorum.expected_votes should override
automatically calculated expected_votes from nodelist.

Also simplify the code to handle expected_votes. "silly defaults" is now
unnecessary because votequorum does config sanity checks upfront.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

votequorum: two_node should enable wait_for_all by default

This avoids fencing races at startup of a cluster.

It is still possible to override WFA by explicitly setting
wait_for_all: 0

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Angus Salkeld <asalkeld@redhat.com>

CTS: make basic tests config-generic

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: fix starting/stopping of test_agents

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: tidy up the shutdown of cpg_test_agent

it is not neccessary to close the fd and remove it
from the mainloop

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: temp comment out quorum tests

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: fix quourm command

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

CTS: fix up the formt strings

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Add debug log messages to corosync for join/leave

This is needed by cts.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

LOG: make sure that debug works to syslog

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

totemiba: Remove unused wthread.h include

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Angus Salkeld <asalkeld@redhat.com>

Make xmlconf in SPEC conditional

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Angus Salkeld <asalkeld@redhat.com>

Change the last references from objctl to cmapctl

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>

Update TODO list

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

votequorum: add support for nodelist config bits

expected votes is now calculated automatically and quorum.expected_votes
can be used to override nodelist calculation. The highest of the two
value is used for runtime.

quorum_votes can be specified either in the node list or in quorum.votes.
The node list has priority over global.

propagate votequorum initalization errors (due to config inconsistencies)
back to vsf_quorum.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>

Remove all unneccessary "\n" from log messages

These look ugly, are inconsistently done and just have
to be removed later in libqb before calling syslog.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

Shorten some really long lines in main.c

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

cmap: add a delete with prefix (needed by cts)

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

cmap: change -t and -T around (capital == with prefix)

I want to add a prefic delete option and then these will
not be consistent.

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

cmap: tweek the usage text

1) It wasn't obvious to me what -b did
2) -a has been removed

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>