add pmxcfs restart detection heuristic for IPCC

add pmxcfs restart detection heuristic for IPCC

Allow clean pmxcfs restarts to be fully transparent for the IPCC
using perl stack aboce.

A restart of pmxcfs invalidates the connection cache, we only set the
cached connection to NULL for this case and the next call to
ipcc_send_rec would connect newly.
Further, such a restart may need quite a bit time (seconds).

Thus write a status file to flag a possible restart when terminating
from pmxcfs. Delete this flag file once we're up and ready again.
Error case handling is described further below.

If a new connections fails and this flag file exists then retry
connecting for a certain period (for now five seconds).

If a cached connection fails always retry once, as every pmxcfs
restart makes the cached connection invalid, even if IPCC would be
fully up and ready again and then also follow the connection polling
heuristic if the restart flag exists, as new connections do.

We use the monotonic clock to avoid problems if the (system) time
changes and to keep things as easy as possible.

We delete the flag file if a IPCC call could not connect in the grace
period, but only if the file is still the same, i.e., no one else has
deleted and recreated it in the meantime (e.g. a second cfs restart).
This guarantees that IPCC calls try this heuristic only for a limited
time (5 seconds until the first failed one) if the cfs does not
starts again.

Further, as the flag resided in /run/... - which is always a tmpfs
(thus in memory and thus cleaned upon reboot) we may not run into
leftover flag files on a node reset, e.g. done by the HA watchdog for
self-fencing.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>