sanlock/wdmd: remove global connection
As long as the sanlock daemon was running, it kept a
constant connection to wdmd, even when no lockspaces
existed. This prevented sanlock and wdmd from being
restarted independently, even when they were unused.
Independent restarting is necessary for upgrades, so
remove the global connection from sanlock to wdmd and
leave only the per-lockspace connections. The lockspace
connections now need to hold a refcount on wdmd which
prevents wdmd restarts.
Also in wdmd, if a client connection is closed, the
refcount must not be cleared on it, otherwise wdmd
could possibly be cleanly shutdown from sigterm while
an expired connection was awaiting a watchdog reset.
Also log at the error level when we kill a pid for
recovery, when that pid exits, and when all pids
are clear for recovery. This makes it much simpler
to see exactly what led up to a watchdog reset
after the fact.
Signed-off-by: David Teigland <teigland@redhat.com>