fix validation of self-signed cert chains

fix validation of self-signed cert chains

The interface here is a bit weird - if the verify callback returns 1
for a certificate higher up in the chain, this will propagate to the
next invocation of the callback for the next certificate, even if
openssl on its own would not trust the certificate.

By re-ordering the checks and keeping track of the fact that we
returned 1 despite openssl failing its own validation, the validation
logic should now cover all combinations of certificate count and
self-signed/system trust status.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>