cachefiles: Wait rather than BUG'ing on "Unexpected object collision"

cachefiles: Wait rather than BUG'ing on "Unexpected object collision"

BugLink: https://bugs.launchpad.net/bugs/1776254
If we meet a conflicting object that is marked FSCACHE_OBJECT_IS_LIVE in
the active object tree, we have been emitting a BUG after logging
information about it and the new object.

Instead, we should wait for the CACHEFILES_OBJECT_ACTIVE flag to be cleared
on the old object (or return an error).  The ACTIVE flag should be cleared
after it has been removed from the active object tree.  A timeout of 60s is
used in the wait, so we shouldn't be able to get stuck there.

Fixes: 9ae326a69004 ("CacheFiles: A cache that backs onto a mounted filesystem")
Signed-off-by: Kiran Kumar Modukuri <kiran.modukuri@gmail.com>
Signed-off-by: David Howells <dhowells@redhat.com>
(cherry picked from commit c2412ac45a8f8f1cd582723c1a139608694d410d)
Signed-off-by: Daniel Axtens <daniel.axtens@canonical.com>
Acked-by: Seth Forshee <seth.forshee@canonical.com>
Acked-by: Kamal Mostafa <kamal@canonical.com>
Signed-off-by: Khalid Elmously <khalid.elmously@canonical.com>