Eventually delete the CA paths lock files

Mark the lockfiles as having to eventually be deleted so that they don’t
stay laying around in the store at the end of the build

Fix #4936