Clear the flag immediately before cache operation to not depend on
the previous state.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>