In case the controller is not initialized, we shall not de-initialize it.
As the control structure will not be filled, we will produce a null ptr
dereference if the controller is not inited.

Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Simon Glass <sjg@chromium.org>