ARM: uniphier: delete unneeded input enable for low-level debug

The UART I/O ports for PH1-Pro4 has no input enable controlling.
This code is useless.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>