arm: Remove include files from common.h

With a small tweak we can avoid including these files for all boards.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>