U-boot is unable to actually use that memory and it can
cause problems with relocation if it tries to.

Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>