phys_addr_t is designed for physical addresses that's why
use it.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>