infiniband: addr: Consolidate code to fetch neighbour hardware address from dst.