usb: Prevent dead ports when xhci is not enabled