I have the exact same issue even with ipad pro. Only when ireal pro switches to background audio latency becomes normal. There should be a a button to switch it to manually I suppose...