Slow shipping rating and buying via API

Anyone else have issues with slow shipping API responses? We ship in-motion and 4000ms to rate + buy is too long and causes rejection on our shipping line. We need a response within 3500ms.

I've tried filtering the requests, tried the new shipping/getRates vs. merchantfulfillment/getEligibleShipmentServices, and opening a support case all to no avail.

One thing that stands out to me is the HUGE response we get back with a slew of carriers that we don't have enabled. It seems like that would be the first place to start to increase performance. Support says 'This API is working as designed and we will not be able to make any adjustment.' Yikes.

We had moved away from Buy Shipping for quite a while, but moved back recently for OTDR protection.