Supplier Requests Take Turns, and None Are Left Out
The more supplier connections you have, the more you need someone keeping order at the door.
A streaming hotel search sends each batch of hotels to every supplier connection at once. Each batch now sends at most a fixed number of supplier requests at a time and queues the rest; every supplier still gets asked.
A buyer with many supplier connections searches a busy city. The system splits the hotels into batches, sends each batch to every supplier connection the buyer can use, and streams results to the page as they arrive, first come, first shown. More connections and bigger batches mean more requests in flight at the same moment.
The original code sent one parallel request per connection, all fired together. With a few connections the difference is invisible. With many, a long line of requests leaves at the same moment, and nothing limited how many supplier requests one batch could have in flight; that depended only on how many connections the buyer had.
Each batch has a ceiling of eight supplier requests in flight, with the rest waiting in line and a new one starting as soon as one returns. The suppliers asked are the same as before, and so are the timeouts. Queuing can make one search slightly slower, but every supplier still gets asked.
Nothing changes on screen. Results still stream in batch by batch, and the diagnostic record of each supplier call is still there to inspect. What changes is that each batch of a search has a ceiling on supplier requests in flight: however many connections a buyer has, a batch sends at most eight at once and the rest follow in order.