Dynamic Windowed Batching of Inference Requests
Intermediate
The problem, from first principles
Running requests through a model one at a time under-uses the hardware -- a GPU can often process a batch of several requests almost as fast as one. But waiting to fill a large batch before running anything trades throughput for latency: the first request in the batch sits idle until enough others arrive to justify running it.
The task
Write batch_requests(requests, max_batch_size, max_wait_ms). Group arrival-ordered (arrival_time_ms, request_id) pairs into batches: keep adding to the current batch as long as it isn't at max_batch_size and the next request arrived within max_wait_ms of the batch's first request; otherwise close the batch and start a new one.
Keywords
production-systemsinference-optimization
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.