Online vs. Batch Inference: Request-by-Request vs. Scheduled Bulk Scoring
The problem, from first principles
A trained model has to serve real requests eventually, but "serving" isn't one single pattern. Some applications need a prediction the instant a request arrives (a fraud check on a live transaction); others just need millions of predictions computed sometime before tomorrow morning (scoring every customer for a weekly churn report). These two patterns trade off in a genuinely quantifiable way: fixed per-call overhead versus how long any individual request has to wait.
From theory to code
Implement online_inference and batch_inference (the two calling patterns themselves), plus online_total_overhead, batch_total_overhead, and worst_case_batch_wait_time, quantifying the real tradeoff between them.
Constraints
online_inference(model_fn, request)calls the model on a batch of size 1 and returns that single result.batch_inference(model_fn, requests)calls the model once on the whole list.online_total_overhead(num_requests, overhead_per_call)returnsnum_requests * overhead_per_call.batch_total_overhead(num_requests, overhead_per_call, batch_size)returnsceil(num_requests / batch_size) * overhead_per_call.worst_case_batch_wait_time(batch_size, per_request_arrival_interval)returns(batch_size - 1) * per_request_arrival_interval.
Hints
Open one at a time. Each gives away a little more than the last.
Hint 1
Every model call, online or batched, pays some fixed overhead regardless of how many items are in the batch (framework dispatch cost, a network round trip, a GPU kernel launch). Fewer, bigger calls means that fixed cost gets paid fewer times.
Hint 2
worst_case_batch_wait_time is about the FIRST request into an otherwise-empty batch: it has to sit and wait for batch_size - 1 MORE requests to show up (each taking roughly per_request_arrival_interval to arrive) before the whole batch actually gets processed.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.