Note: TorchScript and ONNX Export, Why Production Serving Does Not Run Eager Python

The problem, from first principles

05-acceleration showed a Python for loop pays real, fixed overhead per iteration, purely for being interpreted. Eager-mode PyTorch has an analogous cost at a different level: every single tensor operation call pays its own fixed Python-and-framework "dispatch" overhead — deciding which underlying kernel to actually run, checking types and devices — completely separate from the actual math that operation performs. A model with hundreds of layers pays that dispatch tax hundreds of separate times, every single forward pass.

From theory to code

Implement eager_python_dispatch_overhead(num_ops, per_op_dispatch_seconds), graph_mode_dispatch_overhead(per_op_dispatch_seconds), and graph_mode_speedup_estimate(...) — quantifying exactly the dispatch-overhead savings that exporting a model to TorchScript or ONNX (running as a compiled graph, not live Python) provides.

Constraints

• eager_python_dispatch_overhead returns num_ops * per_op_dispatch_seconds — every op pays the dispatch cost separately.
• graph_mode_dispatch_overhead returns per_op_dispatch_seconds directly — paid exactly once, regardless of how many ops the underlying graph actually contains.
• graph_mode_speedup_estimate returns eager_python_dispatch_overhead / graph_mode_dispatch_overhead.

Hints

Open one at a time. Each gives away a little more than the last.