Training Pipelines: Turning a Notebook Into a Reproducible, Scheduled DAG

The problem, from first principles

A notebook's cells run in whatever order you happen to click "run": every dependency between steps (train can't run before featurize, which can't run before clean_data) exists only implicitly, in the author's head. The moment that pipeline needs to run unattended, on a schedule, reliably, every one of those implicit dependencies has to become an EXPLICIT graph: a DAG (Directed Acyclic Graph), where a scheduler can compute a valid execution order automatically, and detect, before ever running anything, if the dependencies as written don't actually make sense.

From theory to code

Implement add_task (registering a task and its explicit dependencies), topological_order (computing a valid execution order via Kahn's algorithm), and has_cycle.

Constraints

• add_task(dag, task_name, depends_on) records which tasks must run BEFORE the given task.
• topological_order(dag) returns a list where every task appears AFTER all of its dependencies; raises ValueError if the DAG contains a cycle.
• Ties (multiple tasks simultaneously ready to run) are broken alphabetically, for a fully deterministic result.
• has_cycle(dag) returns True exactly when topological_order would raise.

Hints

Open one at a time. Each gives away a little more than the last.