Modern Orchestrators: Dagster & Prefect

Get hands-on: step by step

1. Choose one observable output

   Pick a pipeline that transforms a small file or table and name its final output. Write down what a correct output looks like and where a teammate should be able to inspect it. Keep the first exercise local; it should teach the framework model without introducing cloud credentials or production data.

2. Scaffold a Dagster project from its quickstart

   Follow Dagster's current quickstart for prerequisites and project creation. Its documented command is shown below. Keep the generated project separate from any production repository while learning, then add a small fixture and define one asset that turns it into a second file.

   uvx create-dagster@latest project dagster-quickstart

3. Inspect asset definitions and dependencies

   Use the quickstart's asset example as a reference, then list the project definitions with the documented CLI command. Find the asset in the output and trace its input and output names back to the code. If a pipeline has several outputs, sketch their dependencies before adding more logic.

   dg list defs

4. Run and verify the Dagster result

   Start the local development interface as described by the quickstart, materialize the asset, and inspect its run. Verify the resulting file directly as the guide demonstrates. A readable output file gives you a simple way to separate orchestration behavior from assumptions hidden in a dashboard.

   dg dev

5. Express the same work as a Prefect flow

   In a separate scratch project with Prefect installed according to its official setup guide, wrap the same small transformation in a flow. Prefect documents calling the decorated function like an ordinary Python function; begin there, then decide which functions deserve task boundaries. Avoid adding remote workers until you have a clear deployment requirement.

   from prefect import flow
   
   @flow
   def transform_fixture():
       return "inspect this result"
   
   transform_fixture()

6. Compare the run records and write a decision note

   Run both local examples with the same fixture. Compare how each represents dependencies, outputs, retries, logs, and the work required to operate a scheduled deployment. Record one reason to keep the existing tool and one requirement that could justify a migration. Link the note to a real pipeline so the comparison stays grounded in team work.

7. Review the migration boundary with the operators

   Ask the people who support the current scheduler to review the proposed pipeline and identify the operational work a prototype left out. Discuss deployment packaging, secrets, permissions, schedules, alert routing, backfills, and ownership during an incident. Confirm which concerns belong to the orchestrator and which belong to the surrounding platform. Update the decision note with concrete examples from an existing workflow, a migration sequence that can be reversed, and a way to compare reliability after rollout. A framework demo is evidence about authoring and local execution; the operating plan is separate evidence about whether the team can run it. Keep both in view before proposing a production migration. Repeat the review after the first production workflow has run and revise the assumptions using actual run history and support work.