Prefect

Prefect is the dataflow automation platform: it orchestrates, observes, and secures your mission-critical workflows.

Prefect is the intelligent infrastructure for data, ML, and platform engineering teams: it powers over 100K mission-critical workflows every minute. The platform enables engineers to build and deploy complex pipelines with Python, using decorators instead of rigid DAGs. Key features like Durable Execution allow workflows to automatically resume from failure, eliminating expensive re-runs. Its native observability provides a clear control plane for debugging, replacing flat logs with a visual map of task failures. With a community of 25,000+ engineers, Prefect delivers the reliability and scale needed for production environments, often resulting in significant cost reductions (e.g., 73.78% infrastructure savings cited by one user).

https://www.prefect.io