The road ahead: What’s coming in Airflow 3 and beyond? - Airflow Summit 2024

Presented by Vikram Koka at Airflow Summit 2024
This talk introduces the vision behind Airflow 3, including the emerging technology trends in the industry and how Airflow will evolve in response. It includes an overview of the architectural changes in Airflow to support emerging use cases and distributed data infrastructure models. This talk also introduces the major features and the desired outcomes of the release. Airflow 3 will be a foundational release and therefore this talk will similarly introduce the new concepts being introduced as part of Airflow 3, which may be fully realized in follow-on 3.x releases.
The goal of this talk is to raise awareness about Airflow 3 and to get feedback from the Airflow community while the release is still in the development phase.
----
(GenAI summary ahead)
*Key takeaways:*
*Airflow 3 addresses top user requests:* The presentation highlights how Airflow 3 incorporates feedback from the annual Airflow survey, focusing on user requests like DAG versioning, UI improvements, and enhanced security.
*Task isolation and security:* A major architectural change in Airflow 3 introduces a Task Execution Interface to isolate tasks from the Airflow metadatabase, addressing security concerns and enabling independent upgrades for server and task components.
*Run Anywhere, Anytime, in Any Language:* This is the overarching theme of Airflow 3, with features enabling multi-cloud deployments, event-driven scheduling, data partitioning, ad hoc scheduling, and multilingual support.
*Multi-language support:* Airflow 3 introduces a language-agnostic Task Execution Interface, allowing for the development of task SDKs in various languages like Python, Go, and potentially TypeScript and Kotlin. This allows users to build pipelines incorporating tasks written in different languages, eliminating language lock-in.
*Community involvement is crucial:* The presentation emphasizes the importance of community involvement in Airflow's success, encouraging users to contribute by testing beta versions, adding support for new languages, and sharing their experiences with Airflow 3 deployments at future events.
*Timeline:*
Beta releases are expected in January.
Release candidates are planned for the end of March.
General availability will follow after the Apache voting process.
*Call to action:*
Vikram urges the community to get involved by:
Trying out Airflow 3 betas, especially the "Run at Any Time" features.
Contributing to the development of task SDKs for additional languages.
Deploying Airflow 3 in production and sharing their experiences.
*Overall, Airflow 3 represents a significant step forward for the platform, promising greater flexibility, security, and ease of use. The emphasis on community involvement underscores the collaborative spirit driving Airflow's continued evolution.*