Hi can I get confirmation? Around 11:35 or so you seem to say that only when the DAG is triggered and begins to run, the task mapping occurs? So I cannot retrieve rows in previous tasks and trigger a dynamic task using that information? It has to be available at runtime? So I should create a Orchestrator DAG that would trigger and input these rows of data into the child DAG to then have immediately available for my Dynamic Tasks?
No you can retrieve the rows from a previous task! You would just pass that previous tasks output as a mapping param to the .expand() method and it will dynamically map those at runtime! Definitely do not create an Orchestrator DAG, just set the relationship as shown in these docs: docs.astronomer.io/learn/dynamic-tasks and you should be able to accomplish it within that DAG.
It should work. Generally you can map over any type of operator, but not that some parameters can't be mapped over (e.g. BaseOperator params). More here: docs.astronomer.io/learn/dynamic-tasks
Hello,This is very well explained. Thank you! The map index under the mapped tasks are currently numbered from 0. Would it be possible to update the map index to something meaningful ?
Hmmmm good question, since the Airflow UI can be customized, I imagine you could, but would need to add a filter on top of the task instance to read its properties and then render one of them as the index name/#, but might be more trouble than it's worth!
Each Dynamically generated task instance will have its own separate log that you can view in isolation after the dag has finished, just click on the main task and select the task instance you'd like to interact with!
Totally, the process is pretty much the same for every operator where you'll just add the .expand parameter with any fields you want to be used dynamically
It depends which executor you are using. If you are testing on a local instance, the default executor is SequentialExecutor which only executes one task instance at a time as opposed to other executors which will allow parallelism.
Great guide, very well explained, thank you!
Thank you for the kind words!
Hi can I get confirmation? Around 11:35 or so you seem to say that only when the DAG is triggered and begins to run, the task mapping occurs? So I cannot retrieve rows in previous tasks and trigger a dynamic task using that information? It has to be available at runtime?
So I should create a Orchestrator DAG that would trigger and input these rows of data into the child DAG to then have immediately available for my Dynamic Tasks?
No you can retrieve the rows from a previous task! You would just pass that previous tasks output as a mapping param to the .expand() method and it will dynamically map those at runtime! Definitely do not create an Orchestrator DAG, just set the relationship as shown in these docs: docs.astronomer.io/learn/dynamic-tasks and you should be able to accomplish it within that DAG.
great video. I'm trying to make this work with LivyOperator do you know if it can be expanded or partial arguments supplied to it?
It should work. Generally you can map over any type of operator, but not that some parameters can't be mapped over (e.g. BaseOperator params). More here: docs.astronomer.io/learn/dynamic-tasks
Hello,This is very well explained. Thank you! The map index under the mapped tasks are currently numbered from 0. Would it be possible to update the map index to something meaningful ?
Hmmmm good question, since the Airflow UI can be customized, I imagine you could, but would need to add a filter on top of the task instance to read its properties and then render one of them as the index name/#, but might be more trouble than it's worth!
When you perform the dynamic tasks that could scale in parallel based on your input dataset, what happens to the task log after the dag is finished?
Each Dynamically generated task instance will have its own separate log that you can view in isolation after the dag has finished, just click on the main task and select the task instance you'd like to interact with!
Thanks for very helpful video. could I get any other guide to apply dynamic tasks on other operators like athenaoperator?
Totally, the process is pretty much the same for every operator where you'll just add the .expand parameter with any fields you want to be used dynamically
Awesome guide, thanks!
No problem, glad you enjoyed!
Really helpful thank you😍
is the presentation shared anywhere?
All of our past webinars are here: www.astronomer.io/events/webinars/
Expand Operator will execute sequentially or parallel?
It depends which executor you are using. If you are testing on a local instance, the default executor is SequentialExecutor which only executes one task instance at a time as opposed to other executors which will allow parallelism.
Great video. Very helpful.
Glad it was helpful!