Airflow DAG: create dag with bash operator!

Поділитися
Вставка
  • Опубліковано 7 жов 2024

КОМЕНТАРІ • 70

  • @JoyantoGhoshJG
    @JoyantoGhoshJG 3 роки тому +8

    I am following your instructions and have been able to understand Airflow at an intermediate level.
    I would love to see your writing DAGs for PythonOperator with Xcoms. Eagerly waiting for it!

    • @coder2j
      @coder2j  3 роки тому +3

      Nice to hear that. Python operator is already planned. Stay tuned! 😉

    • @Akashkumar-ei5tm
      @Akashkumar-ei5tm 3 роки тому +1

      Same here 😊

  • @helovesdata8483
    @helovesdata8483 2 роки тому +1

    I really enjoyed this video. I learned more today than the past two weeks with another youtube video. Also I reinstalled Airflow with your instructions from another video and it has never been this stable. Thank you!

    • @coder2j
      @coder2j  2 роки тому

      Glad it was helpful!

  • @khalidsultani1896
    @khalidsultani1896 2 роки тому

    Thanks for posting far easy and clearly explained then paid material . Please keep postings if possible on airflow integeration with aws services such as red shift and databricks .

    • @coder2j
      @coder2j  2 роки тому

      Thanks for the feedback ☺️

  • @brunodupont8494
    @brunodupont8494 Рік тому

    Amazing videos, very simply explained

  • @ぺんぎん太郎-p5i
    @ぺんぎん太郎-p5i 2 роки тому

    this is the best video ever.

    • @coder2j
      @coder2j  2 роки тому

      Thanks for your nice comment.

  • @manimaran.v3099
    @manimaran.v3099 3 роки тому

    Nice AirFlow video ever in you Tube, heartfelt Thanks

    • @coder2j
      @coder2j  3 роки тому

      You are welcome!

  • @SudhirKumar-rl4wt
    @SudhirKumar-rl4wt Рік тому

    Thankyou for this this tutorial.

  • @looklook6075
    @looklook6075 9 місяців тому

    OMG, you are my hero!

    • @coder2j
      @coder2j  9 місяців тому

      🤗🤗

  • @caiofelixreis8970
    @caiofelixreis8970 2 роки тому

    Awsomee!!! Thank you for all this broo, its really helpful

    • @coder2j
      @coder2j  2 роки тому

      Thanks. Have fun learning.

  • @Akashkumar-ei5tm
    @Akashkumar-ei5tm 3 роки тому +3

    Nicee .. please create one for passing one variable from 1st function to another function using these python operators

    • @coder2j
      @coder2j  3 роки тому +2

      Thanks for the feedback. Will include that in the coming episode. Subscribe and Stay tuned! 😉

    • @Akashkumar-ei5tm
      @Akashkumar-ei5tm 3 роки тому +2

      Always man 🙂.. would like to see you covering more stuff's like kafta flume spark and all

  • @simonr-i8i
    @simonr-i8i 2 місяці тому

    Nice content thanks a lot

  • @francosterrantino5208
    @francosterrantino5208 10 місяців тому

    Thanks man.

  • @VitaliyZlobin
    @VitaliyZlobin 9 місяців тому

    Hello) Cool videos! Thank you!
    Why is the old version of DAG not rebooting, but a new one is created?
    And how to remove old versions, leaving only the latter?

    • @coder2j
      @coder2j  9 місяців тому +1

      Because a new dag ID is created. You can simply delete the old dag on the UI.

  • @bimerap5735
    @bimerap5735 3 роки тому +3

    Please include how we can install more python package?

    • @coder2j
      @coder2j  3 роки тому +2

      Thanks for your great feedback. We, as data engineers/scientists need a lot of python packages. It definitely deserves an episode! Stay tuned! 😉

    • @leamon9024
      @leamon9024 3 роки тому

      @@coder2j Also interested in this topic. XD Maybe dockerizing an application with all its dependencies, then run it using docker operator?

    • @coder2j
      @coder2j  3 роки тому

      @@leamon9024 Will cover it in the future episode! Stay tuned! 😉

  • @thomasripoll3438
    @thomasripoll3438 2 роки тому

    Thx

  • @fiannafailgalway8446
    @fiannafailgalway8446 3 роки тому

    More please

  • @afolakebaiyewu476
    @afolakebaiyewu476 2 роки тому

    Please include branchpyhtonoperator

  • @simre5476
    @simre5476 3 роки тому

    Good stuff! Thanks for the tuto! I am running the dag which is status success but the task 1, 2, and 3 are blanks and if I select one task to run it independently, it tells me ''Only works with the Celery or Kubernetes executors, sorry'' any clues how to fix this? I should add some settings in the .yaml file?

    • @coder2j
      @coder2j  3 роки тому

      If you want to rerun the dag or one of the task, you need to click the dag or task run, then click the clear button. Wait a second, the scheduler will schedule it and run it again. Check out the airflow tutorial part6, in which I also dodi the same.

  • @dhanasekar7068
    @dhanasekar7068 3 роки тому

    Hi, really good videos. Shall I know how to dm you. I want to ask a doubt regarding sql sensor and hive operator

    • @coder2j
      @coder2j  3 роки тому

      Please leave it directly in the comment section. So that we can share the info in case others may also have the same question.

  • @Akshay-dn7ni
    @Akshay-dn7ni 3 роки тому +1

    i am getting error as- Import "airflow" could not be resolved. possible solutions plz

    • @coder2j
      @coder2j  3 роки тому

      Did you get this error from the airflow webserver or from local python lint? And are you running airflow locally or in docker?

    • @Akshay-dn7ni
      @Akshay-dn7ni 3 роки тому

      @@coder2j I get this error in vscode whenever I write " from airflow import DAG". I am running using docker. I think it is not able to find airflow module which is installed in docker.

    • @coder2j
      @coder2j  3 роки тому

      @Akshay That's normal. Because the airflow package is installed in the docker container. The vs code will complain it since there is no airflow package installed. One tip to bypass this error is to create a local python environment, install airflow in that and tell the vs code the path of the python environment you created. But you can ignore this as long as you can launch airflow webserver successfully!

  • @follygee4667
    @follygee4667 2 роки тому

    how do i import a json config file that is dependent on another python script in airflow

  • @universe2dimension
    @universe2dimension 2 роки тому

    大哥,用visual studio编辑DAG,然后在airflow portal refresh,我这里创建不了新的DAG,而且,也删除不了原有的31个DAG。怎么解决呢?我用的mac

  • @akmalhafiz7830
    @akmalhafiz7830 Рік тому

    when I try to import airflow, it says that "Import "airflow" could not be resolved". I looked up over the internet and it seems like the problem is my local python interpreter doesnt have airflow installed. How can I use python interpretor from the container instead in visual studio code so that I wont get this error?

    • @coder2j
      @coder2j  Рік тому

      You can install vscode remote Container Plugin and select python interpreter within the container.

  • @pythonmathui3057
    @pythonmathui3057 Рік тому

    Sir the airflow webserver has already running but when i tried to import DAG from airflow it says no module named airflow, should i just pip install it? or did i miss something that make the airflow module not automatically installed?

    • @coder2j
      @coder2j  Рік тому

      Is the error coming from the IDE lint or runtime?

    • @pythonmathui3057
      @pythonmathui3057 Рік тому

      @@coder2j ide, but i alreadY just pip install apache-airflow in the terminal

  • @Uxdkkkhcddvbh
    @Uxdkkkhcddvbh 3 роки тому

    Hi this has been really helpful. However the error in the logfiles I get when I run my dag is that says it doesn't have access to my python file - INFO - python: can't open file '. I am using the Bash Operator and the bash command is 'python fulldirectorypath/myscript.py' how can I fix this. I have tried all version of the writing the full path. How do I give airflow permission?

    • @coder2j
      @coder2j  3 роки тому

      If you want to execute python functions, it is recommended to use PythonOperator. I covered that in the Airflow Tutorial Part6.

  • @hayathbasha4519
    @hayathbasha4519 3 роки тому

    Hi is it possible to get task/dag duration from airflow context

    • @coder2j
      @coder2j  3 роки тому

      Yes, you can use the on_success_callback in operators. Check the documentation here: bit.ly/3uxpxER

  • @trantheluc3580
    @trantheluc3580 2 роки тому

    i use BashOperator run bash_command='date > ./date.txt' but when i run task i cannot find 'date.txt' in folder. so where is it?

    • @coder2j
      @coder2j  2 роки тому +1

      You can save it under dags folder: date > dags/date.txt

    • @trantheluc3580
      @trantheluc3580 2 роки тому

      how can i find where it's saved when i run task in webserver?

    • @coder2j
      @coder2j  2 роки тому

      It should be under your dags folder.

    • @trantheluc3580
      @trantheluc3580 2 роки тому

      @@coder2j i tried to run task, it was successfull but i can find this file under dags folder

  • @vilw4739
    @vilw4739 Рік тому

    Hi, am using bash operator and passing multiple commands inside bash_command.Its a mixture of if else loops and commands.I don't know how to correctly incorporate it.l am using" to separate it. Could you guide me here if you have worked with something similar please thanks in advance

    • @coder2j
      @coder2j  Рік тому

      Can you try wrapping the multiple lines command into 3 double quotes? Like """
      """

    • @vilw4739
      @vilw4739 Рік тому

      @@coder2j yes but I have nested if loops and inside one loop I'm executing a s3a command.So my qn is do I put all of these ifs and else's in one ''' ''' or after each if else?also do I need to use + with '''? Like- ''' echo''' + '''if else'''

    • @coder2j
      @coder2j  Рік тому +1

      Hi, you can save your bash commands in a .sh file under the dags folder, then just put the relative path of .sh file as the value of the bash_command parameter.

    • @vilw4739
      @vilw4739 Рік тому

      @@coder2j if I have to pass arguments which has macro values can I do so by-
      Date=some macros #global variable
      Bash_command='''
      "File.sh Date "
      '''

    • @coder2j
      @coder2j  Рік тому +1

      You can use Jinja template to insert parameters. airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html
      bash_task = BashOperator(
      task_id="bash_task",
      bash_command="echo \"here is the message: '$message'\"",
      env={"message": '{{ dag_run.conf["message"] if dag_run else "" }}'},
      )