Hello everybody this is Koda2J.
In this tutorial series we are going to learn Apache air flow.
So what is Apache air flow?
It is an open-sourced tool which we can create schedule and monitor many kinds of work flow.
It is the perfect tool to use when we have a workflow with Taskers 123 which should be run periodically in specific order.
Airflow is based on Python.
If you already have some coding experience with Python, you are good to go.
First of all, let's create a Python project and in-store airflow.
I'm going to open Visual Studio Code to create a project folder and open it.
And I will name it airflow tutorial.
on the desktop directory.
Let's open a terminal and check whether I am in the right directory or not by command PWD.
Since airflow 2.1 requires a Python version above 3.6.
Let's check my Python 3 version by command Python 3 minus minus version.
As we can see, my local Python 3 version is 3.6.7 and meets the requirements.
You can also use version 3.7 or 3.8.
Let's create a Python environment with command Python 3 minus m v inf with environment folder named py underscore inf Once created successfully,
we can see the py underscore info in the explore section.
Now let's active the Python environment, use command source py underscore info bin active.
When activated, we can see the py, underscore inf on the very left of the terminal prompts.
Next step, I'm going to install airflow locally.
Let's open browser and search Apache airflow official GitHub repository.
and navigate to the in-store section.
I will copy the in-store command and paste it in the terminal before I the-store section.
I have to change the dependency to many match my local Python version.
Instead of 3.7, we change to 3.6.
There is an error of missing GCC.
To solve this problem, I have to run command Xcode minus select in store to in store Mac OS command line tools.
The installation takes a of minutes.
After we execute the Pips in-store command again.
This time, there's no error.
Therefore, Airflow is in-store successfully.
Okay, next step, I'm going to initialize the database for Airflow.
Before that, we have to indicate the Airflow home directory.
By default, it will create a folder Airflow in Home Directory.
However, I would like to have everything in my project directory, so I export the Airflow Home Environment variable to my current directory.
Then I will initialize the database with command airflow DB init.
This command will create an SQLite database, a log folder, and some configuration files.
Next, I will start airflow web server by command airflow web server minus P8080 by default 8080 is the port to be used.
If you want, you can also use other port for example 8088, 8082 as you wish.
Then let's open the link in the browser.
It requires me username and password to login.
Okay, let's go back into the terminal, stop the airflow web server and create one user
by comment, airflow users create and give value to parameters as shown in the help output.
By setting the power password, we start the air flow web server again and we are ready to sign in.
I can see all the example decks, but over that it says there's no schedule running.
Okay, in order to execute the decks we have to start the air flow scheduler.
Let's go back to VS Code, open another terminal.
First, we have to make sure we export the airflow home environment variable as we did in the first terminal.
Then we execute the command airflow sketch.
Now, let's go back to browser and refresh the page.
We can see, boom, the message is gone.
It means the scheduler is running properly.
Let's turn on one example deck by clicking the button before the deck.
From the tree view we can see this deck has multiple tasks.
When we click the refresh button on the right hand side.
We can see the tasks have been scheduled and executed.
And it has been marked as dark green once it has been run successfully.
