[Airflow] Celery executor, Flower

[Airflow] Celery executor, Flower

https://heywantodo.tistory.com/68

[Airflow] 멀티 클러스터 환경 구성 (Celery Executer)

이전 포스팅에서 Docker composer를 통해 Celery executor를 구성하는 방법을 다뤄봤다.

이번엔 기존에 설치된 Airflow 서버에 Celery executor를 구성하는 방법을 다뤄보겠다

 

😎 master node

필수 요소

• Redis (RabbitQ, queue 도 가능)
• MySQL/PostgreSQL

 

1. MySQL & Redis 패키지 설치

sudo apt-get install python3-dev gcc libmysqlclient-dev -y
pip3 install mysql-connector-python
pip3 install mysqlclient
pip3 install redis

2. Celery, Flower 설치

pip3 install celery
pip3 install flower

3. config 파일 수정

vi airflow/airflow.cfg

# full import path to the class when using a custom executor.
executor = CeleryExecutor

# The SqlAlchemy connection string to the metadata database.
sql_alchemy_conn = mysql+mysqldb://<user>:<passwd>@<host>:3306/<db>

# The Celery result_backend. When a job finishes, it needs to update the
result_backend = db+mysql+mysqlconnector://<user>:<passwd>@<host>:3306/<db>

# a sqlalchemy database. Refer to the Celery documentation for more information.
broker_url = redis://<host>:6379/0

4. Airflow DB 초기화 & Celery/Flower 실행

airflow db init
airflow shceduler -D
airflow webserver -D
airflow celery -D
airflow celery flower -D

ip:5555로 접속하여 Flower web확인

 

👩‍💻 Worker node

1. airflow 설치

[Airflow] Airflow 설치

2. MySQL & Redis 패키지 설치

sudo apt-get install python3-dev gcc libmysqlclient-dev -y
pip3 install mysql-connector-python
pip3 install mysqlclient
pip3 install redis

3. Celery 설치

pip3 install celery

4. config 파일 수정

vi airflow/airflow.cfg

# full import path to the class when using a custom executor.
executor = CeleryExecutor

# The SqlAlchemy connection string to the metadata database.
sql_alchemy_conn = mysql+mysqldb://<user>:<passwd>@<host>:3306/<db>

# The Celery result_backend. When a job finishes, it needs to update the
result_backend = db+mysql+mysqlconnector://<user>:<passwd>@<host>:3306/<db>

# a sqlalchemy database. Refer to the Celery documentation for more information.
broker_url = redis://<host>:6379/0

5. Worker 실행

airflow celery worker -q queue_1 -D

아래처럼 뜬다면 성공

flower를 재시작하면 웹에서도 확인이 가능하다

 

참조

https://mightytedkim.tistory.com/32