[Airflow] Airflow 설치

Airflow 설치

🔎 설치환경  AWS Amazonlinux 2

 

2.5.1 버전 Airflow 설치 

공식 Documents 참조

https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

Installation from PyPI — Airflow Documentation

 

 

1. 가상환경 구성 (miniconda)

공식 사이트에서 자신의 OS 버전에 맞는 파일을 wget으로 받아옴

https://docs.conda.io/en/latest/miniconda.html

Miniconda — conda documentation

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh

./Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc

conda config --set auto_activate_base false
conda deactivate

conda create -n airflow python=3.7.16
conda activate airflow

 

2. Airflow 설치 환경 설정 & 설치 확인

export AIRFLOW_HOME=~/airflow

AIRFLOW_VERSION=2.5.1
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

airflow version

 

❗ airflow version에서 오류가 발생하는 경우

Amazon linux 같은 경우엔 sqlite 버전 업그레이드 필요

https://airflow.apache.org/docs/apache-airflow/2.3.2/howto/set-up-database.html#setting-up-a-sqlite-database

Set up a Database Backend — Airflow Documentation

sudo yum -y install wget tar gzip gcc make expect

wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
cd sqlite/

export CFLAGS="-DSQLITE_ENABLE_FTS3 \
    -DSQLITE_ENABLE_FTS3_PARENTHESIS \
    -DSQLITE_ENABLE_FTS4 \
    -DSQLITE_ENABLE_FTS5 \
    -DSQLITE_ENABLE_JSON1 \
    -DSQLITE_ENABLE_LOAD_EXTENSION \
    -DSQLITE_ENABLE_RTREE \
    -DSQLITE_ENABLE_STAT4 \
    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
    -DSQLITE_SOUNDEX \
    -DSQLITE_TEMP_STORE=3 \
    -DSQLITE_USE_URI \
    -O2 \
    -fPIC"
    
export PREFIX="/usr/local"
LIBS="-lm" ./configure --disable-tcl --enable-shared --enable-tempstore=always --prefix="$PREFIX"

make
sudo make install

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH