๋„์ปค ์ปดํฌ์ฆˆ ๋ชฉ์ 

๋ชฉ์ ๊ณผ ์„œ๋น„์Šค

  • ๋ชฉ์ : 1๊ฐœ ์ด์ƒ ๋„์ปค ์ปจํ…Œ์ด๋„ˆ ์ƒ์„ฑ ์‹œ ์Šคํฌ๋ฆฝํŠธ ํ•˜๋‚˜(docker-compose.yaml)๋กœ ์ปจํ…Œ์ด๋„ˆ๋“ค ์„ค์ •์„ ๊ด€๋ฆฌํ•ด์ฃผ๋Š” Docker์˜ ๊ธฐ๋Šฅ
  • ์„œ๋น„์Šค: yaml ํŒŒ์ผ์ด ์žˆ๋Š” ์œ„์น˜์—์„œ sudo docker compose up๋ช…๋ น์–ด ์ž…๋ ฅ

๋„์ปค ์ปดํฌ์ฆˆ ํ•ด์„

1) ๊ณ„์ธต์  ๊ตฌ์กฐ

  • json, xml๊ณผ ๊ฐ™์ด key, value๋กœ ๊ตฌ์„ฑ๋˜๋ฉฐ ๊ณ„์ธก์  ๊ตฌ์กฐ

2) 1 Level

  • version : yaml ํŒŒ์ผ์˜ ๋ฒ„์ „ ์ •๋ณด
  • x-airflow-common: Extention Fields > ์ปจํ…Œ์ด๋„ˆ์— ๊ณตํ†ต ์ ์šฉ๋  ํ•ญ๋ชฉ๋“ค ์ •์˜
  • services: ์ปจํ…Œ์ด๋„ˆ ์‹คํ–‰ํ•  ์„œ๋น„์Šค ์ •์˜
  • volumes: ์ปจํ…Œ์ด๋„ˆ ํ• ๋‹นํ•  volume ์ •์˜
  • networks: ์ปจํ…Œ์ด๋„ˆ์— ์—ฐ๊ฒฐํ•  network ์ •์˜
    • ๊ธฐ๋ณธ์ ์œผ๋กœ ์ž‘์„ฑ X
    • IP ๋˜๋Š” ๋„คํŠธ์›Œํฌ ์ ์šฉ์‹œ ์ž‘์„ฑ
2-1) x-airflow-common
  • ๊ณตํ†ต ์ง€์ •ํ•  ํ•ญ๋ชฉ์„ &๋ถ™์—ฌ์„œ ์ง€์ •
  • ์ปจํ…Œ์ด๋„ˆ ์ƒ์„ฑ ์‹œ, airflow-common, airflow-common-env, airflow-common-depends-on์„ ๋ฐ›์Œ center
2-2) services
  • ์ปจํ…Œ์ด๋„ˆ์— ์˜ฌ๋ฆด ์„œ๋น„์Šค ์ง€์ •

    • ์˜ˆ) ์›น์„œ๋ฒ„, postgres ๋“ฑ
  • airflow-webserver์•  x-airflow-common์—์„œ ์„ค์ •ํ•œ image, environment, depends_on ๋“ฑ์ด ๋“ค์–ด๊ฐ

  • ์ด ๋•Œ, depends_on์ด ๋‹ค์‹œ ๋“ฑ์žฅํ•˜๋Š” ๊ฒฝ์šฐ ์œ„์—์„œ ์ž‘์„ฑ๋œ ๊ฑด ๋ฌด์‹œ๋˜๊ณ  ๋ฎ์–ด์”Œ์–ด์ง

    • ์ปจํ…Œ์ด๋„ˆ ์‹คํ–‰ ์ˆœ์„œ ์ •์˜์— ํ™œ์šฉ
    • ์œ„์—์„œ ์ง€์ •๋œ airflow_common-depends-on์—์„œ ์„ค์ •ํ•œ redis, postgres์˜ ์ปจ๋””์…˜์„ ๊ฐ€์ ธ์˜ด
    • ์ฆ‰, depends_on์—์„œ ์„ค์ •ํ•œ ์ปจํ…Œ์ด๋„ˆ(redis, postgres, codition)์„ ๋„์šด ํ›„ ๋ณธ ์ปจํ…Œ์ด๋„ˆ์ธ airflow-webserver๊ฐ€ ๋„์–ด์ง! center
  • postgres ์˜ ์ปจํ…Œ์ด๋„ˆ ์„œ๋น„์Šค

    • image: ์‚ฌ์šฉํ•  postgres ๋ฒ„์ „
    • environment: postgres์—์„œ ์‚ฌ์šฉํ•  OS ํ™˜๊ฒฝ๋ณ€์ˆ˜
    • volumes: ์ปจํ…Œ์ด๋„ˆ์™€ ์—ฐ๊ฒฐํ•  Local ํŒŒ์ผ ์‹œ์Šคํ…œ ๊ฒฝ๋กœ
      • postgres-db-volume: ๋กœ์ปฌ ํŒŒ์ผ ์‹œ์Šคํ…œ ๊ฒฝ๋กœ
        • Level 1์˜ volumes ์—์„œ ์ •์˜๋จ
      • /var/lib/postgresql/data: ์—ฐ๊ฒฐํ•  ์ปจํ…Œ์ด๋„ˆ ๊ฒฝ๋กœ
      • Why? ์ปจํ…Œ์ด๋„ˆ๋ฅผ ๋‚ด๋ ธ๋‹ค๊ฐ€ ์˜ฌ๋ฆด ๋•Œ DB์— ์ €์žฅ๋œ ํŒŒ์ผ์„ ๋กœ์ปฌ์— ์˜ฎ๊ฒจ์•ผ DB๋‚ด ๋ฐ์ดํ„ฐ๊ฐ€ ์‚ฌ๋ผ์ง€์ง€ ์•Š์Œ!
    • ports: ๋กœ์ปฌ๊ณผ docker์— ํฌํŠธ ์—ฐ๊ฒฐ ๋ฐฉ๋ฒ•
      • ์˜ˆ๋ฅผ ๋“ค์–ด, WSL์—์„œ์˜ ํฌํŠธ๋Š” 5432๊ฐ€ ์—ฌ๋Ÿฌ ๊ฐœ์ธ ๊ฒฝ์šฐ, ๋กœ์ปฌ์—์„œ๋Š” 5432, 5431 ๊ฐ๊ธฐ ์„ค์ •ํ•ด์•ผํ•จ
      • ์ฆ‰, ๋กœ์ปฌport:์ปจํ…Œ์ด๋„ˆport

  • redis
    • expose: 6379
      • ๋‚ด๋ถ€ ์ปจํ…Œ์ด๋„ˆ ๊ฐ„ ์—ฐ๋™ > ์™ธ๋ถ€์—์„œ ์—ฐ๋™ X
      • ports๋Š” ์™ธ๋ถ€์™€ ์ปจํ…Œ์ด๋„ˆ ์—ฐ๊ฒฐ

center

2-3) volumes
  • ์ปจํ…Œ์ด๋„ˆ์— ์‚ฌ์šฉ๋  ๋ณผ๋ฅจ์„ ๋ฏธ๋ฆฌ ์ •์˜
  • postgres-db-volume์ด ์ƒˆ๋กœ ๋งŒ๋“ค ๋ณผ๋ฅจ ์ด๋ฆ„
  • ๋ณผ๋ฅจ ์ •๋ณด ํ™•์ธ ๋ฐฉ๋ฒ•
    • ๋ณผ๋ฅจ ๋ฆฌ์ŠคํŠธ ํ™•์ธ; sudo docker volume ls
    • ๋ณผ๋ฅจ ์ƒ์„ธ ๋ณด๊ธฐ: sudo docker volume inspect {volume_id}
2-4) networks
  • ์ปจํ…Œ์ด๋„ˆ ๋„คํŠธ์›Œํฌ ์ •๋ณด ๊ตฌ์„ฑ
  • ์ค‘์š”๋ณ€์ˆ˜
    • subnet:
    • gateway:
  • ๋„คํŠธ์›Œํฌ ์ •๋ณด ํ™•์ธ ๋ฐฉ๋ฒ•
    • ๋„คํŠธ์›Œํฌ ๋ฆฌ์ŠคํŠธ: sudo docker network ls
    • ๋„คํŠธ์›Œํฌ ์ƒ์„ธ ๋ณด๊ธฐ: sudo docker network inspect {network_id}