Fulandi Hudza Grahitama
← projects

Weather Data Pipeline: OpenWeatherMap to BigQuery

Data Engineer (solo project) · [duration — the CV only shows Nov 2026; no start date yet] · 2026

An Airflow pipeline that loads weather data for 5 Indonesian cities into BigQuery every 6 hours, with dbt models and a data quality audit.

Pipeline flow

TL;DR

  • Automated batch pipeline every 6 hours from OpenWeatherMap to BigQuery.
  • dbt transformations (staging → dimension → partitioned fact) with 24 tests passing.
  • Data quality audit (freshness, completeness, range) scored 0–100 and logged to BigQuery; critical failures stop the pipeline.

Metrics

  • 24dbt tests passing
  • 5Indonesian cities monitored
  • 6 hrdata load interval (Airflow DAG)

Stack

  • Python
  • Airflow
  • dbt
  • BigQuery
  • Docker

Problem

[situation before the project / motivation — not in the CV, fill in yourself]

Architecture & technical decisions

The Airflow DAG runs every 6 hours; transformations use dbt (staging → dimension → partitioned fact); the data quality audit scores each run 0–100 and stops the pipeline on critical failures. [add why these tools were chosen + 2–3 trade-offs]

Data model

dbt models: staging, dimension, and partitioned fact. [short explanation + store the ERD at public/images/projects/weather-data-pipeline-erd.png]

Data quality & monitoring

Freshness, completeness, and range checks, scored 0–100 and logged to BigQuery; critical failures stop the pipeline. Of 16 runs, 2 failed and were traced to local network loss.

Code snippet

-- [5–20 lines of the most representative SQL/Python — pick from the repo]

[why this code is interesting]

Challenge

Traced the 2 failed runs (of 16) and confirmed the cause was local network loss, not a pipeline bug.

Result

The pipeline runs automatically every 6 hours with 24 passing dbt tests and a data quality audit that stops the pipeline on critical failures.

What I learned

[1–2 lessons — fill in yourself]