Weather Data Pipeline: OpenWeatherMap to BigQuery
Data Engineer (solo project) · [duration — the CV only shows Nov 2026; no start date yet] · 2026
An Airflow pipeline that loads weather data for 5 Indonesian cities into BigQuery every 6 hours, with dbt models and a data quality audit.
Pipeline flow
Source
OpenWeatherMap API
weather data for 5 Indonesian cities
Ingest
Airflow
DAG scheduled every 6 hours
Transform
dbt
staging, dimension, and partitioned fact models (24 tests passing)
Storage
BigQuery
partitioned fact table
TL;DR
- Automated batch pipeline every 6 hours from OpenWeatherMap to BigQuery.
- dbt transformations (staging → dimension → partitioned fact) with 24 tests passing.
- Data quality audit (freshness, completeness, range) scored 0–100 and logged to BigQuery; critical failures stop the pipeline.
Metrics
- 24dbt tests passing
- 5Indonesian cities monitored
- 6 hrdata load interval (Airflow DAG)
Stack
- Python
- Airflow
- dbt
- BigQuery
- Docker
Problem
[situation before the project / motivation — not in the CV, fill in yourself]
Architecture & technical decisions
The Airflow DAG runs every 6 hours; transformations use dbt (staging → dimension → partitioned fact); the data quality audit scores each run 0–100 and stops the pipeline on critical failures. [add why these tools were chosen + 2–3 trade-offs]
Data model
dbt models: staging, dimension, and partitioned fact. [short explanation + store the
ERD at public/images/projects/weather-data-pipeline-erd.png]
Data quality & monitoring
Freshness, completeness, and range checks, scored 0–100 and logged to BigQuery; critical failures stop the pipeline. Of 16 runs, 2 failed and were traced to local network loss.
Code snippet
-- [5–20 lines of the most representative SQL/Python — pick from the repo]
[why this code is interesting]
Challenge
Traced the 2 failed runs (of 16) and confirmed the cause was local network loss, not a pipeline bug.
Result
The pipeline runs automatically every 6 hours with 24 passing dbt tests and a data quality audit that stops the pipeline on critical failures.
What I learned
[1–2 lessons — fill in yourself]