wiki/tasks/flightradar24/ingest/preprocess

Preprocess Service

Reads unprocessed raw_packets from PostgreSQL and builds aircraft, flights, tracks, and track_points.

What it does

1. Waits for PostgreSQL to be ready
2. Reads fr24.processing_state to find the last processed raw_packet_id
3. Fetches the next batch of unprocessed packets
4. For each packet: upserts aircraft, creates/reuses a flight, appends a track_point
5. Advances the cursor in processing_state
6. Touches /tmp/preprocess-ready for Docker healthcheck

Data is fake/test for this stage — real ADS-B decoding comes later.

Dependencies

• psycopg2-binary

Environment variables

Variable	Default	Description
POSTGRES_HOST	required	DB host
POSTGRES_PORT	5432	DB port
POSTGRES_DB	required	DB name
POSTGRES_USER	required	DB user
POSTGRES_PASSWORD	required	DB password
POLL_INTERVAL_SECONDS	5.0	How often to poll for new packets
BATCH_SIZE	20	Packets per processing batch

Run locally

pip install -r requirements.txt
export POSTGRES_HOST=localhost POSTGRES_DB=fr24 POSTGRES_USER=fr24 POSTGRES_PASSWORD=change-me
python main.py

Build & run via Docker

docker build -t fr24-preprocess .
docker run --env-file ../../compose/.env fr24-preprocess