Real-Time Revenue Dashboard (Data Engineering Project)

A mini data engineering pipeline that streams FX rates via Kafka, stores transactions in PostgreSQL, converts them to USD, and visualizes insights with Streamlit.

Components Overview

Script	Description
1_setup_postgres_transactions.py	Generates and inserts transaction data into PostgreSQL using API FX rates
2_fx_rate_producer.py	Simulates FX rate streaming via Kafka
3_fx_rate_consumer.py	Listens to Kafka and saves FX rates to local JSON
5_convert_fx_live.py	Converts transaction amounts to USD using latest FX from JSON
4_streamlit_dashboard.py	Displays real-time revenue insights in Streamlit using transactions_converted table

How to Run the Project Automaicalluy

Run with Docker : Automatically runs PostgreSQL, Kafka, Streamlit app, and Python scripts

1. Build and launch services:

./run_all.sh

How to Run the Project manually using Python scripts.

0. Install Dependencies

pip install -r requirements.txt

1. Start Kafka

kafka-topics --list --bootstrap-server localhost:9092

2. Simulate Transactions

python 1_setup_postgres_transactions.py

3. Stream FX Rates

python 2_fx_rate_producer.py

# Another terminal !!
python 3_fx_rate_consumer.py

4. Convert to USD

python 4_convert_fx_live.py

5. Launch Dashboard

streamlit run 5_streamlit_dashboard.py

PostgreSQL Tables

revenue_dashboard=# \dt
                     List of relations
 Schema |          Name          | Type  |      Owner      
--------+------------------------+-------+-----------------
 public | transactions           | table | kritsadakruapat
 public | transactions_converted | table | kritsadakruapat

http://localhost:8501