Solar Consumer

This code can be used to download solar forecasts and save them to the data platform (or CSV). It fetches solar generation estimates for embedded solar farms and processes the data for analysis. We currently collect

• UK: Forecast can be retreived from NESO. Generation Data can be retrevied from PVLive.
• NL: Generation values from Ned NL, both national and region. National Forecast values from Ned NL too.
• DE: Generation values from ENTSOE for several TSOs. National forecast values from ENTSOE too.
• BE: Solar PV forecast data (national and regional) from Elia Open Data API.
• India (Rajasthan): Real-time solar and wind generation data from RUVNL (Rajasthan Urja Vikas Nigam Limited).

India – RUVNL (Rajasthan)

The solar consumer supports RUVNL (Rajasthan, India) real-time generation data.

• Both solar and wind generation are supported
• Data is fetched via solar-consumer
• Generation data is saved to the data platform
• Separate locations are created for solar and wind generation
• No separate ruvnl-consumer service is required

Here are the different sources of data, and which methods can be used to save the results

Source	Country	CSV	Data Platform
PVLive	🇬🇧	✅	✅
NESO forecast	🇬🇧	✅	✅
Ned-nl	🇳🇱	✅	✅
Ned-nl forecast	🇳🇱	✅	✅
Germany (ENTSOE)	🇩🇪	✅	✅
Germany (ENTSOE) forecast	🇩🇪	✅	✅
Elia Open Data	🇧🇪	✅	✅
RUVNL (Rajasthan SLDC)	🇮🇳	✅	✅

Requirements

• Docker
• Docker Compose

Installation & Running

1. Clone the repository:

git clone https://github.com/openclimatefix/neso-solar-consumer.git
cd neso-solar-consumer

2. Copy the example environment file:

cp .example.env .env

3. Start the application:

docker compose up -d

The above command will:

• Build and start the NESO Solar Consumer application

To stop the application:

docker compose down

To view logs:

docker compose logs -f

Documentation

The package provides three main functionalities:

1. Data Fetching: Retrieves solar forecast data from the NESO API
2. Data Formatting: Processes the data into a standardized forecast objects
3. Data Storage: Saves the data to CSV or the data platform

Key Components:

• fetch_data.py: Handles API data retrieval
• save/: Save methods (CSV and data platform)
• app.py: Orchestrates the entire pipeline

Environment Variables: (Can be found in the .example.env / .env file)

• COUNTRY="gb" : Country code for fetching data. Currently, other options are ["be", "de", "ind_rajasthan", "nl"]
• SAVE_METHOD: Ways to store the data. Options are ["csv", "data-platform"].
  data-platform is supported for UK (PVLive, NESO forecast), NL (Ned-nl), BE (Elia), DE (ENTSOE), and India (RUVNL).
• CSV_DIR=None : Directory to save CSV files if SAVE_METHOD="csv".
• UK_PVLIVE_REGIME=in-day: For UK PVLive, the regime. Can be "in-day" or "day-after"
• UK_PVLIVE_MAX_GSP_ID=342: For UK PVLive, the amount of gsps we pull data for.
• UK_PVLIVE_BACKFILL_HOURS=2: For UK PVLive, the amount of backfill hours we pull, when regime="in-day"
• UK_PVLIVE_DOMAIN_URL: For UK PVLive, the domain URL to fetch data from. Defaults to "api.pvlive.uk"
• DE_BACKFILL_HOURS=24: For DE (ENTSOE), the amount of backfill hours we pull. ENTSOE only publishes settled values, so we always pull a window of recent history and let the save step drop anything already stored.
• DE_FORECAST_TYPE=day_ahead: For DE (ENTSOE) forecasts, which of the three ENTSOE forecasts to collect. Can be "day_ahead", "intraday" or "current". Each one is saved by its own forecaster, for example entsoe-de-day-ahead.
• NL_POTENTIAL_GENERATION: boolen, to create and save a potential solar generation
• APIKEY_ENTSOE: The ENTSOE api key. Needed for DE solar generation, and for NL day-ahead prices (used for NL potential generation).

Adding a New Country

This guide explains how to add a new country data source to Solar Consumer.

Overview

Adding a country typically involves:

• Identifying a reliable data source or API
• Implementing a country-specific fetch function
• Adding tests
• Saving data locally (CSV) and/or to the data platform

---

Step 1: Find a Data Source / API

Identify a reliable data source for the country:

• Prefer official grid operators or government-backed APIs
• Ensure timestamps, units, and generation values are clearly defined

If the API requires credentials:

• Add the variable to .example.env
• Document the required environment variable name

---

Step 2: Create a Fetch Function

Add a new country-specific fetch module inside the solar_consumer package.

Example naming convention:

solar_consumer/data/fetch_<country>.py

Next Steps

After adding the fetch function:

• Register the country in the main fetch dispatcher
• Add unit and integration tests under tests/
• Verify the data runs locally and can be saved to CSV
• If supported, ensure data can be saved to the data platform
• Open a pull request for review

Saving Data to the Data Platform

If the country supports saving data to the data platform:

1. Clone the data platform repository:

   git clone https://github.com/openclimatefix/data-platform.git

2. Follow the data-platform README to start it locally (Docker-based setup).
3. Configure Solar Consumer to point to the local data-platform instance.
4. Run the consumer and verify data is ingested successfully.

Development

1. Set up the development environment:

pip install ".[dev]"

2. Run tests:

pytest

3. Format code:

black .

4. Run linter:

ruff check .

Running the Test Suite

The test suite includes unit tests and integration tests:

# Run all tests
pytest

# Run specific test file
pytest tests/test_fetch_data.py

# Run with coverage
pytest --cov=neso_solar_consumer

Continuous Integration (CI)

This reposistory has 2 main CI workflows - branch-ci and merged-ci.

• branch-ci is triggered on all pushes to any branch except main, and on any pull request that is opened, reopened or updated. It runs the tests suite, lints the project, and builds and pushes a dev image.
• merged-ci is triggered on any pull request merged into main. It bumps the git tag, and builds and pushes a container with that tag.

FAQ

Q: What format is the data stored in? A: The data is saved to the data platform (or CSV), with timestamps in UTC.

Q: How often should I run the consumer? A: This depends on your use case and the NESO API update frequency. The consumer can be scheduled using cron jobs or other scheduling tools.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing and community

• PR's are welcome! See the Organisation Profile for details on contributing
• Find out about our other projects in the OCF Meta Repo
• Check out the OCF blog for updates
• Follow OCF on LinkedIn

Contributors

Peter Dudfield 🤔	Seao7 💻	Siddharth 👀 🚇 💻	Conor O Callaghan 📖	Ali Rashid ⚠️	Manzoor Ahmed Shaikh 💻	Anas Khan 📖
Peter Ireland 📖	vashisthrahul13 💻	rahul-ahuja 💻	Rahul Joon 💻	michael-gendy 🚇	Shohail Ismail 💻	Prafful Vyas 💻
Ramkumar R 💻	Mohit 💻	Vishal Goyal 💻	冰水泡枸杞 💻

Part of the Open Climate Fix community.