Streaming ETL Patterns for Petabyte-Scale Meter Data with Energy API and Apache Iceberg

Building reliable, scalable energy data pipelines is hard. Electricity auctions publish day-ahead curves on their own timetables, natural gas benchmarks like TTF and Henry Hub trade on different calendars and currencies, and carbon markets and grid carbon intensity arrive from separate agencies with incompatible formats. If your job is to ship analytics, forecasting, or ESG dashboards, you’ve probably spent weeks reverse-engineering government portals, retrying brittle scrapers, and harmonizing a zoo of CSVs into something your warehouse can query. Meanwhile, product teams are waiting for features you can’t deliver until every last edge case is normalized.

This post lays out a pragmatic path to streaming ETL that ingests multi-commodity market and grid data at scale, normalizes everything into a unified schema, and lands it into open table formats like Apache Iceberg for petabyte-scale analytics. We’ll use Energy API as the normalized data surface that replaces direct integrations with OMIE, ENTSO-E, EIA/FRED, ESIOS, and more. You’ll see how to request consistent JSON across electricity, gas, oil, coal, carbon allowances, and carbon intensity, and how to wire those responses into streaming sinks that keep your tables fresh without complicated per-provider logic.

Whether you maintain Kafka/Flink streams, Spark Structured Streaming jobs, or a dbt-powered lakehouse on Iceberg, the patterns here help you cut integration time from weeks to hours, align metrics across commodities, and reliably publish intraday curves, day-ahead auctions, historical series, and OHLC aggregates to any downstream consumer.

Why Energy API

Developers and data engineers need one normalized surface they can trust across energy commodities. Energy API provides exactly that: a single REST interface with a shared JSON schema across electricity, gas, oil, coal, carbon, and carbon intensity. Here’s why that matters in a production-grade streaming ETL:

• Uniform schema and semantics across commodities. You no longer special-case OMIE electricity vs EIA oil vs EU ETS carbon. That means a single ingestion function can parse prices, dates, frequencies, and currencies the same way, dramatically reducing ETL branching and failure modes.
• Multi-symbol batching in one call. Fetch BRENT_CRUDE, TTF_GAS, and EUA_CO2 simultaneously. This synchronizes time alignment for cross-commodity features, simplifies idempotent upserts in your lake tables, and cuts API round-trips.
• Intraday electricity curves and day-ahead auction data where sources publish them. Your real-time and day-ahead workloads can consume hourly or 15-minute series from the same API you use for daily commodities, which eliminates parallel, inconsistent feeds for curves and dailies.
• Consistent endpoints for discovery, latest values, historical snapshots, long-range timeseries, OHLC aggregations, fluctuation deltas, forecasts, category-specific queries, and provider status. The breadth lets you build everything from a price alert microservice to a petabyte-scale meter and market data lakehouse without stitching together bespoke pipelines.

The net effect: less code, fewer retries and parsing bugs, and a dramatically simpler path from an idea (e.g., “Add a cross-commodity hedging widget” or “Ship a carbon intensity overlay for our grid insights”) to a tested feature in production.

Quick Start

Base URL:

https://energy-api.com/api/v1

Let’s fetch the most recent values for three commodities in a single call. This is a perfect first step for validating connectivity and seeing the unified shape of the response.

curl -G https://energy-api.com/api/v1/latest \
--data-urlencode "symbols=BRENT_CRUDE,TTF_GAS,EUA_CO2" \
--data-urlencode "api_key=YOUR_API_KEY"

Example JSON response:

{
"success": true,
"date": "2026-06-11",
"base": "MIXED",
"rates": {
"BRENT_CRUDE": 74.82,
"TTF_GAS": 38.15,
"EUA_CO2": 67.40
},
"dates": {
"BRENT_CRUDE": "2026-06-11",
"TTF_GAS": "2026-06-11",
"EUA_CO2": "2026-06-11"
},
"currencies": {
"BRENT_CRUDE": "USD",
"TTF_GAS": "EUR",
"EUA_CO2": "EUR"
}
}

What to notice:

• success signals the call succeeded and the rates payload is valid.
• rates is a map keyed by symbol with numeric values you can write directly to your warehouse.
• dates provides per-symbol timestamps — critical for idempotent upserts into partitioned Iceberg tables.
• currencies shows per-symbol currency codes so you can standardize pricing or present mixed-currency dashboards without guesswork.

Core Endpoints

This section focuses on a core path you’ll likely use in a streaming ETL for market and meter-aligned analytics: discover symbols, fetch latest snapshots for streaming sinks, pull long historical windows for backfills, and retrieve intraday electricity curves. We’ll also preview rolling volatility/aggregation via OHLC and deltas via fluctuation.

1) Discoverable universe: GET /symbols

Purpose: discover active symbols with metadata so you can configure ingestion topics, create Iceberg tables per category, and display friendly names.

Key params:

• category: filter by commodity (gas|electricity|oil|coal|carbon_intensity|carbon).
• provider: optional filter by underlying source.
• base: optional currency filter.

curl -G https://energy-api.com/api/v1/symbols \
--data-urlencode "category=gas" \
--data-urlencode "api_key=YOUR_API_KEY"

Example JSON:

{
"success": true,
"count": 3,
"symbols": [
{
"symbol": "TTF_GAS",
"name": "TTF Natural Gas Day-Ahead",
"category": "gas",
"country_code": "EU",
"currency_code": "EUR",
"frequency": "daily",
"description": "TTF day-ahead price published by EEX."
}
]
}

How to use:

• symbol and name seed UI dropdowns and documentation in your repo.
• category and frequency guide your partitioning strategy (e.g., daily vs hourly).
• currency_code and country_code inform normalization jobs and geo filters.

2) Streaming snapshots: GET /latest

Purpose: marry high-level “ticker” convenience with cross-commodity batching. Ideal for micro-batches in Spark Structured Streaming or small messages in Kafka for state stores.

Key params:

• symbols: comma-separated list (e.g., BRENT_CRUDE,TTF_GAS,EUA_CO2).
• category and base: optional filters/normalization at the edge.

curl -G https://energy-api.com/api/v1/latest \
--data-urlencode "symbols=OMIE_ES_DA,EUA_CO2,TTF_GAS" \
--data-urlencode "api_key=YOUR_API_KEY"

Implementation notes:

• Write each symbol’s row keyed by (symbol, date). In Iceberg, use partitioning by date and optionally symbol for query pruning.
• Leverage dates per-symbol to avoid duplicates when late data appears. Use MERGE INTO semantics to upsert by primary key.
• For currency conversion, store both native currency and a normalized value (e.g., convert USD to EUR) in derived columns for fast reporting.

3) Backfills and training windows: GET /timeseries

Purpose: load long historical ranges to backfill Iceberg tables and train models. Because the response is keyed by date per symbol, it’s perfect for batch ingest into partitioned tables.

Key params:

• start, end: YYYY-MM-DD inclusive.
• symbols: comma-separated list.
• base: optional filter/normalization.

curl -G https://energy-api.com/api/v1/timeseries \
--data-urlencode "start=2025-01-01" \
--data-urlencode "end=2025-03-31" \
--data-urlencode "symbols=BRENT_CRUDE,TTF_GAS" \
--data-urlencode "api_key=YOUR_API_KEY"

Example JSON:

{
"success": true,
"base": "MIXED",
"start_date": "2025-01-01",
"end_date": "2025-03-31",
"rates": {
"BRENT_CRUDE": {
"2025-01-02": 76.30,
"2025-01-03": 75.90
},
"TTF_GAS": {
"2025-01-02": 46.80,
"2025-01-03": 47.10
}
},
"frequencies": {
"BRENT_CRUDE": "daily",
"TTF_GAS": "daily"
},
"currencies": {
"BRENT_CRUDE": "USD",
"TTF_GAS": "EUR"
}
}

How to use:

• rates is a nested map per symbol keyed by date. Flatten it into rows: (symbol, date, value, currency, frequency).
• frequencies and currencies help enforce schema and quality checks before writes.
• For Iceberg, write with partition spec: PARTITIONED BY (date). For cross-commodity marts, add symbol in the partition spec to improve selective scans.

4) Intraday electricity curves: GET /electricity/hourly

Purpose: get the full intraday curve (hourly or 15-minute, depending on the source) for one electricity symbol on a specific date. This is essential when aligning market curves with meter reads and forecasting models.

Key params:

• symbol: electricity symbol, e.g., OMIE_ES_DA, EPEX_DE_DA, AEMO_NSW1.
• date: YYYY-MM-DD for the curve session.

curl -G https://energy-api.com/api/v1/electricity/hourly \
--data-urlencode "symbol=OMIE_ES_DA" \
--data-urlencode "date=2026-06-11" \
--data-urlencode "api_key=YOUR_API_KEY"

Example JSON (shape shown; values illustrative):

{
"success": true,
"symbol": "OMIE_ES_DA",
"date": "2026-06-11",
"unit": "EUR/MWh",
"frequency": "hourly",
"curve": [
{ "start": "2026-06-11T00:00:00+02:00", "end": "2026-06-11T01:00:00+02:00", "price": 56.12 },
{ "start": "2026-06-11T01:00:00+02:00", "end": "2026-06-11T02:00:00+02:00", "price": 54.95 }
],
"count": 24
}

How to use:

• Store each interval as a row keyed by (symbol, start) with columns for end and price. Partition by date(start) to co-locate intervals.
• If your meter data is 15-minute, resample or align the curve to 15-minute boundaries, or join per-interval using overlap logic.
• For forecasting workflows, compute features like peak/off-peak averages, ramp rates, and shape factors and store them in a derived table for fast BI.

5) Volatility-friendly aggregates: GET /ohlc

Purpose: fetch weekly, monthly, or quarterly OHLC candles. These are ideal for factor modeling, volatility dashboards, and backtesting signal stability.

Key params:

• symbols: list of symbols to aggregate.
• period: weekly|monthly|quarterly (default monthly).
• start, end: optional date filters.

curl -G https://energy-api.com/api/v1/ohlc \
--data-urlencode "symbols=BRENT_CRUDE,TTF_GAS" \
--data-urlencode "period=monthly" \
--data-urlencode "start=2025-01-01" \
--data-urlencode "end=2025-06-30" \
--data-urlencode "api_key=YOUR_API_KEY"

Example JSON (abbreviated):

{
"success": true,
"symbols": {
"BRENT_CRUDE": [
{ "period": "2025-01", "open": 76.30, "high": 80.10, "low": 72.55, "close": 77.45, "data_points": 21 },
{ "period": "2025-02", "open": 77.45, "high": 82.20, "low": 76.80, "close": 80.05, "data_points": 20 }
],
"TTF_GAS": [
{ "period": "2025-01", "open": 46.80, "high": 49.75, "low": 44.10, "close": 47.35, "data_points": 23 }
]
}
}

How to use:

• Write one row per symbol-period with open, high, low, close, data_points. Partition by period for efficient factor scans.
• Use close for month-end navs; high/low for VaR bounds; data_points as a quality metric.

6) Quick deltas: GET /fluctuation

Purpose: reduce client-side math for change detection and alerting. Handy for P&L deltas, watchlists, and daily reports.

Key params:

• start, end: comparison window.
• symbols: comma-separated list.

curl -G https://energy-api.com/api/v1/fluctuation \
--data-urlencode "start=2026-06-01" \
--data-urlencode "end=2026-06-11" \
--data-urlencode "symbols=EUA_CO2,TTF_GAS,BRENT_CRUDE" \
--data-urlencode "api_key=YOUR_API_KEY"

Typical fields per symbol include start_value, end_value, change, change_pct. Persist these for timeboxed analytics or trigger alerts when change_pct crosses thresholds.

Real-World Use Cases

1) Petabyte-scale meter + market alignment in an Iceberg lakehouse

Problem: Utilities and energy retailers maintain massive meter datasets (hourly or 15-minute) spread across regions and products. Product teams want a unified view with day-ahead auction curves, carbon intensity overlays, and gas/oil hedging context — all queryable in minutes across petabytes.

Solution: Use GET /electricity/hourly to land intraday curves into an Iceberg table partitioned by date and symbol. Use GET /carbon-intensity for country-level intensity overlays and GET /latest + GET /timeseries for multi-commodity reference prices. Join meter reads to interval curves and intensity by time-window overlap. Analysts can now run portfolio optimization queries and scenario modeling with consistent, up-to-date market context.

Endpoints: /electricity/hourly, /carbon-intensity, /latest, /timeseries, /symbols.

2) Trading P&L and factor models with OHLC and fluctuations

Problem: Trading and risk teams need fast, daily P&L drivers, along with longer-horizon volatility factors and drawdown analytics without manually deriving candles.

Solution: Stream daily snapshots via /latest for the real-time view, backfill monthly candles with /ohlc for factor models, and use /fluctuation to compute change_pct without bespoke math. Store results in Iceberg for consistent scans and downstream dashboards.

Endpoints: /latest, /ohlc, /fluctuation, /timeseries.

3) ESG dashboards with grid carbon intensity and retail price context

Problem: Sustainability teams must track carbon intensity trends by country and correlate them with retail reference benchmarks (e.g., PVPC in Spain) and day-ahead wholesale curves to explain emissions drivers to customers.

Solution: Pull /carbon-intensity for the region of interest, pair with /electricity/pvpc on the same date, and enrich with /electricity/hourly for wholesale curve shape. Store daily aggregates in Iceberg and expose an analytics API to BI.

Endpoints: /carbon-intensity, /electricity/pvpc, /electricity/hourly, /timeseries.

FAQ

How often does the TTF gas price update?

TTF_GAS is delivered as a daily benchmark. Use /latest for the most recent available value and /historical or /timeseries for prior days. If a requested date is a non-publishing day, /historical returns the most recent value before it.

Can I retrieve electricity intraday curves at 15-minute resolution?

Yes, where the source publishes sub-hourly curves, /electricity/hourly returns the full intraday series at 15-minute or hourly intervals. The response carries start and end timestamps so you can align precisely with meter intervals.

Does the API support multiple commodities in a single request?

Yes. /latest, /timeseries, /historical, /fluctuation, and /ohlc accept multiple symbols. This is ideal for cross-commodity features and reduces orchestration overhead in streaming jobs.

Can I get historical energy prices going back multiple years?

Yes. Use /timeseries with start and end to request multi-year ranges. If you need date-specific snapshots, /historical gives you a single-day view across many symbols at once.

How can I monitor data provider health in my pipelines?

Use /status to see the last fetch status per provider. Ingest that into your observability stack to pause or fail gracefully when a provider is temporarily delayed.

Conclusion + CTA

For teams unifying meter data with wholesale markets, the hardest parts aren’t your models — it’s building a stable, normalized firehose across electricity, gas, oil, coal, carbon allowances, and grid carbon intensity, then landing it in a table format that scales to petabytes. With Energy API, you collapse weeks of ETL into hours: one normalized surface, consistent JSON across all commodities, robust endpoints for discovery, intraday curves, historical windows, OHLC, fluctuations, and provider health. Your data platform can focus on business logic and governance instead of brittle scrapers and format juggling.

If you’re ready to ship price alerting, hedging analytics, ESG dashboards, or cost calculators faster — and keep them reliable in production — start with the patterns here and wire them into your Iceberg or lakehouse architecture. Build once, reuse everywhere, and eliminate per-provider code paths.

Explore the endpoints and start integrating today at Energy API. Want to validate the fit in your stack and accelerate your roadmap? Try Energy API for free and turn market complexity into developer-friendly data your apps can trust.

---

Extended Reference and Implementation Guidance

Below is a broader tour through the available endpoints you can mix and match to build resilient, production-grade streaming ETL and analytics for energy workloads. We’ll also touch on implementation details for Apache Iceberg, error handling, and observability that keep your pipelines healthy at scale.

Endpoint Catalogue at a Glance

• GET /symbols — Discover active symbols and metadata.
• GET /latest — Latest price for one or more symbols.
• GET /historical — Prices for a specific date across symbols, rolling back when that date is non-publishing.
• GET /timeseries — Historical series for date ranges, ideal for backfills and training sets.
• GET /fluctuation — Start/end, absolute and percent change for quick deltas and alerts.
• GET /ohlc — Weekly, monthly, or quarterly candles, great for volatility analytics.
• GET /electricity/latest — Latest for all electricity symbols, optionally filtered by country.
• GET /electricity/hourly — Intraday curves at hourly or 15-minute granularity.
• GET /electricity/pvpc — Hourly Spanish PVPC retail reference prices.
• GET /gas/latest — TTF_GAS and HENRY_HUB in one call.
• GET /emissions/latest — EUA_CO2 (EU ETS allowance) price.
• GET /coal/latest — COAL_ROTTERDAM (API2) and COAL_NEWCASTLE.
• GET /carbon-intensity — Grid carbon intensity (gCO2eq/kWh) by country.
• GET /forecast — Next published day-ahead auction price for applicable electricity symbols.
• POST /cost-estimate — Simple wholesale electricity cost estimate for a symbol or country and monthly kWh.
• GET /status — Last fetch status per provider for monitoring.

Category Highlights and Examples

Electricity: /electricity/latest and /electricity/pvpc

When you need a fleet-wide electricity view, /electricity/latest returns the most recent prices for all supported electricity symbols. Filter by country when building regional dashboards or normalizing offerings.

curl -G https://energy-api.com/api/v1/electricity/latest \
--data-urlencode "country=DE" \
--data-urlencode "api_key=YOUR_API_KEY"

For Spain’s PVPC, /electricity/pvpc returns hourly retail reference prices, often used to explain consumer bills or to power savings calculators. Combine it with /electricity/hourly for a wholesale vs retail spread analysis and with /carbon-intensity for emissions-aware cost narratives.

curl -G https://energy-api.com/api/v1/electricity/pvpc \
--data-urlencode "date=2026-06-11" \
--data-urlencode "api_key=YOUR_API_KEY"

Gas, Oil, Coal: Category rollups for dashboards and hedging

Instead of hand-picking symbols in every query, the category rollups /gas/latest, /coal/latest, and /emissions/latest give you faster reads for common watchlists and risk snapshots:

curl -G https://energy-api.com/api/v1/gas/latest \
--data-urlencode "api_key=YOUR_API_KEY"

curl -G https://energy-api.com/api/v1/coal/latest \
--data-urlencode "api_key=YOUR_API_KEY"

curl -G https://energy-api.com/api/v1/emissions/latest \
--data-urlencode "api_key=YOUR_API_KEY"

These calls simplify SLO-driven microservices that power dashboards without asking your UI to juggle multiple endpoints or symbol lists.

Carbon Intensity and Forecasts

Grid carbon intensity is a powerful operational driver for ESG products. /carbon-intensity provides per-country intensity values in gCO2eq/kWh:

curl -G https://energy-api.com/api/v1/carbon-intensity \
--data-urlencode "country=DE" \
--data-urlencode "api_key=YOUR_API_KEY"

For auction-driven electricity symbols, /forecast returns the next published day-ahead price — a deterministic lookup for production planning and “tomorrow’s rate” explanation. If a symbol is not auction-based, it returns 404, which you can handle gracefully in your pipeline.

curl -G https://energy-api.com/api/v1/forecast \
--data-urlencode "symbol=OMIE_ES_DA" \
--data-urlencode "api_key=YOUR_API_KEY"

Cost Estimation

POST /cost-estimate lets you compute a simple monthly wholesale cost estimate as latest_price × kWh per month for either a specific symbol or a country. It’s a great starting point for calculators that will be later refined with taxes, network charges, and profile-based adjustments.

curl -X POST https://energy-api.com/api/v1/cost-estimate \
-H "Content-Type: application/json" \
-d '{
"country": "ES",
"kwh_per_month": 350
}'

Example JSON:

{
"success": true,
"input": { "country": "ES", "kwh_per_month": 350 },
"symbol": "OMIE_ES_DA",
"unit": "EUR/MWh",
"latest_price": 56.12,
"estimate_eur": 19.64,
"note": "Wholesale-only estimate; excludes taxes, network charges, and profile effects."
}

Provider Status and Observability

Use /status to pull a heartbeat of provider fetches for your health monitors and circuit breakers:

curl -G https://energy-api.com/api/v1/status \
--data-urlencode "api_key=YOUR_API_KEY"

Example JSON:

{
"success": true,
"providers": [
{ "name": "OMIE", "last_fetch": "2026-06-11T12:04:31Z", "status": "ok" },
{ "name": "ENTSO-E", "last_fetch": "2026-06-11T12:01:12Z", "status": "ok" },
{ "name": "EIA", "last_fetch": "2026-06-10T22:15:01Z", "status": "ok" }
]
}

Build alerting that watches for stale last_fetch or non-ok statuses and temporarily pausing downstream joins that depend on delayed sources to prevent partial, misleading dashboards.

Symbols You’ll Use Often

• Gas: TTF_GAS (EUR/MWh, EU), HENRY_HUB (USD/MMBtu, US)
• Oil: BRENT_CRUDE (USD/barrel), WTI_CRUDE (USD/barrel)
• Electricity: OMIE_ES_DA (Spain day-ahead), EPEX_DE_DA (Germany), PVPC_ES_2TD (Spain retail), AEMO_NSW1 (Australia)
• Carbon (ETS): EUA_CO2 (EUR/MT)
• Coal: COAL_ROTTERDAM (API2), COAL_NEWCASTLE
• Carbon Intensity: CARBON_INT_DE, CARBON_INT_EU (gCO2eq/kWh)

Python Example: Normalize and Write to Iceberg

The snippet below demonstrates pulling multi-symbol timeseries and writing normalized rows. Replace the write function with your Iceberg sink (e.g., Spark, Flink, or an Iceberg REST catalog writer).

import requests
from datetime import date
from uuid import uuid4

BASE = "https://energy-api.com/api/v1"

def fetch_timeseries(symbols, start, end, api_key):
r = requests.get(
f"{BASE}/timeseries",
params={"symbols": ",".join(symbols), "start": start, "end": end, "api_key": api_key},
timeout=30
)
r.raise_for_status()
return r.json()

def normalize_timeseries(ts_json):
rows = []
rates = ts_json["rates"]
freqs = ts_json.get("frequencies", {})
currs = ts_json.get("currencies", {})
for symbol, series in rates.items():
freq = freqs.get(symbol)
curr = currs.get(symbol)
for d, v in series.items():
rows.append({
"id": str(uuid4()),
"symbol": symbol,
"date": d,
"value": float(v),
"currency": curr,
"frequency": freq
})
return rows

def write_to_iceberg(rows):
# Pseudocode: replace with your Spark/Flink/Iceberg writer
# spark.createDataFrame(rows).writeTo("lake.energy.prices").append()
pass

if __name__ == "__main__":
symbols = ["BRENT_CRUDE", "TTF_GAS", "EUA_CO2"]
ts = fetch_timeseries(symbols, "2025-01-01", "2025-12-31", "YOUR_API_KEY")
rows = normalize_timeseries(ts)
write_to_iceberg(rows)

Best practices:

• Use partitioning by date (and optionally symbol) in Iceberg to speed up pruning for range filters.
• Store currency and frequency as first-class columns for downstream normalization and QA.
• Upsert by (symbol, date) keys when backfilling or when late-arriving corrections occur.

JavaScript Example: Multi-Commodity Snapshot Service

A simple HTTP service that fetches multi-commodity snapshots with a single call and re-exports a clean JSON. This is useful when front-ends must minimize latency and calls.

import express from "express";
import fetch from "node-fetch";

const app = express();
const BASE = "https://energy-api.com/api/v1";

app.get("/snapshots", async (req, res) => {
const symbols = req.query.symbols || "BRENT_CRUDE,TTF_GAS,EUA_CO2";
const url = new URL(`${BASE}/latest`);
url.searchParams.set("symbols", symbols);
url.searchParams.set("api_key", process.env.ENERGY_API_KEY);

const r = await fetch(url);
if (!r.ok) {
const err = await r.text();
return res.status(r.status).json({ success: false, error: err });
}
const json = await r.json();
res.json({
success: true,
as_of: json.date,
data: Object.entries(json.rates).map(([symbol, value]) => ({
symbol,
value,
currency: json.currencies?.[symbol],
date: json.dates?.[symbol]
}))
});
});

app.listen(8080, () => console.log("snapshot service on :8080"));

Streaming ETL Patterns for Apache Iceberg

When landing market and grid data into Iceberg at scale, your goal is to preserve temporal semantics, minimize duplicates, and keep tables compact for interactive analytics. Here are patterns that work in practice:

• Topic-per-domain architecture. Produce normalized JSON events into Kafka topics like market.latest, market.timeseries, electricity.curves. Consumers (Flink, Spark) converge those into Iceberg tables.
• Idempotent writes. Use MERGE INTO keyed by (symbol, date) for daily series and by (symbol, start) for intraday curves. If a provider republishes corrected values, your job should update rows deterministically.
• Partitioning strategy. For daily tables, partition by date; for intraday curves, partition by date(start). If you support hundreds of symbols, either add symbol to the partition spec or rely on Iceberg’s hidden partitioning with data filters at read time.
• Schema evolution. Keep value as a double, add normalized_value if you convert against a base currency. Carry source fields like frequency and currency to preserve provenance.
• Compaction and clustering. Schedule Iceberg compaction for curve tables to maintain file sizes conducive to interactive scans. If using Flink, tune checkpointing to flush reasonably sized files.
• Late data and watermarks. For curve data, set watermarks conservatively to permit late-arriving intervals without dropping them. Configure your aggregations to recompute daily rollups when new late data appears.

Example: Spark Structured Streaming pseudocode for merging latest snapshots into an Iceberg table:

// Scala or PySpark pseudocode
val latestDF = spark.readStream
.format("kafka")
.option("subscribe", "market.latest")
.load()
.select(from_json(col("value").cast("string"), schema).as("j"))
.selectExpr("j.symbol", "j.date", "j.value", "j.currency")

// Write to a staging table, then MERGE
latestDF.writeStream
.format("iceberg")
.option("fanout-enabled", "true")
.option("checkpointLocation", "s3://.../chk/latest")
.toTable("lake.energy.prices_latest_staging")

// Periodic MERGE job (batch):
spark.sql("""
MERGE INTO lake.energy.prices_latest t
USING lake.energy.prices_latest_staging s
ON t.symbol = s.symbol AND t.date = s.date
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
""")

Handling Errors and Quality Controls

Energy API returns structured errors you can route through your observability pipeline.

• 401 — Missing or invalid credentials. Verify configuration and secret management.
• 404 — No data for given symbols or date. For /forecast, this is expected for non-auction symbols; fall back to /latest or skip for that run.
• 422 — Validation errors such as missing parameters or invalid formats. Validate inputs before requests; in streaming, dead-letter bad messages and alert.
• 429 — Exponential backoff on retry. In streaming frameworks, use built-in retry policies and jitter.

Example error shape:

{
"success": false,
"error": "Human-readable message."
}

Quality best practices:

• Add invariants: value must be non-negative; currency and frequency not null for applicable endpoints.
• Track null ratios per symbol/date to catch upstream publication anomalies early.
• Use /status to correlate anomalies with upstream provider conditions.

Complete, Realistic End-to-End Example

Suppose you need to align OMIE ES day-ahead hourly curve with carbon intensity and compare it against TTF gas and EUA carbon on the same date, then publish both a curve table and a daily summary table in Iceberg.

1. Discover symbols you’ll ingest and cache their metadata via /symbols.
2. Fetch the OMIE_ES_DA hourly curve with /electricity/hourly and write intervals keyed by (symbol, start).
3. Fetch /carbon-intensity for country=ES and write a per-date intensity row.
4. Fetch /latest for TTF_GAS and EUA_CO2; also store the per-symbol dates for audit and backfills.
5. Aggregate a daily view: average_curve_price, peak_price, min_price, carbon_intensity, gas_price, carbon_price — and write to a fact table keyed by (date, region).

You’ve now produced an integrated daily entity across commodities and grid metrics from a single normalized API with consistent types and metadata, making BI and data science substantially simpler.

Additional JSON Examples: /historical and Multi-Commodity Daily Join

/historical provides a specific past-date snapshot for many symbols at once:

curl -G https://energy-api.com/api/v1/historical \
--data-urlencode "date=2025-09-15" \
--data-urlencode "symbols=BRENT_CRUDE,TTF_GAS" \
--data-urlencode "api_key=YOUR_API_KEY"

{
"success": true,
"date": "2025-09-15",
"base": "MIXED",
"rates": {
"BRENT_CRUDE": 71.45,
"TTF_GAS": 36.20
},
"currencies": {
"BRENT_CRUDE": "USD",
"TTF_GAS": "EUR"
}
}

Use it for:

• Point-in-time backtesting for strategies requiring same-day cross-commodity prices.
• Aligning cost estimates for historical billing analysis.

A daily join row you might store after consolidating multiple endpoints:

{
"date": "2026-06-11",
"region": "ES",
"symbol_day_ahead": "OMIE_ES_DA",
"avg_hourly_wholesale_eur_mwh": 58.22,
"peak_hourly_wholesale_eur_mwh": 73.10,
"min_hourly_wholesale_eur_mwh": 44.91,
"pvpc_avg_eur_mwh": 61.05,
"carbon_intensity_gco2_per_kwh": 168,
"ttf_gas_eur_mwh": 38.15,
"eua_co2_eur_ton": 67.40
}

This record becomes the backbone of a daily operations dashboard and feeds forecasting features that consider price shape, retail context, and emissions.

Performance Tips

• Batch multi-symbol requests where possible (e.g., /latest, /timeseries) to reduce network overhead and improve synchronization across commodities.
• Implement client-side caching of /symbols metadata and refresh on a fixed cadence.
• Parallelize per-country curve pulls for intraday via a worker pool sized to your downstream write throughput.
• Plan compaction for high-churn curve tables and consider snapshot expiration policies in Iceberg to maintain performance.

Governance and Observability Patterns

• Separate ingestion and serving tables. Keep raw-normalized rows immutable, and produce curated marts and aggregates via scheduled jobs. This clarifies lineage and allows reproducible reprocessing.
• Per-app or per-domain API usage segregation lets you attribute workloads and filter logs. Combine with audit logs on write side (Iceberg metadata) for full lineage.
• Use health signals from /status to gate merges. If a provider is delayed, hold your daily finalize step to avoid mixing old and new data.

Commodity Mix in One Call: A Differentiator in Practice

Energy workloads frequently need cross-commodity context. With Energy API, you can fetch BRENT_CRUDE, TTF_GAS, EUA_CO2, and an electricity day-ahead indicator in a single request to /latest or /timeseries. This simplifies:

• Atomic writes of a coherent snapshot across markets, protecting downstream analytics from mismatched timestamps.
• Fewer microservice round-trips and reduced retry surfaces.
• Cleaner feature pipelines for ML that expect aligned time slices across covariates.

A Note on Forecasts and Determinism

/forecast for auction-driven electricity symbols gives the next published day-ahead price via deterministic lookup — not a model. This is essential for production-grade planning and for explaining “tomorrow’s cost” without probabilistic error bars. When /forecast returns 404, you know the symbol does not have an auction-based next-day publish, and you can fall back gracefully.

Putting It All Together

With a handful of endpoints — /symbols for discoverability, /timeseries for backfills, /latest for snapshots, /electricity/hourly for curves, /ohlc and /fluctuation for analytics, /carbon-intensity for ESG overlays, and /status for health — you can construct a streaming ETL that feeds an Apache Iceberg lakehouse supporting petabyte-scale queries. You’ll spend your time modeling, optimizing, and explaining data rather than maintaining one-off scrapers and adapters for each provider and format.

Energy API makes this possible with one unified JSON surface spanning electricity, gas, oil, coal, carbon allowances, and carbon intensity — the multi-commodity backbone your products need. Explore the documentation and start integrating at Energy API, and when you’re ready to validate the approach end-to-end in your stack, Try Energy API for free.