Data acquisition · Part 2 of 6
How to get trustworthy India grid data with the Modeler Data Shop
A complete Data Shop tutorial: choose datasets, scope states and dates, create or attach a model, verify lineage, and reproduce a Gujarat demand plus IEX evidence workflow.
- For
- Analysts, BESS developers, data-centre teams and agents preparing inputs
- Time
- 60 minutes including QA
- Updated
- 20 Jul 2026
Direct answer
The Modeler Data Shop turns a documented slice of Energy Atlas Intelligence into a model or an attachment. You select the dataset, geography, coverage window, resolution and target; the server re-validates the request and records an idempotent ingest order.
Solved result first
What the completed analysis must expose
Treat Data Shop ingestion as a data-engineering run with acceptance tests. For a Gujarat FY26 hourly pull, validate the normalized scope, expected 8,760-hour calendar, unit, missingness, duplicate timestamps, aggregate-energy reconciliation, component mapping and immutable receipt before using the series in a solve. The IEX attachment is a bounded derived calibration index; it must never be interpreted as raw ₹/MWh, a tariff or a nodal price.
| Control | Expected evidence | Acceptance benchmark | Failure meaning |
|---|---|---|---|
| Normalized scope | Gujarat · 2025-04-01 → 2026-04-01 · hourly | Exactly one state and one declared interval | Wrong geography or silent default |
| Hourly coverage | Snapshot ledger and receipt | 8,760 timestamps; 0 duplicates; 0 unexplained gaps | Incomplete or duplicated energy volume |
| Demand unit | Load p_set in MW | All mapped series share MW and timezone | Scale or dimensional error |
| Energy reconciliation | Σ MW × 1 h | Difference from source interval total ≤0.1% or explained | Transformation changed annual energy |
| IEX derived band | Attached index series | Every value in [0,1]; unit = index | Raw-price or normalization contract broken |
| Replay | Ingest receipt / normalized key | Same scope produces the same idempotent ledger outcome | Duplicate or non-reproducible materialization |
Analytical method
- 01Write the analytical variable needed by the study before choosing a catalogue card.
- 02Inspect source, lineage, unit, geography, time resolution, coverage and allowed target on the card.
- 03Normalize state, start/end, resolution and target; save the request and receipt as the data contract.
- 04Run structural, temporal and statistical QA before attaching a second dataset or launching a solve.
- 05Join datasets only on an explicit common clock and geography; document resampling, normalization and missing-value rules.
- 06Benchmark derived series against source aggregates and bounded-domain tests, then freeze the data receipt with the study manifest.
Who should interrogate what
Electrical engineer
Will this series map to the modeled components?
Check bus/load keys, MW units, clock alignment and whether structural network elements already exist.
Environmental engineer
Can carbon or weather evidence be aligned hourly?
Inspect temporal coverage, normalization and whether the environmental signal is observed, derived or modeled.
Energy economist
Did transformation change the economic quantity?
Reconcile energy and price statistics, preserve units and separate indices from executable prices or tariffs.
Policy maker
Is the geographic scope comparable?
Confirm state, region and national series are not mixed without an allocation rule and coverage statement.
Energy trader
Is this market evidence raw, normalized or forecast?
Use source window and product definition; never convert a 0–1 band into ₹/MWh without a documented calibration.
Before you start
- • An active Modeler project.
- • Enterprise access for live Data Shop ingestion; lower tiers can inspect the locked catalogue preview.
- • A decision about whether the dataset should create a new model or attach to an existing one.
What you will have
You will create a Gujarat FY26 demand model, attach a derived IEX price band to the active model, and save an ingest receipt with dataset, scope, lineage and target.
Step-by-step
Complete the workflow
Decide whether to create or attach
Choose New model for structural data; choose Attach for a time series or calibration signal that belongs on an existing model.
Do this
- 1.Use New model when the dataset supplies the starting loads, fleet or corridor network.
- 2.Use Attach when the active model already has compatible components and needs a demand, price, generation, carbon or weather series.
- 3.Check the dataset card because not every dataset supports both targets.
Expected evidence
- ✓ A target choice that matches the dataset’s published target matrix.
Trust check
Attaching an incompatible series does not create missing buses, generators or mapping keys.
Read the catalogue before selecting data
Each card states title, description, source table, unit, lineage, supported target and resolution.
Do this
- 1.Open File → Browse datasets, the store icon, or Command Palette → Open Data Shop.
- 2.Read the unit and lineage badge before opening a card.
- 3.Confirm whether the dataset is state-scoped or national.
Expected evidence
- ✓ You can explain what will be persisted without relying on the card title alone.
| Dataset | Unit | Target | Resolution / scope | Lineage |
|---|---|---|---|---|
| State Demand — hourly | MW | New or attach | State / zonal | derived-B |
| Power Plant Fleet | MW | New | State | derived-A |
| State Net Interchange — hourly | MW | New or attach | State / zonal | derived-A |
| Inter-Regional Corridor ATC | MW | New | Zonal / national | derived-A |
| IEX Price Bands — derived | 0–1 index | Attach | State / zonal | derived-B |
| Modeled Demand — hourly | MW | New or attach | State / zonal | derived-B |
| Generation by Fuel — hourly | 0–1 index | Attach | State | derived-B |
| Carbon Intensity — hourly | 0–1 index | Attach | State | derived-A |
| Weather — solar availability | 0–1 index | Attach | State | derived-A |
| IEX Area Price Bands — derived | 0–1 index | Attach | Zonal / national | derived-B |
| IEX Green-DAM Band — derived | 0–1 index | Attach | Zonal / national | derived-B |
| IEX GTAM Band — derived | 0–1 index | Attach | Zonal / national | derived-B |
| Demand Envelope — stress peaks | MW | New | State | derived-B |
Trust check
A source table name documents origin; it does not mean raw warehouse rows will leave the server.
Solved example: configure Gujarat hourly demand
Use State Demand — hourly, Gujarat, the declared FY26 window, state resolution and New model.
Do this
- 1.Select State Demand — hourly.
- 2.Tick Gujarat and leave other states unselected.
- 3.Set From to 2025-04-01 and To to 2026-04-01.
- 4.Choose State resolution and New model.
- 5.Use a name such as ‘Gujarat FY26 demand — Data Shop’ and click Ingest.
Expected evidence
- ✓ A completed ingest toast with an Open model action.
- ✓ A new active model whose load p_set series carries the selected scope.
Trust check
The profile is reconciled and labeled derived-B; do not describe every hourly value as a directly metered Gujarat observation.
Inspect the ingest receipt as a data contract
The receipt should let another authorised human or agent replay the exact pull without guessing defaults.
Do this
- 1.Record dataset ID, source table, normalized states, from/to timestamps, resolution, target, model ID and lineage.
- 2.Record the ingest order or receipt ID, completion time and any source or transformation hash shown.
- 3.Compare the normalized scope with the submitted form and investigate any server-side correction.
- 4.Attach the receipt to the study note before opening the new model.
Expected evidence
- ✓ A replayable scope with explicit date-boundary semantics.
- ✓ A persistent link between the source request and created or attached model.
How to read the result
- • The friendly dataset title is not a stable identifier; agents should use the dataset ID and normalized scope.
- • A completed receipt proves materialization, not analytical suitability; QA begins after completion.
Trust check
Do not accept a toast as provenance. Save the normalized request, receipt and destination model together.
Verify the new model before attaching more data
Open Tables and verify state, units, snapshot count, model name and source metadata.
Do this
- 1.Open the model from the ingest toast.
- 2.Use Tables → Loads to confirm MW units and Gujarat mapping.
- 3.Record the model ID, date range, resolution and lineage in the study note.
Expected evidence
- ✓ The model’s visible metadata matches all five scope selections made in the Data Shop.
Trust check
If scope or units do not match, stop before layering additional data; otherwise the provenance error becomes harder to isolate.
Run temporal and energy QA on the demand series
Prove calendar completeness, ordering, uniqueness, units and energy conservation before using the profile in dispatch.
Do this
- 1.Confirm timestamps are monotonic, hourly and expressed in one declared timezone.
- 2.For 1 April 2025 to 1 April 2026, expect 8,760 hourly records after applying the receipt’s boundary convention.
- 3.Count nulls, duplicates, negative demand and discontinuities; document every repair or retained anomaly.
- 4.Compute annual GWh as the sum of hourly MW divided by 1,000 and reconcile against the source aggregate.
- 5.Inspect peak, minimum, load factor, p50, p95 and top-ten ramp hours.
Expected evidence
- ✓ Zero unexplained gaps and duplicates.
- ✓ Energy reconciliation within 0.1% or an explicit transformation explanation.
- ✓ A summary-statistics table stored with the ingest receipt.
Worked calculation
Annual energy (GWh) = Σ hourly demand (MW) × 1 h ÷ 1,000 Load factor = mean demand / peak demand Hourly ramp (MW/h) = demandₜ − demandₜ₋₁
How to read the result
- • A plausible annual total can still hide shifted peaks or duplicated hours; inspect both aggregate and chronology.
- • India has no daylight-saving transition, so a Gujarat FY26 hourly series should not contain DST-created 23/25-hour days.
| Check | Pass condition | Why it matters |
|---|---|---|
| Coverage | 8,760 expected hours; 0 unexplained missing | A missing peak can understate capacity stress |
| Duplicates | 0 duplicate timestamps | Duplicates overstate energy and weighted demand |
| Physical range | No negative demand; reviewed extreme ramps | Catches sign, timezone and unit errors |
| Energy | Source-vs-materialized difference ≤0.1% or explained | Protects aggregate policy and cost arithmetic |
| Clock | One timezone and documented interval boundary | Prevents price/carbon misalignment |
Trust check
If the time axis does not pass, do not attach market, carbon or weather series—the join will produce precise-looking nonsense.
Solved example: attach the derived IEX price band
IEX Price Bands is attach-only and materializes a 0–1 calibration index, not raw ₹/MWh prices.
Do this
- 1.With the Gujarat demand model active, reopen the Data Shop.
- 2.Select IEX Price Bands — derived.
- 3.Choose Gujarat, 2025-04-01 to 2026-04-01, state resolution and Attach to the active model.
- 4.Click Ingest and reopen the model when the receipt completes.
Expected evidence
- ✓ The active model remains the target and gains an attached calibration series.
- ✓ The persisted unit is index and lineage is derived-B.
Trust check
Never label the 0–1 band as a tariff, nodal LMP, procurement quote or raw IEX price series.
Align demand and market evidence on one clock
A market-derived index is useful only when every timestamp maps deterministically to the demand snapshot it calibrates.
Do this
- 1.Compare demand and IEX coverage windows, resolution and timezone from their receipts.
- 2.Choose the intersection of timestamps; do not forward-fill outside observed coverage.
- 3.Document whether any resampling uses mean, sum, last, maximum or weighted aggregation.
- 4.Verify all IEX band values remain within [0,1] after alignment.
- 5.Save overlap count and excluded-hour count in the study manifest.
Expected evidence
- ✓ One common timestamp index with no many-to-many join.
- ✓ A bounded index and explicit count of retained observations.
Worked calculation
Coverage overlap = |timestamps(demand) ∩ timestamps(market)| / |timestamps(demand)| Require 100% for a full-horizon paired study or shorten the study horizon explicitly.
How to read the result
- • High correlation after a one-hour shift can signal timezone or interval-ending confusion; test lagged correlations during QA.
- • A normalized band preserves relative calibration information, not the rupee level needed for settlement or tariff analysis.
Trust check
Never silently pad a derived market series to make the model horizon fit.
Understand repeat pulls and receipts
The same organisation, dataset, normalized scope and target resolve through an idempotent order ledger.
Do this
- 1.Save the dataset ID, state list, from/to dates, resolution, target and target model ID.
- 2.If you repeat the exact request, verify that the service returns the existing materialization rather than silently duplicating it.
- 3.Change the scope intentionally when you need a distinct artifact.
Expected evidence
- ✓ A reproducible ingest identity instead of duplicate unnamed series.
Trust check
A changed display name does not necessarily change the normalized data scope.
Use Data Shop inputs in a study without overclaiming them
Data Shop evidence calibrates or builds the model; the subsequent solve is still a separate modeled result.
Do this
- 1.Keep the source and ingest receipt beside the scenario result IDs.
- 2.Describe transformations between the Data Shop unit and the solver parameter.
- 3.Separate observed-source discussion from derived inputs and modeled outputs in the final table.
Expected evidence
- ✓ A result table with a truth label on every row.
Trust check
Observed market evidence does not turn a dispatch result into a forecast of future market prices.
Benchmark the joined dataset before solving
Use descriptive and counterfactual benchmarks to detect a bad join or normalization before it influences model results.
Do this
- 1.Calculate demand peak, mean, load factor and p95; calculate IEX-band min, p50, p95 and maximum.
- 2.Inspect contemporaneous and ±1/±24-hour correlations as QA diagnostics, not causal claims.
- 3.Plot one high-demand week and one high-index week with missingness and source-window annotations.
- 4.Compare materialized energy with an independent aggregate when available.
- 5.Freeze the benchmark table and chart alongside the receipt and model ID.
Expected evidence
- ✓ Statistics another engineer or agent can recompute from the materialized series.
- ✓ No out-of-range index values, impossible load factors or unexplained temporal shifts.
How to read the result
- • A high lagged correlation may expose clock misalignment; it does not establish a forecasting relationship.
- • Benchmarks should be recomputed whenever the source window or transformation version changes.
| Variable | Statistics | Plot / test | Decision use |
|---|---|---|---|
| Demand MW | min, mean, p95, max, load factor | Chronology + duration curve | Capacity and energy scale |
| IEX index | min, p50, p95, max | Chronology; [0,1] bound | Relative market-stress calibration |
| Coverage | retained / expected hours | Gap heatmap | Whether full-horizon comparison is valid |
| Alignment | correlation at lag −24…+24 | Lag diagnostic | Detect clock shifts; not causal inference |
| Energy | ΣMW·h / 1,000 | Independent aggregate comparison | Preserve volume through transformation |
Trust check
A dataset becomes study-ready only after the receipt, structural QA and statistical benchmark all agree.
Recover from common Data Shop failures
Most failures come from plan gates, missing project/model targets, unsupported scopes or temporary materializer availability.
Do this
- 1.If cards are locked, confirm the organisation’s Enterprise plan.
- 2.If Attach is disabled, activate a model first.
- 3.If Ingest is disabled, complete every required state/date/resolution/target field.
- 4.If the service is unavailable, preserve the exact scope and retry; do not replace it with an undocumented nearby dataset.
Expected evidence
- ✓ A specific correction path without losing the intended source or scope.
Trust check
Availability problems are not permission to substitute a proxy while keeping the original dataset label.
Before you share
Completion checklist
- Dataset ID and card title are recorded.
- Source table, unit and lineage are recorded.
- State or national scope is explicit.
- From/to dates and resolution match the model’s intended horizon.
- New-model versus attach target is justified.
- The ingest receipt and target model ID are saved.
- Raw restricted prices are never represented as materialized output.
Answer desk
Frequently asked questions
Does the Data Shop expose raw IEX price data?
No. The IEX products materialize derived 0–1 price-band indices for calibration. Raw restricted ₹/MWh rows remain server-side.
Why is Attach disabled for an IEX price band?
IEX price bands are attach-only. Select an active model first so Modeler has a target for the series.
What date range does the current Data Shop declare?
The current catalogue validates pulls inside Indian FY26, from 1 April 2025 through 1 April 2026. Always read the live card because coverage can expand.
Can an agent reproduce a Data Shop ingest?
Yes when it is given the recorded dataset ID, normalized scope and target. The public guide is deliberately structured so agents can extract those fields; the product still re-validates requests server-side.