45 new variable statistics final energy by carriernatural gas

[maxnutz]

Goal

Statistics-function for variable Final Energy [by Carrier]|Natural Gas

Definition: Energy consumption of fossil methane excluding transmission/distribution losses

To Dos

• add a comprehensive docstring to your function
• add the mapping variable_name <> function_name to mapping.default.yaml (and your personal mapping-file)
• Add a testing routine for your Function to tests/ - stick to the testing-README
• make sure, that the newest version of main is merged into your feature Branch

Validation

variable	PyPSA-output	IAMC Energy Balance	🟢 🟡 🟠 🔴
Final Energy [by Carrier|Natural Gas	221754	196259	🟠 too high

• carrier gas for industry includes non-energetic use of gas in industry-sector > needs ratio to substract

Summary by Sourcery

Add a statistics function to compute final energy consumption from fossil natural gas and integrate it into the variable mapping and unit handling.

New Features:

• Introduce Final_Energy_by_Carrier__Natural_Gas statistics function to calculate final energy consumption from fossil natural gas across sectors.

Enhancements:

• Extend unit mapping to explicitly support MWh unit strings in statistics results.

Tests:

• Add a dedicated test suite covering typical and edge-case scenarios for the Final_Energy_by_Carrier__Natural_Gas statistics function.

[sourcery-ai]

Reviewer's Guide

Adds a new statistics function and corresponding tests and mappings to compute IAMC variable 'Final Energy [by Carrier]|Natural Gas', including logic to exclude pipeline gas flows, separate residential/commercial and industry gas demand, and adjust for non-fossil gas shares.

Flow diagram for Final_Energy_by_Carrier__Natural_Gas statistics function

flowchart TD
    A[Call Final_Energy_by_Carrier__Natural_Gas] --> B[Compute non_fossil_gas_prod using n.statistics.supply on bus_carrier gas and non fossil gas carriers]
    B --> C[Compute all_gas using n.statistics.withdrawal for bus_carrier gas and components Link]
    C --> D[Build forbidden_pattern for forbidden_parts pipeline]
    D --> E[Filter out pipeline related carriers from all_gas to get total_gas_usage]
    E --> F[Group total_gas_usage by kwargs.groupby and sum]
    F --> G[Compute non_fossil_fraction = non_fossil_gas_prod divided by total_gas_usage]
    G --> H[Replace inf values in non_fossil_fraction with NaN]
    H --> I[Clip non_fossil_fraction at upper bound 1]
    I --> J[Group non_fossil_fraction by kwargs.groupby and take mean]
    J --> K[Rename index of non_fossil_fraction using UNITS_MAPPING]

    A --> L[Compute rescom withdrawal for bus_carrier gas and residential commercial gas boiler carriers]
    A --> M[Compute industry withdrawal for bus_carrier gas for industry and industry carriers]

    L --> N[Assemble series_list = rescom and industry]
    M --> N
    N --> O[Remove empty series from series_list]
    O --> P{series_list is empty}

    P -->|Yes| Q[Return empty pd.Series with MultiIndex using kwargs.groupby]
    P -->|No| R[Concatenate series_list into total]

    R --> S[Rename index of total using UNITS_MAPPING and group by kwargs.groupby and sum]
    S --> T[Align non_fossil_fraction to total using reindex_like and fill NaN with 0]
    T --> U[Compute result = total multiplied by 1 minus non_fossil_fraction]
    U --> V[Return result]

Loading

File-Level Changes

Change	Details	Files
Introduce Final_Energy_by_Carrier__Natural_Gas statistics function to compute fossil natural gas final energy excluding pipeline flows and accounting for renewable gas share.	Implement supply-based calculation of non-fossil gas production per region via statistics.supply with specific renewable gas carriers. Compute total gas usage from statistics.withdrawal, filter out pipeline-related carriers using a regex mask, and aggregate by the configured groupby index. Derive a non-fossil gas fraction, sanitize infinities, cap at 1, aggregate by groupby, and align it to demand totals. Collect residential/commercial gas demand from specific gas boiler carriers and industrial gas demand from gas-for-industry loads, then sum and adjust by (1 - non_fossil_fraction) to obtain fossil natural gas final energy. Return an empty MultiIndex series with the expected index structure when there is no relevant gas demand.	pypsa_validation_processing/statistics_functions.py
Register the new natural gas statistics function in the IAMC variable mapping and ensure unit renaming covers plain MWh.	Add mapping for 'Final Energy [by Carrier]	Natural Gas' to Final_Energy_by_Carrier__Natural_Gas in the default config mapping file. Extend UNITS_MAPPING to include identity mapping from 'MWh' to 'MWh' to support new statistics index renaming.
Add a dedicated test suite for Final_Energy_by_Carrier__Natural_Gas using a minimal natural-gas statistics accessor and network stub to validate core behavior and edge cases.	Create a _NaturalGasStatisticsAccessor helper that simulates supply and withdrawal for different gas scenarios (mixed, no_gas, no_fossil, no_renewable). Add tests that validate output type and index structure of the new function. Add behavioral tests to ensure pipeline flows are excluded from the denominator used to compute the non-fossil share and confirm correct numerical result for the mixed scenario. Add edge-case tests for scenarios with no gas usage, only renewable gas, and only fossil gas, asserting empty series, zero result, and full demand attribution respectively.	tests/test_statistics_functions.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#45	Implement a new statistics function Final_Energy_by_Carrier__Natural_Gas in statistics_functions.py (including a docstring) to calculate the IAMC variable Final Energy [by Carrier]|Natural Gas as defined.	✅
#45	Register the new variable-function mapping for Final Energy [by Carrier]|Natural Gas in the default mapping configuration file.	✅
#45	Add tests under tests/ to validate the behavior of Final_Energy_by_Carrier__Natural_Gas.	✅

Possibly linked issues

• New Variable Statistics: Final Energy [by Carrier]|Natural Gas #45: The PR implements the Natural Gas final energy statistics function, adds its mapping, and introduces targeted tests as requested.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.