Merge pull request #23 from OSeMOSYS/refactor_filter

Author: willu47

.github/workflows/pytest.yml

@@ -32,11 +32,19 @@ jobs:
         python-version: ${{ matrix.python-version }}
 
     - name: Install pytest
-      run: pip install pytest
-           pip install pytest-cov
+      run: |
+        pip install pytest
+        pip install pytest-cov
 
     - name: Install the package
-      run: pip install .
+      run: |
+        pip install .
+
+
+    - name: Install editable openentrance
+      run: |
+        pip uninstall --yes openentrance
+        pip install -e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance
 
     - name: Test with pytest
       run: pytest

.gitignore

@@ -58,4 +58,7 @@ MANIFEST
 
 # Outputs
 *.csv
-!tests/fixtures/*.csv
+!tests/fixtures/*.csv
+
+# Dependencies
+src/openentrance

LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2022 Will Usher
+Copyright (c) 2022 Will Usher, Hauke Henke
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -4,13 +4,21 @@ Convert OSeMOSYS results to IAMC format
 
 ## Install from Github repository
 
-    pip install git+https://github.com/osesmosys/osemosys2iamc@main#egg=osemosys2iamc
+It is currently necessary to install the OpenEntrance dependency as an editable installation.
+See [issue](https://github.com/openENTRANCE/openentrance/issues/202)
 
-## Run the demo
+    pip install git+https://github.com/osemosys/osemosys2iamc@main#egg=osemosys2iamc
+    pip install -e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance
 
-    osemosys2iamc results config.yaml  test.csv
+## Run the package
 
-Check out the IAMC formatted results in `test.csv` and plots `emissions.csv`
+    $ osemosys2iamc --help
+    osemosys2iamc <inputs_path> <results_path> <config_path> <output_path>
+
+`inputs_path`: Path to a folder of csv files (OSeMOSYS inputs). File names should correspond to OSeMOSYS parameter names.
+`results_path`: Path to a folder of csv files (OSeMOSYS results). File names should correspond to OSeMOSYS variable names.
+`config_path`: Path to the configuration file (see below)
+`output_path`: Path to the .xlsx file you wish to write out
 
 ## The IAMC format
 
@@ -30,7 +38,70 @@ in the contribution by Working Group III to the IPCC's Sixth Assessment Report (
 Please refer to the Horizon 2020 project [openENTRANCE](https://github.com/openENTRANCE/openentrance#data-format-structure)
 for more information about the format and its usage in that project.
 
-## Usage for OSeMOSYS
+## Writing a configuration file
+
+Write a configuration file in YAML format. A simple configuration file with one result variable looks like this:
+
+    model: OSeMBE v1.0.0
+    scenario: DIAG-C400-lin-ResidualFossil
+    region: ['Austria']  # select countries to plot summary results
+    results:
+    - iamc_variable: 'Carbon Capture|Biomass'
+      tech_emi: ['(?=^.{2}(BM))^.{4}(CS)']
+      emissions: [CO2]
+      unit: kt CO2/yr
+      transform: abs
+    osemosys_param: AnnualTechnologyEmissions
+
+The first section of the configuration file with the keys `model`, `scenario`, and `region` are used to define the metadata for
+the IAMC template.
+
+The second section `results:` is where you describe each of the IAMC variables and provide instructions to osemosys2iamc on how
+to compute the values.
+
+`iamc_variable` - this should match one of the IAMC variable names
+`unit` - provide the units of the OSeMOSYS results
+`transform` - only `abs` is currently available. This returns the absolute value of the results
+`osemosys_param` - provide the name of the result file from which the script should extract the result data
+
+One or more of the following filter keys. These filter the
+results by one or more columns found in OSeMOSYS results.
+Following the fitering, the remaining columns except region
+ and year are discarded and rows are summed.
+
+`tech_emi` - filter the results by TECHNOLOGY and EMISSION columns using the provide regular expression and an `emissions` entry
+`emissions` - a list of emissions to filter the results by the EMISSION column
+`fuel` - filter by the FUEL column
+`capacity` - filter the TECHNOLOGY column
+`primary_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key)
+`excluded_prod_tech` - filter the TECHNOLOGY column (can be replaced by `capacity` key)
+`el_prod_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key)
+`demand` - filters by the FUEL column (final energy)
+
+The value for each of these keys is a list of regular expressions. These regular expressions are used to filter the rows of data in the chosen column to those that match the
+regular expression.
+
+Writing regular expressions can be tricky, but there are [useful tools](https://regexr.com/) to help.
+Below we provide some examples:
+
+`^.{2}(WI)` match rows with any two characters followed by `WI`
+
+`(?=^.{2}(HF))^((?!00).)*$` match rows with `HF` in 3rd and 4th position which do not include `00`
+
+`(?=^.{2}(NG))^((?!(00)|(CS)).)*$` match rows with `NG` in 3rd and 4th position that do not include `00` or `CS`
+
+`^.{6}(I0)` match rows which contain any 6 characters followed by `IO` in the 7th and 8th position
+
+Putting this together, the following entry extracts results from the result file `ProductionByTechnologyAnnual.csv`, filters out the rows
+by matching values in the TECHNOLOGY column with a list of 6 regular expressions (this is an OR operation)
+and assigns the unit `PJ/yr` and adds the aggregated (summing over region and year) total under `Primary Energy` in the IAMC template format.
+
+    - iamc_variable: 'Primary Energy'
+      primary_technology: ['^.{6}(I0)','^.{6}(X0)','^.{2}(HY)','^.{2}(OC)','^.{2}(SO)','^.{2}(WI)']
+      unit: PJ/yr
+      osemosys_param: ProductionByTechnologyAnnual
+
+## List of relevant IAMC variables for OSeMOSYS
 
 ### Primary Energy
 

requirements.txt

@@ -1,3 +1,4 @@
 matplotlib
 pandas
 pyam-iamc >= 1.0  # the pyam package is released on pypi under this name
+-e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance