Logical draft of how the metric processors shall work

[ArneTR]

This PR outlines the concept of introducing two new stages to the GMT:

• Post Metric Processors
• Post Phases Stat Processors

They run in the pipeline as follows:
Metric Providers -> Post Metric Processors -> Phase Stats -> Post Phase Stats Processors

The already commited config.yml.example outlines how they are defined in the config.

Simplifications and Domain Logic

This PR simplifies the mental load on where processing happens in the GMT and makes it more explicit and more modular.

• The phase_stats.py will then really only create avg and aggregate values of metric data
• SCI will not be all over the place but in a separate stage. Making it also more easy to get SCI values per phase
• Overloaded Providers like XGBoost and SDIA are then in their separate stage. Not having the need to be processed last in the metric provider stage and loading in "hopefully" generated values by other providers ... which is currently only by accident :)

When we implement this approach the following refactorings need to be done:

• Rip out the calculate_co2_intensity from the phase_stats.py and bring it to a Post Metric Processor (

  green-metrics-tool/lib/phase_stats.py

  Line 81 in d0a6215

  def calculate_co2_intensity(run_id):

  )
• Migrate the XGBoost and SDIA providers to be Post Metric Processors
• Migrate the SCI generation code from the phase_stats and bring it to the Post Phases Stats Processors
• Look at https://if.greensoftware.foundation/pipelines/ to see if we can take some good ideas from how IF thinks about pipelines. We want it less complex. But still might find some good work there

[ArneTR]

@ribalba Please add what you think we also need

[ribalba]

Perfect. Is what we discussed. The more I think about this the more I like how clean it separates concerns.

[ribalba]

Why are you creating a new key carbon_input? Wouldn't this just be another input?

[ribalba]

Doesn't the SDIA output a linear model and not XGBoost?

[ribalba]

I would still add a processor script here that then has the SCI values as parameters. Like with metric providers.

[ribalba]

I would make clear that they are called in order they are defined here. This is important if a "carbon" processor relies on an "energy" processor.