Add steady-state tolerance probes for better functionality

[shyimo]

Hi guys,
At the moment it seems that we are pretty limited when query prometheus on steady-state.
we are using jsonpath tolerance type in order to verify our promQL result.
for example:

{
    "type": "probe",
    "name": "prometheus steady state  consumer rate query - 1 min ago",
    "tolerance": {
        "expect": "1",
        "type": "jsonpath",
        "path": "$.data.result[0].value[1]"
    },
    "provider": {
        "type": "python",
        "module": "chaosprometheus.probes",
        "func": "query",
        "arguments": {
            "query": "${prometheus_ts_consumer_rate_query}",
            "when": "1 minutes ago"
        }
    }
}

this type of query can only be boolean and we need a way to do more in depth verification on the results. for example:

• checks if the Prometheus results are below a given threshold
• checks if Prometheus results are higher than a given threshold

this features will able prometheus probes to be much more flexible :)

[zumic96]

I know it's few years later but I have just hit the same issue and got some basic function created so maybe it helps someone working with this in the future.
Saved this in my_tolerances.py and added path to PYTHONPATH.

def is_metric_under_threshold(threshold: float, value: dict = None):
  """
  `value` is expected to be the JSON dict from Prometheus:
  {
	"status": "success",
	"data": {
        "resultType": "vector",
        "result": [
            { "metric": { … }, "value": [ <timestamp>, "<latency_str>" ] }
        ]
    }
  }
  """

  # Basic sanity checks
  if not value:
    return False

  # If Prometheus returned error in status, reject
  status = value.get("status")
  if status != "success":
    return False

  data = value.get("data", {})
  result = data.get("result", [])
  if not result:
    return False

  # We take only the first element
  first = result[0]
  val_list = first.get("value")
  if not (isinstance(val_list, list) and len(val_list) >= 2):
    return False

  metric_str = val_list[1]
  try:
    metric = float(metric_str)
  except (TypeError, ValueError):
    return False

  # Compare
  return (metric < threshold)

Then use it in experiment's steady state hypothesis like this.

"steady-state-hypothesis": {
   "title":"Service is healthy",
   "probes":[
      {
         "type":"probe",
         "name":"httpbin-latency-p95",
         "provider":{
            "type":"python",
            "module":"chaosprometheus.probes",
            "func":"query",
            "arguments":{
               "query":"histogram_quantile(0.95, sum(rate(istio_request_duration_milliseconds_bucket{destination_workload=\"httpbin\"}[1m])) by (le))"
            }
         },
         "tolerance":{
            "type":"probe",
            "name":"check-p95-under-1000",
            "provider":{
               "type":"python",
               "module":"my_tolerances",
               "func":"is_metric_under_threshold",
               "arguments":{
                  "threshold":1000
               }
            }
         }
      }
   ]
}