Skip to content

Custom Metric

In case the monitoring metrics provided by Aporia are insufficient for your use-case, you can define a metric based on custom logic.

Detection Methods

For this monitor type, you can select the following detection methods:

Custom Metric Detection Methods

  • Absolute Values - The metric value is lower or higher than a specific value.
  • Anomaly Detection - Detects anomalies in the value of the metric in the inspected data and its value in a time period before the data was collected.
  • Change In Percentage - Detects change in the ratio between the metric value of the inspected data and its value in a time period before the data was collected.

Configuration

Choose a custom metric that you defined previously, or create a new one

Custom Metric Configuration

Custom Metric Definition Language

Custom metrics are defined using a syntax that is similar to Python code.

Custom Metric Definition

A custom metric is a single expression, which can be any of the following:

  • A constant, numeric value (e.g. 5)
  • A function call - the currently available functions are sum and prediction_count
  • The sum function must receive a single parameter, which must be a valid field expression
    • The field expression must specify a field with a numeric value
    • The function returns a sum aggregation on the specified field
  • The prediction_count function can receive up to one parameter, which must be a valid segment filter
    • If no parameters were passed, the function returns the total number of predictions reported by the model
    • If a segment filter was passed, the function returns the number of predictions that fall within the data segment described by the filter.
  • A binary operation (+, -, *, /, **), in which the operands are constants or function calls (e.g. sum(features.Annual_Premium) / 100)

Field Expressions

A field expression describes a single field (feature, raw_input, prediction value or actual value), and allows operations on the values of that field across all reported predictions, or within a specific data segment

Field expressions have the following format: <field_category>.<field_name>[<segment_filter>]:

  • field_category is one of features, predictions, raw_inputs and actuals.
  • The available field categories and fields are based on the schema you defined when creating a model version.
  • The actuals category uses the fields defined for the predictions category
  • The segment filter is optional

Data Segment Filters

A data segment filter is an expression that restricts the results of an action to a specific data segment.

Segment filters are boolean expressions - each boolean condition in a segment filter is a comparison between a field and a constant value, and conditions can be combined using and, or.

The following comparisons are possible:

  • All fields can be checked for missing values using is None and is not None - for example: features.Annual_Premium is not None
  • Categorical, Boolean and String features can only be compared to constants using the == and != operators
  • Numeric fields can be compared to constants using the <, <=, >, >=, == and != operators
  • Datetime Fields are currently not supported

Examples

sum(features.Annual_Premium[features.Driving_License == True]) / prediction_count()
prediction_count(raw_inputs.Age <= 35 and raw_inputs.Region_Code == 28) * 3
prediction_count(features.Age > 27) / (sum(features.Annual_Premium) + sum(features.Vintage))