Skip to main content

Monitors

Monitors allow you to define metric queries that trigger alerts based on specific conditions. The system periodically evaluates these queries and generates alerts when the results violate the defined conditions.

Key Components of a Monitor​

Alert Name​

A descriptive name that identifies the monitor. You can use template variables in the name using Prometheus templates syntax.

Query​

A PromQL query that the system evaluates periodically. You can use any valid PromQL query syntax.

Conditions​

Conditions specify the threshold values that trigger alerts. There are two severity levels:

  • Critical
  • Warning

You must define at least one condition, and you can set different thresholds for each severity level. The following operators are available for conditions:

  • Equal to
  • Not equal to
  • Above
  • Above or Equal to
  • Below
  • Below or Equal to

Message​

A customizable message that accompanies alert notifications. You can enhance messages using template variables with Prometheus templates syntax.

For example, with a query like max by (job) (up), you can access the job name in your message using the template variable {{ $labels.job }}.

note

Oodle notifications automatically include the following information, so you don't need to add them to your message:

  • Alert name
  • Severity
  • Threshold
  • Metric Value (the current value when the alert fires)
  • Labels and Annotations

Notification Policy​

Notification policies determine where alert notifications are sent. You can configure multiple notifiers as destinations for your alerts.

Monitors support sophisticated routing logic based on both alert conditions and query labels:

  • Label-based routing: Route alerts to different notifiers based on their labels. For example, send alerts with environment: production to Opsgenie and alerts with environment: dev to Slack.
  • Severity-based routing: Route alerts to different notifiers based on their severity level. For example, send Critical alerts to different destinations than Warning alerts.

Grouping​

Grouping controls how notifications are consolidated. You can choose from three options:

  • By Monitor: Receive a single notification per monitor (default behavior)
  • By Labels: Receive a notification for each unique set of specified labels
  • Disabled: Receive a notification for each individual timeseries

For example, consider a monitor with this query that alerts when the up metric for any job is 0:

max by (job, instance) (up{})

Here's how different grouping options affect notifications:

  • By Monitor: You receive one notification that includes all affected jobs and instances
  • By Labels (with job specified): You receive one notification per affected job
  • Disabled: You receive one notification per affected timeseries (job-instance combination)

Notification Timings​

These parameters control the timing of grouped notifications:

Group Wait​

Group Wait defines how long to wait before sending the first notification for a new group of alerts. The default is 30 seconds.

Group Interval​

Group Interval defines how long to wait before sending a notification for new alerts added to a group that has already sent notifications. The default is 5 minutes.

Repeat Interval​

The repeat interval specifies how often notifications are resent for active alerts. For example, if you have an active alert for the timeseries up with job=foo and instance=bar, setting a repeat interval of 10 minutes means the notification will be sent every 10 minutes until the alert is resolved.

info

For example, with the up metric monitor grouped "By Monitor", consider a group wait of 30 seconds and a group interval of 5 minutes: If two jobs go down within the first 30 seconds, you'll receive one notification covering both jobs. If a third job goes down a minute later, its notification will be delayed by the 5-minute group interval.

Labels​

Labels add metadata to alerts. You can use Prometheus templates syntax to create dynamic labels.

A common use case is tagging monitors with owner teams to identify responsible teams for each monitor.

Annotations​

Annotations provide additional context for alerts, such as descriptions or runbook links. Like labels, annotations support Prometheus templates syntax.

Creating a Monitor​

  1. Navigate to Alerts (Bell icon in sidebar)
  2. Click New Alert
  3. Configure the following settings:
    • Name: Enter a descriptive name for your monitor
    • Query: Specify the PromQL query to evaluate
    • Conditions: Define the alerting conditions
    • Message: (Optional) Add a message for alert notifications
    • Notification Policy: (Optional) Select routing rules for alerts
    • Grouping: (Optional) Configure how you want to group alerts. Additionally, you can configure various timing parameters by clicking on the Clock icon.
    • Labels: (Optional) Add metadata labels to the alerts
    • Annotations: (Optional) Include additional contextual information
  4. Click Save to create the monitor

Support​

If you have any questions or need any assistance, please contact us via our help chat app located at the bottom-right of the page or by reaching out to support@oodle.ai.