oy-periodic-queries

usage: oy-periodic-queries [<flags>] <rule-file>...


Flags:
  -h, --[no-]help                Show context-sensitive help (also try
                                 --help-long and --help-man).
      --[no-]web.systemd-socket  Use systemd socket activation listeners instead
                                 of port listeners (Linux only).
      --web.listen-address=:9099 ...  
                                 Addresses on which to expose metrics and web
                                 interface. Repeatable for multiple addresses.
      --web.config.file=""       Path to configuration file that can
                                 enable TLS or authentication. See:
                                 https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md
      --web.telemetry-path="/metrics"  
                                 Path under which to expose metrics.
      --[no-]web.disable-exporter-metrics  
                                 Exclude metrics about the exporter itself
                                 (promhttp_*, process_*, go_*).
      --[no-]prefetch            Cache metrics before exposing them. Avoid
                                 scrape timeout.
      --prometheus.url="http://127.0.0.1:9090"  
                                 URL of the Prometheus server.
      --client.config=CLIENT.CONFIG  
                                 Path to a HTTP client configuration.
      --log.level=info           Only log messages with the given severity or
                                 above. One of: [debug, info, warn, error]
      --log.format=logfmt        Output format of log messages. One of: [logfmt,
                                 json]
      --[no-]version             Show application version.

Args:
  <rule-file>  File to read periodic recording rules from.

oy-periodic-queries is a tool that allows you to evaluate Prometheus recording rules and export the results as metrics, with defined boundaries such as monthly or weekly (Only monthly is implemented at the moment).

It uses calendar months.

With this exporter, you can easily calculate monthly resource usage, or any other metric that requires periodic evaluation. Simply specify the recording rule and time boundaries, and the exporter will run the rule and provide you with accurate, timely metrics.

In the recording rule and labels, you have access to the following variables, that will be replaced:

Other values in ${} are formatted with Go’s Time.Format, e.g. "${2006-01}" is turned into the month and day of the end of the range. Prepend _ to have it relative to the start of the range: "${_2006-01}".

The syntax of a rule file is:

groups:
  [ - <rule_group> ]

# The name of the group. Must be unique within a file.
name: <string>

# The time period covered by the metrics.
time_period: [ <string> | default = "monthly" ]

# The maximum time to look back.
lookback: [ <duration> | default = "12w" ]

rules:
  [ - <rule> ... ]

The syntax for recording rules is:

# The name of the time series to output to. Must be a valid metric name.
record: <string>

# The PromQL expression to evaluate. Every evaluation cycle this is
# evaluated at the current time, and the result recorded as a new set of
# time series with the metric name as given by 'record'.
expr: <string>

# Labels to add or overwrite before storing the result.
labels:
  [ <labelname>: <labelvalue> ]

A simple example rules file would be:

groups:
- name: prometheus network usage
  time_period: monthly
  lookback: 365d
  include_incomplete_ranges: true
  rules:
    - expr: |
        increase(node_network_receive_bytes_total{instance=~"prometheus.*"}[$RANGE])
        and last_over_time(node_network_receive_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${1_PC_TIMESTAMP})
        and last_over_time(node_network_receive_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${100_PC_TIMESTAMP})
      record: prometheus:node_network_receive_bytes_total:monthly
      labels:
        month: "${2006-01}"
    - expr: |
        increase(node_network_transmit_bytes_total{instance=~"prometheus.*"}[$RANGE])
        and last_over_time(node_network_transmit_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${1_PC_TIMESTAMP})
        and last_over_time(node_network_transmit_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${100_PC_TIMESTAMP})
      record: prometheus:node_network_transmit_bytes_total:monthly
      labels:
        month: "${2006-01}"

In the queries, the following part is used to only select the metrics that existed in the beginning and the end of the different months.

        and last_over_time(node_network_transmit_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${1_PC_TIMESTAMP})
        and last_over_time(node_network_transmit_bytes_total{instance=~"prometheus.*"}[${1_PC_RANGE}] @ ${100_PC_TIMESTAMP})

Output of /metrics, when launched with --web.disable-exporter-metrics:

# HELP periodic_rule_queries_success 
# TYPE periodic_rule_queries_success gauge
periodic_rule_queries_success 1
# HELP prometheus:node_network_receive_bytes_total:monthly 
# TYPE prometheus:node_network_receive_bytes_total:monthly untyped
prometheus:node_network_receive_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 2.2096172024710458e+11
prometheus:node_network_receive_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 1.4455463702554254e+11
prometheus:node_network_receive_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 9.635369117702129e+08
prometheus:node_network_receive_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 2.4574080228794675e+09
# HELP prometheus:node_network_transmit_bytes_total:monthly 
# TYPE prometheus:node_network_transmit_bytes_total:monthly untyped
prometheus:node_network_transmit_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 1.8763866363794147e+11
prometheus:node_network_transmit_bytes_total:monthly{device="ens3",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 3.813239344424051e+11
prometheus:node_network_transmit_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus02.example.com",job="node",month="2023-01"} 9.635369117702129e+08
prometheus:node_network_transmit_bytes_total:monthly{device="lo",environment="o11ylab",instance="prometheus11.example.com",job="node",month="2023-01"} 2.4574080228794675e+09

wget https://github.com/o11ydev/oy-toolkit/releases/download/main/oy-periodic-queries -O oy-periodic-queries && chmod +x oy-periodic-queries && ./oy-periodic-queries --help

docker run quay.io/o11y/oy-toolkit:oy-periodic-queries --help

nix run github:o11ydev/oy-toolkit#oy-periodic-queries -- --help