oy-scrape-jitter

usage: oy-scrape-jitter [<flags>]

Flags:
  -h, --help                 Show context-sensitive help (also try --help-long
                             and --help-man).
      --metric="up"          Metric to use to determine jitter.
      --query.timestamp=QUERY.TIMESTAMP  
                             Timestamp of the query.
      --plot.file=file.png   Path to a file to write an image of the results.
      --plot.log-y           Use logarithmic Y axis.
      --lookback=1h          How much time to look in the past for scrapes.
      --divisor=1s           Divisor to use to determine if a scrape is aligned.
      --log.unaligned-only   Only take unaligned targets in logging.
      --plot.unaligned-only  Only take unaligned targets in plot.
      --log.results-only     Only log final result.
      --prometheus.url="http://127.0.0.1:9090"  
                             URL of the Prometheus server.
      --client.config=CLIENT.CONFIG  
                             Path to a HTTP client configuration.
      --log.level=info       Only log messages with the given severity or above.
                             One of: [debug, info, warn, error]
      --log.format=logfmt    Output format of log messages. One of: [logfmt,
                             json]
      --version              Show application version.

oy-scrape-jitter queries a Prometheus server to see how regular scrapes are. Perfect scrape alignment happens when the distance between all the scrapes are exactly the same. It enables Prometheus delta-of-delta encoding to reduce significantly the size of the blocks.

$ ./oy-scrape-jitter --prometheus.url=https://prometheus.demo.do.prometheus.io/ --log.results-only
ts=2022-04-26T12:11:34.659Z caller=main.go:117 level=info msg="overall results" aligned_targets=0 unaligned_targets=10 max_ms=25

This means that the maximum deviation seen in your scrape jobs is 25ms. You could set --scrape.timestamp-tolerance=25ms to reduce your disk usage over time, by enabling Prometheus to correct timestamps up to 25ms.

Prometheus will only apply timestamp tolerance up to 1%. If your scrape interval is 30s, you can only adjust timestamps up to 300ms. Setting a 500ms tolerance will have no effects on jobs with a scrape interval lower than 500s, even if the deviation is tiny.

By using --plot.file=scrape.png, you can generate a PNG file which shows the scrape timestamps jitter with an histogram.

wget https://github.com/o11ydev/oy-toolkit/releases/download/main/oy-scrape-jitter -O oy-scrape-jitter && chmod +x oy-scrape-jitter && ./oy-scrape-jitter --help

docker run quay.io/o11y/oy-toolkit:oy-scrape-jitter --help

nix run github:o11ydev/oy-toolkit#oy-scrape-jitter -- --help