Metric#

The ngx_http_metric_module module allows creating arbitrary real-time calculated metrics. These metric values are stored in shared memory and displayed in real-time within the /status/http/metric_zones/ API branch. Various data aggregation types are supported (counters, histograms, moving averages, etc.) with grouping by arbitrary keys.

Configuration Example#

Counting API requests:

http {
    metric_zone api_requests:1m count;

    server {
        listen 80;

        location /api/ {
            allow 127.0.0.1;
            deny all;
            api /;

            metric api_requests $http_user_agent on=request;
        }
    }
}

If a request is made to /api/ with this configuration:

$ curl 127.0.0.1/api/ --user-agent "Firefox"

The api_requests metric is updated in real-time:

{
    "status": {
        "http": {
            "metric_zones": {
                "api_requests": {
                    "discarded": 0,
                    "metrics": {
                        "Firefox": 1
                    }
                }
            }
        }
    }
}

Directives#

metric_zone#

Syntax

metric_zone name:size [expire=on|:samp:off] [discard_key=name] mode [parameters];

Default

—

Context

http

Creates a shared memory zone of the specified size with the given name to store metrics. The zone name serves as a node in the /status/http/metric_zones/ branch.

Parameters:

  • expire=<on|off> — behavior when the zone is full:

    • If on, the oldest metrics (by update time) are discarded to

      free memory for new ones;

    • If off (default) — new incoming metrics are discarded,

      preserving existing entries.

  • discard_key=<name> — defines a metric with the key name

    where values of discarded metrics are accumulated. By default, no such metric is created. Reserved key cannot be updated manually.

  • mode — data processing algorithm (see the Operation Modes section);

  • parameters — additional settings for the selected mode

    (e.g., factor for average exp).

Example usage:

metric_zone request_time:1m max;

In the API tree, the shared memory zone template looks as follows:

{
    "discarded": 0,
    "metrics": {
        "key1": 123,
        "key2": 10.5,
    }
}

discarded

Number; the count of discarded metrics in the shared memory zone

metrics

Object; its members are metrics with defined keys and calculated values

Note

In a 1 MB zone, with a key size of 39 bytes and a single metric mode, approximately 8,000 unique key entries can be stored.

metric_complex_zone#

Syntax

metric_complex_zone name:size [expire=on|:samp:off] [discard_key=name] { ... }

Default

—

Context

http

Defines a complex metric — a set of metrics with independent modes. Each line in the block body defines a submetric name, a mode, and optional mode parameters.

Example usage:

metric_complex_zone requests:1m expire=on discard_key="old" {
    # submetric name   mode          parameters
    min_time           min;
    avg_time           average exp   factor=60;
    max_time           max;
    total              count;
}

In the API tree, such a complex metric template looks as follows:

{
    "discarded": 3,
    "metrics": {
        "key1": {
            "min_time": 20,
            "avg_time": 50,
            "max_time": 80,
            "total" 2
        },
        "old": {
             "min_time": 3,
             "avg_time": 40,
             "max_time": 152,
             "total": 80
        }
    }
}

discarded

Number; the count of discarded metrics in the shared memory zone

metrics

Object; its members are complex metrics with set keys. They are objects containing a set of submetrics with calculated values

metric#

Syntax

metric name key=value [on=request| response| end];

Default

—

Context

http, server, location

Calculates the value of the metric for the specified shared memory zone name.

Parameters:

  • key — an arbitrary string (often a variable) used for grouping values.

    Maximum length is 255 bytes. If the key is longer, it will be truncated to 255 bytes and appended with an ellipsis ...;

  • value — a number (can be a variable) processed by the selected mode.

    If omitted, it defaults to 0. If the parameter cannot be converted to a number, it defaults to 1;

  • on — an optional parameter specifying when the metric is calculated:

    • If on=request, calculation occurs when the request is received;

    • If on=response, calculation occurs during response preparation;

    • If on=end (default), calculation occurs after the response is sent.

Note

In the case of internal redirection, metrics at the on=request stage are calculated in the original location. However, on=response and on=end metrics will be calculated in the new location.

Example usage:

metric requests $http_user_agent=$request_time;

Note

Metrics with an empty key or an invalid key=value pair are ignored. An omitted value is treated as 0:

metric foo $bar;  # Equivalent to $bar=0

This is useful, for example, for the count mode, which ignores numerical values and simply reacts to the fact that a metric was updated.

Note

Remember that variables are evaluated in different phases. For example, it is impossible to use $bytes_sent (bytes sent to the client) with on=request (when the request is received).

Operation Modes#

List of available metric operation modes:

  • count — counter;

  • gauge — gauge (increment/decrement);

  • last — the last received value;

  • min — minimum value;

  • max — maximum value;

  • average exp — exponential moving average (EMA) (parameter factor);

  • average mean — mean over a window (parameters window and count);

  • histogram — distribution across "buckets" (a list of threshold values).

count#

The counter increases its value by 1 with every metric update.

Default value — 0.

Note

Any metric update (with any value) monotonically increases the counter by 1.

Examples:

metric_zone count:1m count;

# As part of a complex metric:
#
# metric_complex_zone count:1m {
#     some_metric_name  count;
# }

server {
    listen 80;

    location /metric/ {
        metric count KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric count KEY=1;
    }

    location /api/ {
        api /status/http/metric_zones/count/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/set/23
$ curl 127.0.0.1/metric/set/-32

Expected metric value in the API:

{
    "KEY": 4
}

gauge#

The gauge increases or decreases its value depending on the sign of the passed number. A positive value increases the counter, while a negative value decreases it. A value of 0 does not change the counter.

Default value — 0.

Examples:

metric_zone gauge:1m gauge;

# As part of a complex metric:
#
# metric_complex_zone gauge:1m {
#     some_metric_name  gauge;
# }

server {
    listen 80;

    location /metric/ {
        metric gauge KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric gauge KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/gauge/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": 0
}

Further updates:

$ curl 127.0.0.1/metric/set/5
$ curl 127.0.0.1/metric/set/-5
$ curl 127.0.0.1/metric/set/8

Expected metric value in the API:

{
    "KEY": 8
}

last#

Stores the last received value without any aggregation. If value is omitted, 0 is used.

Examples:

metric_zone last:1m last;

# As part of a complex metric:
#
# metric_complex_zone last:1m {
#     some_metric_name  last;
# }

server {
    listen 80;

    location /metric/ {
        metric last KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric last KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/last/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": 0
}

Further updates:

$ curl 127.0.0.1/metric/set/8000
$ curl 127.0.0.1/metric/set/37
$ curl 127.0.0.1/metric/set/-3.5

Expected metric value in the API:

{
   "KEY": -3.5
}

min#

Saves the minimum of two values — the currently stored value and the new one.

Examples:

metric_zone min:1m min;

# As part of a complex metric:
#
# metric_complex_zone min:1m {
#     some_metric_name  min;
# }

server {
    listen 80;

    location /metric/ {
        metric min KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric min KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/min/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/set/42.999
$ curl 127.0.0.1/metric/set/-512
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": -512
}

max#

Saves the maximum of two values — the currently stored value and the new one.

Examples:

metric_zone max:1m max;

# As part of a complex metric:
#
# metric_complex_zone max:1m {
#     some_metric_name  max;
# }

server {
    listen 80;

    location /metric/ {
        metric max KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric max KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/max/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/set/42.999
$ curl 127.0.0.1/metric/set/-512
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": 42.999
}

average exp#

Calculates the average value using the exponential smoothing algorithm.

Accepts an optional parameter factor=<number> — a coefficient determining how much the new value influences the average. Integer values from 0 to 99 are allowed. Default is 90.

The higher the coefficient, the more weight new values have. If you specify 90, the result will be 90% of the new value and 10% of the previous average.

Examples:

metric_zone avg_exp:1m average exp factor=60;

# As part of a complex metric:
#
# metric_complex_zone avg_exp:1m {
#     some_metric_name  average exp  factor=60;
# }

server {
    listen 80;

    location ~ ^/metric/set/(.+)$ {
        metric avg_exp KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/avg_exp/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/set/100
$ curl 127.0.0.1/metric/set/200
$ curl 127.0.0.1/metric/set/0
$ curl 127.0.0.1/metric/set/8
$ curl 127.0.0.1/metric/set/30

Expected metric value in the API:

{
    "KEY": 30.16
}

average mean#

Calculates the arithmetic mean. Accepts optional parameters window=<off|time> and count=<number>, defining the time interval and sample size for averaging, respectively. Defaults: window=off (entire sample used) and count=10.

Note

For example, window=5s will only consider events from the last 5 seconds. The window parameter cannot be 0. The count=number parameter controls the sample size (cached values) for a smoother mean calculation.

Examples:

metric_zone avg_mean:1m average mean window=5s count=8;

# As part of a complex metric:
#
# metric_complex_zone avg_mean:1m {
#     some_metric_name  average mean  window=5s count=8;
# }

server {
    listen 80;

    location ~ ^/metric/set/(.+)$ {
        metric avg_mean KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/avg_mean/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/set/0.1
$ curl 127.0.0.1/metric/set/0.1
$ curl 127.0.0.1/metric/set/0.4
$ curl 127.0.0.1/metric/set/10
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/set/1

Expected metric value in the API:

{
    "KEY": 2.1
}

If you wait 5 seconds from the last update, the expected value will be:

{
    "KEY": 0
}

histogram#

Creates a set of "buckets," incrementing the relevant counter if the new value does not exceed the bucket threshold. Parameters are provided as a list of numerical thresholds. Useful for analyzing distributions, such as response times.

Mandatory parameters are numbers — the threshold values of the buckets, listed in ascending order.

Note

The bucket value inf or +Inf can be used to capture all values exceeding the highest specified bucket.

Examples:

metric_zone hist:1m histogram 0.1 0.2 0.5 1 2 inf;

# As part of a complex metric:
#
# metric_complex_zone hist:1m {
#     some_metric_name  histogram  0.1 0.2 0.5 1 2 inf;
# }

server {
    listen 80;

    location ~ ^/metric/set/(.+)$ {
        metric histogram KEY=$1;
    }

    location /api/ {
        api /status/http/metric_zones/hist/metrics/;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/set/0.25

Expected metric value in the API:

{
    "KEY": {
        "0.1": 0,
        "0.2": 0,
        "0.5": 1,
        "1": 1,
        "2": 1,
        "inf": 1
    }
}

Further updates:

$ curl 127.0.0.1/metric/set/2

Expected metric value in the API:

{
    "KEY": {
        "0.1": 0,
        "0.2": 0,
        "0.5": 1,
        "1": 1,
        "2": 2,
        "inf": 2
    }
}

Further update:

$ curl 127.0.0.1/metric/set/1000

Expected metric value in the API:

{
    "KEY": {
       "0.1": 0,
       "0.2": 0,
       "0.5": 1,
       "1": 1,
       "2": 2,
       "inf": 3
    }
}

Built-in Variables#

Variables are created for each metric:

  • $metric_<name>

  • $metric_<name>_key

  • $metric_<name>_value

For complex metrics, an additional variable is added:

  • $metric_<name>_value_<metric>

$metric_<name>#

Similar to the metric directive, the $metric_<name> variable setter can be used to update a metric. Calculation occurs during the Rewrite phase, allowing metric processing from the njs module, for example.

The value used for setting the variable must follow the key=value structure. Both key and value can consist of text, variables, and combinations thereof. The key is an arbitrary string for grouping values. The value is a number processed by the selected mode. If omitted, it defaults to 0. If the parameter cannot be converted to a number, it defaults to 1.

Example usage:

http {
    metric_zone counter:1m count;

    # At this point, the $metric_counter variable is added

    server {
        listen 80;

        location /metric/ {
            set $metric_counter $http_user_agent;  # Equivalent to $http_user_agent=0
        }

        location /api/ {
            allow 127.0.0.1;
            deny all;
            api /status/http/metric_zones/counter/;
        }
    }
}

Calculating metrics using the njs module:

http {
    js_import metrics.js;

    resolver 127.0.0.53;

    metric_complex_zone requests:1m {
        min_time        min;
        max_time        max;
        total           count;
    }

    location /metric/ {
        js_content metrics.js_request;
        js_fetch_trusted_certificate /path/to/ISRG_Root_X1.pem;
    }

    location /api/ {
        allow 127.0.0.1;
        deny all;
        api /static/http/metric_zones/requests/;
    }
}

File metrics.js:

async function js_request(r) {
    let start_time = Date.now();

    let results = await Promise.all([ngx.fetch('https://google.com/'),
                                     ngx.fetch('https://google.ru/')]);

    // Using the $metric_requests variable setter
    r.variables.metric_requests = `google={Date.now() - start_time}`;
}

export default {js_request};

After several requests to location /metric/, the values might look like this:

{
    "discarded": 0,
    "metrics": {
        "google": {
            "min_time": 70,
            "max_time": 432,
            "total": 6
        }
    }
}

Note

After setting the variable, you can retrieve its value; it will equal the specified key=value pair.

Additionally, the value stored in the $metric_<name>_key variable will change to the specified key.

$metric_<name>_key and $metric_<name>_value#

The $metric_<name>_key and $metric_<name>_value variables define the key and value respectively. The metric update occurs when the $metric_<name>_value is set, provided that the key in $metric_<name>_key has already been defined.

Note

For complex metrics, the submetric values in the $metric_<name>_value variable are joined using a ", " separator.

Example usage:

http {
    metric_zone gauge:1m gauge;

    # The variables $metric_gauge, $metric_gauge_key, and $metric_gauge_value are added here.

    metric_complex_zone complex:1m {
        hist histogram 1 2 3;
        avg  average exp;
    }

    # $metric_complex, $metric_complex_key, and $metric_complex_value are added here.

    server {
        listen 80;

        location /gauge/ {
            set $metric_gauge_key "foo";
            set $metric_gauge_value 1;

            # Or: set $metric_gauge foo=1;

            return 200 "Updated with '$metric_gauge'\nValue='$metric_gauge_value'\n";
        }

        location /complex/ {
            set $metric_complex_key "foo";
            set $metric_complex_value 3;

            # Or: set $metric_complex foo=3;

            return 200 "Updated with '$metric_complex'\nValue='$metric_complex_value'\n";
        }
    }
}

With this configuration, a request to /gauge/ yields:

$ curl 127.0.0.1/gauge/
Updated with 'foo=1'
Value='1'

For /complex/:

$ curl 127.0.0.1/complex/
Updated with 'foo=3'
Value='0 0 1, 3'

Note

If an empty string is assigned to $metric_<name>_value, the value is recognized as 0. If the string consists of characters that cannot be converted to a number, it is recognized as 1.

Calculation occurs only after both $metric_<name>_key and $metric_<name>_value have been set.

In this case, the value stored in $metric_<name> becomes equal to the new key=value pair.

The value in $metric_<name>_key represents the last key specified via variables.

The value in $metric_<name>_value represents the last calculated value for the key set in $metric_<name>_key.

$metric_<name>_value_<metric>#

For complex metrics, the value of a specific submetric can be retrieved using the $metric_<name>_value_<metric> variable, where <metric> is the name of the submetric.

Example usage:

http {
    metric_complex_zone foo:1m {
        count count;
        min   min;
        avg   average exp;
    }

    # Adds $metric_foo, $metric_foo_key, $metric_foo_value,
    # and $metric_foo_value_count, $metric_foo_value_min, $metric_foo_value_avg.

    server {
        listen 80;

        location /foo/ {
            set $metric_foo_key   bar;
            set $metric_foo_value 9;

            # Or: set $metric_foo bar=9;

            return 200 "Updated with '$metric_foo'\nValues='$metric_foo_value'\nCount='$metric_foo_value_count'\n";
        }
    }
}

With this configuration, a request to /foo/ yields:

$ curl 127.0.0.1/foo/
Updated with 'bar=9'
Values='1, 9, 9'
Count='1'

Additional Examples#

Monitoring HTTP Methods#

metric_zone http_methods:1m count;

server {
    listen 80;

    location / {
        metric http_methods $request_method;
    }

    location /metrics/ {
        allow 127.0.0.1;
        deny all;
        api /status/http/metric_zones/http_methods/metrics/;
    }
}

Response:

{
    "GET": 65,
    "POST": 20,
    "PUT": 10,
    "DELETE": 5
}

Upstream Response Time Distribution#

metric_zone upstream_time:10m expire=on histogram
    0.05 0.1 0.3 0.5 1 2 5 10 inf;

server {
    listen 80;

    location /backend/ {
        proxy_pass http://backend;
        metric upstream_time $upstream_addr=$upstream_response_time on=end;
    }

    location /metrics/ {
        allow 127.0.0.1;
        deny all;
        api /status/http/metric_zones/upstream_time/;
    }
}

Response:

{
    "discarded": 0,
    "metrics": {
        "backend1:8080": {
            "0.05": 12,
            "0.1": 28,
            "0.3": 56,
            "0.5": 78,
            "1": 92,
            "2": 97,
            "5": 99,
            "10": 100,
            "inf": 100
        }
    }
}

Active Connections#

metric_zone active_connections:2m gauge;

server {
    listen 80;
    server_name site1.com;

    location / {
        # Увеличиваем при подключении
        metric active_connections site1=1 on=request;

        # Уменьшаем при завершении
        metric active_connections site1=-1 on=end;
    }
}

server {
    listen 80;
    server_name site2.com;

    location / {
        metric active_connections site2=1 on=request;
        metric active_connections site2=-1 on=end;
    }
}

server {
    listen 8080;

    location /connections/ {
        allow 127.0.0.1;
        deny all;
        api /status/http/metric_zones/active_connections/metrics;
    }
}

Response:

{
    "site1": 42,
    "site2": 17
}

Prometheus Support#

Angie includes a built-in module for displaying metrics in Prometheus format, which supports custom metrics.

As an integration example, consider the following configuration:

http {
    # Creating the "upload" metric
    metric_complex_zone upload:1m discard_key="other" {
        stats    histogram 64 256 1024 4096 16384 +Inf;
        sum      gauge;
        count    count;
        avg_size average exp;
    }

    # Describing the Prometheus template for the "upload" metric
    prometheus_template upload_metric {
        'stats{le="$1"}' $p8s_value
                         path=~^/http/metric_zones/upload/metrics/angie/stats/(.+)$
                         type=histogram;

        'stats_sum'      $p8s_value
                         path=/http/metric_zones/upload/metrics/angie/sum;
        'stats_count'    $p8s_value
                         path=/http/metric_zones/upload/metrics/angie/count;

        'avg_size'       $p8s_value
                         path=/http/metric_zones/upload/metrics/angie/avg_size;
    }

    server {
        listen 80;

        # Updating the metric
        location ~ ^/upload/(.*)$ {
            api /status/http/metric_zones/upload/metrics/angie/;
            metric upload angie=$1 on=request;
        }

        # Target for metric scraping
        location /prometheus/upload_metric/ {
            prometheus upload_metric;
        }
    }
}

After several requests to /upload/...:

$ curl 127.0.0.1/upload/16384
$ curl 127.0.0.1/upload/64448
$ curl 127.0.0.1/upload/64
$ curl 127.0.0.1/upload/1028
$ curl 127.0.0.1/upload/1028

The metric values will be:

{
    "stats": {
        "64": 1,
        "256": 1,
        "1024": 1,
        "4096": 3,
        "16384": 4,
        "+Inf": 5
    },

    "sum": 82952,
    "count": 5,
    "avg_size": 1077.9376
}

In Prometheus format, the metric is available at /prometheus/upload_metric/:

# Angie Prometheus template "upload_metric"
# TYPE stats histogram
stats{le="64"} 1
stats{le="256"} 1
stats{le="1024"} 1
stats{le="4096"} 3
stats{le="16384"} 4
stats{le="+Inf"} 5
stats_sum 82952
stats_count 5
avg_size 1077.9376