Metric#

Added in version 1.11.0.

The ngx_http_metric_module module allows creating arbitrary real-time calculated metrics. These metric values are stored in shared memory and displayed in real-time within the /status/http/metric_zones/ API branch. Various data aggregation types are supported (counters, histograms, moving averages, etc.) with grouping by arbitrary keys.

For the response schema, see the HTTP metric zone API reference.

When building from the source code, the module is built by default; it can be disabled with the build option --without-http_metric_module.

In packages and images from our repositories, the module is included in the build.

Note

On a configuration reload, a zone created by metric_zone or metric_complex_zone keeps its accumulated values only while its storage layout is unchanged. The zone starts empty if the zone size, the number of metrics, or a mode differs from the previous configuration, if the average mean count changes, if the average mean window is switched to or from off, or if the number of histogram buckets changes; the reason is then reported in the error log at the info level. Other parameters, such as the average exp factor, the average mean window interval, or the histogram bucket boundaries, can be changed without losing the accumulated values.

Configuration Example#

Counting API requests:

http {
    metric_zone api_requests:1m count;

    server {
        listen 80;

        location /api/ {
            allow 127.0.0.1;
            deny all;
            api /status/;

            metric api_requests $http_user_agent on=request;
        }
    }
}

If a request is made to /api/ with this configuration:

$ curl 127.0.0.1/api/ --user-agent "Firefox"

The api_requests metric is updated in real-time:

{
    "http": {
       "metric_zones": {
           "api_requests": {
               "discarded": 0,
               "metrics": {
                   "Firefox": 1
               }
           }
       }
    }
}

Directives#

metric#

Syntax	`metric` name key=value [`on`=`request`\| `response`\| `end`];
Default	—
Context	http, server, location

Calculates the value of the metric for the specified shared memory zone name.

Parameters:

key — an arbitrary string (often a variable) used for grouping values. Maximum length is 255 bytes; longer keys are truncated. In API output, Angie appends ... to every 255-byte key, including one whose original length was exactly 255 bytes. If the key itself contains one or more = characters, only the substring after the last = is treated as value; everything before it becomes part of the key;
value — a number (can be a variable) processed by the selected mode. If omitted, it defaults to 0. If the parameter cannot be converted to a number, it defaults to 1;
on — an optional parameter specifying when the metric is calculated:
- If on=request, calculation occurs when the request is received;
- If on=response, calculation occurs during response preparation;
- If on=end (default), calculation occurs after the response is sent.

Note

In the case of internal redirection, metrics at the on=request stage are calculated in the original location. However, on=response and on=end metrics will be calculated in the new location.

Several metric directives can be specified on the same configuration level. These directives are inherited from the previous configuration level if and only if there are no metric directives defined on the current level. A single metric in a location block therefore cancels every rule inherited from server or http, at all three on stages and not just at its own.

Example usage:

metric requests $http_user_agent=$request_time;

Note

Metrics with an empty key or an invalid key=value pair are ignored. An omitted value is treated as 0:

metric foo $bar;  # Equivalent to $bar=0

This is useful, for example, for the count mode, which ignores numerical values and simply reacts to the fact that a metric was updated.

Note

Remember that variables are evaluated in different phases. For example, it is impossible to use $bytes_sent (bytes sent to the client) with on=request (when the request is received).

metric_complex_zone#

Syntax	`metric_complex_zone` name:size [`expire`=`on`\| `off`] [`discard_key`=`name`] { ... }
Default	—
Context	http

Defines a complex metric — a set of metrics with independent modes. Each line in the block body defines a submetric name, a mode, and optional mode parameters. The zone size must be at least eight system memory pages: 32K where the page size is 4K.

Example usage:

metric_complex_zone requests:1m expire=on discard_key="old" {
    # submetric name   mode          parameters
    min_time           min;
    avg_time           average exp   factor=60;
    max_time           max;
    total              count;
}

In the API tree, such a complex metric template looks as follows:

{
    "discarded": 3,
    "metrics": {
        "key1": {
            "min_time": 20,
            "avg_time": 50,
            "max_time": 80,
            "total": 2
        },
        "old": {
             "min_time": 3,
             "avg_time": 40,
             "max_time": 152,
             "total": 80
        }
    }
}

metric_zone#

Syntax	`metric_zone` name:size [`expire`=`on`\| `off`] [`discard_key`=`name`] mode [parameters];
Default	—
Context	http

Creates a shared memory zone of the specified size with the given name to store metrics. The zone name serves as a node in the /status/http/metric_zones/ branch. The zone size must be at least eight system memory pages: 32K where the page size is 4K.

Parameters:

expire=<on|off> — behavior when the zone is full:
- If on, the oldest metrics (by update time) are discarded to free memory for new ones;
- If off (default) — new incoming metrics are discarded, preserving existing entries.
discard_key=<name> — defines a metric with the key name where values of discarded metrics are accumulated. By default, no such metric is created. Reserved key cannot be updated manually.
mode — data processing algorithm (see the Operation Modes section);
parameters — additional settings for the selected mode (e.g., factor for average exp).

Example usage:

metric_zone request_time:1m max;

In the API tree, the shared memory zone template looks as follows:

{
    "discarded": 0,
    "metrics": {
        "key1": 123,
        "key2": 10.5
    }
}

Note

In a 1 MB zone, with a key size of 39 bytes and a single metric mode, approximately 8,000 unique key entries can be stored.

Operation Modes#

List of available metric operation modes:

count — counter;
gauge — gauge (increment/decrement);
last — the last received value;
min — minimum value;
max — maximum value;
average exp — exponential moving average (EMA) (parameter factor);
average mean — mean over a window (parameters window and count);
histogram — distribution across "buckets" (a list of threshold values).

The examples below use the /api/ endpoint from the configuration example. Each result is available under /api/http/metric_zones/<zone>/metrics/.

count#

Each metric update increments the counter by 1, regardless of the supplied value.

Default value — 0.

Examples:

metric_zone count:1m count;

# As part of a complex metric:
#
# metric_complex_zone count:1m {
#     some_metric_name  count;
# }

server {
    listen 80;

    location /metric/ {
        metric count KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric count KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/set/23
$ curl 127.0.0.1/metric/set/-32

Expected metric value in the API:

{
    "KEY": 4
}

gauge#

The gauge increases or decreases its value depending on the sign of the passed number. A positive value increases the counter, while a negative value decreases it. A value of 0 does not change the counter.

Default value — 0.

Examples:

metric_zone gauge:1m gauge;

# As part of a complex metric:
#
# metric_complex_zone gauge:1m {
#     some_metric_name  gauge;
# }

server {
    listen 80;

    location /metric/ {
        metric gauge KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric gauge KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": 0
}

Further updates:

$ curl 127.0.0.1/metric/set/5
$ curl 127.0.0.1/metric/set/-5
$ curl 127.0.0.1/metric/set/8

Expected metric value in the API:

{
    "KEY": 8
}

last#

Stores the last received value without any aggregation. If value is omitted, 0 is used.

Examples:

metric_zone last:1m last;

# As part of a complex metric:
#
# metric_complex_zone last:1m {
#     some_metric_name  last;
# }

server {
    listen 80;

    location /metric/ {
        metric last KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric last KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": 0
}

Further updates:

$ curl 127.0.0.1/metric/set/8000
$ curl 127.0.0.1/metric/set/37
$ curl 127.0.0.1/metric/set/-3.5

Expected metric value in the API:

{
   "KEY": -3.5
}

min#

Saves the minimum of two values — the currently stored value and the new one.

Examples:

metric_zone min:1m min;

# As part of a complex metric:
#
# metric_complex_zone min:1m {
#     some_metric_name  min;
# }

server {
    listen 80;

    location /metric/ {
        metric min KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric min KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/set/42.999
$ curl 127.0.0.1/metric/set/-512
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": -512
}

max#

Saves the maximum of two values — the currently stored value and the new one.

Examples:

metric_zone max:1m max;

# As part of a complex metric:
#
# metric_complex_zone max:1m {
#     some_metric_name  max;
# }

server {
    listen 80;

    location /metric/ {
        metric max KEY;
    }

    location ~ ^/metric/set/(.+)$ {
        metric max KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/set/42.999
$ curl 127.0.0.1/metric/set/-512
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/

Expected metric value in the API:

{
    "KEY": 42.999
}

average exp#

Calculates the average value using the exponential smoothing algorithm.

Accepts an optional parameter factor=<number> — a coefficient determining how much the new value influences the average. Integer values from 0 to 99 are allowed. Default is 90.

The higher the coefficient, the more weight new values have. If you specify 90, the result will be 90% of the new value and 10% of the previous average.

Examples:

metric_zone avg_exp:1m average exp factor=60;

# As part of a complex metric:
#
# metric_complex_zone avg_exp:1m {
#     some_metric_name  average exp  factor=60;
# }

server {
    listen 80;

    location ~ ^/metric/set/(.+)$ {
        metric avg_exp KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/set/100
$ curl 127.0.0.1/metric/set/200
$ curl 127.0.0.1/metric/set/0
$ curl 127.0.0.1/metric/set/8
$ curl 127.0.0.1/metric/set/30

Expected metric value in the API:

{
    "KEY": 30.16
}

average mean#

Calculates the arithmetic mean. Accepts optional parameters window=<off|time> and count=<number>, defining the time interval and sample size for averaging, respectively. Defaults: window=off (entire sample used) and count=10.

Note

For example, window=5s will only consider events from the last 5 seconds. The window parameter cannot be 0. The count=number parameter controls the sample size (cached values) for a smoother mean calculation.

Examples:

metric_zone avg_mean:1m average mean window=5s count=8;

# As part of a complex metric:
#
# metric_complex_zone avg_mean:1m {
#     some_metric_name  average mean  window=5s count=8;
# }

server {
    listen 80;

    location ~ ^/metric/set/(.+)$ {
        metric avg_mean KEY=$1;
    }

}

Updating the metric:

$ curl 127.0.0.1/metric/set/0.1
$ curl 127.0.0.1/metric/set/0.1
$ curl 127.0.0.1/metric/set/0.4
$ curl 127.0.0.1/metric/set/10
$ curl 127.0.0.1/metric/set/1
$ curl 127.0.0.1/metric/set/1

Expected metric value in the API:

{
    "KEY": 2.1
}

If you wait 5 seconds from the last update, the expected value will be:

{
    "KEY": 0
}

histogram#

Creates a set of "buckets," incrementing the relevant counter if the new value does not exceed the bucket threshold. Parameters are provided as a list of numerical thresholds. Useful for analyzing distributions, such as response times.

Mandatory parameters are numbers — the threshold values of the buckets, typically listed in ascending order.

Note

The bucket value inf or +Inf can be used to capture all values exceeding the highest specified bucket.

Examples:

metric_zone hist:1m histogram 0.1 0.2 0.5 1 2 inf;

# As part of a complex metric:
#
# metric_complex_zone hist:1m {
#     some_metric_name  histogram  0.1 0.2 0.5 1 2 inf;
# }

server {
    listen 80;

    location ~ ^/metric/set/(.+)$ {
        metric hist KEY=$1;
    }
}

Updating the metric:

$ curl 127.0.0.1/metric/set/0.25

Expected metric value in the API:

{
    "KEY": {
        "0.1": 0,
        "0.2": 0,
        "0.5": 1,
        "1": 1,
        "2": 1,
        "inf": 1
    }
}

Further updates:

$ curl 127.0.0.1/metric/set/2

Expected metric value in the API:

{
    "KEY": {
        "0.1": 0,
        "0.2": 0,
        "0.5": 1,
        "1": 1,
        "2": 2,
        "inf": 2
    }
}

Further update:

$ curl 127.0.0.1/metric/set/1000

Expected metric value in the API:

{
    "KEY": {
       "0.1": 0,
       "0.2": 0,
       "0.5": 1,
       "1": 1,
       "2": 2,
       "inf": 3
    }
}

Built-in Variables#

Variables are created for each metric:

$metric_<name>
$metric_<name>_key
$metric_<name>_value

For complex metrics, an additional variable is added:

$metric_<name>_value_<metric>

`$metric_<name>`#

Similar to the metric directive, the $metric_<name> variable setter can be used to update a metric. Calculation occurs during the Rewrite phase, allowing metric processing from the njs module, for example.

The setter accepts a key with an optional =value suffix. Without the suffix, the value is 0. The key and value can contain text, variables, or combinations of both.

Example usage:

http {
    metric_zone counter:1m count;

    # At this point, the $metric_counter variable is added

    server {
        listen 80;

        location /metric/ {
            set $metric_counter $http_user_agent;  # Equivalent to $http_user_agent=0
        }

        location /api/ {
            allow 127.0.0.1;
            deny all;
            api /status/http/metric_zones/counter/;
        }
    }
}

Calculating metrics using the njs module:

http {
    js_import metrics.js;

    resolver 127.0.0.53;

    metric_complex_zone requests:1m {
        min_time        min;
        max_time        max;
        total           count;
    }

    location /metric/ {
        js_content metrics.js_request;
        js_fetch_trusted_certificate /path/to/ISRG_Root_X1.pem;
    }

    location /api/ {
        allow 127.0.0.1;
        deny all;
        api /static/http/metric_zones/requests/;
    }
}

File metrics.js:

async function js_request(r) {
    let start_time = Date.now();

    let results = await Promise.all([ngx.fetch('https://google.com/'),
                                     ngx.fetch('https://google.ru/')]);

    // Using the $metric_requests variable setter
    r.variables.metric_requests = `google=${Date.now() - start_time}`;
}

export default {js_request};

After several requests to location /metric/, the values might look like this:

{
    "discarded": 0,
    "metrics": {
        "google": {
            "min_time": 70,
            "max_time": 432,
            "total": 6
        }
    }
}

Note

After setting the variable, you can retrieve its value; it will equal the key together with the newly calculated metric value, in the key=value format — not the literal string that was assigned.

Additionally, the value stored in the $metric_<name>_key variable will change to the specified key.

`$metric_<name>_key` and `$metric_<name>_value`#

The $metric_<name>_key and $metric_<name>_value variables define the key and value respectively. The metric update occurs when the $metric_<name>_value is set, provided that the key in $metric_<name>_key has already been defined.

Note

For complex metrics, the submetric values in the $metric_<name>_value variable are joined using a ", " separator.

Example usage:

http {
    metric_zone gauge:1m gauge;

    # The variables $metric_gauge, $metric_gauge_key, and $metric_gauge_value are added here.

    metric_complex_zone complex:1m {
        hist histogram 1 2 3;
        avg  average exp;
    }

    # $metric_complex, $metric_complex_key, and $metric_complex_value are added here.

    server {
        listen 80;

        location /gauge/ {
            set $metric_gauge_key "foo";
            set $metric_gauge_value 1;

            # Or: set $metric_gauge foo=1;

            return 200 "Updated with '$metric_gauge'\nValue='$metric_gauge_value'\n";
        }

        location /complex/ {
            set $metric_complex_key "foo";
            set $metric_complex_value 3;

            # Or: set $metric_complex foo=3;

            return 200 "Updated with '$metric_complex'\nValue='$metric_complex_value'\n";
        }
    }
}

With this configuration, a request to /gauge/ yields:

$ curl 127.0.0.1/gauge/
Updated with 'foo=1'
Value='1'

For /complex/:

$ curl 127.0.0.1/complex/
Updated with 'foo=0 0 1, 3'
Value='0 0 1, 3'

Note

If an empty string is assigned to $metric_<name>_value, the value is recognized as 0. If the string consists of characters that cannot be converted to a number, it is recognized as 1.

Calculation occurs only after both $metric_<name>_key and $metric_<name>_value have been set.

In this case, the value stored in $metric_<name> becomes equal to the key together with the newly calculated metric value, in the key=value format.

The value in $metric_<name>_key represents the last key specified via variables.

The value in $metric_<name>_value represents the last calculated value for the key set in $metric_<name>_key.

`$metric_<name>_value_<metric>`#

For complex metrics, the value of a specific submetric can be retrieved using the $metric_<name>_value_<metric> variable, where <metric> is the name of the submetric.

Example usage:

http {
    metric_complex_zone foo:1m {
        count count;
        min   min;
        avg   average exp;
    }

    # Adds $metric_foo, $metric_foo_key, $metric_foo_value,
    # and $metric_foo_value_count, $metric_foo_value_min, $metric_foo_value_avg.

    server {
        listen 80;

        location /foo/ {
            set $metric_foo_key   bar;
            set $metric_foo_value 9;

            # Or: set $metric_foo bar=9;

            return 200 "Updated with '$metric_foo'\nValues='$metric_foo_value'\nCount='$metric_foo_value_count'\n";
        }
    }
}

With this configuration, a request to /foo/ yields:

$ curl 127.0.0.1/foo/
Updated with 'bar=1, 9, 9'
Values='1, 9, 9'
Count='1'

Additional Examples#

Monitoring HTTP Methods#

metric_zone http_methods:1m count;

server {
    listen 80;

    location / {
        metric http_methods $request_method;
    }

    location /metrics/ {
        allow 127.0.0.1;
        deny all;
        api /status/http/metric_zones/http_methods/metrics/;
    }
}

Response:

{
    "GET": 65,
    "POST": 20,
    "PUT": 10,
    "DELETE": 5
}

Upstream Response Time Distribution#

metric_zone upstream_time:10m expire=on histogram
    0.05 0.1 0.3 0.5 1 2 5 10 inf;

server {
    listen 80;

    location /backend/ {
        proxy_pass http://backend;
        metric upstream_time $upstream_addr=$upstream_response_time on=end;
    }

    location /metrics/ {
        allow 127.0.0.1;
        deny all;
        api /status/http/metric_zones/upstream_time/;
    }
}

Response:

{
    "discarded": 0,
    "metrics": {
        "backend1:8080": {
            "0.05": 12,
            "0.1": 28,
            "0.3": 56,
            "0.5": 78,
            "1": 92,
            "2": 97,
            "5": 99,
            "10": 100,
            "inf": 100
        }
    }
}

Active Connections#

metric_zone active_connections:2m gauge;

server {
    listen 80;
    server_name site1.com;

    location / {
        # Увеличиваем при подключении
        metric active_connections site1=1 on=request;

        # Уменьшаем при завершении
        metric active_connections site1=-1 on=end;
    }
}

server {
    listen 80;
    server_name site2.com;

    location / {
        metric active_connections site2=1 on=request;
        metric active_connections site2=-1 on=end;
    }
}

server {
    listen 8080;

    location /connections/ {
        allow 127.0.0.1;
        deny all;
        api /status/http/metric_zones/active_connections/metrics;
    }
}

Response:

{
    "site1": 42,
    "site2": 17
}

Prometheus Support#

Angie includes a built-in Prometheus module for displaying metrics in Prometheus format, which supports custom metrics.

As an integration example, consider the following configuration:

http {
    # Creating the "upload" metric
    metric_complex_zone upload:1m discard_key="other" {
        stats    histogram 64 256 1024 4096 16384 +Inf;
        sum      gauge;
        count    count;
        avg_size average exp;
    }

    # Describing the Prometheus template for the "upload" metric
    prometheus_template upload_metric {
        'stats{le="$1"}' $p8s_value
                         path=~^/http/metric_zones/upload/metrics/angie/stats/(.+)$
                         type=histogram;

        'stats_sum'      $p8s_value
                         path=/http/metric_zones/upload/metrics/angie/sum;
        'stats_count'    $p8s_value
                         path=/http/metric_zones/upload/metrics/angie/count;

        'avg_size'       $p8s_value
                         path=/http/metric_zones/upload/metrics/angie/avg_size;
    }

    server {
        listen 80;

        # Updating the metric
        location ~ ^/upload/(.*)$ {
            api /status/http/metric_zones/upload/metrics/angie/;
            metric upload angie=$1 on=request;
        }

        # Target for metric scraping
        location /prometheus/upload_metric/ {
            prometheus upload_metric;
        }
    }
}

After several requests to /upload/...:

$ curl 127.0.0.1/upload/16384
$ curl 127.0.0.1/upload/64448
$ curl 127.0.0.1/upload/64
$ curl 127.0.0.1/upload/1028
$ curl 127.0.0.1/upload/1028

The metric values will be:

{
    "stats": {
        "64": 1,
        "256": 1,
        "1024": 1,
        "4096": 3,
        "16384": 4,
        "+Inf": 5
    },

    "sum": 82952,
    "count": 5,
    "avg_size": 1077.9376
}

In Prometheus format, the metric is available at /prometheus/upload_metric/:

# Angie Prometheus template "upload_metric"
# TYPE stats histogram
stats{le="64"} 1
stats{le="256"} 1
stats{le="1024"} 1
stats{le="4096"} 3
stats{le="16384"} 4
stats{le="+Inf"} 5
stats_sum 82952
stats_count 5
avg_size 1077.9376

Metric#

Configuration Example#

Directives#

metric#

metric_complex_zone#

metric_zone#

Operation Modes#

count#

gauge#

last#

min#

max#

average exp#

average mean#

histogram#

Built-in Variables#

$metric_<name>#

$metric_<name>_key and $metric_<name>_value#

$metric_<name>_value_<metric>#

Additional Examples#

Monitoring HTTP Methods#

Upstream Response Time Distribution#

Active Connections#

Prometheus Support#

`$metric_<name>`#

`$metric_<name>_key` and `$metric_<name>_value`#

`$metric_<name>_value_<metric>`#