Metric#

Added in version 1.12.0.

The ngx_stream_metric_module module allows creating arbitrary real-time calculated metrics for stream (TCP and UDP) traffic. These metric values are stored in shared memory and displayed in real-time within the /status/stream/metric_zones/ API branch. Various data aggregation types are supported (counters, histograms, moving averages, etc.) with grouping by arbitrary keys.

For the response schema, see the Stream metric zone API reference.

Note

On a configuration reload, a zone created by metric_zone or metric_complex_zone keeps its accumulated values only while its storage layout is unchanged. The zone starts empty if the zone size, the number of metrics, or a mode differs from the previous configuration, if the average mean count changes, if the average mean window is switched to or from off, or if the number of histogram buckets changes; the reason is then reported in the error log at the info level. Other parameters, such as the average exp factor, the average mean window interval, or the histogram bucket boundaries, can be changed without losing the accumulated values.

Configuration Example#

Counting connections by client address:

stream {
    metric_zone connections:1m count;

    server {
        listen 12345;

        metric connections $remote_addr on=connect;
    }
}

http {
    server {
        listen 80;

        location /status/ {
            allow 127.0.0.1;
            deny all;
            api /status/;
        }
    }
}

If a connection is made to port 12345:

$ nc 127.0.0.1 12345 </dev/null

The connections metric is updated in real-time:

{
    "stream": {
       "metric_zones": {
           "connections": {
               "discarded": 0,
               "metrics": {
                   "127.0.0.1": 1
               }
           }
       }
    }
}

Directives#

metric#

Syntax	`metric` name key=value [`on`=`connect`\| `preread`\| `end`];
Default	—
Context	stream, server

Calculates the value of the metric for the specified shared memory zone name.

Parameters:

key — an arbitrary string (often a variable) used for grouping values. Maximum length is 255 bytes; longer keys are truncated. In API output, Angie appends ... to every 255-byte key, including one whose original length was exactly 255 bytes. A key may itself contain = characters — only the text after the last = is treated as the value, so everything before it (including any embedded =) becomes the key;
value — a number (can be a variable) processed by the selected mode. If omitted, it defaults to 0. If the parameter cannot be converted to a number, it defaults to 1;
on — an optional parameter specifying at which point of session processing the metric is calculated:
- If on=connect, calculation occurs during the Pre-access phase, before any data is read from the client;
- If on=preread, calculation occurs once per session, when the first data from the upstream side — received from the server proxied to with proxy_pass, or generated by return — is sent to the client; by that point, modules such as proxy_protocol, ssl_preread, and mqtt_preread have already processed the client's initial data;
- If on=end (default), calculation occurs during the Log phase, when the session ends.

Several metric directives can be specified on the same configuration level. These directives are inherited from the previous configuration level if and only if there are no metric directives defined on the current level. A single metric in a server block therefore cancels every rule inherited from stream, at all three on stages and not just at its own.

Example usage:

metric sessions $remote_addr=$session_time on=end;

Note

Metrics with an empty key or an invalid key=value pair are ignored. An omitted value is treated as 0:

metric foo $bar;  # Equivalent to $bar=0

This is useful, for example, for the count mode, which ignores numerical values and simply reacts to the fact that a metric was updated.

Note

Remember that variables are evaluated in different phases. For example, it is impossible to use $upstream_bytes_sent (bytes sent to the upstream) with on=connect (before the connection has been proxied anywhere).

metric_complex_zone#

Syntax	`metric_complex_zone` name:size [`expire`=`on`\| `off`] [`discard_key`=`name`] { ... }
Default	—
Context	stream

Defines a complex metric — a set of metrics with independent modes. Each line in the block body defines a submetric name, a mode, and optional mode parameters. The zone size must be at least eight system memory pages.

Example usage:

metric_complex_zone sessions:1m expire=on discard_key="old" {
    # submetric name   mode          parameters
    min_time           min;
    avg_time           average exp   factor=60;
    max_time           max;
    total              count;
}

In the API tree, such a complex metric template looks as follows:

{
    "discarded": 3,
    "metrics": {
        "key1": {
            "min_time": 20,
            "avg_time": 50,
            "max_time": 80,
            "total": 2
        },
        "old": {
             "min_time": 3,
             "avg_time": 40,
             "max_time": 152,
             "total": 80
        }
    }
}

metric_zone#

Syntax	`metric_zone` name:size [`expire`=`on`\| `off`] [`discard_key`=`name`] mode [parameters];
Default	—
Context	stream

Creates a shared memory zone of the specified size with the given name to store metrics. The zone name serves as a node in the /status/stream/metric_zones/ branch. The zone size must be at least eight system memory pages.

Parameters:

expire=<on|off> — behavior when the zone is full:
- If on, the oldest metrics (by update time) are discarded to free memory for new ones;
- If off (default) — new incoming metrics are discarded, preserving existing entries.
discard_key=<name> — defines a metric with the key name where values of discarded metrics are accumulated. By default, no such metric is created. Reserved key cannot be updated manually.
mode — data processing algorithm (see the Operation Modes section);
parameters — additional settings for the selected mode (e.g., factor for average exp).

Example usage:

metric_zone session_time:1m max;

In the API tree, the shared memory zone template looks as follows:

{
    "discarded": 0,
    "metrics": {
        "key1": 123,
        "key2": 10.5
    }
}

Note

In a 1 MB zone, with a key size of 39 bytes and a single metric mode, approximately 8,000 unique key entries can be stored.

Operation Modes#

List of available metric operation modes:

count — counter;
gauge — gauge (increment/decrement);
last — the last received value;
min — minimum value;
max — maximum value;
average exp — exponential moving average (EMA) (parameter factor);
average mean — mean over a window (parameters window and count);
histogram — distribution across "buckets" (a list of threshold values).

The examples below use the /status/ endpoint from the configuration example. Each result is available under /status/stream/metric_zones/<zone>/metrics/.

count#

Each metric update increments the counter by 1, regardless of the supplied value.

Default value — 0.

Examples:

metric_zone count:1m count;

# As part of a complex metric:
#
# metric_complex_zone count:1m {
#     some_metric_name  count;
# }

server {
    listen 12345;

    metric count KEY;
}

Updating the metric:

$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12345 </dev/null

Expected metric value in the API:

{
    "KEY": 4
}

gauge#

The gauge increases or decreases its value depending on the sign of the passed number. A positive value increases the counter, while a negative value decreases it. A value of 0 does not change the counter.

Default value — 0.

Examples:

metric_zone gauge:1m gauge;

# As part of a complex metric:
#
# metric_complex_zone gauge:1m {
#     some_metric_name  gauge;
# }

server {
    listen 12351;
    metric gauge KEY;
}

server {
    listen 12352;
    metric gauge KEY=5;
}

server {
    listen 12353;
    metric gauge KEY=-5;
}

server {
    listen 12354;
    metric gauge KEY=8;
}

Updating the metric:

$ nc 127.0.0.1 12351 </dev/null

Expected metric value in the API:

{
    "KEY": 0
}

Further updates:

$ nc 127.0.0.1 12352 </dev/null
$ nc 127.0.0.1 12353 </dev/null
$ nc 127.0.0.1 12354 </dev/null

Expected metric value in the API:

{
    "KEY": 8
}

last#

Stores the last received value without any aggregation. If value is omitted, 0 is used.

Examples:

metric_zone last:1m last;

# As part of a complex metric:
#
# metric_complex_zone last:1m {
#     some_metric_name  last;
# }

server {
    listen 12361;
    metric last KEY;
}

server {
    listen 12362;
    metric last KEY=8000;
}

server {
    listen 12363;
    metric last KEY=37;
}

server {
    listen 12364;
    metric last KEY=-3.5;
}

Updating the metric:

$ nc 127.0.0.1 12361 </dev/null

Expected metric value in the API:

{
    "KEY": 0
}

Further updates:

$ nc 127.0.0.1 12362 </dev/null
$ nc 127.0.0.1 12363 </dev/null
$ nc 127.0.0.1 12364 </dev/null

Expected metric value in the API:

{
   "KEY": -3.5
}

min#

Saves the minimum of two values — the currently stored value and the new one.

Examples:

metric_zone min:1m min;

# As part of a complex metric:
#
# metric_complex_zone min:1m {
#     some_metric_name  min;
# }

server {
    listen 12371;
    metric min KEY=42.999;
}

server {
    listen 12372;
    metric min KEY=-512;
}

server {
    listen 12373;
    metric min KEY=1;
}

Updating the metric:

$ nc 127.0.0.1 12371 </dev/null
$ nc 127.0.0.1 12372 </dev/null
$ nc 127.0.0.1 12373 </dev/null

Expected metric value in the API:

{
    "KEY": -512
}

max#

Saves the maximum of two values — the currently stored value and the new one.

Examples:

metric_zone max:1m max;

# As part of a complex metric:
#
# metric_complex_zone max:1m {
#     some_metric_name  max;
# }

server {
    listen 12381;
    metric max KEY=42.999;
}

server {
    listen 12382;
    metric max KEY=-512;
}

server {
    listen 12383;
    metric max KEY=1;
}

Updating the metric:

$ nc 127.0.0.1 12381 </dev/null
$ nc 127.0.0.1 12382 </dev/null
$ nc 127.0.0.1 12383 </dev/null

Expected metric value in the API:

{
    "KEY": 42.999
}

average exp#

Calculates the average value using the exponential smoothing algorithm.

Accepts an optional parameter factor=<number> — a coefficient determining how much the new value influences the average. Integer values from 0 to 99 are allowed. Default is 90.

The higher the coefficient, the more weight new values have. If you specify 90, the result will be 90% of the new value and 10% of the previous average.

Examples:

metric_zone avg_exp:1m average exp factor=60;

# As part of a complex metric:
#
# metric_complex_zone avg_exp:1m {
#     some_metric_name  average exp  factor=60;
# }

server {
    listen 12391;
    metric avg_exp KEY=100;
}

server {
    listen 12392;
    metric avg_exp KEY=200;
}

server {
    listen 12393;
    metric avg_exp KEY=0;
}

server {
    listen 12394;
    metric avg_exp KEY=8;
}

server {
    listen 12395;
    metric avg_exp KEY=30;
}

Updating the metric:

$ nc 127.0.0.1 12391 </dev/null
$ nc 127.0.0.1 12392 </dev/null
$ nc 127.0.0.1 12393 </dev/null
$ nc 127.0.0.1 12394 </dev/null
$ nc 127.0.0.1 12395 </dev/null

Expected metric value in the API:

{
    "KEY": 30.16
}

average mean#

Calculates the arithmetic mean. Accepts optional parameters window=<off|time> and count=<number>, defining the time interval and sample size for averaging, respectively. Defaults: window=off (entire sample used) and count=10.

Note

For example, window=5s will only consider events from the last 5 seconds. The window parameter cannot be 0. The count=number parameter controls the sample size (cached values) for a smoother mean calculation.

Examples:

metric_zone avg_mean:1m average mean window=5s count=8;

# As part of a complex metric:
#
# metric_complex_zone avg_mean:1m {
#     some_metric_name  average mean  window=5s count=8;
# }

server {
    listen 12401;
    metric avg_mean KEY=0.1;
}

server {
    listen 12402;
    metric avg_mean KEY=0.4;
}

server {
    listen 12403;
    metric avg_mean KEY=10;
}

server {
    listen 12404;
    metric avg_mean KEY=1;
}

Updating the metric:

$ nc 127.0.0.1 12401 </dev/null
$ nc 127.0.0.1 12401 </dev/null
$ nc 127.0.0.1 12402 </dev/null
$ nc 127.0.0.1 12403 </dev/null
$ nc 127.0.0.1 12404 </dev/null
$ nc 127.0.0.1 12404 </dev/null

Expected metric value in the API:

{
    "KEY": 2.1
}

If you wait 5 seconds from the last update, the expected value will be:

{
    "KEY": 0
}

histogram#

Creates a set of "buckets," incrementing the relevant counter if the new value does not exceed the bucket threshold. Parameters are provided as a list of numerical thresholds. Useful for analyzing distributions, such as session durations.

Mandatory parameters are numbers — the threshold values of the buckets, typically listed in ascending order.

Note

The bucket value inf or +Inf can be used to capture all values exceeding the highest specified bucket.

Examples:

metric_zone hist:1m histogram 0.1 0.2 0.5 1 2 inf;

# As part of a complex metric:
#
# metric_complex_zone hist:1m {
#     some_metric_name  histogram  0.1 0.2 0.5 1 2 inf;
# }

server {
    listen 12411;
    metric hist KEY=0.25;
}

server {
    listen 12412;
    metric hist KEY=2;
}

server {
    listen 12413;
    metric hist KEY=1000;
}

Updating the metric:

$ nc 127.0.0.1 12411 </dev/null

Expected metric value in the API:

{
    "KEY": {
        "0.1": 0,
        "0.2": 0,
        "0.5": 1,
        "1": 1,
        "2": 1,
        "inf": 1
    }
}

Further updates:

$ nc 127.0.0.1 12412 </dev/null

Expected metric value in the API:

{
    "KEY": {
        "0.1": 0,
        "0.2": 0,
        "0.5": 1,
        "1": 1,
        "2": 2,
        "inf": 2
    }
}

Further update:

$ nc 127.0.0.1 12413 </dev/null

Expected metric value in the API:

{
    "KEY": {
       "0.1": 0,
       "0.2": 0,
       "0.5": 1,
       "1": 1,
       "2": 2,
       "inf": 3
    }
}

Built-in Variables#

Variables are created for each metric:

$metric_<name>
$metric_<name>_key
$metric_<name>_value

For complex metrics, an additional variable is added:

$metric_<name>_value_<metric>

`$metric_<name>`#

Similar to the metric directive, the $metric_<name> variable setter can be used to update a metric. Calculation occurs when the variable is set — for example, via the set directive, which is evaluated during the Pre-access phase (the same stage as on=connect), or programmatically from the Stream JS module, for example in a js_access handler.

The setter accepts a key with an optional =value suffix. The last = separates the key from the value; without the suffix, the value is 0. The key and value can contain text, variables, or combinations of both. These are the same parsing rules as for metric.

Example usage:

stream {
    metric_zone hits:1m count;

    # At this point, the $metric_hits variable is added

    server {
        listen 12345;

        metric hits client=1 on=connect;
    }

    server {
        listen 12346;

        set $metric_hits client=1;

        return "$metric_hits\n";
    }
}

After three connections to port 12345 and one connection to port 12346:

$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12346 </dev/null
client=4

Note

Reading $metric_<name> back returns the key together with the metric's current calculated value, in key=value form — not the literal string that was assigned. In the example above, the connection assigns the literal value client=1, but since the count mode ignores the assigned value and simply increments on every update, reading $metric_hits afterward returns client=4 — the counter's new total, which already includes the three prior updates from port 12345.

Additionally, the value stored in the $metric_<name>_key variable changes to the specified key.

`$metric_<name>_key` and `$metric_<name>_value`#

The $metric_<name>_key and $metric_<name>_value variables define the key and value respectively. The metric update occurs when the $metric_<name>_value is set, provided that the key in $metric_<name>_key has already been defined.

Note

For complex metrics, the submetric values in the $metric_<name>_value variable are joined using a ", " separator.

Example usage:

stream {
    metric_zone level:1m gauge;

    # The variables $metric_level, $metric_level_key, and $metric_level_value are added here.

    metric_complex_zone stats:1m {
        count count;
        min   min;
        avg   average exp;
    }

    # $metric_stats, $metric_stats_key, and $metric_stats_value are added here.

    server {
        listen 12345;

        metric level sensor=10 on=connect;
    }

    server {
        listen 12346;

        set $metric_level_key   "sensor";
        set $metric_level_value 5;

        # Or: set $metric_level sensor=5;

        return "Updated with '$metric_level'\nValue='$metric_level_value'\n";
    }

    server {
        listen 12347;

        set $metric_stats_key   bar;
        set $metric_stats_value 9;

        # Or: set $metric_stats bar=9;

        return "Updated with '$metric_stats'\nValues='$metric_stats_value'\n";
    }
}

With this configuration, a connection to port 12345 (setting the gauge's starting value to 10) followed by a connection to port 12346 yields:

$ nc 127.0.0.1 12345 </dev/null
$ nc 127.0.0.1 12346 </dev/null
Updated with 'sensor=15'
Value='15'

A connection to port 12347 yields:

$ nc 127.0.0.1 12347 </dev/null
Updated with 'bar=1, 9, 9'
Values='1, 9, 9'

Note

If an empty string is assigned to $metric_<name>_value, the value is recognized as 0. If the string consists of characters that cannot be converted to a number, it is recognized as 1.

Calculation occurs only after both $metric_<name>_key and $metric_<name>_value have been set.

In this case, the value stored in $metric_<name> becomes equal to the new, calculated key=value pair, not the literal values just assigned.

The value in $metric_<name>_key represents the last key specified via variables.

The value in $metric_<name>_value represents the last calculated value for the key set in $metric_<name>_key.

`$metric_<name>_value_<metric>`#

For complex metrics, the value of a specific submetric can be retrieved using the $metric_<name>_value_<metric> variable, where <metric> is the name of the submetric.

Example usage:

stream {
    metric_complex_zone foo:1m {
        count count;
        min   min;
        avg   average exp;
    }

    # Adds $metric_foo, $metric_foo_key, $metric_foo_value,
    # and $metric_foo_value_count, $metric_foo_value_min, $metric_foo_value_avg.

    server {
        listen 12345;

        set $metric_foo_key   bar;
        set $metric_foo_value 9;

        # Or: set $metric_foo bar=9;

        return "Updated with '$metric_foo'\nValues='$metric_foo_value'\nCount='$metric_foo_value_count'\n";
    }
}

With this configuration, a connection to port 12345 yields:

$ nc 127.0.0.1 12345 </dev/null
Updated with 'bar=1, 9, 9'
Values='1, 9, 9'
Count='1'

Additional Examples#

Monitoring Connections by Protocol#

metric_zone protocols:1m count;

server {
    listen 12345;
    listen 12346 udp;

    metric protocols $protocol;
}

Response:

{
    "TCP": 340,
    "UDP": 58
}

Upstream Session Time Distribution#

metric_zone upstream_time:10m expire=on histogram
    0.05 0.1 0.3 0.5 1 2 5 10 inf;

server {
    listen 12345;

    proxy_pass backend;
    metric upstream_time $upstream_addr=$upstream_session_time on=end;
}

Response:

{
    "discarded": 0,
    "metrics": {
        "backend1:8080": {
            "0.05": 12,
            "0.1": 28,
            "0.3": 56,
            "0.5": 78,
            "1": 92,
            "2": 97,
            "5": 99,
            "10": 100,
            "inf": 100
        }
    }
}

Active Connections#

stream {
    metric_zone active_connections:2m gauge;

    server {
        listen 12345;

        metric active_connections service_a=1 on=connect;
        metric active_connections service_a=-1 on=end;
    }

    server {
        listen 12346;

        metric active_connections service_b=1 on=connect;
        metric active_connections service_b=-1 on=end;
    }
}

http {
    server {
        listen 8080;

        location /connections/ {
            allow 127.0.0.1;
            deny all;
            api /status/stream/metric_zones/active_connections/metrics/;
        }
    }
}

Response:

{
    "service_a": 42,
    "service_b": 17
}

Prometheus Support#

Angie includes a built-in Prometheus module for displaying metrics in Prometheus format, which supports custom metrics, including those collected on the stream side. Since Prometheus exposition is an HTTP-only feature, the prometheus_template and prometheus directives are configured in an http block, referencing the stream metric zone through the same unified /status/ API tree.

As an integration example, consider the following configuration:

stream {
    # Creating the "upload" metric
    metric_complex_zone upload:1m discard_key="other" {
        stats    histogram 64 256 1024 4096 16384 +Inf;
        sum      gauge;
        count    count;
        avg_size average exp;
    }

    upstream backend {
        server 127.0.0.1:15001;
    }

    server {
        listen 12345;

        # Updating the metric at session end with the total
        # number of bytes received from the client
        proxy_pass backend;
        metric upload angie=$bytes_received on=end;
    }
}

http {
    # Describing the Prometheus template for the "upload" metric
    prometheus_template upload_metric {
        'stats{le="$1"}' $p8s_value
                         path=~^/stream/metric_zones/upload/metrics/angie/stats/(.+)$
                         type=histogram;

        'stats_sum'      $p8s_value
                         path=/stream/metric_zones/upload/metrics/angie/sum;
        'stats_count'    $p8s_value
                         path=/stream/metric_zones/upload/metrics/angie/count;

        'avg_size'       $p8s_value
                         path=/stream/metric_zones/upload/metrics/angie/avg_size;
    }

    server {
        listen 80;

        # Target for metric scraping
        location /prometheus/upload_metric/ {
            prometheus upload_metric;
        }
    }
}

After five connections to port 12345, sending payloads of 16384, 64448, 64, 1028, and 1028 bytes:

$ head -c 16384 /dev/urandom | nc 127.0.0.1 12345
$ head -c 64448 /dev/urandom | nc 127.0.0.1 12345
$ head -c 64 /dev/urandom | nc 127.0.0.1 12345
$ head -c 1028 /dev/urandom | nc 127.0.0.1 12345
$ head -c 1028 /dev/urandom | nc 127.0.0.1 12345

The metric values will be:

{
    "stats": {
        "64": 1,
        "256": 1,
        "1024": 1,
        "4096": 3,
        "16384": 4,
        "+Inf": 5
    },

    "sum": 82952,
    "count": 5,
    "avg_size": 1077.9376
}

In Prometheus format, the metric is available at /prometheus/upload_metric/:

# Angie Prometheus template "upload_metric"
# TYPE stats histogram
stats{le="64"} 1
stats{le="256"} 1
stats{le="1024"} 1
stats{le="4096"} 3
stats{le="16384"} 4
stats{le="+Inf"} 5
stats_sum 82952
stats_count 5
avg_size 1077.9376

Metric#

Configuration Example#

Directives#

metric#

metric_complex_zone#

metric_zone#

Operation Modes#

count#

gauge#

last#

min#

max#

average exp#

average mean#

histogram#

Built-in Variables#

$metric_<name>#

$metric_<name>_key and $metric_<name>_value#

$metric_<name>_value_<metric>#

Additional Examples#

Monitoring Connections by Protocol#

Upstream Session Time Distribution#

Active Connections#

Prometheus Support#

`$metric_<name>`#

`$metric_<name>_key` and `$metric_<name>_value`#

`$metric_<name>_value_<metric>`#