Stream Module#

The core stream module implements basic functionality for handling TCP and UDP connections: this includes defining server blocks, traffic routing, configuring proxying, SSL/TLS support, and managing connections for streaming services, such as databases, DNS, and other protocols that operate over TCP and UDP.

The other modules in this section extend this functionality, allowing you to flexibly configure and optimize the stream server for various scenarios and requirements.

When building from the source code, this module isn't built by default; it should be enabled with the ‑‑with‑stream build option. In packages and images from our repos, the module is included in the build.

Configuration Example#

worker_processes auto;

error_log /var/log/angie/error.log info;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash $remote_addr consistent;

        server backend1.example.com:12345 weight=5;
        server 127.0.0.1:12345            max_fails=3 fail_timeout=30s;
        server unix:/tmp/backend3;
    }

    upstream dns {
       server 192.168.0.1:53535;
       server dns.example.com:53;
    }

    server {
        listen 12345;
        proxy_connect_timeout 1s;
        proxy_timeout 3s;
        proxy_pass backend;
    }

    server {
        listen 127.0.0.1:53 udp reuseport;
        proxy_timeout 20s;
        proxy_pass dns;
    }

    server {
        listen [::1]:12345;
        proxy_pass unix:/tmp/stream.socket;
    }
}

Directives#

listen#

Syntax

listen address[:port] [ssl] [udp] [proxy_protocol] [setfib=number] [fastopen=number] [backlog=number] [rcvbuf=size] [sndbuf=size] [accept_filter=filter] [deferred] [bind] [ipv6only=on|off] [reuseport] [so_keepalive=on|off|[keepidle]:[keepintvl]:[keepcnt]];

Default

Context

server

Sets the address and port for the socket on which the server will accept connections. It is possible to specify just the port. The address can also be a hostname, for example:

listen 127.0.0.1:12345;
listen *:12345;
listen 12345;     # same as *:12345
listen localhost:12345;

IPv6 addresses are specified in square brackets:

   listen [::1]:12345;
   listen [::]:12345;

UNIX domain sockets are specified with the :samp:`unix:` prefix
listen unix:/var/run/angie.sock;

Port ranges are specified with the first and last port separated by a hyphen:

listen 127.0.0.1:12345-12399;
listen 12345-12399;

Important

Different servers must listen on different address:port pairs.

ssl

allows specifying that all connections accepted on this port should work in SSL mode.

udp

configures a listening socket for working with datagrams. In order to handle packets from the same address and port in the same session, the reuseport parameter should also be specified.

proxy_protocol

allows specifying that all connections accepted on this port should use the PROXY protocol.

The listen directive can have several additional parameters specific to socket-related system calls.

setfib=number

sets the associated routing table, FIB (the SO_SETFIB option) for the listening socket. This currently works only on FreeBSD.

fastopen=number

enables "TCP Fast Open" for the listening socket and limits the maximum length for the queue of connections that have not yet completed the three-way handshake.

Caution

Do not enable this feature unless the server can handle receiving the same SYN packet with data more than once.

backlog=number

sets the backlog parameter in the listen() call that limits the maximum length for the queue of pending connections. By default, backlog is set to -1 on FreeBSD, DragonFly BSD, and macOS, and to 511 on other platforms.

rcvbuf=size

sets the receive buffer size (the SO_RCVBUF option) for the listening socket.

sndbuf=size

sets the send buffer size (the SO_SNDBUF option) for the listening socket.

accept_filter=filter

Sets the name of accept filter (the SO_ACCEPTFILTER option) for the listening socket that filters incoming connections before passing them to accept(). This works only on FreeBSD and NetBSD 5.0+. Acceptable values are dataready and httpready.

deferred

instructs to use a deferred accept() (the TCP_DEFER_ACCEPT socket option) on Linux.

bind

this parameter instructs to make a separate bind() call for a given address:port pair. The fact is that if there are several listen directives with the same port but different addresses, and one of the listen directives listens on all addresses for the given port (*:port), Angie will bind() only to *:port. It should be noted that the getsockname() system call will be made in this case to determine the address that accepted the connection. If the setfib, fastopen, backlog, rcvbuf, sndbuf, accept_filter, deferred, ipv6only, reuseport, or so_keepalive parameters are used then for a given address:port pair a separate bind() call will always be made.

ipv6only=on | off

this parameter determines (via the IPV6_V6ONLY socket option) whether an IPv6 socket listening on a wildcard address [::] will accept only IPv6 connections or both IPv6 and IPv4 connections. This parameter is turned on by default. It can only be set once on start.

reuseport

this parameter instructs to create an individual listening socket for each worker process (using the SO_REUSEPORT socket option on Linux 3.9+ and DragonFly BSD, or SO_REUSEPORT_LB on FreeBSD 12+), allowing a kernel to distribute incoming connections between worker processes. This currently works only on Linux 3.9+, DragonFly BSD, and FreeBSD 12+.

Caution

Inappropriate use of this option may have its security implications.

so_keepalive=on | off | [keepidle]:[keepintvl]:[keepcnt]
configures the "TCP keepalive" behavior for the listening socket.

''

if this parameter is omitted then the operating system's settings will be in effect for the socket

on

the SO_KEEPALIVE option is turned on for the socket

off

the SO_KEEPALIVE option is turned off for the socket

Some operating systems support setting of TCP keepalive parameters on a per-socket basis using the TCP_KEEPIDLE, TCP_KEEPINTVL, and TCP_KEEPCNT socket options. On such systems (currently, Linux 2.4+, NetBSD 5+, and FreeBSD 9.0-STABLE), they can be configured using the keepidle, keepintvl, and keepcnt parameters. One or two parameters may be omitted, in which case the system default setting for the corresponding socket option will be in effect.

For example,

so_keepalive=30m::10

will set the idle timeout (TCP_KEEPIDLE) to 30 minutes, leave the probe interval (TCP_KEEPINTVL) at its system default, and set the probes count (TCP_KEEPCNT) to 10 probes.

preread_buffer_size#

Syntax

preread_buffer_size size;

Default

preread_buffer_size 16k;

Context

stream, server

Specifies a size of the preread buffer.

preread_timeout#

Syntax

preread_timeout timeout;

Default

preread_timeout 30s;

Context

stream, server

Specifies a timeout of the preread phase.

proxy_protocol_timeout#

Syntax

proxy_protocol_timeout timeout;

Default

proxy_protocol_timeout 30s;

Context

stream, server

Specifies a timeout for reading the PROXY protocol header to complete. If no entire header is transmitted within this time, the connection is closed.

resolver#

Syntax

resolver address ... [valid=time] [ipv4=on|off] [ipv6=on|off] [status_zone=zone];

Default

Context

stream, server, upstream

Configures name servers used to resolve names of upstream servers into addresses, for example:

resolver 127.0.0.53 [::1]:5353;

The address can be specified as a domain name or IP address, with an optional port. If port is not specified, the port 53 is used. Name servers are queried in a round-robin fashion.

By default, Angie caches answers using the TTL value of a response. The optional valid parameter allows overriding it:

valid

optional valid parameter allows overriding cached entry validity

resolver 127.0.0.53 [::1]:5353 valid=30s;

By default, Angie will look up both IPv4 and IPv6 addresses while resolving.

ipv4=off

disables looking up of IPv4 addresses

ipv6=off

disables looking up of IPv6 addresses

status_zone

optional parameter; enables the collection of DNS server request and response metrics (/status/resolvers/<zone>) in the specified zone.

Tip

To prevent DNS spoofing, it is recommended configuring DNS servers in a properly secured trusted local network.

resolver_timeout#

Syntax

resolver_timeout time;

Default

resolver_timeout 30s;

Context

stream, server, upstream

Sets a timeout for name resolution, for example:

resolver_timeout 5s;

server#

Syntax

server { ... }

Default

Context

stream

Sets the configuration for a server.

server_name#

Syntax

server_name name ...;

Default

server_name "";

Context

server

Sets names of a virtual server, for example:

server {
    server_name example.com www.example.com;
}

The first name becomes the primary server name.

Server names can include an asterisk (*) to replace the first or last part of a name:

server {
    server_name example.com *.example.com www.example.*;
}

These names are called wildcard names.

The first two examples mentioned above can be combined into one:

server {
    server_name .example.com;
}

You can also use regular expressions in server names by preceding the name with a tilde (~):

server {
    server_name www.example.com ~^www\d+\.example\.com$;
}

Regular expressions may include captures that can be used in other directives:

server {
    server_name ~^(www\.)?(.+)$;

    proxy_pass www.$2:12345;
}

Named captures in regular expressions create variables that can be used in other directives:

server {
    server_name ~^(www\.)?(?<domain>.+)$;

    proxy_pass www.$domain:12345;
}

If the directive's parameter is set to $hostname, the machine's hostname is inserted.

When searching for a virtual server by name, if the name matches more than one of the specified variants (e.g., both a wildcard name and a regular expression match), the first matching variant will be chosen in the following order of priority:

  • The exact name

  • The longest wildcard name starting with an asterisk, e.g., *.example.com

  • The longest wildcard name ending with an asterisk, e.g., mail.*

  • The first matching regular expression (in order of appearance in the configuration file)

Attention

For TLS connections, use the SSL Preread module instead.

server_names_hash_bucket_size#

Syntax

server_names_hash_bucket_size size;

Default

server_names_hash_bucket_size 32|64|128;

Context

stream

Sets the bucket size for the server names hash tables. The default value depends on the size of the processor's cache line.

server_names_hash_max_size#

Syntax

server_names_hash_max_size size;

Default

server_names_hash_max_size 512;

Context

stream

Sets the maximum size of the server names hash tables.

status_zone#

Syntax

status_zone zone;

Default

Context

server

Allocates a shared memory zone to collect metrics for /status/stream/server_zones/<zone>.

Multiple server contexts can share the same zone for data collection.

stream#

Syntax

stream { ... }

Default

Context

main

Provides the configuration file context in which the stream server directives are specified.

tcp_nodelay#

Syntax

tcp_nodelay on | off;

Default

tcp_nodelay on;

Context

stream, server

Enables or disables the use of the TCP_NODELAY option. The option is enabled for both client and proxied server connections.

variables_hash_bucket_size#

Syntax

variables_hash_bucket_size size;

Default

variables_hash_bucket_size 64;

Context

stream

Sets the bucket size for the variables hash table. The details of setting up hash tables are provided in a separate document.

variables_hash_max_size#

Syntax

variables_hash_max_size size;

Default

variables_hash_max_size 1024;

Context

stream

Sets the maximum size of the variables hash table. The details of setting up hash tables are provided in a separate document.

Built-in Variables#

The stream core module supports following variables:

$angie_version#

Angie version

$binary_remote_addr#

client address in a binary form, value's length is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses

$bytes_received#

number of bytes received from a client

$bytes_sent#

number of bytes sent to a client

$connection#

connection serial number

$hostname#

host name

$msec#

current time in seconds with the milliseconds resolution

$pid#

PID of the worker process

$protocol#

protocol used to communicate with the client: TCP or UDP

$proxy_protocol_addr#

client address from the PROXY protocol header
The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

$proxy_protocol_port#

client port from the PROXY protocol header
The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

$proxy_protocol_server_addr#

server address from the PROXY protocol header
The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

$proxy_protocol_server_port#

server port from the PROXY protocol header
The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

$proxy_protocol_tlv_ name#

TLV from the PROXY Protocol header. The name can be a TLV type name or its numeric value. In the latter case, the value is hexadecimal and should be prefixed with 0x:

$proxy_protocol_tlv_alpn
$proxy_protocol_tlv_0x01

SSL TLVs can also be accessed by TLV type name or its numeric value, both prefixed by ssl_:

$proxy_protocol_tlv_ssl_version
$proxy_protocol_tlv_ssl_0x21

The following TLV type names are supported:

  • alpn (0x01) - upper layer protocol used over the connection

  • authority (0x02) - host name value passed by the client

  • unique_id (0x05) - unique connection id

  • netns (0x30) - name of the namespace

  • ssl (0x20) - binary SSL TLV structure

The following SSL TLV type names are supported:

  • ssl_version (0x21) - SSL version used in client connection

  • ssl_cn (0x22) - SSL certificate Common Name

  • ssl_cipher (0x23) - name of the used cipher

  • ssl_sig_alg (0x24) - algorithm used to sign the certificate

  • ssl_key_alg (0x25) - public-key algorithm

Also, the following special SSL TLV type name is supported:

  • ssl_verify - client SSL certificate verification result, 0 if the client presented a certificate and it was successfully verified, non-zero otherwise.

The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

$remote_addr#

client address

$remote_port#

client port

$server_addr#

an address of the server which accepted a connection
Computing a value of this variable usually requires one system call. To avoid a system call, the listen directives must specify addresses and use the bind parameter.

$server_port#

port of the server which accepted a connection

$session_time#

session duration in seconds with a milliseconds resolution

$status#

session status, can be one of the following:

200

session completed successfully

400

client data could not be parsed, for example, the PROXY protocol header

403

access forbidden, for example, when access is limited for certain client addresses

500

internal server error

502

bad gateway, for example, if an upstream server could not be selected or reached

503

service unavailable, for example, when access is limited by the number of connections

$time_iso8601#

local time in the ISO 8601 standard format

$time_local#

local time in the Common Log Format