TransportServer#

The TransportServer resource allows you to configure load balancing for TCP, UDP, and TLS Passthrough protocols. It is implemented as a custom resource.

This is the reference documentation for the TransportServer resource.

Prerequisites#

For TCP and UDP, the TransportServer resource must be used in conjunction with the GlobalConfiguration resource, which must be created separately.
For TLS Passthrough, make sure to enable the command-line parameter -enable-tls-passthrough in ANIC.

TransportServer Specification#

The TransportServer resource defines the load balancing configuration for TCP, UDP, or TLS Passthrough traffic. Below are some examples:

TCP Load Balancing:

apiVersion: k8s.angie.software/v1alpha1
kind: TransportServer
metadata:
  name: dns-tcp
spec:
  listener:
    name: dns-tcp
    protocol: TCP
  tls:
    secret: cafe-secret
  upstreams:
  - name: dns-app
    service: dns-service
    port: 5353
    action:
      pass: dns-app

UDP Load Balancing:

apiVersion: k8s.angie.software/v1alpha1
kind: TransportServer
metadata:
  name: dns-udp
spec:
  listener:
    name: dns-udp
    protocol: UDP
  upstreams:
  - name: dns-app
    service: dns-service
    port: 5353
    upstreamParameters:
      udpRequests: 1
      udpResponses: 1
    action:
      pass: dns-app

TLS Passthrough Load Balancing:

apiVersion: k8s.angie.software/v1alpha1
kind: TransportServer
metadata:
  name: secure-app
spec:
  listener:
    name: tls-passthrough
    protocol: TLS_PASSTHROUGH
  host: app.example.com
  upstreams:
  - name: secure-app
    service: secure-app
    port: 8443
    action:
      pass: secure-app

Field	Description	Type	Required
`listener`	The listener through which Angie will accept incoming connections and datagrams.	Listener	Yes
`host`	The host (domain name) of the server. This must be a valid subdomain as defined in RFC 1123, such as `my-app` or `hello.example.com`. Wildcard domains like `*.example.com` are not allowed. Required for TLS Passthrough load balancing.	string	No
tls	TLS termination configuration. Not supported for TLS Passthrough load balancing.	TLS	No
`upstreams`	A list of upstreams.	[]upstream	Yes
`upstreamParameters`	Upstream parameters.	UpstreamParameters	No
`action`	The action to be taken for the client connection or datagram.	Action	Yes
`ingressClassName`	Specifies which instance of ANIC should handle the TransportServer resource.	`string`	No
`streamSnippets`	Specifies a custom snippet in the context of `stream`.	`string`	No
`serverSnippets`	Specifies a custom snippet in the context of `server`.	`string`	No

Listener#

Refers to the listener through which Angie will receive incoming traffic to the TransportServer. For TCP and UDP, the listener must be defined in the GlobalConfiguration resource. When referencing the listener, both the name and protocol must match. For TLS Passthrough, use the built-in listener named tls-passthrough with the protocol TLS_PASSTHROUGH.

Example:

listener:
  name: dns-udp
  protocol: UDP

Field	Description	Type	Required
`name`	The name of the listener.	`string`	Yes
`protocol`	The protocol of the listener.	`string`	Yes

TLS#

The tls field defines the TLS configuration for the TransportServer. Note that the current implementation supports TLS termination on multiple ports, where each application has its own dedicated port. In this case, ANIC terminates TLS connections on each port, where each application uses its own certificate or key, and directs connections to the appropriate application (service) based on the incoming port (i.e., any TLS connection regardless of SNI settings on the port will be redirected to the application corresponding to that port). An example configuration is shown below:

secret: cafe-secret

Field	Description	Type	Required
`secret`	The name of the secret containing the TLS certificate and key. The secret must belong to the same namespace as the transport server. The secret must be of type `kubernetes.io/tls` and contain keys named `tls.crt` and `tls.key`, containing the certificate and private key, as described here.	`string`	No

Upstream#

Defines the endpoint for the TransportServer. For example:

name: secure-app
service: secure-app
port: 8443
maxFails: 3
maxConns: 100
failTimeout: 30s
loadBalancingMethod: least_conn

Field	Description	Type	Required
`name`	The name of the upstream. It should be a valid DNS label as defined in RFC 1035. For example, valid values are `hello` and `upstream-123`. The name must be unique among all upstreams of the resource.	`string`	Yes
`service`	The name of the service. The service must belong to the same namespace as the resource. If the service does not exist, Angie will assume it has no endpoints and will close client connections and ignore datagrams.	`string`	Yes
`port`	The service port. If this port is not set for the service, Angie will assume it has no endpoints and will close client connections and ignore datagrams. The value must be in the range `1..65535`.	`int`	Yes
`maxFails`	Specifies the number of failed attempts to connect to the server that must occur within the time specified by the failTimeout parameter for the server to be considered unavailable. Default value: `1`.	`int`	No
`maxConns`	Specifies the maximum number of connections to the proxied server. The default value is zero, which means no limits. Default value is `0`.	`int`	No
`failTimeout`	Specifies the time, during which the specified number of failed attempts to connect to the server must occur for the server to be considered unavailable, and the period during which the server will be considered unavailable. The default value is `10` seconds.	`string`	No
`loadBalancingMethod`	The load balancing method between upstream servers. By default, connections are distributed among servers using the weighted round-robin method. Available methods and details can be found in the Upstream section.	`string`	No

UpstreamParameters#

Various upstream parameters:

upstreamParameters:
  udpRequests: 1
  udpResponses: 1
  connectTimeout: 60s
  nextUpstream: true
  nextUpstreamTimeout: 50s
  nextUpstreamTries: 1

Field	Description	Type	Required
`udpRequests`	The number of datagrams after which the next datagram from the same client starts a new session. See the proxy_requests directive. Default value is `0`.	`int`	No
`udpResponses`	The number of datagrams expected from the proxied server in response to the client datagram. See the proxy_responses. By default, the number of datagrams is unlimited.	`int`	No
`connectTimeout`	The timeout for establishing a connection with the proxied server. See the proxy_connect_timeout directive. Default value is `60` seconds.	`string`	No
`nextUpstream`	If a connection to the proxied server fails, determines whether the client connection will be passed to the next server. See the proxy_next_upstream directive. Default value is `true`.	bool	No
`nextUpstreamTries`	The number of attempts before passing the connection to the next server. See the proxy_next_upstream_tries directive. Default value is `0`.	`int`	No
`nextUpstreamTimeout`	The time allocated for passing the connection to the next server. See the proxy_next_upstream_timeout. Default value is `0`.	`string`	No

SessionParameters#

Various parameters for TCP connections and UDP sessions.

sessionParameters:
  timeout: 50s

Field	Description	Type	Required
`timeout`	The timeout between two consecutive read or write operations in connections with the client or the proxied server. See the proxy_timeout. Default value is `10m`.	`string`	No

Action#

The action to be taken for the client connection or datagram.

In the example below, client connections and datagrams are passed to the upstream dns-app:

action:
  pass: dns-app

Field	Description	Type	Required
`pass`	Passes connections and datagrams to the upstream. An upstream with that name must be defined in the resource.	`string`	Yes

Using TransportServer#

You can use standard kubectl commands to work with TransportServer resources, similar to Ingress resources.

For example, the following command creates a TransportServer resource defined in transport-server-passthrough.yaml with the name secure-app:

kubectl apply -f transport-server-passthrough.yaml

transportserver.k8s.angie.software/secure-app created

You can retrieve the resource by executing:

kubectl get transportserver secure-app

NAME         AGE
secure-app   46s

In kubectl get and similar commands, you can also use the short name ts instead of transportserver.

Using Snippets#

Snippets allow you to insert Angie configuration elements into various contexts of Angie configuration. In the example below, we use snippets to configure access control on the TransportServer:

apiVersion: k8s.angie.software/v1alpha1
kind: TransportServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  serverSnippets: |
    deny  192.168.1.1;
    allow 192.168.1.0/24;
  upstreams:
  - name: tea
    service: tea-svc
    port: 80

Snippets can also be specified for streams. In the example below, we use snippets for limiting the number of connections :

apiVersion: k8s.angie.software/v1alpha1
kind: TransportServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  streamSnippets: limit_conn_zone $binary_remote_addr zone=addr:10m;
  serverSnippets: limit_conn addr 1;
  upstreams:
  - name: tea
    service: tea-svc
    port: 80

Snippets are intended for advanced Angie users who require more control over the generated Angie configuration.

However, due to the drawbacks described below, snippets are disabled by default. To use snippets, set the command-line argument enable-snippets.

Drawbacks of using snippets:

Complexity. Using snippets requires:
- Understanding Angie configuration primitives and implementing the correct Angie configuration.
- Understanding how ANIC generates Angie configuration so that the snippet does not interfere with other configuration features.
Reduced reliability. An incorrect snippet makes the Angie configuration invalid, which will result in an error during reload. This will prevent any configuration updates from being applied, including updates for another TransportServer resource, until the snippet is corrected.
Security implications. Snippets provide access to Angie configuration primitives, and these primitives are not validated by ANIC itself.

Note

While the Angie configuration contains an invalid snippet, Angie will continue to operate with the last valid configuration.

Note

To configure snippets in the context of stream, use the stream-snippets key in the ConfigMap.

Validation#

There are two types of validation available for the TransportServer resource:

Structural validation using kubectl and the Kubernetes API server.
Comprehensive validation using ANIC.

Structural Validation#

The custom resource definition for TransportServer includes an OpenAPI structural schema that describes the type of each field of the resource.

If you try to create (or update) a resource that violates the structural schema (for example, using a string value for the upstream port field), the kubectl server and Kubernetes API will reject such a resource:

Example of kubectl validation:

kubectl apply -f transport-server-passthrough.yaml

  error: error validating "transport-server-passthrough.yaml": error validating data: ValidationError(TransportServer.spec.upstreams[0].port): invalid type for software.angie.k8s.v1alpha1.TransportServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false

Example of Kubernetes API server validation:

kubectl apply -f transport-server-passthrough.yaml --validate=false

  The TransportServer "secure-app" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list:
  spec.upstreams.port in body must be of type integer: "string"

If the resource is not rejected (i.e., does not violate the structural schema), ANIC will further validate it.

Comprehensive Validation#

ANIC checks the fields of the TransportServer resource. If the resource is invalid, ANIC will reject it: the resource will continue to exist in the cluster, but ANIC will ignore it.

You can check whether ANIC successfully applied the TransportServer configuration. For the example TransportServer secure-app, we can run:

kubectl describe ts secure-app

. . .
Events:
  Type    Reason          Age   From                      Message
-----
Normal  AddedOrUpdated  3s    angie-ingress-controller  Configuration for default/secure-app was added or updated

Note that the "Events" section includes a Normal event with the reason AddedOrUpdated, which informs us that the configuration was successfully applied.

If you create an invalid resource, ANIC will reject it and issue a Rejected event. For example, if you create a TransportServer secure-app with a pass action that references a non-existent upstream, you will receive:

kubectl describe ts secure-app

. . .
Events:
  Type     Reason    Age   From                      Message
-----
Warning  Rejected  2s    angie-ingress-controller  TransportServer default/secure-app is invalid and was rejected: spec.action.pass: Not found: "some-app"

Note that the events section includes a Warning event with the reason Rejected.

Note

If you make an error in an already existing resource, the ingress controller will reject it and remove the corresponding configuration from Angie.

Configuration using ConfigMap#

The keys in ConfigMap (except for stream-snippets, stream-log-format, resolver-addresses, resolver-ipv6, resolver-valid, and resolver-timeout) do not affect TransportServer resources.