Troubleshooting

If you encounter a technical issue and can't find a solution in other sections, ask a question on the community forum or in the Telegram channel.

Technical support for clients:

• https://support.angie.software

Debug Logging

The debug log should be enabled before performing self-diagnostics or as recommended by technical support.

To do this, run Angie using the executable with debug support:

LinuxFreeBSDDockerBuilding from Source

In the pre-built packages for Linux, the angie-debug file is built with debug support:

$ ls -l /usr/sbin/ | grep angie

   lrwxrwxrwx 1 root root      13 Sep 21 18:58 angie -> angie-nodebug
   -rwxr-xr-x 1 root root 1561224 Sep 21 18:58 angie-debug
   -rwxr-xr-x 1 root root 1426056 Sep 21 18:58 angie-nodebug

Configure running angie-debug:

$ sudo ln -fs angie-debug /usr/sbin/angie
$ sudo angie -t && sudo service angie upgrade

This will initiate a live executable upgrade.

To revert to the regular executable after debugging:

$ sudo ln -fs angie-nodebug /usr/sbin/angie
$ sudo angie -t && sudo service angie upgrade

Note

Using the executable with debug support may slightly reduce performance; enabling the debug log can significantly reduce it and increase disk space usage.

To enable the debug log, set the debug level in the configuration using the error_log directive (templated Docker images use an environment variable instead, as the Docker tab below shows):

error_log /path/to/log debug;

Then apply the change:

LinuxFreeBSDDockerBuilding from Source

$ sudo angie -t && sudo service angie reload

If you switch to the executable without debug support but leave the debug level in the error_log directive, Angie will log entries at the info level.

Directive Location

The location of the error_log directive affects the completeness of debug information collected.

A directive specified at a lower configuration level (for example, inside a server or location block) overrides logging settings specified at a higher level (for example, at the main configuration level or inside an http block).

Debug log disabled for a specific server

If debug logging is enabled globally but error_log is specified for an individual server without the debug level, debug information will not be collected for that server.

error_log /var/log/angie/error.log debug; # Global debug log

http {

    server {

        listen 80;
        server_name example.com;

        error_log /var/log/angie/example.com.error.log;
        # Debug log for example.com is disabled, file contains error level

        # ...
    }

    server {

        listen 80;
        server_name another.com;

        # This server will use the global debug log
        # ...
    }
}

Preserving debug log at server level

To keep collecting debug information for a specific server, remove its error_log override; to direct this information to a different file, also specify the debug level in the override:

error_log /var/log/angie/error.log debug; # Global debug log

http {

    server {

        listen 80;
        server_name example.com;

        error_log /var/log/angie/example.com.error.log debug;
        # Debug log for example.com is enabled but written to a separate file

        # ...
    }
}

Logging Specific Addresses

You can enable debug logging only for specified client addresses:

error_log /path/to/log;

events {
  debug_connection 192.168.1.1;
  debug_connection 192.168.10.0/24;
}

Cyclic Memory Buffer

Debug log can be written to a cyclic memory buffer:

error_log memory:32m debug;

Writing to the memory buffer at the debug level will not significantly impact performance even under high load. In this case, the log can be extracted using a GDB script, for example:

set $log = ngx_cycle->log

while $log->writer != ngx_log_memory_writer
  set $log = $log->next
end

set $buf = (ngx_log_memory_buf_t *) $log->wdata
dump binary memory debug_log.txt $buf->start $buf->end

Core Dumps

Core dumps help investigate crashes. Include them when contacting support. For builds from our repositories, we provide debug symbols in special packages. They have the same names as the original packages with the -dbg suffix added, for example angie-dbg.

Note

This section assumes you are running Angie as the root user (recommended).

Linux: systemd

To enable core dump saving when running Angie as a systemd service (for example, when installed from packages), modify the service settings in the /lib/systemd/system/angie.service file:

[Service]
...
LimitCORE=infinity
LimitNOFILE=65535

Or update the global settings in the /etc/systemd/system.conf file:

[Manager]
...
DefaultLimitCORE=infinity
DefaultLimitNOFILE=65535

Then reload the service configuration and restart Angie to reproduce the crash conditions:

$ sudo systemctl daemon-reload
$ sudo systemctl restart angie.service

After the crash, find the core dump file:

$ sudo coredumpctl -1 # optional

   TIME                           PID   UID   GID SIG COREFILE  EXE
   --- 2026-10-13 11:05:40 GMT   1157     0     0  11 present   /usr/sbin/angie

$ sudo ls -al /var/lib/systemd/coredump/  # default, see also /etc/systemd/coredump.conf and /etc/systemd/coredump.conf.d/*.conf

  ...
  -rw-r----- 1 root root 177662 Jul 27 11:05 core.angie.0.6135489c850b4fb4a74795ebbc1e382a.1157.1590577472000000.lz4

Linux: Manual Configuration

This section covers Angie started without systemd, such as a build from source, and assumes that Angie from this installation is running; the commands use the default paths of such a build under /usr/local/angie.

Where the kernel puts a core dump file depends on /proc/sys/kernel/core_pattern. If the pattern starts with |, the kernel passes the dump to the program named after it, such as systemd-coredump, even for Angie started without systemd; then find the core dump file as described in Linux: systemd and skip the working directory steps below. If the pattern is a plain name such as core, the kernel writes the core dump file to the working directory of the crashed process, which for worker processes is by default the directory that Angie was started from.

Worker processes run under the account that the user directive sets (nobody in a default source build); if this account cannot write to the working directory, the crash leaves no core dump file, so create a directory for core dump files:

$ sudo mkdir -p /var/tmp/angie-cores
$ sudo chown nobody /var/tmp/angie-cores

Set it with the working_directory directive in the main context of /usr/local/angie/conf/angie.conf:

working_directory /var/tmp/angie-cores;

Then raise the core dump size limit. To raise it for worker processes only and avoid a restart, also add the worker_rlimit_core directive, which takes a size rather than unlimited, and reload the configuration:

worker_rlimit_core 1g;

$ sudo /usr/local/angie/sbin/angie -s reload

To raise the limit for the master process as well, restart Angie instead. By default, sudo starts commands with a core dump size limit of 0, and ulimit changes the limit only for the shell that runs it and the commands it starts. Stop the running Angie, wait until its master process exits, then raise the limit and start Angie in one root shell:

$ sudo /usr/local/angie/sbin/angie -s quit
$ while [ -e /usr/local/angie/logs/angie.pid ]; do sleep 1; done
$ sudo sh -c 'ulimit -c unlimited && exec /usr/local/angie/sbin/angie'

A source build configured with --with-debug has debug support in /usr/local/angie/sbin/angie itself; it has no separate angie-debug executable.

Check the limit of a worker process; after the restart, both values are unlimited, and with worker_rlimit_core, they show its size in bytes:

$ grep 'core file' /proc/$(pgrep -n -f 'angie: worker process')/limits
Max core file size        unlimited            unlimited            bytes

Then reproduce the crash. If it left a core dump file, the error log (/usr/local/angie/logs/error.log by default) shows worker process pid exited on signal number (core dumped); the same message without (core dumped) means that the limit or the working directory prevented the core dump file. Find the core dump file:

$ sudo ls -al /var/tmp/angie-cores
  ...
  -rw------- 1 nobody nogroup 177662 Jul 27 11:05 core.1157

FreeBSD

Check the core dump settings in the /etc/sysctl.conf file, modify them if necessary:

kern.coredump=1                             # should be 1
kern.corefile=/path/to/core/files/%N.core   # needs correct path

Or update the settings at runtime:

$ sudo sysctl kern.coredump=1
$ sudo sysctl kern.corefile=/path/to/core/files/%N.core

Either way, the directory in kern.corefile must be writable by the account set by the user directive: worker processes run under this account and write their core dump files there.

Then restart Angie to reproduce the crash conditions. If Angie is installed as a service:

$ sudo service angie restart

If Angie is installed manually, stop it, wait until its master process exits, then raise the core dump size limit and start Angie in a single root shell, as on Linux:

$ sudo <path to Angie installation directory>/sbin/angie -s quit
$ while [ -e <path to Angie installation directory>/logs/angie.pid ]; do sleep 1; done
$ sudo sh -c 'ulimit -c unlimited && exec <path to Angie installation directory>/sbin/angie'

After the crash, find the core dump file:

$ sudo ls -al <path to core dump files>

  ...
  -rw------- 1 nobody nobody 9912320 Jul 27 11:05 angie.core