<!-- review: finished -->

<a id="troubleshooting"></a>

# Troubleshooting

If you encounter a technical issue
and can't find a solution in other sections,
ask a question on the [community forum](https://forum.angie.support/)
or in the [Telegram channel](https://t.me/angie_support).

Technical support for clients:

- [https://support.angie.software](https://support.angie.software)
- [support@angie.software](mailto:support@angie.software)

<a id="debug-logging"></a>

## Debug Logging

The debug log should be enabled
before performing self-diagnostics or as recommended by technical support.

To do this, run Angie using the executable with debug support:

Linux

In the [pre-built packages](https://en.angie.software//angie/docs/installation/index.md#install-packages) for Linux,
the `angie-debug` file is built with debug logging enabled:

```console
$ ls -l /usr/sbin/ | grep angie

   lrwxrwxrwx 1 root root      13 Sep 21 18:58 angie -> angie-nodebug
   -rwxr-xr-x 1 root root 1561224 Sep 21 18:58 angie-debug
   -rwxr-xr-x 1 root root 1426056 Sep 21 18:58 angie-nodebug
```

Configure running `angie-debug`:

```console
$ sudo ln -fs angie-debug /usr/sbin/angie
$ sudo angie -t && sudo service angie upgrade
```

This will initiate a [live executable upgrade](https://en.angie.software//angie/docs/configuration/runtime.md#service-upgrade).

To revert to the regular executable after debugging:

```console
$ sudo ln -fs angie-nodebug /usr/sbin/angie
$ sudo angie -t && sudo service angie upgrade
```

FreeBSD

In the [pre-built packages](https://en.angie.software//angie/docs/installation/index.md#install-packages) for FreeBSD,
the `angie-debug` file is built with debug logging enabled:

```console
$ ls -l /usr/local/sbin/ | grep angie

   lrwxrwxrwx 1 root root      13 Sep 21 18:58 angie -> angie-nodebug
   -rwxr-xr-x 1 root root 1561224 Sep 21 18:58 angie-debug
   -rwxr-xr-x 1 root root 1426056 Sep 21 18:58 angie-nodebug
```

Configure running `angie-debug`:

```console
$ sudo ln -fs angie-debug /usr/local/sbin/angie
$ sudo angie -t && sudo service angie upgrade
```

This will initiate a [live executable upgrade](https://en.angie.software//angie/docs/configuration/runtime.md#service-upgrade).

To revert to the regular executable after debugging:

```console
$ sudo ln -fs angie-nodebug /usr/local/sbin/angie
$ sudo angie -t && sudo service angie upgrade
```

Docker

In [templated Docker images](https://en.angie.software//angie/docs/installation/docker.md#docker-images),
you can switch to the debug version
by overriding the `ANGIE_BINARY` environment variable:

```console
$ docker run -it --rm -e ANGIE_BINARY="angie-debug" \
  docker.angie.software/angie:templated
```

Building from Source

When building Angie from source,
[enable debugging](https://en.angie.software//angie/docs/installation/sourcebuild.md#configure) before compilation:

```console
$ ./configure --with-debug ...
```

After installation, **angie -V** allows verifying
that debug logging is enabled:

```console
$ angie -V

  ...
  configure arguments: --with-debug ...
```

#### NOTE
Using the executable with debug support
may slightly reduce performance;
enabling the debug log can significantly reduce it
and increase disk space usage.

To enable the debug log,
set the `debug` level in the configuration
using the [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log) directive:

```nginx
error_log /path/to/log debug;
```

And reload the configuration:

```console
$ sudo angie -t && sudo service angie reload
```

In [templated Docker images](https://en.angie.software//angie/docs/installation/docker.md#docker-images)
with debug logging enabled,
you can also use the `ANGIE_ERROR_LOG_SEVERITY`
environment variable:

```console
$ docker run -it --rm -e ANGIE_BINARY="angie-debug" \
-e ANGIE_ERROR_LOG_SEVERITY="debug" \
docker.angie.software/angie:templated
```

If you switch to the executable without debug support
but leave the `debug` level in the [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log) directive,
Angie will log entries at the `info` level.

Overriding [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log) in the configuration
without specifying the `debug` level disables the debug log.
Here, overriding the log at the [server](https://en.angie.software//angie/docs/configuration/modules/http/index.md#server) level
disables debug logging for an individual server:

```nginx
error_log /path/to/log debug;

http {
   server {
     error_log /path/to/log;
    # ...
```

To avoid this, remove the line that overrides [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log),
or set the `debug` level in it:

```nginx
error_log /path/to/log debug;

http {
   server {
     error_log /path/to/log debug;
   #  ...
```

<a id="directive-location"></a>

### Directive Location

The location of the [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log) directive
affects the completeness of debug information collected.

A directive specified at a lower configuration level
(for example, inside a `server` or `location` block)
overrides logging settings specified at a higher level
(for example, at the main configuration level or inside an `http` block).

### Debug log disabled for a specific server

If debug logging is enabled globally
but [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log) is specified for an individual server without the `debug` level,
debug information will not be collected for that server.

```nginx
error_log /var/log/angie/error.log debug; # Global debug log

http {

    server {

        listen 80;
        server_name example.com;

        error_log /var/log/angie/example.com.error.log;
        # Debug log for example.com is disabled, file contains info level

        # ...
    }

    server {

        listen 80;
        server_name another.com;

        # This server will use the global debug log
        # ...
    }
}
```

### Preserving debug log at server level

To preserve debug information collection for a specific server
but direct it to a different file,
you must also specify the `debug` level:

```nginx
error_log /var/log/angie/error.log debug; # Global debug log

http {

    server {

        listen 80;
        server_name example.com;

        error_log /var/log/angie/example.com.error.log debug;
        # Debug log for example.com is enabled but written to a separate file

        # ...
    }
}
```

Therefore, to enable debug logging globally
but override the log file for individual blocks,
also specify the `debug` level in those overrides.
Otherwise, if no logging level is specified in the [error_log](https://en.angie.software//angie/docs/configuration/modules/core.md#error-log) directive,
the `error` level will be used by default
and debug information for those blocks will be lost.

<a id="logging-specific-addresses"></a>

### Logging Specific Addresses

You can enable debug logging only for
[specified client addresses](https://en.angie.software//angie/docs/configuration/modules/core.md#debug-connection):

```nginx
error_log /path/to/log;

events {
  debug_connection 192.168.1.1;
  debug_connection 192.168.10.0/24;
}
```

<a id="cyclic-memory-buffer"></a>

### Cyclic Memory Buffer

Debug log can be written to a cyclic memory buffer:

```nginx
error_log memory:32m debug;
```

Writing to the memory buffer at the `debug` level
will not significantly impact performance even under high load.
In this case, the log can be extracted using a GDB script, for example:

```console
set $log = ngx_cycle->log

while $log->writer != ngx_log_memory_writer
  set $log = $log->next
end

set $buf = (ngx_log_memory_buf_t *) $log->wdata
dump binary memory debug_log.txt $buf->start $buf->end
```

<a id="core-dumps"></a>

## Core Dumps

Core dumps help investigate crashes.
Include them when [contacting support](#troubleshooting).
For builds from [our repositories](https://en.angie.software//angie/docs/installation/index.md#install-packages),
we provide debug symbols in special packages.
They have the same names as the original packages
with the `-dbg` suffix added, for example `angie-dbg`.

#### NOTE
This section assumes
you are running Angie as the `root` user (recommended).

<a id="linux-systemd"></a>

### Linux: systemd

To enable core dump saving when running Angie as a **systemd** service
(for example, when [installed from packages](https://en.angie.software//angie/docs/installation/index.md#install-packages)),
modify the [service settings](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties)
in the `/lib/systemd/system/angie.service` file:

```ini
[Service]
...
LimitCORE=infinity
LimitNOFILE=65535
```

Or update the [global settings](https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html)
in the `/etc/systemd/system.conf` file:

```ini
[Manager]
...
DefaultLimitCORE=infinity
DefaultLimitNOFILE=65535
```

Then reload the service configuration and restart Angie
to reproduce the crash conditions:

```console
$ sudo systemctl daemon-reload
$ sudo systemctl restart angie.service
```

After the crash, find the core dump file:

```console
$ sudo coredumpctl -1 # optional

   TIME                           PID   UID   GID SIG COREFILE  EXE
   --- |sampledateshort| 11:05:40 GMT   1157     0     0  11 present   /usr/sbin/angie

$ sudo ls -al /var/lib/systemd/coredump/  # default, see also /etc/systemd/coredump.conf and /etc/systemd/coredump.conf.d/*.conf

  ...
  -rw-r----- 1 root root 177662 Jul 27 11:05 core.angie.0.6135489c850b4fb4a74795ebbc1e382a.1157.1590577472000000.lz4
```

<a id="linux-manual-configuration"></a>

### Linux: Manual Configuration

Check the [core dump settings](https://man7.org/linux/man-pages/man5/limits.conf.5.html)
in the `/etc/security/limits.conf` file, modify them if necessary:

```none
root soft core 0          # by default disables core dumps
root hard core unlimited  # allows increasing the size limit
```

Then increase the core dump size limit using [ulimit](https://man7.org/linux/man-pages/man1/ulimit.1p.html),
then restart Angie to reproduce the crash conditions:

```console
$ sudo ulimit -c unlimited
$ sudo cd <path to Angie installation directory>
$ sudo sbin/angie  # or sbin/angie-debug
```

After the crash, find the core dump file:

```console
$ sudo ls -al <path to Angie working directory>  # default, see /proc/sys/kernel/core_pattern
  ...
  -rw-r----- 1 root root 177662 Jul 27 11:05 core.1157
```

<a id="freebsd"></a>

### FreeBSD

Check the [core dump settings](https://man.freebsd.org/cgi/man.cgi?query=sysctl.conf&sektion=5)
in the `/etc/sysctl.conf` file, modify them if necessary:

```ini
kern.coredump=1                             # should be 1
kern.corefile=/path/to/core/files/%N.core   # needs correct path
```

Or update the settings at runtime:

```console
$ sudo sysctl kern.coredump=1
$ sudo sysctl kern.corefile=/path/to/core/files/%N.core
```

Then restart Angie to reproduce the crash conditions.
If Angie is installed as a service:

```console
$ sudo service angie restart
```

If Angie is installed manually:

```console
$ sudo cd <path to Angie installation directory>
$ sudo sbin/angie
```

After the crash, find the core dump file:

```console
$ sudo ls -al <path to core dump files>

  ...
  -rw------- 1 root root 9912320 Jul 27 11:05 angie.core
```