Overview

First of all, this scenario is not in a working production environment, so the operation is rather random, but I mainly want to introduce some things related to docker logs.

While running a project, I found that the business was not working properly, and then I wanted to see what errors the logs output.

  1. [root@liqiang.io]# docker logs -f xxx

Surprisingly found no log content, which is rather confusing, because even if there are no errors, at least the startup information these or i should be there, so is the application not up?

Problem location

First of all, I confirmed that the Docker Container is running normally, which is fine; then I looked at the port the application is listening on to see if it is running.

  1. [root@liqiang.io]# ss -anlp | grep 222
  2. tcp LISTEN 0 4096 127.0.0.1:2222 *:* users:(("server",pid=27069,fd=10))

The port is also listening, at least in the business to this step, then the application must be output logs, then the problem is at the infrature level.

Next, let’s see if Docker itself is abnormal, let’s look at the docker logs first.

  1. [root@liqiang.io]# journalctl -xu docker
  2. Journal file /var/log/journal/ae6603ff6b1849af8525fbf6688811fd/system.journal uses an unsupported feature, ignoring file.
  3. Use SYSTEMD_LOG_LEVEL=debug journalctl --file=/var/log/journal/ae6603ff6b1849af8525fbf6688811fd/system.journal to see the details.
  4. -- No entries --

As it turns out, the problem is with Docker’s own journal, which looks like it’s related to the journal, and then the good thing is that there are hints here, so you can look at.

  1. [root@liqiang.io]# SYSTEMD_LOG_LEVEL=debug journalctl --file=/var/log/journal/ae6603ff6b1849af8525fbf6688811fd/system.journal
  2. Journal effective settings seal=no keyed_hash=yes compress=no compress_threshold_bytes=8B
  3. Journal file /var/log/journal/ae6603ff6b1849af8525fbf6688811fd/system.journal uses incompatible flag xz-compressed disabled at compilation time.
  4. Failed to open journal file /var/log/journal/ae6603ff6b1849af8525fbf6688811fd/system.journal: Protocol not supported
  5. mmap cache statistics: 0 hit, 1 miss
  6. Failed to open files: Protocol not supported

OK, the problem here looks like I recompiled systemd (CentOS7 upgrade systemd 242/243 ), and then it looks like I missed some compression options, so I need to recompile systemd.

docker logging configuration

I found the reason why Docker’s own logging was not working, but why was the container not logging, so I had to look at the logging configuration in docker, and after some searching I determined that it was here.

  1. [root@liqiang.io]# grep -ir log /etc/sysconfig/docker
  2. OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
  3. # Controls the /etc/cron.daily/docker-logrotate cron job status.
  4. # LOGROTATE=false

It turns out that the default docker log uses journal logs, so journal logs are not available, and docker’s own logs are not viewable. So now, in addition to solving the journal logging problem, you can also look at other logging drivers supported by docker (Driver list supported-logging-drivers). Then you may want to try it by selecting Local logging first.

After checking the documentation, I found that instead of modifying the global configuration, it was possible to modify the configuration of a single container, so I ran a

  1. [root@liqiang.io]# docker run \
  2.       --log-driver local --log-opt max-size=10m \
  3.       alpine echo hello world
  4. /usr/bin/docker-current: Error response from daemon: logger: no log driver named 'local' is registered.
  5. See '/usr/bin/docker-current run --help'.

However, the reality is that the run failed, and then I looked at the detailed description in the documentation and found no special instructions, but after looking at my docker version, I found that it was already very old, so I simply upgraded to the latest version 20.10.5 and found that the docker logs were back to normal.

Summary

This article is pretty much like this, not too many amazing things, but for the processing of docker logs really did not pay too much attention before, this time is also considered a posture up. The official documentation says that if you don’t set the rotate of log, it won’t clean up the log by default, so you need to pay attention to it yourself.

Ref