Docker Syslog

Why use syslog-ng for collecting Docker logs? Docker already provides many drivers for logging, even for central log collection. On the other hand, remote logging drivers arrive with a minimalist feature set and you are not able to use the “doc. Jan 07, 2018 specialised container to build doc for the rsyslog.com website. Jun 28, 2017 Why use syslog-ng for collecting Docker logs? Docker already provides many drivers for logging, even for central log collection. On the other hand, remote logging drivers arrive with a minimalist feature set and you are not able to use the “doc. Syslog-ng reads the journals and sends the processed messages to Elasticsearch, which in fact runs in the same Docker environment. The logging daemon stores the logs both on local filesystem and in Elasticsearch. Therefore in case Elastic goes down, no logs will be lost. Configuring Docker daemon to store logs of containers in journald. As syslog-ng is running in the container, the docker daemon is collecting the logs from the running container and sending it over the network to the syslog server. This is why the filtering has to be performed on the network source which is just as you have done.

When you’re troubleshooting an application, the first instinct is to read the logs to understand what’s happening. When working with containers, though, we need to consider the ephemeral nature of a container. By the time you find out there’s a problem with your application, the container and the logs might be gone. Fortunately, Docker has a native way of reading logs from a container.

However, we live in a world where we won’t have only one container running, and our systems have multiple dependencies. We need to have a centralized location for our logs to have a better context of the problem. Also, if the container has gone away, we can read the logs collected while it was running.

In this post, I’ll provide a quick overview of why having a syslog server might be a good option. Then, I’ll show you how you can integrate syslog and Docker to improve your troubleshooting strategy through logs.

Reading Logs From a Container

Docker comes with a native command, docker logs, to read logs from a container. The following command runs a container that simply prints a text indefinitely.

If you want to read the logs from that container, you need either the name or the ID of the container. Then, you can read the logs with the docker logs CONTAINER, and you can add the -f flag to see logs live. See below for an example of the commands you’d usually run:

That’s initially good because you can read the logs from a container. However, if the container goes away, you can’t read its logs anymore using the logs command. This command only works while the container is running. We need a better way to collect container logs, so we don’t lose them when the container stops—containers come and go all the time.

There are many ways you can process and retain Docker container logs. But this time, I’ll use a syslog server. You’ll notice, though, there’s going to be a missing piece once logs are there—I’ll come back to this later.

Why Syslog for Logging in Docker?

Using a syslog server might not be the first option you think of when you think about collecting container logs. However, it’s a cheap and straightforward strategy to help you get started. It’s likely you already have a logging architecture using syslog and the missing piece was how to collect logs from Docker.

For instance, you could use a central syslog server as an aggregation location and then process all logs for removal of sensitive or private information, aggregating data, or forwarding logs to other places. Additionally, for performance reasons at the networking level, you could decide to have a syslog server locally as a buffer to network traffic. Or, perhaps you’re even planning on removing syslog soon, but you need to process Docker logs in the meantime. Nonetheless, troubleshooting without proper logs is challenging, so syslog provides a quick and easy way to start.

Configuring a Syslog Server

I’m using an Ubuntu 18.04 server in AWS, which already comes with rsyslog (the syslog server) installed. Otherwise, you’ll install it with the sudo apt install -y rsyslog command if you’re using Ubuntu. Once you have rsyslog, it’s good practice to create a configuration file specifically for the Docker daemon. We’d like to have a dedicated location for these types of logs, including the Docker daemon and container logs.

First, create a configuration file to collect all logs from the Docker daemon. Create a file at “/etc/rsyslog.d/49-docker-daemon.conf” using sudo with the following content:

Now, create a configuration file to collect logs from containers in a separate file. Create a file at “/etc/rsyslog.d/48-docker-containers.conf” using sudo with the following content:

And let’s restart the rsyslog server to confirm everything is working:

Configuring Docker for Sending Logs to Syslog

Once rsyslog is ready to process the Docker logs, let’s configure the Docker daemon to send the logs to syslog.

Modify (or create it, as I did) the Docker daemon configuration (“/etc/docker/daemon.json”) with sudo permissions, and add the following content:

In this case, I’m including a tag with the value dev to identify the logs I’m collecting in this server. Also, I’m using the local syslog server (through sockets), but this could be a remote server as well. For instance, using the syslog-address property, you could have something like this:

You might use the local syslog server to avoid losing logs for network connectivity and to forward logs to a centralized server.

Now, let’s restart the Docker daemon:

To make sure everything is working, let’s create a new container:

This time, if you try to read the logs from the container using the log command, you’ll get an error:

So, to read the logs from the container, you can go to the file generated by rsyslog:

Both the daemon and container file logs are there. Let’s look at each one of them:

Syslog

Docker Syslog

As you can see, the name of the files and the log format complies with the structure I defined in the rsyslog configuration.

And that’s how you can configure the Docker daemon to send logs to a syslog server.

There’s a Missing Piece

Docker Syslog Option

What’s the missing piece I previously mentioned? Well, once logs land at the syslog server, you need to do something with them. Even if logs are now centralized and it doesn’t matter if the container source of logging goes away, you still need to have a better way of reading logs. Going to a server and exploring logs manually with a grep or tail command won’t scale (in addition to being notoriously dynamic, containers and their logs tend to be quite prolific). What if you use syslog as an intermediary location? For instance, once you collect logs, you can send them to a system in which to run search queries, identify patterns, or create dashboards and alerts.

Installing Syslog On The Raspberry Pi Using Syslog-NG

Using SolarWinds® Loggly®, there are several ways you can process logs. For instance, you could have an agent running in the syslog server or sidecar container forwarding the logs to Loggly. In Loggly, you can then analyze the logs you’re collecting from your systems, even if they’re not running in containers. Give it a try in Loggly today.

Loggly

See unified log analysis and monitoring for yourself.

This post was written by Christian Meléndez. Christian is a technologist who started as a software developer and has more recently become a cloud architect focused on implementing continuous delivery pipelines with applications in several flavors, including .NET, Node.js, and Java, often using Docker containers.

What Is The System Log (Syslog)? - Definition From Techopedia

Related Posts