Stop Managing Log Aggregators: Use the Docker AWS Logs Driver

01The "Agent Fatigue" Incident

A few years ago, I was responsible for a fleet of EC2 instances running Dockerized microservices. For logging, we followed the industry-standard "best practice": install a heavy-duty logging agent like Fluentd or Filebeat on every single host. We spent more time debugging the agent—monitoring its memory usage, patching its config, and watching it crash during high-traffic bursts—than we did managing the actual application services.

Then, one afternoon, the agent process on our primary batch-processing node hung. It stopped shipping logs, buffered them to disk, and silently filled up the entire root partition until the OS panicked. The service went down, not because the application was flawed, but because the "observability infrastructure" decided to cannibalize the host it was supposed to be watching.

That was the day I stopped treating logging as a separate infrastructure problem and started treating it as a native component of the container runtime.

02Why Sidecar Agents Are a Maintenance Debt

Sidecar logging agents are incredibly powerful, but they are also incredibly high-maintenance. When you install an agent on your host, you are effectively running a whole new application that requires its own lifecycle management, security patching, and resource orchestration.

The hidden costs of the agent approach:

• Resource Overhead: Agents consume CPU and RAM that could be used by your actual application code.
• State Management: You are responsible for handling rotation, buffer overflow, and backpressure if the aggregator service has issues.
• Configuration Drift: Keeping the config files in sync across 50+ EC2 instances eventually leads to missing logs in your dashboard.
• Fragility: If the agent fails, it doesn't just stop logging—it can become a bottleneck that prevents the node from functioning correctly.

If you are using Docker on AWS, you don't need a middleman. The awslogs driver is built directly into the Docker engine.

03Stream Logs Directly to CloudWatch

The awslogs driver allows you to send container logs directly from the Docker daemon to Amazon CloudWatch Logs. It does exactly one thing, and it does it without requiring any extra processes to manage, patch, or monitor.

Configuring Docker Compose

To move away from agents, define the logging driver inside your docker-compose.yml file. This tells the Docker engine to take everything sent to stdout and stderr and push it straight to your defined Log Group.

services:  web-app:    image: my-app:latest    logging:      driver: "awslogs"      options:        awslogs-group: "/docker/my-app-logs"        awslogs-region: "us-east-1"        awslogs-stream: "web-app-container"        awslogs-create-group: "true"

When you run docker-compose up, the Docker daemon creates the Log Group if it doesn't exist (thanks to awslogs-create-group) and attaches the stream to the specified group. You don't need to touch the EC2 host at all.

04Security and IAM Scoping

Since your Docker daemon is now acting as an IAM client, it needs permissions to talk to the CloudWatch API. You should never use static access keys here.

Instead, attach an IAM Role to your EC2 instance (or use ECS Task Roles if you move to Fargate). Your policy needs minimal access:

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "logs:CreateLogGroup",        "logs:CreateLogStream",        "logs:PutLogEvents",        "logs:DescribeLogStreams"      ],      "Resource": "arn:aws:logs:*:*:*"    }  ]}

By assigning this role to the EC2 instance profile, the Docker engine automatically picks up the temporary credentials via the instance metadata service. Your logs flow securely without you ever handling a secret.

05What I Actually Do / My Take

I have completely banned sidecar logging agents from our EC2-based clusters. If a container needs logs, it goes to stdout/stderr. If it needs to go to CloudWatch, the awslogs driver handles it.

My standard playbook for production logging:

1. Strictly stdout: I enforce that all applications write logs to standard output. If an app writes to files, it creates a local state that is hard to manage. If the app crashes, those local files are gone.
2. Use the Driver: For our core Docker Compose setups on EC2, the awslogs driver is mandatory. It is the most "cloud-native" way to handle logs without adding a maintenance layer.
3. Global Configuration: For production, I often set the awslogs driver globally in /etc/docker/daemon.json so that even if a developer forgets to add the logging block to a docker-compose.yml, the logs are still captured by default. This creates a safety net where every container is logged by default.
4. Keep it simple: If we ever reach a point where we need complex log filtering, transformation, or routing to non-AWS destinations, only then will I reconsider a dedicated agent. But 99% of the time, that complexity is premature optimization.

Stop treating your logging infra like a service that needs its own life-support. By using native drivers, you shift the burden to the platform providers, allowing you to focus on the code that actually provides business value.

06Closing / TL;DR

Sidecar logging agents like Fluentd are a common source of instability and maintenance overhead on EC2. Replace them with the native Docker awslogs driver to stream directly to CloudWatch. It is more secure, requires zero process management, and keeps your containers ephemeral and stateless. Let AWS handle the log buffer; you have better things to do.