Writing to a container's main process stdout and stderr streams

A quick trick to make background and exec'd process output show up in docker logs and kubectl logs, using Linux's /proc filesystem.

Share
Writing to a container's main process stdout and stderr streams
Photo by Timelab / Unsplash

When you run a process inside a container, say via kubectl exec or from a cron job, its output usually won't show up in kubectl logs or docker logs. That's essentially because container runtimes capture logs from the stdout/stderr pipe of the container's main process (PID 1). Any process that doesn't share that pipe is invisible to the log collector.

There's a simple trick to fix this: redirect your process output to /proc/1/fd/1 (stdout) and /proc/1/fd/2 (stderr).

Container logging

When a container starts, the runtime (Docker, containerd, CRI-O) creates a pipe and connects it to the stdout and stderr of the entrypoint process, i.e., the process running as PID 1 inside the container's PID namespace. Everything written to that pipe is what docker logs and kubectl logs show you.

When you start a container with something like:

CMD ["nginx", "-g", "daemon off;"]

The nginx process becomes PID 1. Anything it writes to stdout or stderr goes through the pipe and gets picked up by the runtime.

Actually any child process that PID 1 forks also inherits those file descriptors; so a child's stdout is captured too, as long as it doesn't explicitly close or redirect it. This is standard POSIX behavior: fork() copies all open file descriptors.

Now the problem is when processes don't inherit PID 1's file descriptors. For example:

  • a process started via kubectl exec or docker exec would get its own stdio connected back to your terminal session, not to the container's log pipe
  • a cron job or a process launched by a supervisor that sets up fresh stdio
  • a daemon that explicitly closes its stdout/stderr on startup

The /proc/1/fd trick

Linux procfs exposes the file descriptors of every process under /proc/<pid>/fd/.
For example, a process with PID 1 would use/proc/1/fd/1 for stdout and /proc/1/fd/2 for stderr. These are actually symlinks to the actual file descriptors. Any process in the same PID namespace can open and write to them. So by redirecting output there, we're
effectively piping our process output into the container's log stream.

Basic usage

Shell redirect

echo "Hello from a background job" > /proc/1/fd/1
echo "Something went wrong" > /proc/1/fd/2

Running a command with redirected output

/bin/sh -c '(my-background-task) 1>>/proc/1/fd/1 2>>/proc/1/fd/2'

Note the use of >> (append) rather than > (truncate). If the container runtime backs the log stream with a file, > would truncate it and wipe all prior log content.

From a cron job or supervisor

Since cron jobs and supervised processes don't inherit PID 1's file descriptors, their output won't appear in docker logs by default. The redirect fixes that:

# In a crontab inside a container
* * * * * /usr/local/bin/cleanup.sh 1>>/proc/1/fd/1 2>>/proc/1/fd/2

Note that child processes forked directly from the entrypoint script already inherit PID 1's stdout, which means that they don't need this redirect unless they explicitly close or redirect their own stdio.

A real-world example: odo and Devfile commands

This technique is exactly what we recently used in odo (a Golang-based CLI tool for fast iterative development on Kubernetes/OpenShift) to make Devfile command output appear in odo logs and kubectl logs.

When odo runs a Devfile command (e.g., a build or run command) inside a container, the command is executed via kubectl exec. Without redirection, the output would only go back to the exec session and never appear in the container's log stream.

Here's the pattern odo uses (overly simplified):

/bin/sh -c "cd /projects && (npm start) 1>>/proc/1/fd/1 2>>/proc/1/fd/2"

The full command is more involved as it also stores the process PID for lifecycle
management:

/bin/sh -c "echo $$ > /shared/.odo_cmd_run.pid && \
  cd /projects && \
  (npm start) 1>>/proc/1/fd/1 2>>/proc/1/fd/2; \
  echo $? >> /shared/.odo_cmd_run.pid"

This way:

  • The command runs in the background inside the container
  • Its output is captured by the container runtime's log collector
  • kubectl logs and odo logs show the output seamlessly
  • The PID is tracked so the process can be stopped and restarted later

Caveats

Permissions

The process writing to /proc/1/fd/1 must have permission to open those file descriptors. If PID 1 is running as a different user, you may get a Permission denied error. Actually, this may happen even as root in some configurations, like the common cases below:

  • Rootless podman: container UIDs are mapped to unprivileged host UIDs, causing ownership mismatches on the fd entries
  • OpenShift / restricted security contexts: containers run with a random non-root UID, which typically differs from the UID that owns PID 1's fds
  • Certain runc configurations: the runtime may restrict access to PID 1's stdio based on the container user

PID namespace sharing

In Kubernetes, if you enable shared process namespaces across containers in a pod, PID 1 becomes the infrastructure pause container rather than your application. Writing to /proc/1/fd/1 sends output to the pause container's stdout, which kubectl logs doesn't expose for any app container. As a consequence, your output effectively goes to a black hole.

In this case, you need to find your application's actual PID and write to /proc/<app-pid>/fd/1 instead.

Not all PID 1 processes have a useful stdout

If PID 1 is a daemon that closes its stdout or redirects it to /dev/null, writing to /proc/1/fd/1 will fail or go nowhere. Init wrappers like tini or dumb-init are fine, so they're transparent and keep stdout open. But a traditional daemon that detaches from its terminal and closes stdio will leave /proc/1/fd/1 pointing at nothing useful. Check with ls -l /proc/1/fd/1 if you hit unexpected behavior.

Some container images set up stdout and stderr logging by symlinking log files to /dev/stdout and /dev/stderr at build time. This is what the official nginx image used to do for instance:

RUN ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log

This approach works well when you control the image, but /proc/1/fd/1 is the
escape hatch when you don't.

Try it yourself

Spin up a container and see it in action:

# Terminal 1: run a container

docker run --name test-container -d alpine sleep 3600

# Terminal 2: follow the container logs
docker logs -f test-container

# Terminal 3: exec into the container and write to PID 1's stdout
docker exec test-container sh -c 'echo "visible in docker logs!" > /proc/1/fd/1'

You'll see visible in docker logs! appear in Terminal 2. That's the entire
trick.

Wrapping up

/proc/1/fd/1 and /proc/1/fd/2 give you a direct pipe to the container's log stream from any process in the container. It's a simple Linux mechanism that solves a common container problem: getting output from background or exec'd processes into the standard log collection pipeline.

The next time you're scratching your head wondering why kubectl logs is empty
even though your process is clearly running and producing output, this is probably what you need.

Thanks for reading, and, as usual, feel free to leave your thoughts in the comments.