Great write up, but I fear this only the tip of the iceberg. And that’s not a criticism of your article — rather I’m lamenting the degree of peril that exists in this space. Here’s a hazard I happened upon recently — if a pod is short lived enough, its log file may disappear from /var/log/containers before fluentd (or other agent) notices it and begins to tail it. Similarly, if a container produces logs at a great enough rate, the log file can be rotated by the kubelet before tailing by the agent starts — in which many log entries could be overlooked.

I may be coming to view logging in k8s as a best-effort proposition, so I appreciated your emphasis on focusing on the logs that really matter and considering alternative methods of reporting on critical errors / events.