Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust Kubelet garbage collection and logging retention #4

Open
vigevenoj opened this issue Nov 7, 2017 · 3 comments
Open

Adjust Kubelet garbage collection and logging retention #4

vigevenoj opened this issue Nov 7, 2017 · 3 comments

Comments

@vigevenoj
Copy link
Owner

Currently if a node experiences disk pressure, Kubelet GC removes infrequently-used images. Our usual case is either kube-proxy or owntracks2db. kube-proxy can be downloaded from public repositories, but owntracks2db is side-loaded.

Fixing #3 will make this less painful to recover from, but docker logs should be truncated periodically and logging should be adjusted to be less noisy. traefik is set to debug and could be changed to a less-noisy logging level as a quick win.

@vigevenoj
Copy link
Owner Author

An overview of Kubelet GC is available in https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

@vigevenoj
Copy link
Owner Author

Temporary workaround, look at biggest log and truncate them

find /var/lib/docker/containers -name "*-json.log" -exec ls -lh {} ; | sort -h -k5

@vigevenoj
Copy link
Owner Author

We should configure the docker logging to rotate. 10 100m files should be enough to troubleshoot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant