An active Kubernetes cluster can collect old containers and images. Ensuring discarded resources when redundant frees up resources on the nodes of your cluster. Here’s how to access garbage collection in Kubernetes.
Kubernetes has a built-in garage collection system that can clean up unused images. It is managed by Kubelet, the Kubernetes worker process that runs on every node.
Kubelet automatically checks for unused images and will periodically delete them. Decisions about deletion are made by assessing the disk usage of the image and the time it was last used. A large image that has not been used for a week is usually cleaned up for a small one that has been used yesterday.
You can customize when garbage collection is performed by specifying high and low disk usage thresholds. Disk usage above the “high”
The thresholds are defined using two Kubelet flags:
image-gc-high-threshold– Sets the high threshold; standard 85%.
image-gc-low-threshold– Sets the low threshold; standard 80%.
These settings should already be active in your cluster. Kubelet will try to reduce disk usage to 80% after it is 85% full.
You can set Kubectl flags in
Restart Kubectl after editing the file:
systemctl daemon-reload systemctl restart kubelet
Clean up old containers
Kubelet also takes care of cleaning up excess containers. All containers that are stopped or not identified are eligible for disposal.
You can give old containers a grace period before they are deleted by defining a minimum age for containers. Additional flags allow you to determine the total number of dead containers that can exist in a single pod and on the node:
maximum-dead-containers– Maximum number of old containers to be kept. If set to
-1(the default), no limit applies.
maximum-dead-containers-per-container– Set the number of older copies to keep per container. If a container is replaced by a newer instance, that many older versions can be kept.
minimum-container-ttl-duration– Grace period for waste collection for dead containers. Once a container is that many minutes old, it is eligible for garbage collection. The default value of
0means that no grace period applies.
You can configure these settings with Kubelet flags using the same procedure as described above.
Do I have to intervene manually?
You shouldn’t make any manual attempts to delete dead containers or images. If disk space fills up, or if garbage collection doesn’t seem to work, try adjusting your Kubelet flags to more aggressive settings.
Kubernetes warns against performing external garbage collection. Do not manually delete resources using cluster management APIs or third-party tools. This risks creating an inconsistent condition that could affect the operation of Kubelet.
Kubelet is responsible for managing the containers assigned to each node. When a new container is scheduled, Kubelet downloads its image. Successful cluster operations depend on meeting Kubelet’s expectations. A missing image or container can lead to Kubelet problems.
The Future: Evictions
The settings described above are supported in current Kubernetes versions. However, they are being phased out in favor of a more robust “expulsion” system. Stakes are a unified way to clean up Kubernetes resources; they will eventually replace garbage collection.
An expulsion can take place for various reasons. Kubelet monitors multiple factors, including available hardware resources and user configuration for retention periods.
This new system facilitates the disposal of waste collection as a special mechanism. The same process that terminates pods due to a low memory scenario will delete redundant images as disk space becomes limited.
Two types of expulsion are defined: difficult and soft. A hard haul will take immediate action to remove the target resource. There is no grace period. A soft evacuation has a user-configured grace period; the resource will be targeted once the grace period has expired. If the cause of the shutdown is resolved during the grace period, such as by freeing up more disk space, the deletion can be terminated.
The eviction system is not yet fully supported for container clean-up. The
dead-containers flags are already obsolete, ready for the future. You can already use it with container images – set
--eviction-soft instead of the
In this example, Kubelet is instructed to delete all unused container images if the available disk space for image storage falls below 1 GB.
This second example shows how a "soft" evacuation can be used instead. In this case, images are not deleted unless the available disk space is less than 1 GB for at least five minutes.
For Kubernetes, garbage collection is enabled by default. Dead containers and redundant images are periodically cleaned up. The default scheme targets disk usage of 80% or less; containers are cleared quite aggressively after they stop. You can use Kubelet flags to adjust the thresholds in the process.
Waste collection as a concept will eventually be disposed of in favor of evictions. Stakes have a simplified configuration that is more consistent with other forms of resource disposal. You can set up stakeouts in your cluster today. Keep in mind that the garbage collection flags do not point directly to their eviction counterparts.