Resource handling

Kubernetes uses YAML files to specify resource requirements for pods and containers, including CPU and memory resources. CPU resources are allocated using CPU requests and CPU limits in millicores. If a container exceeds the CPU limit, it will be throttled by the kernel. Memory resources are allocated using memory requests and memory limits in units such as bytes, kilobytes, megabytes, or gigabytes. If a container exceeds its memory limit, Kubernetes will terminate and restart it, which is known as an OOMKilled event. Properly configuring memory limits is important to avoid degraded performance and downtime.

Resource requirements for pods and containers in Kubernetes can be specified in the YAML file. Kubernetes uses this information to schedule and manage the deployment of the pod or container across the available nodes in the cluster.

CPU Resources

In Kubernetes, CPU resources can be allocated to Pods and containers to ensure that they have the required processing power to run their applications. CPU resources are defined using the CPU resource units (millicores or milliCPU) and can be specified as CPU requests and CPU limits.

CPU Requests and Limits with YAML

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      requests:
        cpu: "100m"
      limits:
        cpu: "200m"

This example specifies a pod with a single container that has a CPU request of 100 millicores (100m) and a CPU limit of 200 millicores (200m):

Impact of Exceeding CPU Limits

If a container tries to exceed its CPU request, it will be scheduled on a node that has enough CPU resources to satisfy the request. However, if a container tries to exceed its CPU limit, it will be throttled by the kernel. This can cause the container to become unresponsive, leading to degraded performance and potentially impacting other containers running on the same node.

Memory Resources

In Kubernetes, memory resources can be allocated to Pods and containers to ensure that they have the required memory to run their applications. Memory resources are defined using the memory resource units (such as bytes, kilobytes, megabytes, or gigabytes) and can be specified as memory requests and memory limits.

Memory Requests and Limits with YAML

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      requests:
        memory: "64Mi"
      limits:
        memory: "128Mi"

This example specifies a pod with a single container that has a memory request of 64 megabytes (64Mi) and a memory limit of 128 megabytes (128Mi).

Impact of Exceeding Memory Limits

When a container exceeds its memory limit in Kubernetes, it is terminated and restarted by Kubernetes. This event is known as OOMKilled, which stands for Out Of Memory Killed. This happens when a container or a pod consumes all the available memory resources allocated to it, and the kernel terminates it to prevent it from consuming more memory and impacting the stability of the node. It is important to properly configure memory limits for containers and pods to avoid OOMKilled events, as they can lead to degraded performance and downtime.

Commands

Check the CPU and memory usage for all Pods

kubectl top pods -A

Output columns:

NAMESPACE     NAME    CPU(cores)   MEMORY(bytes)

Find the pod that consumes the most CPU resources

kubectl top pods -A --no-headers | sort --reverse --key 3 | head -n 1
kubectl top pods -A --no-headers | sort -nr -k3 | head -1

Find the pod that consumes the most memory resources

kubectl top pods -A --no-headers | sort --reverse --key 4 | head -n 1
kubectl top pods -A --no-headers | sort -nr -k4 | head -1

Check the CPU and memory usage of Nodes

kubectl top nodes

Output columns:

NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%

Real-Life Examples

Example 1: Setting Resource Requests and Limits for a Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: nginx:1.25
        resources:
          requests:
            cpu: "250m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Example 2: Observing OOMKilled Events

If a container exceeds its memory limit, it will be restarted with an OOMKilled event. To check for this:

kubectl get pods --all-namespaces | grep OOMKilled
kubectl describe pod <pod-name> | grep -A5 State

Example 3: Troubleshooting High Resource Usage

Find the top CPU and memory consumers:

kubectl top pods -A --sort-by=cpu | head -n 10
kubectl top pods -A --sort-by=memory | head -n 10

Example 4: Enforcing Resource Quotas (Namespace Level)

apiVersion: v1
kind: ResourceQuota
metadata:
  name: devops-quota
  namespace: devops-tools
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.cpu: "4"
    limits.memory: 4Gi

Apply with:

kubectl apply -f resource-quota.yaml

Best Practices

Always set both requests and limits for CPU and memory in production workloads.
Use kubectl top and monitoring tools (Prometheus, Grafana, Datadog) to observe real usage.
Set namespace-level ResourceQuotas to prevent noisy neighbor problems.
Avoid setting limits too low (causes throttling/OOMKilled) or too high (wastes resources).
Use Vertical Pod Autoscaler for dynamic resource tuning.
Document resource requirements in your Git repository for reproducibility.

References

Tip: Use CI/CD (GitHub Actions, GitLab CI, Azure Pipelines) to validate resource settings and catch misconfigurations before deploying to production.

PreviousStatus in Pods NextPod Troubleshooting Commands

Last updated 1 month ago

Table of Contents