JVM memory limits
Last updated
Last updated
Disclaimer. This post describes hotfix for Java less than 17. If you are using Java 17+ skip this post and just use -XX:UseContainerSupport . Don’t remember to set limits for containers.
Note that Heap is not a single memory consumer in JVM. JVM Memory = Heap + NonHeap where NonHeap = Metaspace + CodeHeap (non-nmethods) + Compressed Class Space + CodeHeap (non-profiled nmethods) . While we can limit Heap size using a single parameter like -Xmx we can not do the same thing for nonheap.
We can specify memory request and limits for each container using spec.containers[].resources.limits.memory and spec.containers[].resources.requests.memory . If the container will consume more memory than it was permitted then the container will be killed.
The main reason why you should specify requests and limits in the same time is QoS classes: policy of resource assignment in case of lack of resources.
Let’s look at QOS description from Kubernetes Github issue:
Guaranteed — pods are guaranteed to have the requested resources when they are scheduled, and the least likely to be evicted if the node running the pod is overcommitted.
Burstable — pods are guaranteed to have the requested resources, but are not guaranteed to have the full resources specified in the resource `limits` when scheduled. If a node is overcommitted, Burstable pods will be evicted before Guaranteed pods are evicted.
BestEffort — pods are not given any guarantees with respect to allocated resources when scheduled. If a node is overcommitted, BestEffort pods are the first ones considered for eviction.
If you would like to understand memory qos more deeply I recommend you to read a KEP-2570 “KEP-2570: Support Memory QoS with cgroups v2” and KEP-1769 “KEP-1769: Memory Manager”.
Let’s look at the memory profile of some Java application using Yourkit profiler. There are 3 frames: heap memory, non-heap and classes. Heap memory structure is the most well known by regular Java developers because it’s found in stories about GC (Garbage Collection). Look at the “limit” label in Heap Memory and Non-Heap Memory. You can see that Non-Heap memory limit can be equal or higher then Heap memory. That means that we have to limit non-heap memory too in case of k8s application.
A lesser known part of JVM memory is Non-Heap. Non heap memory consists of the following parts:
Meta space — contains meta information about jvm.
Compressed Class Space — separate space for class metadata.
CodeHeap (non-nmethods) is new name of Code Cache that contains complited bytecode into native code.
CodeHeap (non-profiled nmethods) is new name of Code Cache that contains complited bytecode into native code.
You can apply memory flags adding to java {PUT_FLAGS} {OTHER_COMMAND} like java -Mmx1024m -jar app.jar . Each option consists of avalue and unit like g (GB), m (MB), k (KB).
Let’s look at all memory limit related flags (option -XX:+PrintFlagsFinal shows all JVM flags):
Xmx
MaxMetaspaceSize
CompressedClassSpaceSize
ProfiledCodeHeapSize
NonProfiledCodeHeapSize
NonNMethodCodeHeapSize
We can limit only Heap size (Xmx), Metaspace (MaxMetaspaceSize), Compressed Class (CompressedClassSpaceSize). Other parts of memory we can mark as other and assume that it will be less than 250MB.
Using this assumptions we can evaluate heap size using kubernetes container limit:
HEAP_SIZE = KUBE_MEMORY_LIMIT-METASPACE_SIZE-COMPRESSED_CLASS_SIZE-OTHER_NON_HEAP_SIZE
KUBE_MEMORY_LIMIT comes from container limits. You can pass this limit to container using valueFrom env (value -> valueFrom.resourceFieldRef.containerName:{yourContainerName} + valueFrom.resourceFieldRef.resource=limits.memory )
METASPACE_SIZE depends on your application size. I suggest using 200 MB as a starting point. If metaspace is not enough then new classes can not be loaded.
COMPRESSED_CLASS_SIZE can be equal 100MB but it is depends on your application
OTHER_NON_HEAP_SIZE can be equal 250MB.
I suggest using entrypoint.sh as the launcher of your application that evaluates all configuration options. Note that you have to print an error if the limit is too low for your application (HEAP_SIZE based on your formula less than zero).
Specify KUBERNETES_MEMORY_LIMITS in env
Start your application with entrypoint.sh that evaluates heap size with fixed predefined meta space and compressed class space.
