You can also use a Job to run multiple Pods in parallel. The Job object will start a new Pod if the first Pod fails or is deleted (for example due to a node hardware failure or a node reboot). A simple case is to create one Job object in order to reliably run one Pod to completion. As per our understanding, the kubernetes-scheduler should not have scheduled a pod (non critical) to a node where there is already disk pressure.Ĭlean composition of Kubelet-level functionality with cluster-level functionality - Kubelet is effectively the "pod controller" high-availability applications, which will expect Pods to be replaced in advance of their termination and certainly in advance of deletion, such as in the case of planned evictions or image prefetching.ĭeleting a Job will clean up the Pods it created.
However, when the pod-2 is evicted it went to node-1 where pod-1 was already running and node-1 was already experiencing node pressure.
The node was low on resource ephemeral storage free#
First, kubelet tries to free node resources, especially disk, by deleting dead pods and its containers, and then unused images. So please take that into consideration when applying this change.At that moment, kubelet starts to reclaim resources, killing containers and declaring pods as failed until the resource usage is under the eviction threshold again. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. If you are reaching this, please note Hakob's recommendation for why it is not suggested best practice to do this. What I needed to do in order to schedule more than one to each node was change it to nodeSelector. Resolution: Based on Hakob's comments, the issue is that I had nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution set, which is a hard requirement directing the scheduler to only schedule one to each node. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this? And even the memory being used shows me it's only 13% used. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. I tainted it like kubectl taint nodes es-data=true:NoScheduleĪccording to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. RequiredDuringSchedulingIgnoredDuringExecution:Īnd here is my node tolerance when I describe the node Taints: es-data=true:NoSchedule Here is what my nodeAffinity looks like nodeAffinity: Here is my resource limits and requests for the data pods Limits: Warning FailedScheduling 56s (x5 over 3m35s) default-scheduler 0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector. Here is the events when I describe pod Events:
Prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc 1/1 Running 0 22h Here is what my cluster looks like kubectl get pods -n es (Total limits may be over 100 percent, i.e., overcommitted.)
Here is the output of one of these r5 nodes. They are r5.2Xlarge's which have 64G of memory. Right now I have 3 memory optimized ec2 instances for these data pods. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. I have an elasticsearch cluster in a kubernetes cluster.
0 kommentar(er)
