Pod优先级与抢占机制深度解析让关键业务永不掉线生产环境的核心服务被驱逐Pod优先级和抢占机制是你必须掌握的救命稻草。为什么需要Pod优先级想象这个场景凌晨2点电商大促高峰期订单服务Pod因为节点资源不足被驱逐meanwhile测试环境的垃圾Pod还在占用资源这就是没有优先级管理的后果。Pod优先级确保关键业务在资源紧张时优先获得调度权。PriorityClass基础配置1. 定义优先级类# priority-classes.yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: critical value: 1000000 globalDefault: false description: 系统关键服务最高优先级 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high value: 100000 globalDefault: false description: 生产环境核心业务 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: medium value: 10000 globalDefault: false description: 生产环境一般业务 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: low value: 1000 globalDefault: false description: 开发测试环境 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: background value: 100 globalDefault: true description: 后台任务最低优先级 preemptionPolicy: PreemptLowerPriority2. 应用优先级到Pod# critical-service.yaml apiVersion: apps/v1 kind: Deployment metadata: name: payment-service namespace: production spec: replicas: 3 selector: matchLabels: app: payment template: metadata: labels: app: payment priority: critical spec: priorityClassName: critical containers: - name: payment image: payment-service:v2.1.0 resources: requests: memory: 2Gi cpu: 1000m limits: memory: 4Gi cpu: 2000m livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 10 periodSeconds: 5 readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 5 periodSeconds: 3抢占机制详解抢占流程高优先级Pod创建 → 调度器检查资源 → 资源不足 → 寻找可牺牲Pod → 驱逐低优先级Pod → 调度高优先级Pod非抢占式优先级某些场景下你不希望Pod去抢占其他PodapiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: non-preempting-high value: 50000 globalDefault: false description: 高优先级但不抢占 preemptionPolicy: Never适用场景批处理任务数据分析作业非紧急的定时任务实战构建完整的优先级体系1. 系统级关键服务# system-critical.yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: system-cluster-critical value: 2000000000 globalDefault: false description: 集群关键系统组件 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: system-node-critical value: 2000001000 globalDefault: false description: 节点关键系统组件 preemptionPolicy: PreemptLowerPriority2. 业务分级策略# business-priority.yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: business-critical value: 1000000 globalDefault: false description: 核心业务支付、订单、用户 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: business-standard value: 100000 globalDefault: false description: 标准业务商品、库存 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: business-supporting value: 10000 globalDefault: false description: 支撑业务报表、统计 preemptionPolicy: PreemptLowerPriority3. 环境分级# environment-priority.yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: env-production value: 100000 globalDefault: false description: 生产环境 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: env-staging value: 10000 globalDefault: false description: 预发环境 preemptionPolicy: PreemptLowerPriority --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: env-development value: 1000 globalDefault: false description: 开发环境 preemptionPolicy: PreemptLowerPriority优先级与PodDisruptionBudget保护关键业务不被驱逐# pdb.yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payment-pdb namespace: production spec: minAvailable: 2 selector: matchLabels: app: payment --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: order-pdb namespace: production spec: maxUnavailable: 1 selector: matchLabels: app: order优先级与PDB的交互高优先级Pod需要抢占 → 检查目标Pod的PDB → 如果违反PDB → 寻找其他可抢占Pod → 无法找到则Pending监控与告警1. 优先级分布监控# priority-metrics.yaml apiVersion: v1 kind: ConfigMap metadata: name: priority-recording-rules namespace: monitoring data: rules.yml: | groups: - name: pod_priority rules: - record: pod:priority:count expr: | count by (priority_class) ( kube_pod_info * on(pod, namespace) group_left(priority_class) kube_pod_labels{label_priority_class!} ) - record: pod:priority:preemption_rate expr: | rate(pod_preemption_victims_total[5m]) - record: pod:priority:pending_high expr: | count by (priority_class) ( kube_pod_status_phase{phasePending} * on(pod, namespace) group_left(priority_class) kube_pod_labels{label_priority_class~critical|high} )2. 抢占告警规则# priority-alerts.yaml apiVersion: v1 kind: ConfigMap metadata: name: priority-alerts namespace: monitoring data: alerts.yml: | groups: - name: priority_alerts rules: - alert: HighPriorityPodPending expr: pod:priority:pending_high 0 for: 5m labels: severity: critical annotations: summary: 高优先级Pod长时间Pending description: 优先级{{ $labels.priority_class }}的Pod pending超过5分钟 - alert: FrequentPreemption expr: rate(pod_preemption_victims_total[10m]) 0.1 for: 5m labels: severity: warning annotations: summary: 频繁发生Pod抢占 description: 过去10分钟内发生{{ $value }}次Pod抢占 - alert: CriticalPodEvicted expr: | increase(kube_pod_container_status_terminated_reason{reasonEvicted}[5m]) 0 * on(pod, namespace) group_left(priority_class) kube_pod_labels{label_priority_classcritical} for: 0m labels: severity: critical annotations: summary: Critical Pod被驱逐 description: Critical优先级的Pod {{ $labels.pod }} 被驱逐最佳实践1. 优先级设计原则# 推荐优先级范围 # system-cluster-critical: 2000000000 (kube-system核心组件) # system-node-critical: 2000001000 (kubelet等) # critical: 1000000 (业务核心) # high: 100000 (重要业务) # medium: 10000 (一般业务) # low: 1000 (非关键) # background: 100 (批处理)2. 命名空间隔离策略# namespace-priority.yaml apiVersion: v1 kind: ResourceQuota metadata: name: priority-quota namespace: development spec: hard: pods: 50 scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: [low, background] --- apiVersion: v1 kind: LimitRange metadata: name: priority-limit namespace: development spec: limits: - default: cpu: 500m memory: 512Mi defaultRequest: cpu: 100m memory: 128Mi type: Container3. 自动化优先级分配# priority-mutating-webhook.yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: priority-webhook webhooks: - name: priority.webhook.k8s.io rules: - apiGroups: [] apiVersions: [v1] operations: [CREATE] resources: [pods] clientConfig: service: name: priority-webhook namespace: kube-system path: /mutate admissionReviewVersions: [v1] sideEffects: NoneWebhook逻辑伪代码def assign_priority(pod): namespace pod.metadata.namespace # 根据命名空间自动分配优先级 if namespace production: if is_critical_service(pod): pod.spec.priorityClassName critical else: pod.spec.priorityClassName high elif namespace staging: pod.spec.priorityClassName medium else: pod.spec.priorityClassName low return pod常见问题排查1. 高优先级Pod无法调度# 查看调度事件 kubectl describe pod pod-name | grep -A 10 Events # 检查资源是否充足 kubectl top nodes # 查看低优先级Pod分布 kubectl get pods --all-namespaces \ -o custom-columnsNAMESPACE:.metadata.namespace,NAME:.metadata.name,PRIORITY:.spec.priorityClassName2. 抢占不生效# 检查kube-scheduler日志 kubectl logs -n kube-system -l componentkube-scheduler | grep -i preempt # 验证PriorityClass存在 kubectl get priorityclass # 检查PDB限制 kubectl get pdb --all-namespaces3. 优先级冲突# 查看Pod实际优先级 kubectl get pod pod-name -o jsonpath{.spec.priority} # 对比PriorityClass定义 kubectl get priorityclass name -o yaml性能优化1. 大规模集群优化# scheduler-config.yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler plugins: preFilter: enabled: - name: NodeResourcesFit filter: enabled: - name: NodeResourcesFit - name: PodTopologySpread postFilter: enabled: - name: DefaultPreemption score: enabled: - name: NodeResourcesFit weight: 100 pluginConfig: - name: DefaultPreemption args: minCandidateNodesPercentage: 10 minCandidateNodesAbsolute: 1002. 抢占性能调优# 限制候选节点数量提高调度速度 apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler pluginConfig: - name: DefaultPreemption args: # 至少检查10%的节点 minCandidateNodesPercentage: 10 # 至少检查100个节点 minCandidateNodesAbsolute: 100总结Pod优先级和抢占机制是保障关键业务稳定性的重要手段合理设计优先级层级从系统级到业务级层次分明配合PDB使用保护关键业务不被过度驱逐监控告警及时发现优先级相关问题自动化分配减少人为配置错误记住优先级不是万能的合理的资源规划才是根本。