K8s 节点故障排查从症状定位到根因分析生产环境的系统化排障方法论一、K8s 节点故障的复杂性症状与根因的距离K8s 节点故障的表现多种多样Pod 处于 Pending 状态无法调度、节点状态变为 NotReady、Pod 频繁 OOMKilled、服务响应延迟飙升。这些症状的根因可能相距甚远——Pod Pending 可能是资源不足也可能是污点容忍问题节点 NotReady 可能是 kubelet 崩溃也可能是网络分区OOMKilled 可能是内存泄漏也可能是资源限额设置不当。更棘手的是级联故障一个节点故障导致 Pod 迁移到其他节点增加了其他节点的负载引发更多故障。排查时如果只关注最终症状可能找不到真正的根因。系统化的排障方法论比直觉猜测更可靠——从症状出发逐层缩小范围最终定位根因。二、K8s 节点故障排查的决策树节点故障排查遵循从宏观到微观的原则先确认集群和节点状态再深入 Pod 和容器层面最后检查系统资源。flowchart TD A[故障现象] -- B{节点是否 Ready?} B --|NotReady| C[检查 kubelet 状态] C -- C1[kubelet 进程是否运行?] C -- C2[kubelet 日志有无错误?] C -- C3[节点网络是否正常?] C1 --|未运行| C1a[重启 kubelet: systemctl restart kubelet] C2 --|有错误| C2a[根据错误类型修复] C3 --|异常| C3a[检查网络配置和 DNS] B --|Ready| D{Pod 状态异常?} D --|Pending| E[调度问题排查] E -- E1[kubectl describe pod: 查看事件] E -- E2[节点资源是否充足?] E -- E3[污点/容忍是否匹配?] D --|CrashLoopBackOff| F[容器问题排查] F -- F1[kubectl logs: 查看容器日志] F -- F2[应用启动是否成功?] F -- F3[健康检查是否通过?] D --|OOMKilled| G[资源问题排查] G -- G1[内存限制是否合理?] G -- G2[是否存在内存泄漏?] G -- G3[节点内存压力?] D --|ImagePullBackOff| H[镜像问题排查] H -- H1[镜像地址是否正确?] H -- H2[镜像仓库是否可达?] H -- H3[认证凭据是否配置?] style C fill:#ffcdd2 style E fill:#fff3e0 style F fill:#fff3e0 style G fill:#fff3e02.1 节点状态检查脚本#!/bin/bash # node-diagnose.sh — K8s 节点故障诊断脚本 # 设计意图一键收集节点状态信息快速定位 NotReady 节点的根因 set -euo pipefail NODE_NAME${1:?用法: $0 node-name} echo 节点状态诊断: ${NODE_NAME} # 1. 节点基础状态 echo -e \n--- 节点状态 --- kubectl get node ${NODE_NAME} -o wide # 2. 节点条件Conditions echo -e \n--- 节点条件 --- kubectl get node ${NODE_NAME} -o jsonpath{range .status.conditions[*]}{.type}: {.status} ({.reason}: {.message}){\n}{end} # 3. 节点资源使用 echo -e \n--- 节点资源 --- kubectl top node ${NODE_NAME} 2/dev/null || echo metrics-server 未安装无法获取资源使用 # 4. 节点上的 Pod 状态 echo -e \n--- 节点上的 Pod --- kubectl get pods --all-namespaces --field-selector spec.nodeName${NODE_NAME} \ -o wide --sort-by.status.phase # 5. 节点事件 echo -e \n--- 节点事件最近 10 条--- kubectl get events --field-selector involvedObject.kindNode,involvedObject.name${NODE_NAME} \ --sort-by.lastTimestamp | tail -10 # 6. SSH 到节点检查系统状态 echo -e \n--- 远程系统检查 --- echo 正在通过 kubectl debug 在节点上执行诊断... kubectl debug node/${NODE_NAME} -it --imagebusybox:1.36 -- \ sh -c echo 系统负载 uptime echo -e \n 内存使用 free -h echo -e \n 磁盘使用 df -h / /var/lib/kubelet /var/lib/docker 2/dev/null echo -e \n 网络连通性 ping -c 2 -W 2 kubernetes.default.svc.cluster.local 2/dev/null echo DNS: OK || echo DNS: FAIL echo -e \n kubelet 进程 ps aux | grep kubelet | grep -v grep || echo kubelet 进程未找到 echo -e \n 系统日志最近 10 行 tail -10 /var/log/messages 2/dev/null || journalctl -n 10 --no-pager 2/dev/null || echo 无法读取日志 echo -e \n 诊断完成 2.2 Pod 故障排查工具# pod_troubleshooter.py — Pod 故障排查工具 # 设计意图自动分析 Pod 状态和事件给出可能的根因和修复建议 from dataclasses import dataclass from typing import Optional import subprocess import json dataclass class DiagnosisResult: pod_name: str namespace: str status: str reason: str root_cause: str recommendation: str severity: str # critical/warning/info class PodTroubleshooter: def diagnose(self, pod_name: str, namespace: str default) - DiagnosisResult: 诊断 Pod 故障 pod_data self._get_pod_data(pod_name, namespace) if not pod_data: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusUnknown, reasonPod not found, root_cause无法获取 Pod 数据, recommendation检查 Pod 名称和命名空间, severitycritical, ) status self._get_pod_status(pod_data) reason self._get_pod_reason(pod_data) # 根据状态和原因进行诊断 if status Pending: return self._diagnose_pending(pod_data, pod_name, namespace) elif status Failed or CrashLoopBackOff in reason: return self._diagnose_crash(pod_data, pod_name, namespace) elif OOMKilled in reason: return self._diagnose_oom(pod_data, pod_name, namespace) elif ImagePullBackOff in reason: return self._diagnose_image_pull(pod_data, pod_name, namespace) else: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusstatus, reasonreason, root_causef未识别的故障状态: {status}/{reason}, recommendation手动检查 Pod 日志和事件, severitywarning, ) def _diagnose_pending(self, pod_data: dict, pod_name: str, namespace: str) - DiagnosisResult: 诊断 Pending 状态 events self._get_pod_events(pod_name, namespace) # 分析事件中的调度失败原因 for event in events: message event.get(message, ) if Insufficient in message: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusPending, reason资源不足, root_causef调度失败: {message}, recommendation增加节点资源或降低 Pod 资源请求, severitywarning, ) if node(s) had taints in message: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusPending, reason污点不匹配, root_causef节点污点阻止调度: {message}, recommendation添加对应的容忍度或移除节点污点, severityinfo, ) if node(s) didnt match Pods node affinity in message: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusPending, reason亲和性不匹配, root_cause节点亲和性规则没有匹配的节点, recommendation检查 nodeSelector 和 nodeAffinity 配置, severityinfo, ) return DiagnosisResult( pod_namepod_name, namespacenamespace, statusPending, reason未知, root_cause无法确定 Pending 原因, recommendation手动检查 kubectl describe pod 输出, severitywarning, ) def _diagnose_crash(self, pod_data: dict, pod_name: str, namespace: str) - DiagnosisResult: 诊断 CrashLoopBackOff containers pod_data.get(status, {}).get(containerStatuses, []) for container in containers: state container.get(state, {}) waiting state.get(waiting, {}) last_state container.get(lastState, {}) terminated last_state.get(terminated, {}) exit_code terminated.get(exitCode, 0) reason waiting.get(reason, ) or terminated.get(reason, ) if exit_code ! 0: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusCrashLoopBackOff, reasonreason, root_causef容器退出码 {exit_code}: {terminated.get(message, 无错误信息)}, recommendationf检查容器日志: kubectl logs {pod_name} -n {namespace} --previous, severitycritical, ) if Error in reason or Completed in reason: return DiagnosisResult( pod_namepod_name, namespacenamespace, statusCrashLoopBackOff, reasonreason, root_cause容器启动后立即退出可能是启动命令错误, recommendation检查容器启动命令和入口点配置, severitycritical, ) return DiagnosisResult( pod_namepod_name, namespacenamespace, statusCrashLoopBackOff, reason未知, root_cause无法确定崩溃原因, recommendation查看容器日志和事件, severitycritical, ) def _diagnose_oom(self, pod_data: dict, pod_name: str, namespace: str) - DiagnosisResult: 诊断 OOMKilled containers pod_data.get(spec, {}).get(containers, []) for container in containers: resources container.get(resources, {}) limits resources.get(limits, {}) memory_limit limits.get(memory, 未设置) return DiagnosisResult( pod_namepod_name, namespacenamespace, statusOOMKilled, reason内存超限, root_causef容器内存使用超过限制 ({memory_limit}), recommendation增加内存限制或排查内存泄漏, severitycritical, ) def _diagnose_image_pull(self, pod_data: dict, pod_name: str, namespace: str) - DiagnosisResult: 诊断镜像拉取失败 containers pod_data.get(spec, {}).get(containers, []) image containers[0].get(image, unknown) if containers else unknown return DiagnosisResult( pod_namepod_name, namespacenamespace, statusImagePullBackOff, reason镜像拉取失败, root_causef无法拉取镜像: {image}, recommendation检查镜像地址、仓库认证凭据和网络连通性, severitycritical, ) def _get_pod_data(self, pod_name: str, namespace: str) - Optional[dict]: try: result subprocess.run( [kubectl, get, pod, pod_name, -n, namespace, -o, json], capture_outputTrue, textTrue, timeout10, ) return json.loads(result.stdout) if result.returncode 0 else None except Exception: return None def _get_pod_status(self, pod_data: dict) - str: return pod_data.get(status, {}).get(phase, Unknown) def _get_pod_reason(self, pod_data: dict) - str: containers pod_data.get(status, {}).get(containerStatuses, []) for c in containers: state c.get(state, {}) waiting state.get(waiting, {}) if waiting: return waiting.get(reason, ) terminated state.get(terminated, {}) if terminated: return terminated.get(reason, ) return def _get_pod_events(self, pod_name: str, namespace: str) - list[dict]: try: result subprocess.run( [kubectl, get, events, -n, namespace, --field-selector, finvolvedObject.name{pod_name}, -o, json], capture_outputTrue, textTrue, timeout10, ) if result.returncode 0: return json.loads(result.stdout).get(items, []) except Exception: pass return []四、边界分析与架构权衡诊断脚本的生产安全诊断脚本在节点上执行命令可能影响生产环境。必须确保脚本只读取信息不修改状态且使用 kubectl debug 的临时容器而非 SSH 直连避免在节点上留下残留进程。CrashLoopBackOff 的日志获取容器崩溃后重启当前日志可能已被覆盖。必须使用kubectl logs --previous获取上一次容器的日志。但如果容器快速重启多次之前的日志可能已经丢失。OOMKilled 的内存分析OOMKilled 只告诉容器被杀了不告诉哪个对象占用了内存。需要在容器中配置 JVM 的 -XX:HeapDumpOnOutOfMemoryError 或类似机制在 OOM 前生成堆转储。NotReady 节点的级联影响节点 NotReady 后Pod 不会立即迁移需要等待 pod-eviction-timeout默认 5 分钟。在此期间NotReady 节点上的 Pod 仍可能接收流量如果 kube-proxy 尚未更新规则。需要配合 readinessGates 和节点健康检查确保流量快速切换。五、总结K8s 节点故障排查需要系统化的方法论从节点状态到 Pod 状态从宏观现象到微观根因。诊断脚本自动化收集节点信息Pod 排查工具分析常见故障模式并给出修复建议。落地建议建立标准化的诊断脚本一键收集节点状态Pending 问题先看事件再查资源CrashLoopBackOff 先看上一次日志OOMKilled 配置堆转储便于事后分析NotReady 节点配合 readinessGates 加速流量切换。