Kubernetes 实战：生产级集群管理与监控

Kubernetes 核心概念回顾

集群架构

• Master 节点：API Server、Scheduler、Controller Manager、etcd
• Worker 节点：Kubelet、Kube-proxy、Container Runtime

核心资源

• Pod：最小部署单元
• Deployment：无状态应用管理
• StatefulSet：有状态应用管理
• Service：服务发现和负载均衡
• Ingress：HTTP(S) 路由

生产级部署

1. 高可用 Master

# kubeadm 配置示例
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.29.0
controlPlaneEndpoint: "lb.k8s.local:6443"
etcd:
  external:
    endpoints:
    - https://etcd1.k8s.local:2379
    - https://etcd2.k8s.local:2379
    - https://etcd3.k8s.local:2379
​

2. 网络插件选择

CNI 插件对比

Calico 部署

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
​

3. 存储管理

StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
allowVolumeExpansion: true
reclaimPolicy: Delete
​

PVC 示例

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi
​

资源管理

1. 资源配额

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
spec:
  hard:
    requests.cpu: "100"
    requests.memory: 200Gi
    limits.cpu: "200"
    limits.memory: 400Gi
    persistentvolumeclaims: "10"
​

2. LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
spec:
  limits:
  - default:
      cpu: "1"
      memory: 512Mi
    defaultRequest:
      cpu: "0.5"
      memory: 256Mi
    type: Container
​

安全加固

1. RBAC 配置

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
​

2. Pod 安全策略

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 3000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
​

3. 网络策略

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: nginx
    ports:
    - protocol: TCP
      port: 80
​

监控体系

1. Prometheus 部署

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
​

2. Grafana 仪表板

核心指标：

• 集群资源使用率
• Pod 状态分布
• 节点健康状态
• 网络流量
• 存储使用情况

3. 日志收集（ELK）

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
      </parse>
    </source>
​

自动化运维

1. HPA（水平自动扩缩容）

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
​

2. VPA（垂直自动扩缩容）

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      maxAllowed:
        cpu: "2"
        memory: 4Gi
      minAllowed:
        cpu: "100m"
        memory: 128Mi
​

3. Cluster Autoscaler

kubectl autoscale node --max-nodes=10 --min-nodes=2 --nodes=1:10
​

故障排查

1. Pod 故障

# 查看 Pod 状态
kubectl get pods -n <namespace>

# 查看 Pod 详情
kubectl describe pod <pod-name> -n <namespace>

# 查看 Pod 日志
kubectl logs <pod-name> -n <namespace>

# 进入 Pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
​

2. 服务故障

# 检查 Service
kubectl get endpoints <service-name> -n <namespace>

# 测试服务连通性
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- <service-name>
​

3. 网络故障

# 检查网络策略
kubectl get networkpolicies -n <namespace>

# 使用 Netshoot 调试
kubectl run netshoot --image=nicolaka/netshoot -it --rm -- bash
​

备份与恢复

1. etcd 备份

# 备份
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# 恢复
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
​

2. Velero 备份

# 安装 Velero
velero install --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket velero \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=true

# 备份
velero backup create my-backup --include-namespaces my-app

# 恢复
velero restore create --from-backup my-backup
​

性能优化

1. 节点优化

# 调整 kubelet 参数
cat > /etc/kubernetes/kubelet-config.yaml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250
podPidsLimit: 2048
containerLogMaxSize: 100Mi
containerLogMaxFiles: 5
EOF
​

2. 集群优化

• 节点亲和性：合理调度 Pod
• 污点与容忍：隔离专用节点
• Pod 反亲和性：高可用部署
• 资源限制：防止资源争抢

最佳实践

1. 命名空间管理

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    env: production
    team: platform
​

2. 标签规范

metadata:
  labels:
    app: myapp
    version: v1.0.0
    environment: production
    team: backend
​

3. 配置管理

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  config.yaml: |
    database:
      host: postgres
      port: 5432
---
# Secret
apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=
​

学习路径

1. 基础：理解核心概念和资源
2. 实践：部署实际应用
3. 运维：监控、日志、备份
4. 优化：性能调优、成本优化
5. 进阶：自定义控制器、Operator 开发

总结

Kubernetes 是云原生的核心，掌握它需要时间和实践。建议：

1. 从简单的应用开始部署
2. 建立完善的监控体系
3. 制定清晰的备份策略
4. 持续学习和优化