Kubernetes生产环境最佳实践

Kubernetes生产环境最佳实践

集群架构

多可用区部署

# 节点池设计
nodePools:
  - name: system
    zones: ["us-east-1a", "us-east-1b", "us-east-1c"]
    instanceType: t3.large
    autoscaling:
      min: 3
      max: 10
    
  - name: workload
    zones: ["us-east-1a", "us-east-1b", "us-east-1c"]
    instanceType: m5.xlarge
    autoscaling:
      min: 5
      max: 20
​

控制平面高可用

• 3个或5个Master节点
• 负载均衡器：NLB/ALB分发API请求
• etcd集群：每个Master运行etcd成员

资源管理

资源配额

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
spec:
  hard:
    requests.cpu: "100"
    requests.memory: 200Gi
    limits.cpu: "200"
    limits.memory: 400Gi
    persistentvolumeclaims: "10"
​

LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 250m
      memory: 256Mi
    type: Container
​

HPA自动扩缩容

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
​

安全加固

RBAC权限控制

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
rules:
- apiGroups: ["", "apps"]
  resources: ["pods", "deployments"]
  verbs: ["get", "list", "create", "update"]
​

Pod Security Standards

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
​

网络策略

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
​

监控告警

Prometheus监控栈

# Prometheus配置
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
​

告警规则

groups:
- name: critical-alerts
  rules:
  - alert: PodCrashLooping
    expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Pod {{ $labels.pod }} crash looping"
​

日志管理

EFK架构

• Elasticsearch：日志存储
• Fluentd：日志采集
• Kibana：日志查询

Loki（轻量级）

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-config
data:
  loki.yaml: |
    server:
      http_listen_port: 3100
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory
​

CI/CD集成

GitOps

# ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-app
spec:
  destination:
    namespace: production
    server: https://kubernetes.default.svc
  source:
    repoURL: https://github.com/org/manifests
    targetRevision: main
    path: apps/production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
​

滚动更新

apiVersion: apps/v1
kind: Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
​

故障排查

常见问题

Pod Pending

• 检查资源配额
• 查看调度器事件
• 验证节点亲和性

CrashLoopBackOff

• 查看容器日志
• 检查健康探针
• 验证启动命令

Service无法访问

• 检查Endpoints
• 验证NetworkPolicy
• 查看iptables规则

调试工具

# 查看Pod事件
kubectl describe pod <pod-name>

# 进入容器调试
kubectl exec -it <pod-name> -- /bin/bash

# 查看日志
kubectl logs <pod-name> --tail=100 -f

# 端口转发
kubectl port-forward svc/<service-name> 8080:80
​

成本优化

资源右调

• 分析实际资源使用
• 调整requests和limits
• 使用Vertical Pod Autoscaler

Spot实例

apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    kubernetes.io/arch: amd64
  tolerations:
  - key: cloud.google.com/gke-spot
    operator: Equal
    value: "true"
    effect: NoSchedule
​

最佳实践总结

1. 声明式配置：所有资源YAML化
2. 版本控制：Git管理所有manifest
3. 渐进式发布：金丝雀、蓝绿部署
4. 自动化测试：集成到CI流水线
5. 文档先行：架构设计、运维手册