首页

    监控迁移到kubernetes

    标签:kubernetes,grafana,prometheus

    volumeMounts权限

    修改kubernetes yaml配置,执行kubectl apply -f,会出现err="opening storage failed: lock DB directory: open /data/lock: permission denied",这就需要配置initContainers,在启动真正的image前,修改文件夹权限

    spec:
      initContainers:
        - name: "init-chown-data"
          image: "busybox:latest"
          imagePullPolicy: "IfNotPresent"
          command: ["chown", "-R", "65534:65534", "/data"]
          volumeMounts:
            - name: data-volume
              mountPath: /data
              subPath: ""
      containers:
        - name: prometheus
          ...
          volumeMounts:
            - name: data-volume
              mountPath: /data
    

    migrate grafana data

    kubectl apply -f或迁移grafana数据(grafana.db),会出现

    GF_PATHS_DATA='/var/lib/grafana' is not writable.
    You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later
    mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied
    

    一样是权限问题.注意这里的userid和groupid要改为472(grafana)

    spec:
      initContainers:
        - name: "init-chown-data"
          image: "busybox:latest"
          imagePullPolicy: "IfNotPresent"
          command: ["chown", "-R", "472:472", "/var/lib/grafana"]
          volumeMounts:
            - name: data-volume
              mountPath: /var/lib/grafana
              subPath: ""
    

    可以参考Installing using Docker

    hostPort and hostNetwork

    hostPorthostNetwork类似,都可以让我们在宿主机上使用<host-ip>:<port>访问服务
    同时在kubernetes上的其他服务仍然可以通过<service-name>:<port>访问服务,不过如果配置了hostNetwork:true,宿主机上可以看到端口号,进程;配置hostPort:true则不能

    node exporter监控宿主机

    如果要node-exporter监控宿主机网络,需要配置hostNetwork:true,不能配置hostPort:true
    配置hostNetwork:true可以让pod"看到"宿主机的所有网络接口,而配置hostPort:true类似docker networks中说的,通过iptables规则映射宿主机端口和kubernetes内容器端口,只是暴露(expose)端口,pod并没有"挂在"宿主机上,所以看不到端口号,进程,这时如果用node-exporter监控,看到的是pod的网络接口(lo,eth0)

    DaemonSet update strategy

    node-exporter适合使用DaemonSet作为控制器(Controller),不过需要注意

    spec:
      updateStrategy:
        type: OnDelete
    

    To enable the rolling update feature of a DaemonSet, you must set its .spec.updateStrategy.type to RollingUpdate

    • 更新策略如果设置的是OnDelete,在更新时需要手动删除旧的DaemonSet

    OnDelete: With OnDelete update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.

    • 如果设置的是RollingUpdate,这个rolling update的行为和其他控制器(如Deployment)不一样,DaemonSet的rolling update是直接删除旧的DaemonSet,然后创建新的DaemonSet

    With RollingUpdate update strategy, after you update a DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods will be created automatically, in a controlled fashion.


    不定期更新