首页

    kubernetes nginx ingress尝鲜

    标签:kubernetes,nginx-ingress

    helm

    这里用helm,免得到处找yaml配置,复制粘贴

    注意安装的时候需要配置rbac

    kubectl create serviceaccount --namespace kube-system tiller
    kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
    helm init --service-account tiller
    

    创建my-nginx service

    helm create my-nginx
    helm install --name=my-nginx --set image.repository=nginx,image.tag=alpine,replicaCount=4 ./my-nginx
    

    同理,创建my-openresty service

    helm inspect values my-nginx查看可以设置的key
    helm install --name=my-nginx --set image.repository=nginx,image.tag=alpine --debug --dry-run ./my-nginx查看生成release的yaml配置

    nginx ingress controller

    执行helm install stable/nginx-ingress可能会找不到安装位置,可以执行helm fetch stable/nginx-ingress,这个命令会把chart下载到当前目录,这里是nginx-ingress-0.23.0.tgz.然后执行helm install --name=nginx-ingress ./nginx-ingress-0.23.0.tgz解压并安装,这时在当前目录可看到nginx-ingress目录

    helm fetch stable/nginx-ingress
    helm install --name=nginx-ingress ./nginx-ingress-0.23.0.tgz
    

    Ingress Resource

    nginx ingress controller安装好后,还需要定义ingress rule.在nginx-ingress/templates目录创建ingress.yaml

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: example
      annotations:
        nginx.ingress.kubernetes.io/ssl-redirect: "false"
    spec:
      rules:
      - http:
          paths:
          - path: /nginx
            backend:
              serviceName: my-nginx
              servicePort: 80
          - path: /openresty
            backend:
              serviceName: my-openresty
              servicePort: 80
    

    然后helm upgrade nginx-ingress .更新,当然也可以在执行helm install前就创建这个文件,然后再helm install.
    注意这里的ingress rule没定义host,可以直接127.0.0.1:<port>访问,但是需要设置annotations

    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    

    否则nginx ingress controller会输出308 Permanent Redirect

    如果定义了host

    ...
    spec:
      rules:
      - host: example.com
        http:
          paths:
          - path: /nginx
            backend:
              serviceName: my-nginx
              servicePort: 80
          - path: /openresty
            backend:
              serviceName: my-openresty
              servicePort: 80
    

    则需要把host(example.com),绑定到本地,编辑/etc/hosts,添加

    127.0.0.1 example.com
    

    验证

    kubectl get svc

    NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
    nginx-ingress-controller        LoadBalancer   10.99.205.50    <pending>     80:32743/TCP,443:32089/TCP   12h
    nginx-ingress-default-backend   ClusterIP      10.102.31.7     <none>        80/TCP                       12h
    

    nginx-ingress-controller像LoadBalancer,不管在master node还是worker node,都能使用节点的IP加端口访问该服务,即这里的curl example.com:<port>访问,只不过没有云供应商提供的EXTERNAL-IP

    • curl example.com:32743
    default backend - 404
    
    • curl example.com:32743/nginx
    <html>
    <head><title>404 Not Found</title></head>
    <body bgcolor="white">
    <center><h1>404 Not Found</h1></center>
    <hr><center>nginx/1.15.1</center>
    </body>
    </html>
    
    • curl example.com:32743/openresty
    <html>
    <head><title>404 Not Found</title></head>
    <body bgcolor="white">
    <center><h1>404 Not Found</h1></center>
    <hr><center>openresty/1.13.6.2</center>
    </body>
    </html>
    

    worker node and pod load balance

    Worker node有2个:hehe,hehe1

    NAME      STATUS    ROLES     AGE       VERSION
    haha      Ready     master    1d        v1.10.5
    hehe      Ready     <none>    1d        v1.10.5
    hehe1     Ready     <none>    23h       v1.10.5
    

    kubectl get svc

    NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
    kubernetes                      ClusterIP      10.96.0.1       <none>        443/TCP                      1d
    my-nginx                        ClusterIP      10.110.112.57   <none>        80/TCP                       17h
    my-openresty                    ClusterIP      10.99.34.170    <none>        80/TCP                       14h
    nginx-ingress-controller        LoadBalancer   10.99.205.50    <pending>     80:32743/TCP,443:32089/TCP   12h
    nginx-ingress-default-backend   ClusterIP      10.102.31.7     <none>        80/TCP                       12h
    

    kubectl get pod

    注意测试用的my-openresty有4个pod,也就是说每个worker node有2个pod

    NAME                                            READY     STATUS    RESTARTS   AGE
    my-nginx-84d496b9c-2mwx5                        1/1       Running   2          17h
    my-nginx-84d496b9c-4rfnk                        1/1       Running   1          17h
    my-nginx-84d496b9c-bst49                        1/1       Running   1          17h
    my-nginx-84d496b9c-ghk9c                        1/1       Running   4          17h
    my-nginx-84d496b9c-kfknr                        1/1       Running   1          17h
    my-nginx-84d496b9c-z96vc                        1/1       Running   2          17h
    my-openresty-6c5c7f4745-bp4r9                   1/1       Running   0          39m
    my-openresty-6c5c7f4745-qrbh5                   1/1       Running   0          39m
    my-openresty-6c5c7f4745-s5bxg                   1/1       Running   0          39m
    my-openresty-6c5c7f4745-t22l9                   1/1       Running   0          39m
    nginx-ingress-controller-6d4d467669-nfs9d       1/1       Running   0          41m
    nginx-ingress-default-backend-d676cbb5f-w869s   1/1       Running   1          13h
    

    随便进入1个pod,比如kubectl exec -ti my-nginx-84d496b9c-2mwx5 sh,执行

    for i in `seq 1 12`;do
      curl my-openresty
    done
    

    效果

    可以看到,12个请求分布在2个worker node的4个pod,worker node和其中的pod都有请求

    host port

    上面的nginx-ingress-controller service还是用的非80,443端口访问,我们当然想外部通过80,443端口就能访问,可以

    • socat转发
    NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
    nginx-ingress-controller        LoadBalancer   10.103.44.98     <pending>     80:32598/TCP,443:32503/TCP   41m
    
    socat TCP4-LISTEN:80,fork,su=nobody TCP4:127.0.0.1:32598
    socat TCP4-LISTEN:443,fork,su=nobody TCP4:127.0.0.1:32503
    

    127.0.0.1不能去掉

    • daemonset host port

    注意到chart可以设置hostPort,执行

    helm inspect values ~/nginx-ingress
    

    找到kinddaemonset,将chart修改为

    kind: DaemonSet
    daemonset:
      useHostPort: true
      hostPorts:
        http: 80
        https: 443
    

    就能满足要求,可以执行

    # 创建
    helm install --name=nginx-ingress --set controller.kind=DaemonSet,controller.daemonset.useHostPort=true ~/nginx-ingress
    # 更新
    helm upgrade --set controller.kind=DaemonSet,controller.daemonset.useHostPort=true nginx-ingress ~/nginx-ingress
    

    或者直接修改chart文件
    注意iptables更新后,需要重启nginx-ingress-controller,因为host port实际上是通过修改iptables规则实现的
    设置好后,日志里的客户端ip不是--pod-network-cidr设置的10.244.0.0/16,而是真正的客户端ip了
    另外可参考Get Access to the Ingress Controller

    参考

    nginx-ingress-controller always redirect to HTTPS regardless of Ingress annotations if host was not specified
    Get Access to the Ingress Controller

    restrict ip

    nginx-ingress/templates/ingress.yaml

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: test
      annotations:
        ...
        nginx.ingress.kubernetes.io/whitelist-source-range: 'cidr1,cidr2....'
    

    只是设置白名单ip还不够,因为nginx-ingress默认将header里的X-Forwarded-For作为source ip,并不是remote_addr,因此还需要像设置ngx_http_realip_moduleset_real_ip_from一样,设置信任的代理ip

    By default NGINX uses the content of the header X-Forwarded-For as the source of truth to get information about the client IP address. This works without issues in L7 if we configure the setting proxy-real-ip-cidr with the correct information of the IP/network address of trusted external load balancer.

    在ConfigMap配置proxy-real-ip-cidr

    nginx-ingress/templates/controller-configmap.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      ...
    data:
      proxy-real-ip-cidr: 'cidr1,cidr2,...'
    

    注意当前(2019.01.29)nginx-ingress最新版本0.22.0的ConfigMaps添加了一个新配置use-forwarded-headers,默认值是"false",会忽略X-Forwarded-* headers

    By default do not trust any client to extract true client IP address from X-Forwarded-For header using realip module (use-forwarded-headers: "false")

    multiple ingress controller

    在kubernetes里使用多个nginx ingress,需要Ingress Resource的kubernetes.io/ingress.class和nginx-ingress-controller命令的参数--ingress-class设置的值一样

    nginx-ingress/templates/ingress.yaml

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: test
      annotations:
        ...
        kubernetes.io/ingress.class: my-ingress
    

    注意到controller-deployment.yaml,controller-daemonset.yaml里的--ingress-class参数是通过{{ .Values.controller.ingressClass }}设置的

    {{- if eq .Values.controller.kind "DaemonSet" }}
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      ...
    spec:
      template:
        metadata:
          ...
        spec:
          ...
          containers:
            - name: {{ template "nginx-ingress.name" . }}-{{ .Values.controller.name }}
              image: "{{ .Values.controller.image.repository }}:{{ .Values.controller.image.tag }}"
              ...
              args:
                - /nginx-ingress-controller
                - --default-backend-service={{ if .Values.defaultBackend.enabled }}{{ .Release.Namespace }}/{{ template "nginx-ingress.defaultBackend.fullname" . }}{{ else }}{{ .Values.controller.defaultBackendService }}{{ end }}
              {{- if and (semverCompare ">=0.9.0-beta.1" .Values.controller.image.tag) .Values.controller.publishService.enabled }}
                - --publish-service={{ template "nginx-ingress.controller.publishServicePath" . }}
              {{- end }}
              {{- if (semverCompare ">=0.9.0-beta.1" .Values.controller.image.tag) }}
                - --election-id={{ .Values.controller.electionID }}
              {{- end }}
              {{- if (semverCompare ">=0.9.0-beta.1" .Values.controller.image.tag) }}
                - --ingress-class={{ .Values.controller.ingressClass }}
              {{- end }}
              ...
    

    因此只需设置values.yaml

    controller:
      name: contoller
      image:
        repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
        tag: "0.21.0"
        pullPolicy: IfNotPresent
    
      ...
      ## Name of the ingress class to route through this controller
      ##
      ingressClass: my-ingress
    

    SSL Passthrough

    相当于nginx stream block,作为layer 4反向代理,只需

    • 设置Ingress object
    kind: Ingress
    metadata:
      name: my-ingress
      annotations:
        kubernetes.io/ingress.class: 123
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    spec:
      tls:
        - hosts:
          - your domain
          #secretName: your secret
      rules:
      - host: your domain
        http:
          paths:
          - path: /
            backend:
              serviceName: your service
              servicePort: your port
    

    不需要设置secretName

    • 命令添加参数--enable-ssl-passthrough
          containers:
              .....
              args:
                - /nginx-ingress-controller
                - --enable-ssl-passthrough
                - --ingress-class=123
    

    lost remote IP

    上面的my-ingress也可以和nginx.ingress.kubernetes.io/ssl-passthrough: "false"的ingress(no-ssl-ingress)一起使用

    kind: Ingress
    metadata:
      name: no-ssl-ingress
      annotations:
        kubernetes.io/ingress.class: 123
        nginx.ingress.kubernetes.io/ssl-passthrough: "false"
    spec:
      tls:
        - hosts:
          - another domain
          secretName: another secret
      rules:
      - host: another domain
        http:
          paths:
          - path: /
            backend:
              serviceName: your service
              servicePort: your port
    

    但是no-ssl-ingress获取不到remote IP,虽然可以反向代理服务

    TCP/UDP service

    参照文档Exposing TCP and UDP services,只需在--set时加上udp.port=<namespace/service name>:,如

    helm install --name=nginx-ingress-quic --set ...,udp.8444=default/quiche:4430 ~/nginx-ingress
    

    但是执行会报错

    Error: release nginx-ingress-quic failed: Service "nginx-ingress-controller-quic" is invalid: spec.ports: Invalid value: []core.ServicePort{core.ServicePort{Name:"http", Protocol:"TCP", Port:80, TargetPort:intstr.IntOrString{Type:1, IntVal:0, StrVal:"http"}, NodePort:0}, core.ServicePort{Name:"https", Protocol:"TCP", Port:443, TargetPort:intstr.IntOrString{Type:1, IntVal:0, StrVal:"https"}, NodePort:0}, core.ServicePort{Name:"8443-udp", Protocol:"UDP", Port:8443, TargetPort:intstr.IntOrString{Type:1, IntVal:0, StrVal:"8443-udp"}, NodePort:0}, core.ServicePort{Name:"8444-udp", Protocol:"UDP", Port:8444, TargetPort:intstr.IntOrString{Type:1, IntVal:0, StrVal:"8444-udp"}, NodePort:0}}: cannot create an external load balancer with mix protocols
    

    报错说的很清楚,LoadBalancer里,TCP和UDP不能一起用,加上--debug --dry-run,看下生成的配置

    # Source: nginx-ingress/templates/controller-service.yaml
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: nginx-ingress
        chart: nginx-ingress-1.21.0
        component: "controller"
        heritage: Tiller
        release: nginx-ingress-quic
      name: nginx-ingress-controller-quic
    spec:
      clusterIP: ""
      ports:
        - name: http
          port: 80
          protocol: TCP
          targetPort: http
        - name: https
          port: 443
          protocol: TCP
          targetPort: https
        - name: "8444-udp"
          port: 8444
          protocol: UDP
          targetPort: "8444-udp"
      selector:
        app: nginx-ingress
        component: "controller"
        release: nginx-ingress-quic
      type: "LoadBalancer"
    ---
    
    # Source: nginx-ingress/templates/controller-daemonset.yaml
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      labels:
        app: nginx-ingress
        chart: nginx-ingress-1.21.0
        component: "controller"
        heritage: Tiller
        release: nginx-ingress-quic
      name: nginx-ingress-controller-quic
    spec:
      ...
      template:
        metadata:
          labels:
            app: nginx-ingress
            component: "controller"
            release: nginx-ingress-quic
        spec:
          dnsPolicy: ClusterFirst
          containers:
            - name: nginx-ingress-controller
              image: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1"
              imagePullPolicy: "IfNotPresent"
              args:
                - /nginx-ingress-controller
                - --default-backend-service=default/nginx-ingress-quic-default-backend
                - --election-id=ingress-controller-leader
                - --ingress-class=quic
                - --configmap=default/nginx-ingress-quic-controller
                - --udp-services-configmap=default/nginx-ingress-quic-udp
              ...
              ports:
                - name: http
                  containerPort: 80
                  protocol: TCP
                  hostPort: 80
                - name: https
                  containerPort: 443
                  protocol: TCP
                  hostPort: 443
                - name: "8444-udp"
                  containerPort: 8444
                  protocol: UDP
                  hostPort: 8444
              ...
    

    ports里面TCP,UDP确实混用了,那把TCP相关去掉,注意到templates/controller-daemonset.yaml

    # Source: nginx-ingress/templates/controller-daemonset.yaml
    containers:
      - name: nginx-ingress-controller
      ...
      ports:
      {{- range $key, $value := .Values.controller.containerPort }}
      - name: {{ $key }}
        containerPort: {{ $value }}
        protocol: TCP
        {{- if $useHostPort }}
        hostPort: {{ index $hostPorts $key | default $value }}
        {{- end }}
      {{- end }}
    ---
    
    # Source: nginx-ingress/templates/controller-service.yaml
    apiVersion: v1
    kind: Service
    ...
    spec:
      ...
      ports:
        {{- $setNodePorts := (or (eq .Values.controller.service.type "NodePort") (eq .Values.controller.service.type "LoadBalancer")) }}
        {{- if .Values.controller.service.enableHttp }}
        - name: http
          port: {{ .Values.controller.service.ports.http }}
          protocol: TCP
          targetPort: {{ .Values.controller.service.targetPorts.http }}
          {{- if (and $setNodePorts (not (empty .Values.controller.service.nodePorts.http))) }}
          nodePort: {{ .Values.controller.service.nodePorts.http }}
          {{- end }}
        {{- end }}
        {{- if .Values.controller.service.enableHttps }}
        - name: https
          port: {{ .Values.controller.service.ports.https }}
          protocol: TCP
          targetPort: {{ .Values.controller.service.targetPorts.https }}
          {{- if (and $setNodePorts (not (empty .Values.controller.service.nodePorts.https))) }}
          nodePort: {{ .Values.controller.service.nodePorts.https }}
          {{- end }}
        {{- end }}
    

    修改values.yaml

    controller:
      ...
      containerPort: {}
    

    执行helm install时在--set加上,controller.service.enableHttp=false,controller.service.enableHttps=false

    这样暴露的服务也是可以负载均衡的,只是和TCP不一样

    TCP/UDP service PROXY PROTOCOL

    nginx ingress在L4负载均衡时,如果backend需要原始的ip,这时nginx ingress不能以x-forwarded-for header传给backend,只能以PROXY PROTOCOL方式传(需要backend也支持PROXY PROTOCOL的解析),可以

    helm install --name=nginx-ingress --set ...,tcp.8443=default/backend:8443::PROXY ~/nginx-ingress
    

    format:<namespace/service name>::[PROXY]:[PROXY] The two last fields are optional. Adding PROXY in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service

    第一个[PROXY]参数表示nginx ingress decode上面传过来的PROXY PROTOCOL
    第二个[PROXY]参数表示nginx ingress encode成PROXY PROTOCOL,传给下面

    TCP/UDP service whitelist

    nginx ingress被用作L4负载均衡(--enable-ssl-passthrough)时,其具有的功能只有转发,其他的功能(ip whitelist,rate limit等)即使配置了也无效

    Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object.

    Tcp connection support whitelist?


    不定期更新