traefik v2.4 自动配置 https

1、什么是Traefik

前面我写过一遍关于什么是Traefik以及如何在kubernetes中部署Traefik的文章;详细介绍了Traefik的基本概念、kubernetes部署Traefik以及如何配置ingress路由规则等内容。感兴趣的小伙伴可以自行移步:https://blog.z0ukun.com/?p=2048 。但是前面我们部署的Traefik版本是v2.0,而且我们通过ingress方式只配置了HTTP规则;这里我们把Traefik升级到最新版本(目前是v2.4.11)、并通过ACME自动生成TLS证书。

在Traefik v2.4中增加了对 Kubernetes Gateway API 的支持、那么什么是 Kubernetes Gateway API 呢?Gateway API 是由 SIG-NETWORK 社区管理的一个开源项目;该项目的目标是在 Kubernetes 生态系统内发展服务网络 API;网关 API 提供了用于暴露 Kubernetes 应用程序的 Service、Ingress 等。Gateway API 旨在通过提供可表达的,可扩展的,面向角色的接口来改善服务网络,这些接口已由许多供应商实施并获得了广泛的行业支持;网关 API 是 API 资源(服务、网关类、网关、HTTPRoute、TCPRoute等)的集合、这些资源共同为各种网络用例建模。其实 Traefik 除了支持我们手动配置 TLS 证书之外,还支持自动生成 TLS 证书,下面我们就一起来看看如何在 Traefik 2.4 中配置自动化 HTTPS 服务。

2、Traefik部署

2.1、创建CRD资源

下面我们先把 Traefk v2.4 部署到K8S环境中、Traefik从2.0版本以后就开始使用CRD(Custom Resource Definition)来完成路由配置、我们先来创建CRD资源、资源清单文件如下:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: serverstransports.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: ServersTransport
    plural: serverstransports
    singular: serverstransport
  scope: Namespaced

# 用下面的命令创建CRD资源(提前创建一个名为traefik的namespace)
[root@kubernetes01]# kubectl apply -f traefik-crd.yaml -n traefik

2.2、创建RBAC权限

# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: traefik
  name: traefik-ingress-controller
---
# ClusterRole
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-ingress-controller
  namespace: traefik
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
      - ingressroutetcps
      - ingressrouteudps
      - middlewares
      - tlsoptions
      - tlsstores
      - traefikservices
      - serverstransports
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.x-k8s.io
    resources:
      - gatewayclasses
      - gatewayclasses/status
      - gateways
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.x-k8s.io
    resources:
      - gatewayclasses/status
    verbs:
      - get
      - patch
      - update
  - apiGroups:
      - networking.x-k8s.io
    resources:
      - gateways/status
    verbs:
      - get
      - patch
      - update
  - apiGroups:
      - networking.x-k8s.io
    resources:
      - httproutes
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - networking.x-k8s.io
    resources:
      - httproutes/status
    verbs:
      - get
      - patch
      - update
---
# ClusterRoleBinding
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: traefik

# 用下面的命令继续创建RBAC访问控制策略资源
[root@kubernetes01 traefik-deploy]# kubectl apply -f traefik-rbac.yaml -n traefik

注:这里我的kubernetes版本是v1.19.13、请各位小伙伴自行根据自己的kubernetes版本查看API信息。

2.3、Traefik配置文件

Traefik 配置很多,通过 CLI 定义不是很方便,一般时候都会通过配置文件配置 Traefik 参数,然后存入 ConfigMap,将其挂入 Traefik 中。

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
data:
  traefik.yaml: |-
    ping: ""                                ## 启用 Ping
    serversTransport:
      insecureSkipVerify: true              ## Traefik 忽略验证代理服务的 TLS 证书
    api:
      insecure: true                        ## 允许 HTTP 方式访问 API
      dashboard: true                       ## 启用 Dashboard
      debug: false                          ## 启用 Debug 调试模式
    metrics:
      prometheus: ""                        ## 配置 Prometheus 监控指标数据,并使用默认配置
    entryPoints:
      web:
        address: ":80"                      ## 配置 80 端口,并设置入口名称为 web
      websecure:
        address: ":443"                     ## 配置 443 端口,并设置入口名称为 websecure
    providers:
      kubernetesCRD: ""                     ## 启用 Kubernetes CRD 方式来配置路由规则
      kubernetesIngress: ""                 ## 启用 Kubernetes Ingress 方式来配置路由规则
      kubernetesGateway: ""                 ## 启用 Kubernetes Gateway API
    experimental:               
      kubernetesGateway: true               ## 允许使用 Kubernetes Gateway API
    log:
      filePath: ""                          ## 设置调试日志文件存储路径,如果为空则输出到控制台
      level: error                          ## 设置调试日志级别
      format: json                          ## 设置调试日志格式
    accessLog:
      filePath: ""                          ## 设置访问日志文件存储路径,如果为空则输出到控制台
      format: json                          ## 设置访问调试日志格式
      bufferingSize: 0                      ## 设置访问日志缓存行数
      filters:
        #statusCodes: ["200"]               ## 设置只保留指定状态码范围内的访问日志
        retryAttempts: true                 ## 设置代理访问重试失败时,保留访问日志
        minDuration: 20                     ## 设置保留请求时间超过指定持续时间的访问日志
      fields:                               ## 设置访问日志中的字段是否保留(keep 保留、drop 不保留)
        defaultMode: keep                   ## 设置默认保留访问日志字段
        names:                              ## 针对访问日志特别字段特别配置保留模式
          ClientUsername: drop  
        headers:                            ## 设置 Header 中字段是否保留
          defaultMode: keep                 ## 设置默认保留 Header 中字段
          names:                            ## 针对 Header 中特别字段特别配置保留模式
            User-Agent: redact
            Authorization: drop
            Content-Type: keep
    #tracing:                               ## 链路追踪配置,支持 zipkin、datadog、jaeger、instana、haystack 等 
    #  serviceName:                         ## 设置服务名称(在链路追踪端收集后显示的服务名)
    #  zipkin:                              ## zipkin配置
    #    sameSpan: true                     ## 是否启用 Zipkin SameSpan RPC 类型追踪方式
    #    id128Bit: true                     ## 是否启用 Zipkin 128bit 的跟踪 ID
    #    sampleRate: 0.1                    ## 设置链路日志采样率(可以配置0.0到1.0之间的值)
    #    httpEndpoint: http://localhost:9411/api/v2/spans     ## 配置 Zipkin Server 端点

    # 创建ConfigMap资源
    [root@kubernetes01 traefik-deploy]# kubectl apply -f traefik-config.yaml -n traefik

2.4、配置节点Lable标签

由于是 Kubernetes DeamonSet 这种方式部署 Traefik,所以需要提前给节点设置 Label,这样当程序部署时 Pod 会自动调度到设置 Label 的节点上。

kubectl label nodes kubernetes01 IngressProxy=true
kubectl label nodes kubernetes02 IngressProxy=true
kubectl label nodes kubernetes03 IngressProxy=true

[root@kubernetes01]# kubectl get nodes --show-labels
NAME           STATUS   ROLES    AGE     VERSION    LABELS
kubernetes01   Ready    <none>   6d15h   v1.19.13   IngressProxy=true,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes01,kubernetes.io/os=linux
kubernetes02   Ready    <none>   6d15h   v1.19.13   IngressProxy=true,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes02,kubernetes.io/os=linux
kubernetes03   Ready    <none>   6d15h   v1.19.13   IngressProxy=true,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kubernetes03,kubernetes.io/os=linux
[root@kubernetes01]# 

# 如果想删除标签,可以使用 kubectl label nodes kubernetes01 IngressProxy- 命令

2.5、安装 Kubernetes Gateway CRD 资源

Kubernetes 集群上默认没有安装 Service APIs,我们需要提前安装 Gateway API 的 CRD 资源,需要确保在 Traefik 安装之前启用 Service APIs 资源。

gateway-api github地址:https://github.com/kubernetes-sigs/gateway-api

当前版本 v0.3.0、我们通过git cone命令把文件拉取到服务器上。拉取完成以后进入到gateway-api/config/crd/bases目录中执行下面的命令创建资源对象。

[root@kubernetes01 bases]# pwd
/opt/k8s/work/traefik/gateway-api/config/crd/bases
[root@kubernetes01 bases]# kubectl apply -f .

2.6、部署Traefik v2.4

下面我们用 DaemonSet 的方式把 Traefik 部署到所有K8S节点上,便于在多服务器间扩展;然后用 hostport 方式绑定服务器 80、443 端口,方便流量通过物理机进入 Kubernetes 内部。

这里我们如果想使用 Let’s Encrypt 来进行自动化 HTTPS,就需要首先开启 ACME;开启 ACME 需要通过静态配置的方式,即可以通过环境变量、启动参数等方式来提供;这里直接使用启动参数的形式来开启,在 Traefik 的部署文件中添加相应命令行参数、详细资源文件如下:

apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  ports:
    - name: web
      port: 80
    - name: websecure
      port: 443
    - name: admin
      port: 8080
  selector:
    app: traefik
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik-ingress-controller
  labels:
    app: traefik
spec:
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      name: traefik
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 1
      containers:
        - image: traefik:v2.4.11
          name: traefik-ingress-lb
          ports:
            - name: web
              containerPort: 80
              hostPort: 80                  # 将容器端口绑定所在服务器的 80 端口
            - name: websecure
              containerPort: 443            
              hostPort: 443                 # 将容器端口绑定所在服务器的 443 端口
            - name: admin
              containerPort: 8080           # Traefik Dashboard 端口
          resources:
            limits:
              cpu: 2000m
              memory: 1024Mi
            requests:
              cpu: 1000m
              memory: 1024Mi
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
          args:
            - --configfile=/config/traefik.yaml
            - --entrypoints.web.Address=:80
            - --entrypoints.websecure.Address=:443
            # 开启 webui 需要该参数
            - --api.insecure=true
            - --providers.kubernetescrd
            - --api
            - --api.dashboard=true
            - --accesslog
            # 使用 tls 验证这种方式
            - --certificatesresolvers.default.acme.tlsChallenge=true
            # 邮箱配置
            - --certificatesResolvers.default.acme.email="z0ukun@163.com"
            # 保存 ACME 证书的位置
            - --certificatesResolvers.default.acme.storage="acme.json"
            # ca服务测试
            - --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
          volumeMounts:
            - mountPath: "/config"
              name: "config"
          readinessProbe:
            httpGet:
              path: /ping
              port: 8080
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          livenessProbe:
            httpGet:
              path: /ping
              port: 8080
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5    
      volumes:
        - name: config
          configMap:
            name: traefik-config 
      tolerations:
        - operator: "Exists"                   # 设置容忍所有污点,防止节点被设置污点
      nodeSelector:
        IngressProxy: "true"                   # 设置node筛选器,在特定label的节点上启动

上面我们定义了一个名为default的证书解析器,使用 tlschallenge 这种ACME验证方式进行验证。需要注意的是在使用ACME验证的过程中、Let’s Encrypt 到 Traefik 443 端口必须是可达的;当然除了这种 tlsChallenge 这种验证方式以外还有 httpChallenge 和 dnsChallenge 两种验证方式。在这是那种验证方式中、更常用的还是 httpChallenge 。关于这几种验证方式更详细的使用方法、各位小伙伴可以自行移步官方文档。

2.7、Traefik验证

[root@kubernetes01 ~]# kubectl get pod -n traefik
NAME                               READY   STATUS    RESTARTS   AGE
traefik-ingress-controller-hfk8g   1/1     Running   0          16h
traefik-ingress-controller-hz6m8   1/1     Running   0          16h
traefik-ingress-controller-vsv9p   1/1     Running   0          16h
[root@kubernetes01 ~]# kubectl get svc -n traefik
NAME      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
traefik   ClusterIP   10.254.173.152   <none>        80/TCP,443/TCP,8080/TCP   16h
[root@kubernetes01 ~]# 

到此 Traefik v2.4 应用已经部署完成、但是我们该如何访问呢?

3、Traefik路由规则

Traefik 应用已经部署完成,但是想让外部访问 Kubernetes 内部服务,还需要配置路由规则。上面部署 Traefik 时我们开启了 Traefik Dashboard,这是 Traefik 提供的视图看板。所以,首先配置基于 HTTPTraefik Dashboard 路由规则,使外部能够访问 Traefik Dashboard。然后,再配置基于 HTTPSKubernetes Dashboard 的路由规则。这里我们使用 IngressRoute 路由规则、详细如下:

3.1、CRD路由规则

配置HTTP路由:

[root@kubernetes01 traefik-deploy]# cat traefik-ingressroute-http.yaml 
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-webui
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`traefik.z0ukun.com`)
    kind: Rule
    services:
    - name: traefik
      port: 8080

配置HTTPS路由:

[root@kubernetes01 traefik-deploy]# cat traefik-ingressroute-https.yaml 
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-webui-tls
  namespace: traefik
spec:
  entryPoints:
  - websecure  # 注意这里是websecure这个entryPoint,监控443端口
  routes:
  - match: Host(`traefik.z0ukun.com`)
    kind: Rule
    services:
    - name: traefik
      port: 8080
  tls:
    certResolver: default

我们分别创建 http 和 https 的 IngressRoute 资源文件、然后创建该资源对象。然后我们去修改本机的 hosts 文件的IP和域名的对应关系、然后用 traefik.z0ukun.com 去访问 Traefik :

[root@kubernetes01 traefik-deploy]# kubectl apply -f traefik-ingressroute-http.yaml -n traefik
ingressroute.traefik.containo.us/traefik-webui created
[root@kubernetes01 traefik-deploy]# kubectl apply -f traefik-ingressroute-https.yaml -n traefik
ingressroute.traefik.containo.us/traefik-webui-tls created
[root@kubernetes01 traefik-deploy]# kubectl get ingressroute -A
NAMESPACE              NAME                  AGE
harbor                 harbor-webui-tls      2d17h
kube-system            weave-webui           41h
kubernetes-dashboard   dashboard-webui-tls   2d17h
rook-ceph              ceph-webui-tls        2d17h
traefik                ceph-webui-tls        2d17h
traefik                dashboard-webui-tls   2d17h
traefik                traefik-webui         9s
traefik                traefik-webui-tls     6s
[root@kubernetes01 traefik-deploy]# 

image-20210729152008750

可以看到我们已经可以用上面配置的域名通过 http 和 https 来访问 Traefik 了。

3.2、配置 ceph crd路由

我们继续配置一下ceph的crd访问路由、ceph svc 信息和详细的资源文件内容如下、创建完成以后我们就可以用tls域名正常访问了。

[root@kubernetes01 traefik-ingressroute]# kubectl get svc -n rook-ceph
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
csi-cephfsplugin-metrics   ClusterIP   10.254.82.21     <none>        8080/TCP,8081/TCP   7d20h
csi-rbdplugin-metrics      ClusterIP   10.254.199.147   <none>        8080/TCP,8081/TCP   7d20h
rook-ceph-mgr              ClusterIP   10.254.45.66     <none>        9283/TCP            7d20h
rook-ceph-mgr-dashboard    ClusterIP   10.254.248.146   <none>        8443/TCP            7d20h
rook-ceph-mon-a            ClusterIP   10.254.74.230    <none>        6789/TCP,3300/TCP   7d20h
rook-ceph-mon-b            ClusterIP   10.254.8.242     <none>        6789/TCP,3300/TCP   7d20h
rook-ceph-mon-c            ClusterIP   10.254.1.31      <none>        6789/TCP,3300/TCP   7d20h

[root@kubernetes01 traefik-ingressroute]# cat ceph-ingressroute.yaml 
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ceph-webui-tls
  namespace: rook-ceph
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`ceph.z0ukun.com`)
    kind: Rule
    services:
    - name: rook-ceph-mgr-dashboard
      port: 8443
      namespace: rook-ceph
  tls:
    certResolver: default

[root@kubernetes01 traefik-ingressroute]# kubectl get ingressroute -A
NAMESPACE              NAME                  AGE
harbor                 harbor-webui-tls      2d19h
kube-system            weave-webui           42h
kubernetes-dashboard   dashboard-webui-tls   2d19h
rook-ceph              ceph-webui-tls        2d19h
traefik                dashboard-webui-tls   2d19h
[root@kubernetes01 traefik-ingressroute]# 

image-20210729165156702

推荐文章