z0ukun 文章目录
今天我们来介绍一个如何基于青云VPN网络来构建一个Kubernetes高可用集群；先声明一下、这里并没有用到自动化部署的相关内容。我们还是参考了之前的部署方案采用二进制的方式进行部署；和之前有一点区别的是、网络我们采用的是Calico；Kubernetes版本我们升级到了 v1.19.8。容器运行时还是使用的Docker、并没有适配Containerd容器运行时。感兴趣的同学可以自动切换到Containerd容器运行时（Containerd 实现了 kubernetes 的 Container Runtime Interface (CRI) 接口，提供容器运行时核心功能，如镜像管理、容器管理等，相比 dockerd 更加简单、健壮和可移植）；下面我们一起来看看具体的操作步骤。
首先我们需要到青云上注册账号并购买相关资源、比如云主机、公网IP、VPC网络等资源。购买完成以后我们创建一个名为Kubernetes的项目、把所有资源全部放到Kubernetes项目中去进行管理。我们自定义VPC网络为 172.20.0.0/24网段、并给云主机配置IP地址和DNS信息；然后我们通过青云自带的VPN服务连接到Kubernetes项目即可管理所有资源啦（青云相关操作请参考官方文档、这里就不再详细描述了；这里说一点就是配置完VPN之后记得在安全组中开放对应端口）。
1、组件版本说明（参考）
| 序号 | 组件 | 版本 | 备注 |
|---|---|---|---|
| 1 | kubernetes | v1.19.8 | |
| 2 | etcd | v3.4.15 | |
| 3 | docker | v19.03.8 | |
| 4 | calico | v3.18.1 | |
| 5 | coredns | v1.8.3 | |
| 6 | dashboard | v2.2.2 | |
| 7 | k8s-prometheus-adapter | v0.5.0 | |
| 8 | prometheus-operator | v0.38.0 | |
| 9 | prometheus | v2.15.2 | |
| 10 | elasticsearch、kibana | v7.11.2 | |
| 11 | cni-plugins | v3.18.1 | |
| 12 | metrics-server | v0.4.2 | |
| 13 | weave | v2.8.1 | |
| 14 | kubeapps | v2.3.1 | |
| 15 | helm | v3.1.0 | |
| 16 | grafana | v1.13.0 | |
| 17 | traefik | v2.1 |
主要配置策略
kube-apiserver：
- 使用节点本地 nginx 4 层透明代理实现高可用；
- 关闭非安全端口 8080 和匿名访问；
- 在安全端口 6443 接收 https 请求；
- 严格的认证和授权策略 (x509、token、RBAC)；
- 开启 bootstrap token 认证，支持 kubelet TLS bootstrapping；
- 使用 https 访问 kubelet、etcd，加密通信；
kube-controller-manager：
- 3 节点高可用；
- 关闭非安全端口，在安全端口 10252 接收 https 请求；
- 使用 kubeconfig 访问 apiserver 的安全端口；
- 自动 approve kubelet 证书签名请求 (CSR)，证书过期后自动轮转；
- 各 controller 使用自己的 ServiceAccount 访问 apiserver；
kube-scheduler：
- 3 节点高可用；
- 使用 kubeconfig 访问 apiserver 的安全端口；
kubelet：
- 使用 kubeadm 动态创建 bootstrap token，而不是在 apiserver 中静态配置；
- 使用 TLS bootstrap 机制自动生成 client 和 server 证书，过期后自动轮转；
- 在 KubeletConfiguration 类型的 JSON 文件配置主要参数；
- 关闭只读端口，在安全端口 10250 接收 https 请求，对请求进行认证和授权，拒绝匿名访问和非授权访问；
- 使用 kubeconfig 访问 apiserver 的安全端口；
kube-proxy：
- 使用 kubeconfig 访问 apiserver 的安全端口；
- 在 KubeProxyConfiguration 类型的 JSON 文件配置主要参数；
- 使用 ipvs 代理模式；
集群插件：
- DNS：使用功能、性能更好的 coredns；
- Dashboard：支持登录认证；
- Metric：metrics-server，使用 https 访问 kubelet 安全端口；
- Log：Elasticsearch、Fluend、Kibana；
- Registry 镜像库：docker-registry、harbor；
2、系统初始化
集群规划
- Kubernetes-01：172.20.0.20
- Kubernetes-02：172.20.0.21
- Kubernetes-03：172.20.0.22
注：如果没有特殊说明，本小节所有操作需要在所有节点上执行本文档的初始化操作。
配置主机名
# 可以将下面的节点名称替换为自己的主机名称
hostnamectl set-hostname kubernetes-01
hostnamectl set-hostname kubernetes-02
hostnamectl set-hostname kubernetes-03
如果 DNS 不支持主机名称解析，还需要在每台机器的 /etc/hosts 文件中添加主机名和 IP 的对应关系：
cat >> /etc/hosts <<EOF
172.20.0.20 kubernetes-01
172.20.0.21 kubernetes-02
172.20.0.22 kubernetes-03
EOF
然后退出，重新登录 root 账号，可以看到主机名生效。
添加节点SSH互信
本操作只需要在 Kubernetes-01 节点上进行，设置 root 账户可以无密码登录所有节点：
ssh-keygen -t rsa
ssh-copy-id root@kubernetes-01
ssh-copy-id root@kubernetes-02
ssh-copy-id root@kubernetes-03
更新PATH变量
echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc && source /root/.bashrc
注：/opt/k8s/bin 目录保存本文档下载安装的程序。
安装依赖包
yum install -y epel-release
yum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git
注：本文档的 kube-proxy 使用 ipvs 模式，ipvsadm 为 ipvs 的管理工具； etcd 集群各机器需要时间同步，chrony 用于系统时间同步。
关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
注：关闭防火墙，清理防火墙规则，设置默认转发策略。
# 关闭swap分区
swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
# 关闭SELinux
setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
注：关闭 swap 分区，否则kubelet 会启动失败(可以设置 kubelet 启动参数 –fail-swap-on 为 false 关闭 swap 检查)。关闭 SELinux，否则 kubelet 挂载目录时可能报错 Permission denied。
优化内核参数
cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh1=2048
net.ipv4.neigh.default.gc_thresh1=4096
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf
注：关闭 tcp_tw_recycle，否则与 NAT 冲突，可能导致服务不通。
配置系统时区及时钟同步
# 配置系统时区
[root@kubernetes-01 work]# timedatectl set-timezone Asia/Shanghai
# 查看同步状态
[root@kubernetes-01 work]# timedatectl status
Local time: Tue 2021-03-30 14:17:11 CST
Universal time: Tue 2021-03-30 06:17:11 UTC
RTC time: Tue 2021-03-30 06:17:12
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a
[root@kubernetes-01 work]#
# 将当前的 UTC 时间写入硬件时钟
[root@kubernetes-01 work]# timedatectl set-local-rtc 0
# 重启依赖于系统时间的服务
[root@kubernetes-01 work]# systemctl restart rsyslog
[root@kubernetes-01 work]# systemctl restart crond
# 关闭无关服务
[root@kubernetes-01 work]# systemctl stop postfix && systemctl disable postfix
# 创建接下来要使用的相关安装目录
[root@kubernetes-01 work]# mkdir -p /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/cert
注：System clock synchronized: yes，表示时钟已同步； NTP service: active，表示开启了时钟同步服务。
分发集群配置参数
后续使用的环境变量都定义在文件 environment.sh 中，请根据自己的机器、网络情况修改：
#!/usr/bin/bash
# 生成 EncryptionConfig 所需的加密 key
export ENCRYPTION_KEY=(head -c 32 /dev/urandom | base64)
# 集群各机器 IP 数组
export NODE_IPS=(172.20.0.20 172.20.0.21 172.20.0.22)
# 集群各 IP 对应的主机名数组
export NODE_NAMES=(kubernetes-01 kubernetes-02 kubernetes-03)
# etcd 集群服务地址列表
export ETCD_ENDPOINTS="https://172.20.0.20:2379,https://172.20.0.21:2379,https://172.20.0.22:2379"
# etcd 集群间通信的 IP 和端口
export ETCD_NODES="kubernetes-01=https://172.20.0.20:2380,kubernetes-02=https://172.20.0.21:2380,kubernetes-03=https://172.20.0.22:2380"
# kube-apiserver 的反向代理(kube-nginx)地址端口
export KUBE_APISERVER="https://127.0.0.1:8443"
# 节点间互联网络接口名称
export IFACE="eth0"
# etcd 数据目录
export ETCD_DATA_DIR="/data/k8s/etcd/data"
# etcd WAL 目录，建议是 SSD 磁盘分区，或者和 ETCD_DATA_DIR 不同的磁盘分区
export ETCD_WAL_DIR="/data/k8s/etcd/wal"
# k8s 各组件数据目录
export K8S_DIR="/data/k8s/k8s"
## DOCKER_DIR 和 CONTAINERD_DIR 二选一
# docker 数据目录
export DOCKER_DIR="/data/k8s/docker"
# flannel
export FLANNEL_ETCD_PREFIX=”/kubernetes/network”
# containerd 数据目录
# export CONTAINERD_DIR="/data/k8s/containerd"
## 以下参数一般不需要修改
# TLS Bootstrapping 使用的 Token，可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成
BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c"
# 最好使用 当前未用的网段 来定义服务网段和 Pod 网段
# 服务网段，部署前路由不可达，部署后集群内路由可达(kube-proxy 保证)
SERVICE_CIDR="10.254.0.0/16"
# Pod 网段，建议 /16 段地址，部署前路由不可达，部署后集群内路由可达(flanneld 保证)
CLUSTER_CIDR="172.30.0.0/16"
# 服务端口范围 (NodePort Range)
export NODE_PORT_RANGE="30000-32767"
# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"
# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"
# 集群 DNS 域名（末尾不带点号）
export CLUSTER_DNS_DOMAIN="cluster.local"
# 将二进制目录 /opt/k8s/bin 加到 PATH 中
export PATH=/opt/k8s/bin:PATH
把上面的 environment.sh 文件修改完成之后保存到 /opt/k8s/bin/ 目录下面，然后拷贝到所有节点（修改完上面的文件之后再执行下面的操作）：
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp environment.sh root@{node_ip}:/opt/k8s/bin/
ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
done
[root@kubernetes-01 bin]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 bin]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp environment.sh root@{node_ip}:/opt/k8s/bin/
> ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
environment.sh 100% 2275 6.0MB/s 00:00
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
environment.sh 100% 2275 2.4MB/s 00:00
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
environment.sh 100% 2275 2.6MB/s 00:00
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 bin]#
内核升级
CentOS 7.x 系统自带的 3.10.x 内核存在一些 Bugs，导致运行的 Docker、Kubernetes 不稳定，例如：
- 高版本的 docker(1.13 以后) 启用了 3.10 kernel 实验支持的 kernel memory account 功能(无法关闭)，当节点压力大如频繁启动和停止容器时会导致 cgroup memory leak；
- 网络设备引用计数泄漏，会导致类似于报错：”kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1″;
解决方案如下：
- 升级内核到 4.4.X 以上；
- 或者，手动编译内核，disable CONFIG_MEMCG_KMEM 特性；
- 或者，安装修复了该问题的 Docker 18.09.1 及以上的版本。但由于 kubelet 也会设置 kmem（它 vendor 了 runc），所以需要重新编译 kubelet 并指定 GOFLAGS=”-tags=nokmem”；
git clone --branch v1.14.1 --single-branch --depth 1 https://github.com/kubernetes/kubernetes
cd kubernetes
KUBE_GIT_VERSION=v1.14.1 ./build/run.sh make kubelet GOFLAGS="-tags=nokmem"
这里采用升级内核的解决办法：
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# 安装完成后检查 /boot/grub2/grub.cfg 中对应内核 menuentry 中是否包含 initrd16 配置，如果没有，再安装一次！
yum --enablerepo=elrepo-kernel install -y kernel-lt
# 设置开机从新内核启动
grub2-set-default 0
# 执行完上面所有的操作之后我们就可以重启所有主机了
sync
reboot
本小节参考文档：
- 系统内核相关参数参考：https://docs.openshift.com/enterprise/3.2/admin_guide/overcommit.html
- 3.10.x 内核 kmem bugs 相关的讨论和解决办法：
- https://github.com/kubernetes/kubernetes/issues/61937
- https://support.mesosphere.com/s/article/Critical-Issue-KMEM-MSPH-2018-0006
- https://pingcap.com/image/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s/
3、创建CA证书和秘钥
为确保安全，kubernetes 系统各组件需要使用 x509 证书对通信进行加密和认证。 CA (Certificate Authority) 是自签名的根证书，用来签名后续创建的其它证书。 CA 证书是集群所有节点共享的，只需要创建一次，后续用它签名其它所有证书。 本小节使用 CloudFlare 的 PKI 工具集 cfssl 创建所有证书。
注：如果没有特殊指明，本小节所有操作均在 Kubernetes-01 节点上执行。
安装 cfssl 工具集
cfssl github项目地址：https://github.com/cloudflare/cfssl
sudo mkdir -p /opt/k8s/cert && cd /opt/k8s/work
wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64
mv cfssl_1.5.0_linux_amd64 /opt/k8s/bin/cfssl
mv cfssljson_1.5.0_linux_amd64 /opt/k8s/bin/cfssljson
mv cfssl-certinfo_1.5.0_linux_amd64 /opt/k8s/bin/cfssl-certinfo
chmod +x /opt/k8s/bin/* && export PATH=/opt/k8s/bin:$PATH
创建配置文件
CA 配置文件用于配置根证书的使用场景 (profile) 和具体参数 (usage，过期时间、服务端认证、客户端认证、加密等)：
cd /opt/k8s/work
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "876000h"
}
}
}
}
EOF
- signing：表示该证书可用于签名其它证书（生成的 ca.pem 证书中 CA=TRUE）；
- server auth：表示 client 可以用该该证书对 server 提供的证书进行验证；
- client auth：表示 server 可以用该该证书对 client 提供的证书进行验证；
- “expiry”: “876000h”：证书有效期设置为 100 年；
创建证书签名请求文件
cd /opt/k8s/work
cat > ca-csr.json <<EOF
{
"CN": "kubernetes-ca",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "z0ukun"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
- CN：Common Name：kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)，浏览器使用该字段验证网站是否合法；
- O：Organization：kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；
- kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识；
注：不同证书 csr 文件的 CN、C、ST、L、O、OU 组合必须不同，否则可能出现 PEER’S CERTIFICATE HAS AN INVALID SIGNATURE 错误；后续创建证书的 csr 文件时，CN 都不相同（C、ST、L、O、OU 相同），以达到区分的目的。
生成CA证书和私钥
[root@kubernetes-01 work]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*2021/03/30 14:07:56 [INFO] generating a new CA key and certificate from CSR
2021/03/30 14:07:56 [INFO] generate received request
2021/03/30 14:07:56 [INFO] received CSR
2021/03/30 14:07:56 [INFO] generating key: rsa-2048
2021/03/30 14:07:57 [INFO] encoded CSR
2021/03/30 14:07:57 [INFO] signed certificate with serial number 11321818047568877525981512325340936770890720652
[root@kubernetes-01 work]# ls ca*
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
分发证书文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p /etc/kubernetes/cert"
scp ca*.pem ca-config.json root@{node_ip}:/etc/kubernetes/cert
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@{node_ip} "mkdir -p /etc/kubernetes/cert"
> scp ca*.pem ca-config.json root@{node_ip}:/etc/kubernetes/cert
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
ca-key.pem 100% 1675 930.7KB/s 00:00
ca.pem 100% 1322 2.4MB/s 00:00
ca-config.json 100% 293 556.0KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
ca-key.pem 100% 1675 998.5KB/s 00:00
ca.pem 100% 1322 1.7MB/s 00:00
ca-config.json 100% 293 433.7KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
ca-key.pem 100% 1675 870.4KB/s 00:00
ca.pem 100% 1322 1.7MB/s 00:00
ca-config.json 100% 293 305.5KB/s 00:00
[root@kubernetes-01 work]#
4、部署 Kubectl 命令行工具
注：本小节介绍安装和配置 kubernetes 命令行管理工具 kubectl 的步骤。本小节只需要部署一次，生成的 kubeconfig 文件是通用的，可以拷贝到需要执行 kubectl 命令的机器的 ~/.kube/config 位置。
注：如果没有特殊指明，本小节所有操作均在 Kubernetes-01 节点上执行。
下载Kubectl二进制文件
cd /opt/k8s/work
# 自行解决翻墙下载问题
wget https://dl.k8s.io/v1.19.8/kubernetes-client-linux-amd64.tar.gz
tar -xzvf kubernetes-client-linux-amd64.tar.gz
[root@kubernetes-01 work]# wget https://dl.k8s.io/v1.19.8/kubernetes-client-linux-amd64.tar.gz
--2021-03-30 14:08:13-- https://dl.k8s.io/v1.19.8/kubernetes-client-linux-amd64.tar.gz
Resolving dl.k8s.io (dl.k8s.io)... 34.107.204.206, 2600:1901:0:26f3::
Connecting to dl.k8s.io (dl.k8s.io)|34.107.204.206|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://storage.googleapis.com/kubernetes-release/release/v1.19.8/kubernetes-client-linux-amd64.tar.gz [following]
--2021-03-30 14:08:16-- https://storage.googleapis.com/kubernetes-release/release/v1.19.8/kubernetes-client-linux-amd64.tar.gz
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.27.144, 172.217.160.80, 216.58.200.48, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.27.144|:443... connected.
^C
[root@kubernetes-01 work]# tar -zxvf kubernetes-client-linux-amd64.tar.gz
kubernetes/
kubernetes/client/
kubernetes/client/bin/
kubernetes/client/bin/kubectl
[root@kubernetes-01 work]#
分发到所有使用 kubectl 工具的节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kubernetes/client/bin/kubectl root@{node_ip}:/opt/k8s/bin/
ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp kubernetes/client/bin/kubectl root@{node_ip}:/opt/k8s/bin/
> ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kubectl 100% 41MB 122.7MB/s 00:00
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kubectl 100% 41MB 93.2MB/s 00:00
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kubectl 100% 41MB 94.6MB/s 00:00
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
创建admin证书和私钥
kubectl 使用 https 协议与 kube-apiserver 进行安全通信，kube-apiserver 对 kubectl 请求包含的证书进行认证和授权。kubectl 后续用于集群管理，所以这里创建具有最高权限的 admin 证书。
创建证书签名请求：
cd /opt/k8s/work
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "z0ukun"
}
]
}
EOF
- O: system:masters：kube-apiserver 收到使用该证书的客户端请求后，为请求添加组（Group）认证标识 system:masters；
- 预定义的 ClusterRoleBinding cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予操作集群所需的最高权限；
- 该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空。
生成证书和私钥：
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
ls admin*
[root@kubernetes-01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
> -ca-key=/opt/k8s/work/ca-key.pem \
> -config=/opt/k8s/work/ca-config.json \
> -profile=kubernetes admin-csr.json | cfssljson -bare admin
ls admin*2021/03/30 14:10:04 [INFO] generate received request
2021/03/30 14:10:04 [INFO] received CSR
2021/03/30 14:10:04 [INFO] generating key: rsa-2048
2021/03/30 14:10:04 [INFO] encoded CSR
2021/03/30 14:10:04 [INFO] signed certificate with serial number 263807722639968682310482022034641272658670741851
2021/03/30 14:10:04 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@kubernetes-01 work]# ls admin*
admin.csr admin-csr.json admin-key.pem admin.pem
[root@kubernetes-01 work]#
注：忽略警告消息： [WARNING] This certificate lacks a “hosts” field。
创建 kubeconfig 文件
kubectl 使用 kubeconfig 文件访问 apiserver，该文件包含 kube-apiserver 的地址和认证信息（CA 证书和客户端证书）：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server=https://${NODE_IPS[0]}:6443 \
--kubeconfig=kubectl.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials admin \
--client-certificate=/opt/k8s/work/admin.pem \
--client-key=/opt/k8s/work/admin-key.pem \
--embed-certs=true \
--kubeconfig=kubectl.kubeconfig
# 设置上下文参数
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin \
--kubeconfig=kubectl.kubeconfig
# 设置默认上下文
kubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig
- –certificate-authority：验证 kube-apiserver 证书的根证书；
- –client-certificate、–client-key：刚生成的 admin 证书和私钥，与 kube-apiserver https 通信时使用；
- –embed-certs=true：将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl.kubeconfig 文件中(否则，写入的是证书文件路径，后续拷贝 kubeconfig 到其它机器时，还需要单独拷贝证书文件，不方便。)；
- –server：指定 kube-apiserver 的地址，这里指向第一个节点上的服务。
分发 kubeconfig 文件
分发到所有使用 kubectl 命令的节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p ~/.kube"
scp kubectl.kubeconfig root@{node_ip}:~/.kube/config
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@{node_ip} "mkdir -p ~/.kube"
> scp kubectl.kubeconfig root@{node_ip}:~/.kube/config
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kubectl.kubeconfig 100% 6223 6.6MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kubectl.kubeconfig 100% 6223 2.2MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kubectl.kubeconfig 100% 6223 2.3MB/s 00:00
[root@kubernetes-01 work]#
5、部署Etcd集群
etcd 是基于 Raft 的分布式 KV 存储系统，由 CoreOS 开发，常用于服务发现、共享配置以及并发控制（如 leader 选举、分布式锁等）；kubernetes 使用 etcd 集群持久化存储所有 API 对象、运行数据。
本小节介绍如何部署一个三节点高可用 etcd 集群的步骤：
- 下载和分发 etcd 二进制文件；
- 创建 etcd 集群各节点的 x509 证书，用于加密客户端(如 etcdctl) 与 etcd 集群、etcd 集群之间的通信；
- 创建 etcd 的 systemd unit 文件，配置服务参数；
- 检查集群工作状态；
etcd 集群节点名称和 IP 如下：
- Kubernetes-01：172.20.0.20
- Kubernetes-02：172.20.0.21
- Kubernetes-03：172.20.0.22
注：如果没有特殊指明，本小节所有操作均在 Kubernetes-01 节点上执行。
下载etcd二进制文件
etcd下载地址：https://github.com/etcd-io/etcd/releases
cd /opt/k8s/work
wget https://github.com/coreos/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz
tar -xvf etcd-v3.4.15-linux-amd64.tar.gz
分发二进制文件到集群所有节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp /opt/k8s/work/etcd-v3.4.15-linux-amd64/etcd* root@{node_ip}:/opt/k8s/bin
ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
done
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp /opt/k8s/work/etcd-v3.4.15-linux-amd64/etcd* root@{node_ip}:/opt/k8s/bin
> ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
etcd 100% 23MB 133.4MB/s 00:00
etcdctl 100% 17MB 144.4MB/s 00:00
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
etcd 100% 23MB 120.3MB/s 00:00
etcdctl 100% 17MB 134.3MB/s 00:00
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
etcd 100% 23MB 134.1MB/s 00:00
etcdctl 100% 17MB 133.9MB/s 00:00
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 work]#
创建etcd证书和私钥
创建证书签名请求：
cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"172.20.0.20",
"172.20.0.21",
"172.20.0.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "z0ukun"
}
]
}
EOF
- hosts：指定授权使用该证书的 etcd 节点 IP 列表，需要将 etcd 集群所有节点 IP 都列在其中。
生成证书和私钥：
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pem
[root@kubernetes-01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
> -ca-key=/opt/k8s/work/ca-key.pem \
> -config=/opt/k8s/work/ca-config.json \
> -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
2021/03/31 13:23:06 [INFO] generate received request
2021/03/31 13:23:06 [INFO] received CSR
2021/03/31 13:23:06 [INFO] generating key: rsa-2048
2021/03/31 13:23:07 [INFO] encoded CSR
2021/03/31 13:23:07 [INFO] signed certificate with serial number 702521290657600647093264045094244876594460352842
[root@kubernetes-01 work]# ls etcd*pem
etcd-key.pem etcd.pem
[root@kubernetes-01 work]#
分发生成的证书和私钥到各 etcd 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p /etc/etcd/cert"
scp etcd*.pem root@{node_ip}:/etc/etcd/cert/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@{node_ip} "mkdir -p /etc/etcd/cert"
> scp etcd*.pem root@{node_ip}:/etc/etcd/cert/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
etcd-key.pem 100% 1675 1.9MB/s 00:00
etcd.pem 100% 1440 1.8MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
etcd-key.pem 100% 1675 1.5MB/s 00:00
etcd.pem 100% 1440 137.6KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
etcd-key.pem 100% 1675 1.7MB/s 00:00
etcd.pem 100% 1440 1.6MB/s 00:00
[root@kubernetes-01 work]#
创建etcd的systemd unit模板文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory={ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\
--data-dir={ETCD_DATA_DIR} \\
--wal-dir={ETCD_WAL_DIR} \\
--name=##NODE_NAME## \\
--cert-file=/etc/etcd/cert/etcd.pem \\
--key-file=/etc/etcd/cert/etcd-key.pem \\
--trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
--peer-cert-file=/etc/etcd/cert/etcd.pem \\
--peer-key-file=/etc/etcd/cert/etcd-key.pem \\
--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
--peer-client-cert-auth \\
--client-cert-auth \\
--listen-peer-urls=https://##NODE_IP##:2380 \\
--initial-advertise-peer-urls=https://##NODE_IP##:2380 \\
--listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\
--advertise-client-urls=https://##NODE_IP##:2379 \\
--initial-cluster-token=etcd-cluster-0 \\
--initial-cluster={ETCD_NODES} \\
--initial-cluster-state=new \\
--auto-compaction-mode=periodic \\
--auto-compaction-retention=1 \\
--max-request-bytes=33554432 \\
--quota-backend-bytes=6442450944 \\
--heartbeat-interval=250 \\
--election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
- WorkingDirectory、–data-dir：指定工作目录和数据目录为 ${ETCD_DATA_DIR}，需在启动服务前创建这个目录；
- –wal-dir：指定 wal 目录，为了提高性能，一般使用 SSD 或者和 –data-dir 不同的磁盘；
- –name：指定节点名称，当 –initial-cluster-state 值为 new 时，–name 的参数值必须位于 –initial-cluster 列表中；
- –cert-file、–key-file：etcd server 与 client 通信时使用的证书和私钥；
- –trusted-ca-file：签名 client 证书的 CA 证书，用于验证 client 证书；
- –peer-cert-file、–peer-key-file：etcd 与 peer 通信使用的证书和私钥；
- –peer-trusted-ca-file：签名 peer 证书的 CA 证书，用于验证 peer 证书。
为各节点创建和分发 etcd systemd unit 文件
替换模板文件中的变量，为各节点创建 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" etcd.service.template > etcd-{NODE_IPS[i]}.service done
ls *.service
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for (( i=0; i<3; i++ ))
> do
> sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" etcd.service.template>etcd-{NODE_IPS[i]}.service
> done
[root@kubernetes-01 work]# ls *.service
etcd-172.20.0.20.service etcd-172.20.0.21.service etcd-172.20.0.22.service
[root@kubernetes-01 work]#
- NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP。
分发生成的 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp etcd-{node_ip}.service root@{node_ip}:/etc/systemd/system/etcd.service
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp etcd-{node_ip}.service root@{node_ip}:/etc/systemd/system/etcd.service
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
etcd-172.20.0.20.service 100% 1392 2.1MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
etcd-172.20.0.21.service 100% 1392 1.5MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
etcd-172.20.0.22.service 100% 1392 919.0KB/s 00:00
[root@kubernetes-01 work]#
启动etcd服务
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p{ETCD_DATA_DIR} {ETCD_WAL_DIR}"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@{node_ip} "mkdir -p{ETCD_DATA_DIR} {ETCD_WAL_DIR}"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
[3] 1671
>>> 172.20.0.21
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
[4] 1680
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
[5] 1708
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 work]#
[3] Done ssh root@{node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd "
[4] Done ssh root@{node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd "
[5]- Done ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd "
[root@kubernetes-01 work]#
- 切记必须先创建 etcd 数据目录和工作目录;
- etcd 进程首次启动时会等待其它节点的 etcd 加入集群，命令 systemctl start etcd 会卡住一段时间，为正常现象。
检查启动结果
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status etcd|grep Active"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl status etcd|grep Active"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 13:36:34 CST; 33s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 13:36:34 CST; 33s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 13:36:34 CST; 34s ago
[root@kubernetes-01 work]#
注：确保状态为 active (running)，否则可以通过 journalctl -u etcd 查看日志，确认原因。
验证服务状态
部署完 etcd 集群后，在任一 etcd 节点上执行如下命令：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
/opt/k8s/bin/etcdctl \
--endpoints=https://${node_ip}:2379 \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/etcd/cert/etcd.pem \
--key=/etc/etcd/cert/etcd-key.pem endpoint health
done
- 3.4.X 版本的 etcd/etcdctl 默认启用了 V3 API，所以执行 etcdctl 命令时不需要再指定环境变量 ETCDCTL_API=3；
- 从 K8S 1.13 开始，不再支持 v2 版本的 etcd。
预期输出：
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> /opt/k8s/bin/etcdctl \
> --endpoints=https://${node_ip}:2379 \
> --cacert=/etc/kubernetes/cert/ca.pem \
> --cert=/etc/etcd/cert/etcd.pem \
> --key=/etc/etcd/cert/etcd-key.pem endpoint health
> done
>>> 172.20.0.20
https://172.20.0.20:2379 is healthy: successfully committed proposal: took = 11.520721ms
>>> 172.20.0.21
https://172.20.0.21:2379 is healthy: successfully committed proposal: took = 12.447712ms
>>> 172.20.0.22
https://172.20.0.22:2379 is healthy: successfully committed proposal: took = 12.52316ms
[root@kubernetes-01 work]#
输出均为 healthy 时表示集群服务正常。
查看当前的leader
source /opt/k8s/bin/environment.sh
/opt/k8s/bin/etcdctl \
-w table --cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/etcd/cert/etcd.pem \
--key=/etc/etcd/cert/etcd-key.pem \
--endpoints=${ETCD_ENDPOINTS} endpoint status
输出信息如下：
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# /opt/k8s/bin/etcdctl \
> -w table --cacert=/etc/kubernetes/cert/ca.pem \
> --cert=/etc/etcd/cert/etcd.pem \
> --key=/etc/etcd/cert/etcd-key.pem \
> --endpoints=${ETCD_ENDPOINTS} endpoint status
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://172.20.0.20:2379 | 96ff53158e80760 | 3.4.15 | 20 kB | true | false | 148 | 18 | 18 | |
| https://172.20.0.21:2379 | a27a01e650530e07 | 3.4.15 | 20 kB | false | false | 148 | 18 | 18 | |
| https://172.20.0.22:2379 | f868e309e507a67b | 3.4.15 | 20 kB | false | false | 148 | 18 | 18 | |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@kubernetes-01 work]#
可见，当前的 leader 为 172.20.0.20。
6、部署 Master 节点
kubernetes master 节点运行如下组件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行：
- kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例，其它实例处于阻塞模式，当 leader 挂了后，重新选举产生新的 leader，从而保证服务可用性；
- kube-apiserver 是无状态的，可以通过 kube-nginx 进行代理访问，从而保证服务可用性；
注：如果没有特殊指明，本小节所有操作均在 Kubernetes-01 节点上执行。
下载最新版本二进制文件
Kubernetes v1.19.8下载地址：https://github.com/kubernetes/kubernetes/tree/v1.16.8
cd /opt/k8s/work
# 自行解决翻墙问题
wget https://dl.k8s.io/v1.19.8/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes && tar -xzvf kubernetes-src.tar.gz
将二进制文件拷贝到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@{node_ip}:/opt/k8s/bin/
ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@{node_ip}:/opt/k8s/bin/
> ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
apiextensions-apiserver 100% 45MB 110.4MB/s 00:00
kube-apiserver 100% 110MB 124.0MB/s 00:00
kube-controller-manager 100% 102MB 122.9MB/s 00:00
kube-proxy 100% 37MB 111.7MB/s 00:00
kube-scheduler 100% 41MB 112.5MB/s 00:00
kubeadm 100% 37MB 113.0MB/s 00:00
kubectl 100% 41MB 141.4MB/s 00:00
kubelet 100% 105MB 144.7MB/s 00:00
mounter 100% 1596KB 121.0MB/s 00:00
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
apiextensions-apiserver 100% 45MB 102.8MB/s 00:00
kube-apiserver 100% 110MB 109.9MB/s 00:01
kube-controller-manager 100% 102MB 102.3MB/s 00:01
kube-proxy 100% 37MB 105.0MB/s 00:00
kube-scheduler 100% 41MB 96.5MB/s 00:00
kubeadm 100% 37MB 114.6MB/s 00:00
kubectl 100% 41MB 129.1MB/s 00:00
kubelet 100% 105MB 133.4MB/s 00:00
mounter 100% 1596KB 102.6MB/s 00:00
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
apiextensions-apiserver 100% 45MB 103.2MB/s 00:00
kube-apiserver 100% 110MB 109.9MB/s 00:01
kube-controller-manager 100% 102MB 109.2MB/s 00:00
kube-proxy 100% 37MB 102.9MB/s 00:00
kube-scheduler 100% 41MB 41.0MB/s 00:01
kubeadm 100% 37MB 108.8MB/s 00:00
kubectl 100% 41MB 119.0MB/s 00:00
kubelet 100% 105MB 111.5MB/s 00:00
mounter 100% 1596KB 86.1MB/s 00:00
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 work]#
6.1、APIServer集群
注：本小节讲解部署一个三实例 kube-apiserver 集群的步骤。如果没有特殊指明，本小节所有操作均在Kubernetes-01 节点上执行。
创建 kubernetes-master 证书和私钥
创建证书签名请求：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubernetes-csr.json <<EOF
{
"CN": "kubernetes-master",
"hosts": [
"127.0.0.1",
"172.20.0.20",
"172.20.0.21",
"172.20.0.22",
"{CLUSTER_KUBERNETES_SVC_IP}",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local.",
"kubernetes.default.svc.{CLUSTER_DNS_DOMAIN}."
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "z0ukun"
}
]
}
EOF
- hosts 字段指定授权使用该证书的 IP 和域名列表，这里列出了 master 节点 IP、kubernetes 服务的 IP 和域名；
生成证书和私钥：
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
ls kubernetes*pem
[root@kubernetes-01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
> -ca-key=/opt/k8s/work/ca-key.pem \
> -config=/opt/k8s/work/ca-config.json \
> -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
2021/03/31 14:01:12 [INFO] generate received request
2021/03/31 14:01:12 [INFO] received CSR
2021/03/31 14:01:12 [INFO] generating key: rsa-2048
2021/03/31 14:01:13 [INFO] encoded CSR
2021/03/31 14:01:13 [INFO] signed certificate with serial number 161942050792348668140609684865799859487804413770
[root@kubernetes-01 work]# ls kubernetes*pem
kubernetes-key.pem kubernetes.pem
[root@kubernetes-01 work]#
将生成的证书和私钥文件拷贝到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p /etc/kubernetes/cert"
scp kubernetes*.pem root@{node_ip}:/etc/kubernetes/cert/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@{node_ip} "mkdir -p /etc/kubernetes/cert"
> scp kubernetes*.pem root@{node_ip}:/etc/kubernetes/cert/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kubernetes-key.pem 100% 1679 2.2MB/s 00:00
kubernetes.pem 100% 1700 3.6MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kubernetes-key.pem 100% 1679 902.7KB/s 00:00
kubernetes.pem 100% 1700 2.1MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kubernetes-key.pem 100% 1679 795.6KB/s 00:00
kubernetes.pem 100% 1700 2.2MB/s 00:00
[root@kubernetes-01 work]#
创建加密配置文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: ${ENCRYPTION_KEY}
- identity: {}
EOF
将加密配置文件拷贝到所有 master 节点的 /etc/kubernetes 目录下：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp encryption-config.yaml root@{node_ip}:/etc/kubernetes/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp encryption-config.yaml root@{node_ip}:/etc/kubernetes/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
encryption-config.yaml 100% 240 273.9KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
encryption-config.yaml 100% 240 25.7KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
encryption-config.yaml 100% 240 137.7KB/s 00:00
[root@kubernetes-01 work]#
创建审计策略文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# The following requests were manually identified as high-volume and low-risk, so drop them.
- level: None
resources:
- group: ""
resources:
- endpoints
- services
- services/status
users:
- 'system:kube-proxy'
verbs:
- watch
- level: None
resources:
- group: ""
resources:
- nodes
- nodes/status
userGroups:
- 'system:nodes'
verbs:
- get
- level: None
namespaces:
- kube-system
resources:
- group: ""
resources:
- endpoints
users:
- 'system:kube-controller-manager'
- 'system:kube-scheduler'
- 'system:serviceaccount:kube-system:endpoint-controller'
verbs:
- get
- update
- level: None
resources:
- group: ""
resources:
- namespaces
- namespaces/status
- namespaces/finalize
users:
- 'system:apiserver'
verbs:
- get
# Don't log HPA fetching metrics.
- level: None
resources:
- group: metrics.k8s.io
users:
- 'system:kube-controller-manager'
verbs:
- get
- list
# Don't log these read-only URLs.
- level: None
nonResourceURLs:
- '/healthz*'
- /version
- '/swagger*'
# Don't log events requests.
- level: None
resources:
- group: ""
resources:
- events
# node and pod status calls from nodes are high-volume and can be large, don't log responses
# for expected updates from nodes
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
users:
- kubelet
- 'system:node-problem-detector'
- 'system:serviceaccount:kube-system:node-problem-detector'
verbs:
- update
- patch
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
userGroups:
- 'system:nodes'
verbs:
- update
- patch
# deletecollection calls can be large, don't log responses for expected namespace deletions
- level: Request
omitStages:
- RequestReceived
users:
- 'system:serviceaccount:kube-system:namespace-controller'
verbs:
- deletecollection
# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
# so only log at the Metadata level.
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- secrets
- configmaps
- group: authentication.k8s.io
resources:
- tokenreviews
# Get repsonses can be large; skip them.
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
verbs:
- get
- list
- watch
# Default level for known APIs
- level: RequestResponse
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
# Default level for all other requests.
- level: Metadata
omitStages:
- RequestReceived
EOF
分发审计策略文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp audit-policy.yaml root@{node_ip}:/etc/kubernetes/audit-policy.yaml
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp audit-policy.yaml root@{node_ip}:/etc/kubernetes/audit-policy.yaml
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
audit-policy.yaml 100% 4322 4.9MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
audit-policy.yaml 100% 4322 3.5MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
audit-policy.yaml 100% 4322 2.8MB/s 00:00
[root@kubernetes-01 work]#
创建后续访问 metrics-server 或 kube-prometheus 使用的证书
创建证书签名请求:
cd /opt/k8s/work
cat > proxy-client-csr.json <<EOF
{
"CN": "aggregator",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "z0ukun"
}
]
}
EOF
- CN 名称需要位于 kube-apiserver 的 –requestheader-allowed-names 参数中，否则后续访问 metrics 时会提示权限不足。
生成证书和私钥：
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
-ca-key=/etc/kubernetes/cert/ca-key.pem \
-config=/etc/kubernetes/cert/ca-config.json \
-profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client
ls proxy-client*.pem
[root@kubernetes-01 work]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
> -ca-key=/etc/kubernetes/cert/ca-key.pem \
> -config=/etc/kubernetes/cert/ca-config.json \
> -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client
2021/03/31 14:03:13 [INFO] generate received request
2021/03/31 14:03:13 [INFO] received CSR
2021/03/31 14:03:13 [INFO] generating key: rsa-2048
2021/03/31 14:03:13 [INFO] encoded CSR
2021/03/31 14:03:13 [INFO] signed certificate with serial number 519453823699954399345164730015346247955623868178
2021/03/31 14:03:13 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@kubernetes-01 work]# ls proxy-client*.pem
proxy-client-key.pem proxy-client.pem
[root@kubernetes-01 work]#
将生成的证书和私钥文件拷贝到所有 master 节点：
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp proxy-client*.pem root@{node_ip}:/etc/kubernetes/cert/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp proxy-client*.pem root@{node_ip}:/etc/kubernetes/cert/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
proxy-client-key.pem 100% 1675 2.8MB/s 00:00
proxy-client.pem 100% 1399 3.7MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
proxy-client-key.pem 100% 1675 1.7MB/s 00:00
proxy-client.pem 100% 1399 1.9MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
proxy-client-key.pem 100% 1675 1.6MB/s 00:00
proxy-client.pem 100% 1399 1.5MB/s 00:00
[root@kubernetes-01 work]#
创建 kube-apiserver systemd unit 模板文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory={K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
--advertise-address=##NODE_IP## \\
--default-not-ready-toleration-seconds=360 \\
--default-unreachable-toleration-seconds=360 \\
--max-mutating-requests-inflight=2000 \\
--max-requests-inflight=4000 \\
--default-watch-cache-size=200 \\
--delete-collection-workers=2 \\
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
--etcd-cafile=/etc/kubernetes/cert/ca.pem \\
--etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
--etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
--etcd-servers={ETCD_ENDPOINTS} \\
--bind-address=##NODE_IP## \\
--secure-port=6443 \\
--tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
--insecure-port=0 \\
--audit-log-maxage=15 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-truncate-enabled \\
--audit-log-path={K8S_DIR}/kube-apiserver/audit.log \\
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
--profiling \\
--anonymous-auth=false \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--enable-bootstrap-token-auth \\
--requestheader-allowed-names="aggregator" \\
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--service-account-key-file=/etc/kubernetes/cert/ca.pem \\
--authorization-mode=Node,RBAC \\
--runtime-config=api/all=true \\
--enable-admission-plugins=NodeRestriction \\
--allow-privileged=true \\
--apiserver-count=3 \\
--event-ttl=168h \\
--kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
--kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
--kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
--kubelet-https=true \\
--kubelet-timeout=10s \\
--proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
--proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
--service-cluster-ip-range={SERVICE_CIDR} \\
--service-node-port-range=${NODE_PORT_RANGE} \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
- –advertise-address：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）；
- –default-*-toleration-seconds：设置节点异常相关的阈值；
- –max-*-requests-inflight：请求相关的最大阈值；
- –etcd-*：访问 etcd 的证书和 etcd 服务器地址；
- –bind-address： https 监听的 IP，不能为 127.0.0.1，否则外界不能访问它的安全端口 6443；
- –secret-port：https 监听端口；
- –insecure-port=0：关闭监听 http 非安全端口(8080)；
- –tls-*-file：指定 apiserver 使用的证书、私钥和 CA 文件；
- –audit-*：配置审计策略和审计日志文件相关的参数；
- –client-ca-file：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书；
- –enable-bootstrap-token-auth：启用 kubelet bootstrap 的 token 认证；
- –requestheader-*：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用；
- –requestheader-client-ca-file：用于签名 –proxy-client-cert-file 和 –proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用；
- –requestheader-allowed-names：不能为空，值为逗号分割的 –proxy-client-cert-file 证书的 CN 名称，这里设置为 “aggregator”；
- –service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 –service-account-private-key-file 指定私钥文件，两者配对使用；
- –runtime-config=api/all=true： 启用所有版本的 APIs，如 autoscaling/v2alpha1；
- –authorization-mode=Node,RBAC、–anonymous-auth=false： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求；
- –enable-admission-plugins：启用一些默认关闭的 plugins；
- –allow-privileged：运行执行 privileged 权限的容器；
- –apiserver-count=3：指定 apiserver 实例的数量；
- –event-ttl：指定 events 的保存时间；
- –kubelet-：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权；
- –proxy-client-*：apiserver 访问 metrics-server 使用的证书；
- –service-cluster-ip-range： 指定 Service Cluster IP 地址段；
- –service-node-port-range： 指定 NodePort 的端口范围；
注：如果 kube-apiserver 机器没有运行 kube-proxy，则还需要添加 –enable-aggregator-routing=true 参数；
关于 –requestheader-XXX 相关参数，参考：
- https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md
- https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/
注： –requestheader-client-ca-file 指定的 CA 证书，必须具有 client auth and server auth； 如果 –requestheader-allowed-names 不为空,且 –proxy-client-cert-file 证书的 CN 名称不在 allowed-names 中，则后续查看 node 或 pods 的 metrics 失败，提示：
为各节点创建和分发 kube-apiserver systemd unit 文件
替换模板文件中的变量，为各节点生成 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-{NODE_IPS[i]}.service done
ls kube-apiserver*.service
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for (( i=0; i<3; i++ ))
> do
> sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-apiserver.service.template>kube-apiserver-{NODE_IPS[i]}.service
> done
[root@kubernetes-01 work]# ls kube-apiserver*.service
kube-apiserver-172.20.0.20.service kube-apiserver-172.20.0.21.service kube-apiserver-172.20.0.22.service
[root@kubernetes-01 work]#
- NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；
分发生成的 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-apiserver-{node_ip}.service root@{node_ip}:/etc/systemd/system/kube-apiserver.service
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp kube-apiserver-{node_ip}.service root@{node_ip}:/etc/systemd/system/kube-apiserver.service
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-apiserver-172.20.0.20.service 100% 2506 3.6MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-apiserver-172.20.0.21.service 100% 2506 2.8MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-apiserver-172.20.0.22.service 100% 2506 2.5MB/s 00:00
[root@kubernetes-01 work]#
启动 kube-apiserver 服务
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p{K8S_DIR}/kube-apiserver"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "mkdir -p {K8S_DIR}/kube-apiserver"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /etc/systemd/system/kube-apiserver.service.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /etc/systemd/system/kube-apiserver.service.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /etc/systemd/system/kube-apiserver.service.
[root@kubernetes-01 work]#
检查 kube-apiserver 运行状态
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status kube-apiserver |grep 'Active:'"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl status kube-apiserver |grep 'Active:'"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:04:30 CST; 15s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:04:35 CST; 10s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:04:40 CST; 5s ago
[root@kubernetes-01 work]#
确保状态为 active (running)，否则可以通过 journalctl -u kube-apiserver 查看日志，确认原因。
检查集群状态
# 查看集群信息
[root@kubernetes-01 work]# kubectl cluster-info
Kubernetes master is running at https://172.20.0.20:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@kubernetes-01 work]#
# 查看所有的命令空间
[root@kubernetes-01 work]# kubectl get all --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 2m6s
[root@kubernetes-01 work]#
# 查看组件状态
[root@kubernetes-01 work]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
[root@kubernetes-01 work]#
参考文章：https://zhongpan.tech/2019/10/30/018-output-change-kubectl-get-cs/
Github Issue：https://github.com/kubernetes/kubernetes/issues/83024
检查 kube-apiserver 监听的端口
[root@kubernetes-01 work]# sudo netstat -lnpt|grep kube
tcp 0 0 172.20.0.20:6443 0.0.0.0:* LISTEN 3679/kube-apiserver
[root@kubernetes-01 work]#
- 6443: 接收 https 请求的安全端口，对所有请求做认证和授权；
- 由于关闭了非安全端口，故没有监听 8080；
6.2、Controller-Manager集群
本小节介绍部署高可用 kube-controller-manager 集群的步骤。该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。
为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：
- 与 kube-apiserver 的安全端口通信;
- 在安全端口(https，10252) 输出 prometheus 格式的 metrics；
注：如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行。
创建 kube-controller-manager 证书和私钥
创建证书签名请求：
cd /opt/k8s/work
cat > kube-controller-manager-csr.json <<EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"172.20.0.20",
"172.20.0.21",
"172.20.0.22"
],
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:kube-controller-manager",
"OU": "z0ukun"
}
]
}
EOF
- hosts 列表包含所有 kube-controller-manager 节点 IP；
- CN 和 O 均为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。
生成证书和私钥：
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
ls kube-controller-manager*pem
[root@kubernetes-01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
> -ca-key=/opt/k8s/work/ca-key.pem \
> -config=/opt/k8s/work/ca-config.json \
> -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2021/03/31 14:08:34 [INFO] generate received request
2021/03/31 14:08:34 [INFO] received CSR
2021/03/31 14:08:34 [INFO] generating key: rsa-2048
2021/03/31 14:08:35 [INFO] encoded CSR
2021/03/31 14:08:35 [INFO] signed certificate with serial number 167734461365167531618854597958116803518757833256
[root@kubernetes-01 work]# ls kube-controller-manager*pem
kube-controller-manager-key.pem kube-controller-manager.pem
[root@kubernetes-01 work]#
将生成的证书和私钥分发到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-controller-manager*.pem root@{node_ip}:/etc/kubernetes/cert/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp kube-controller-manager*.pem root@{node_ip}:/etc/kubernetes/cert/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-controller-manager-key.pem 100% 1675 1.9MB/s 00:00
kube-controller-manager.pem 100% 1513 2.6MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-controller-manager-key.pem 100% 1675 1.8MB/s 00:00
kube-controller-manager.pem 100% 1513 2.0MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-controller-manager-key.pem 100% 1675 1.4MB/s 00:00
kube-controller-manager.pem 100% 1513 1.9MB/s 00:00
[root@kubernetes-01 work]#
创建和分发 kubeconfig 文件
kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书等信息：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server="https://##NODE_IP##:6443" \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.pem \
--client-key=kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
- kube-controller-manager 与 kube-apiserver 混布，故直接通过节点 IP 访问 kube-apiserver。
分发 kubeconfig 到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
sed -e "s/##NODE_IP##/{node_ip}/" kube-controller-manager.kubeconfig>kube-controller-manager-{node_ip}.kubeconfig
scp kube-controller-manager-{node_ip}.kubeconfig root@{node_ip}:/etc/kubernetes/kube-controller-manager.kubeconfig
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> sed -e "s/##NODE_IP##/{node_ip}/" kube-controller-manager.kubeconfig>kube-controller-manager-{node_ip}.kubeconfig
> scp kube-controller-manager-{node_ip}.kubeconfig root@{node_ip}:/etc/kubernetes/kube-controller-manager.kubeconfig
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-controller-manager-172.20.0.20.kubeconfig 100% 6453 8.1MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-controller-manager-172.20.0.21.kubeconfig 100% 6453 5.1MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-controller-manager-172.20.0.22.kubeconfig 100% 6453 5.2MB/s 00:00
[root@kubernetes-01 work]#
创建 kube-controller-manager systemd unit 模板文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory={K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\
--profiling \\
--cluster-name=kubernetes \\
--controllers=*,bootstrapsigner,tokencleaner \\
--kube-api-qps=1000 \\
--kube-api-burst=2000 \\
--leader-elect \\
--use-service-account-credentials\\
--concurrent-service-syncs=2 \\
--bind-address=##NODE_IP## \\
--secure-port=10252 \\
--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
--port=0 \\
--authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-allowed-names="aggregator" \\
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
--experimental-cluster-signing-duration=876000h \\
--horizontal-pod-autoscaler-sync-period=10s \\
--concurrent-deployment-syncs=10 \\
--concurrent-gc-syncs=30 \\
--node-cidr-mask-size=24 \\
--service-cluster-ip-range={SERVICE_CIDR} \\
--pod-eviction-timeout=6m \\
--terminated-pod-gc-threshold=10000 \\
--root-ca-file=/etc/kubernetes/cert/ca.pem \\
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
- –port=0：关闭监听非安全端口（http），同时 –address 参数无效，–bind-address 参数有效；
- –secure-port=10252、–bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求；
- –kubeconfig：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver；
- –authentication-kubeconfig 和 –authorization-kubeconfig：kube-controller-manager 使用它连接 apiserver，对 client 的请求进行认证和授权。kube-controller-manager 不再使用 –tls-ca-file 对请求 https metrics 的 Client 证书进行校验。如果没有配置这两个 kubeconfig 参数，则 client 连接 kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。
- –cluster-signing-*-file：签名 TLS Bootstrap 创建的证书；
- –experimental-cluster-signing-duration：指定 TLS Bootstrap 证书的有效期；
- –root-ca-file：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验；
- –service-account-private-key-file：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的 –service-account-key-file 指定的公钥文件配对使用；
- –service-cluster-ip-range ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致；
- –leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；
- –controllers=*,bootstrapsigner,tokencleaner：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token；
- –horizontal-pod-autoscaler-*：custom metrics 相关参数，支持 autoscaling/v2alpha1； –tls-cert-file、
- –tls-private-key-file：使用 https 输出 metrics 时使用的 Server 证书和秘钥；
- –use-service-account-credentials=true: kube-controller-manager 中各 controller 使用 serviceaccount 访问 kube-apiserver；
为各节点创建和分发 kube-controller-mananger systemd unit 文件
替换模板文件中的变量，为各节点创建 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-{NODE_IPS[i]}.service done
ls kube-controller-manager*.service
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for (( i=0; i<3; i++ ))
> do
> sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-controller-manager.service.template>kube-controller-manager-{NODE_IPS[i]}.service
> done
[root@kubernetes-01 work]# ls kube-controller-manager*.service
kube-controller-manager-172.20.0.20.service kube-controller-manager-172.20.0.21.service kube-controller-manager-172.20.0.22.service
[root@kubernetes-01 work]#
分发到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-controller-manager-{node_ip}.service root@{node_ip}:/etc/systemd/system/kube-controller-manager.service
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp kube-controller-manager-{node_ip}.service root@{node_ip}:/etc/systemd/system/kube-controller-manager.service
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-controller-manager-172.20.0.20.service 100% 1882 1.9MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-controller-manager-172.20.0.21.service 100% 1882 1.8MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-controller-manager-172.20.0.22.service 100% 1882 1.6MB/s 00:00
[root@kubernetes-01 work]#
启动 kube-controller-manager 服务
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p{K8S_DIR}/kube-controller-manager"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "mkdir -p {K8S_DIR}/kube-controller-manager"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.
[root@kubernetes-01 work]#
检查服务运行状态
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status kube-controller-manager|grep Active"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl status kube-controller-manager|grep Active"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:10:48 CST; 18s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:10:48 CST; 18s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:10:48 CST; 17s ago
[root@kubernetes-01 work]#
确保状态为 active (running)，否则 可以通过 journalctl -u kube-controller-manager 命令查看日志，确认原因。
kube-controller-manager 监听 10252 端口，接收 https 请求：
[root@kubernetes-01 work]# sudo netstat -nltp | grep kube-cont
tcp 0 0 172.20.0.20:10252 0.0.0.0:* LISTEN 4328/kube-controlle
[root@kubernetes-01 work]#
查看输出的 metrics
下面的命令在 kube-controller-manager 节点上执行：
[root@kubernetes-01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.20.0.20:10252/metrics |head
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
[root@kubernetes-01 work]#
查看当前的 leader
[root@kubernetes-01 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-01_bebce482-0a5f-41c3-b1dd-d7d19ca128c2","leaseDurationSeconds":15,"acquireTime":"2021-03-31T06:10:48Z","renewTime":"2021-03-31T06:12:28Z","leaderTransitions":0}'
creationTimestamp: "2021-03-31T06:10:48Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:control-plane.alpha.kubernetes.io/leader: {}
manager: kube-controller-manager
operation: Update
time: "2021-03-31T06:10:48Z"
name: kube-controller-manager
namespace: kube-system
resourceVersion: "534"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: a34fb645-ced8-4f53-84de-c6070e4ae7b0
[root@kubernetes-01 work]#
可见，当前的 leader 为 Kubernetes-01 节点；其他的节点测试请各位小伙伴自行完成。
测试 kube-controller-manager 集群的高可用
停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。
[root@kubernetes-01 work]# systemctl stop kube-controller-manager
[root@kubernetes-02 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-01_bebce482-0a5f-41c3-b1dd-d7d19ca128c2","leaseDurationSeconds":15,"acquireTime":"2021-03-31T06:10:48Z","renewTime":"2021-03-31T06:13:37Z","leaderTransitions":0}'
creationTimestamp: "2021-03-31T06:10:48Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:control-plane.alpha.kubernetes.io/leader: {}
manager: kube-controller-manager
operation: Update
time: "2021-03-31T06:10:48Z"
name: kube-controller-manager
namespace: kube-system
resourceVersion: "623"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: a34fb645-ced8-4f53-84de-c6070e4ae7b0
[root@kubernetes-02 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-03_b71fe527-2fde-4f36-b079-9dfa64175ccd","leaseDurationSeconds":15,"acquireTime":"2021-03-31T06:13:56Z","renewTime":"2021-03-31T06:13:56Z","leaderTransitions":1}'
creationTimestamp: "2021-03-31T06:10:48Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:control-plane.alpha.kubernetes.io/leader: {}
manager: kube-controller-manager
operation: Update
time: "2021-03-31T06:10:48Z"
name: kube-controller-manager
namespace: kube-system
resourceVersion: "631"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: a34fb645-ced8-4f53-84de-c6070e4ae7b0
[root@kubernetes-02 ~]#
- 关于 controller 权限和 use-service-account-credentials 参数：https://github.com/kubernetes/kubernetes/issues/48208
- kubelet 认证和授权：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization
6.3、Scheduler集群
本小节介绍部署高可用 kube-scheduler 集群的步骤。该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。为保证通信安全，本文档先生成 x509 证书和私钥，kube-scheduler 在如下两种情况下使用该证书：
- 与 kube-apiserver 的安全端口通信;
- 在安全端口(https，10251) 输出 prometheus 格式的 metrics；
注：如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行。
创建 kube-scheduler 证书和私钥
创建证书签名请求：
cd /opt/k8s/work
cat > kube-scheduler-csr.json <<EOF
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"172.20.0.20",
"172.20.0.21",
"172.20.0.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:kube-scheduler",
"OU": "z0ukun"
}
]
}
EOF
- hosts 列表包含所有 kube-scheduler 节点 IP；
- CN 和 O 均为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限；
生成证书和私钥：
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
ls kube-scheduler*pem
[root@kubernetes-01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
> -ca-key=/opt/k8s/work/ca-key.pem \
> -config=/opt/k8s/work/ca-config.json \
> -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
2021/03/31 14:15:06 [INFO] generate received request
2021/03/31 14:15:06 [INFO] received CSR
2021/03/31 14:15:06 [INFO] generating key: rsa-2048
2021/03/31 14:15:06 [INFO] encoded CSR
2021/03/31 14:15:06 [INFO] signed certificate with serial number 216169409049839048403577298545882720794791501568
[root@kubernetes-01 work]# ls kube-scheduler*pem
kube-scheduler-key.pem kube-scheduler.pem
[root@kubernetes-01 work]#
将生成的证书和私钥分发到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-scheduler*.pem root@{node_ip}:/etc/kubernetes/cert/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp kube-scheduler*.pem root@{node_ip}:/etc/kubernetes/cert/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-scheduler-key.pem 100% 1675 2.6MB/s 00:00
kube-scheduler.pem 100% 1489 3.1MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-scheduler-key.pem 100% 1675 1.5MB/s 00:00
kube-scheduler.pem 100% 1489 1.6MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-scheduler-key.pem 100% 1675 1.4MB/s 00:00
kube-scheduler.pem 100% 1489 1.3MB/s 00:00
[root@kubernetes-01 work]#
创建和分发 kubeconfig 文件
kube-scheduler 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server="https://##NODE_IP##:6443" \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
--client-certificate=kube-scheduler.pem \
--client-key=kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
分发 kubeconfig 到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
sed -e "s/##NODE_IP##/{node_ip}/" kube-scheduler.kubeconfig>kube-scheduler-{node_ip}.kubeconfig
scp kube-scheduler-{node_ip}.kubeconfig root@{node_ip}:/etc/kubernetes/kube-scheduler.kubeconfig
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> sed -e "s/##NODE_IP##/{node_ip}/" kube-scheduler.kubeconfig>kube-scheduler-{node_ip}.kubeconfig
> scp kube-scheduler-{node_ip}.kubeconfig root@{node_ip}:/etc/kubernetes/kube-scheduler.kubeconfig
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.20.kubeconfig 100% 6385 5.7MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.21.kubeconfig 100% 6385 5.9MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.22.kubeconfig 100% 6385 5.7MB/s 00:00
[root@kubernetes-01 work]#
创建 kube-scheduler 配置文件
cd /opt/k8s/work
cat >kube-scheduler.yaml.template <<EOF
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig"
enableContentionProfiling: false
enableProfiling: true
healthzBindAddress: ##NODE_IP##:10251
leaderElection:
leaderElect: true
metricsBindAddress: ##NODE_IP##:10251
EOF
- –kubeconfig：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver；
- –leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；
替换模板文件中的变量：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-scheduler.yaml.template > kube-scheduler-{NODE_IPS[i]}.yaml
done
ls kube-scheduler*.yaml
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for (( i=0; i<3; i++ ))
> do
> sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-scheduler.yaml.template>kube-scheduler-{NODE_IPS[i]}.yaml
> done
[root@kubernetes-01 work]# ls kube-scheduler*.yaml
kube-scheduler-172.20.0.20.yaml kube-scheduler-172.20.0.21.yaml kube-scheduler-172.20.0.22.yaml
[root@kubernetes-01 work]#
- NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；
分发 kube-scheduler 配置文件到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-scheduler-{node_ip}.yaml root@{node_ip}:/etc/kubernetes/kube-scheduler.yaml
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp kube-scheduler-{node_ip}.yaml root@{node_ip}:/etc/kubernetes/kube-scheduler.yaml
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.20.yaml 100% 324 525.8KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.21.yaml 100% 324 262.3KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.22.yaml 100% 324 353.5KB/s 00:00
[root@kubernetes-01 work]#
- 重命名为 kube-scheduler.yaml；
创建 kube-scheduler systemd unit 模板文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-scheduler.service.template <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-scheduler
ExecStart=/opt/k8s/bin/kube-scheduler \\
--config=/etc/kubernetes/kube-scheduler.yaml \\
--bind-address=##NODE_IP## \\
--secure-port=10259 \\
--port=0 \\
--tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\
--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--logtostderr=true \\
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF
为各节点创建和分发 kube-scheduler systemd unit 文件
替换模板文件中的变量，为各节点创建 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-{NODE_IPS[i]}.service done
ls kube-scheduler*.service
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
for (( i=0; i<3; i++ ))
[root@kubernetes-01 work]# for (( i=0; i<3; i++ ))
> do
> sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-scheduler.service.template>kube-scheduler-{NODE_IPS[i]}.service
> done
[root@kubernetes-01 work]# ls kube-scheduler*.service
kube-scheduler-172.20.0.20.service kube-scheduler-172.20.0.21.service kube-scheduler-172.20.0.22.service
[root@kubernetes-01 work]#
分发 systemd unit 文件到所有 master 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-scheduler-{node_ip}.service root@{node_ip}:/etc/systemd/system/kube-scheduler.service
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> scp kube-scheduler-{node_ip}.service root@{node_ip}:/etc/systemd/system/kube-scheduler.service
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.20.service 100% 985 1.4MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.21.service 100% 985 824.8KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-scheduler-172.20.0.22.service 100% 985 770.5KB/s 00:00
[root@kubernetes-01 work]#
启动 kube-scheduler 服务
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p{K8S_DIR}/kube-scheduler"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "mkdir -p {K8S_DIR}/kube-scheduler"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /etc/systemd/system/kube-scheduler.service.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /etc/systemd/system/kube-scheduler.service.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /etc/systemd/system/kube-scheduler.service.
[root@kubernetes-01 work]#
检查服务运行状态
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status kube-scheduler|grep Active"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl status kube-scheduler|grep Active"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:17:36 CST; 14s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:17:36 CST; 13s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:17:37 CST; 13s ago
[root@kubernetes-01 work]#
确保状态为 active (running)，否则可以通过 journalctl -u kube-scheduler 命令查看日志，确认原因。
查看输出的 metrics
注：以下命令在 kube-scheduler 节点上执行。
kube-scheduler 监听 10251 和 10259 端口：
- 10251：接收 http 请求，非安全端口，不需要认证授权；
- 10259：接收 https 请求，安全端口，需要认证授权；
两个接口都对外提供 /metrics 和 /healthz 的访问。
[root@kubernetes-01 work]# sudo netstat -nltp | grep kube-sch
tcp 0 0 172.20.0.20:10251 0.0.0.0:* LISTEN 5019/kube-scheduler
tcp 0 0 172.20.0.20:10259 0.0.0.0:* LISTEN 5019/kube-scheduler
[root@kubernetes-01 work]#
[root@kubernetes-01 work]# curl -s http://172.20.0.20:10251/metrics |head
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
[root@kubernetes-01 work]#
[root@kubernetes-01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.20.0.20:10259/metrics |head
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
[root@kubernetes-01 work]#
查看当前的 leader
[root@kubernetes-01 work]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-01_feaae043-cf6a-4d22-9227-4c8a3e6fc49b","leaseDurationSeconds":15,"acquireTime":"2021-03-31T06:17:37Z","renewTime":"2021-03-31T06:19:35Z","leaderTransitions":0}'
creationTimestamp: "2021-03-31T06:17:37Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:control-plane.alpha.kubernetes.io/leader: {}
manager: kube-scheduler
operation: Update
time: "2021-03-31T06:17:37Z"
name: kube-scheduler
namespace: kube-system
resourceVersion: "1190"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
uid: 40dc57ef-af3a-4e83-98c7-cdcd79d10704
[root@kubernetes-01 work]#
可见，当前的 leader 为 Kubernetes-01 节点。
测试 kube-scheduler 集群的高可用
随便找一个或两个 master 节点，停掉 kube-scheduler 服务，看其它节点是否获取了 leader 权限。
[root@kubernetes-01 work]# systemctl stop kube-scheduler
[root@kubernetes-02 ~]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-01_feaae043-cf6a-4d22-9227-4c8a3e6fc49b","leaseDurationSeconds":15,"acquireTime":"2021-03-31T06:17:37Z","renewTime":"2021-03-31T06:20:26Z","leaderTransitions":0}'
creationTimestamp: "2021-03-31T06:17:37Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:control-plane.alpha.kubernetes.io/leader: {}
manager: kube-scheduler
operation: Update
time: "2021-03-31T06:17:37Z"
name: kube-scheduler
namespace: kube-system
resourceVersion: "1305"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
uid: 40dc57ef-af3a-4e83-98c7-cdcd79d10704
[root@kubernetes-02 ~]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"kubernetes-03_67d9c4d1-fb75-495a-b628-17ffe90d068b","leaseDurationSeconds":15,"acquireTime":"2021-03-31T06:20:45Z","renewTime":"2021-03-31T06:20:45Z","leaderTransitions":1}'
creationTimestamp: "2021-03-31T06:17:37Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:control-plane.alpha.kubernetes.io/leader: {}
manager: kube-scheduler
operation: Update
time: "2021-03-31T06:17:37Z"
name: kube-scheduler
namespace: kube-system
resourceVersion: "1331"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
uid: 40dc57ef-af3a-4e83-98c7-cdcd79d10704
[root@kubernetes-02 ~]#
7、部署 Worker 节点
kubernetes worker 节点运行如下组件：
- docker
- kubelet
- kube-proxy
- kube-nginx
注：如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行。切记在开始部署Worker集群之前、我们需要先去部署Flannel网络和Docker。
安装依赖包
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "yum install -y epel-release" &
ssh root@{node_ip} "yum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git" &
done
7.1、APIServer高可用
本小节讲解使用 nginx 4 层透明代理功能实现 Kubernetes worker 节点组件高可用访问 kube-apiserver 集群的步骤。
注：如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行。
基于 nginx 代理的 kube-apiserver 高可用方案
- 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署且连接本机的 kube-apiserver，所以只要有一个实例正常，就可以保证高可用；
- 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；
- 在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；
- kubelet、kube-proxy 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用。
下载和编译Nginx
Nginx下载地址：http://nginx.org/
下载源码（这里我们安装Nginx 1.18.0）：
cd /opt/k8s/work && wget http://nginx.org/download/nginx-1.18.0.tar.gz && tar -xzvf nginx-1.18.0.tar.gz
配置编译参数
cd /opt/k8s/work/nginx-1.18.0 &&mkdir nginx-prefix && yum install -y gcc make
./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
- –with-stream：开启 4 层透明转发(TCP Proxy)功能；
- –without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；
编译和安装：
cd /opt/k8s/work/nginx-1.18.0 && make && make install
验证编译的 nginx
[root@kubernetes-01 nginx-1.18.0]# ./nginx-prefix/sbin/nginx -v
nginx version: nginx/1.18.0
[root@kubernetes-01 nginx-1.18.0]#
安装和部署Nginx
创建目录结构：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 work]#
拷贝二进制程序：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
scp /opt/k8s/work/nginx-1.18.0/nginx-prefix/sbin/nginx root@{node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
ssh root@{node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
> scp /opt/k8s/work/nginx-1.18.0/nginx-prefix/sbin/nginx root@{node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
> ssh root@{node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
nginx 100% 1740KB 132.2MB/s 00:00
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
nginx 100% 1740KB 105.4MB/s 00:00
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
nginx 100% 1740KB 106.5MB/s 00:00
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 work]#
- 这里我们重命名二进制文件为 kube-nginx；
配置 nginx，开启 4 层透明转发功能：
cd /opt/k8s/work
cat > kube-nginx.conf << \EOF
worker_processes 1;
events {
worker_connections 1024;
}
stream {
upstream backend {
hash $remote_addr consistent;
server 172.20.0.20:6443 max_fails=3 fail_timeout=30s;
server 172.20.0.21:6443 max_fails=3 fail_timeout=30s;
server 172.20.0.22:6443 max_fails=3 fail_timeout=30s;
}
server {
listen 127.0.0.1:8443;
proxy_connect_timeout 1s;
proxy_pass backend;
}
}
EOF
- upstream backend 中的 server 列表为集群中各 kube-apiserver 的节点 IP，需要根据实际情况修改；
分发配置文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-nginx.conf root@{node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp kube-nginx.conf root@{node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-nginx.conf 100% 461 548.3KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-nginx.conf 100% 461 453.8KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-nginx.conf 100% 461 230.5KB/s 00:00
[root@kubernetes-01 work]#
配置 systemd unit 文件，启动服务
配置 kube-nginx systemd unit 文件：
cd /opt/k8s/work
cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
分发 systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp kube-nginx.service root@{node_ip}:/etc/systemd/system/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp kube-nginx.service root@{node_ip}:/etc/systemd/system/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kube-nginx.service 100% 624 870.6KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kube-nginx.service 100% 624 601.9KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kube-nginx.service 100% 624 518.2KB/s 00:00
[root@kubernetes-01 work]#
启动 kube-nginx 服务：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-nginx.service to /etc/systemd/system/kube-nginx.service.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-nginx.service to /etc/systemd/system/kube-nginx.service.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-nginx.service to /etc/systemd/system/kube-nginx.service.
[root@kubernetes-01 work]#
检查 kube-nginx 服务运行状态
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status kube-nginx |grep 'Active:'"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl status kube-nginx |grep 'Active:'"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:50:37 CST; 17s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:50:37 CST; 17s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:50:37 CST; 16s ago
[root@kubernetes-01 work]#
确保状态为 active (running)，否则可以通过 journalctl -u kube-nginx 查看日志，确认原因。
7.2、部署Docker
docker 运行和管理容器，kubelet 通过 Container Runtime Interface (CRI) 与它进行交互。
注：在开始部署Docker之前一定要记得提前部署Flannel网络；如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行，然后远程分发文件和执行命令；如果之前安装过Docker的相关版本、记得一定要把Docker提前卸载干净。
Docker卸载命令：
# 查询已安装的Docker版本
rpm -qa | grep docker
# 卸载已安装的Docker版本
yum remove -y ***
# 重启主机
reboot
安装依赖包
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "yum install -y epel-release" &
ssh root@{node_ip} "yum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git" &
done
下载Docker二进制文件：
Docker下载地址：https://download.docker.com/linux/static/stable/x86_64/
cd /opt/k8s/work && wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.8.tgz
tar -xvf docker-19.03.8.tgz
分发Docker二进制文件到所有的Worker节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
scp docker/* root@{node_ip}:/opt/k8s/bin/
ssh root@{node_ip} "chmod +x /opt/k8s/bin/*"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp docker/* root@{node_ip}:/opt/k8s/bin/
> ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
containerd 100% 33MB 141.3MB/s 00:00
containerd-shim 100% 5977KB 135.4MB/s 00:00
ctr 100% 18MB 141.5MB/s 00:00
docker 100% 63MB 141.4MB/s 00:00
dockerd 100% 69MB 143.9MB/s 00:00
docker-init 100% 746KB 125.5MB/s 00:00
docker-proxy 100% 2810KB 137.9MB/s 00:00
runc 100% 9403KB 144.1MB/s 00:00
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
containerd 100% 33MB 131.6MB/s 00:00
containerd-shim 100% 5977KB 129.5MB/s 00:00
ctr 100% 18MB 109.1MB/s 00:00
docker 100% 63MB 110.3MB/s 00:00
dockerd 100% 69MB 124.5MB/s 00:00
docker-init 100% 746KB 107.5MB/s 00:00
docker-proxy 100% 2810KB 124.3MB/s 00:00
runc 100% 9403KB 105.8MB/s 00:00
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
containerd 100% 33MB 123.5MB/s 00:00
containerd-shim 100% 5977KB 128.2MB/s 00:00
ctr 100% 18MB 129.6MB/s 00:00
docker 100% 63MB 134.3MB/s 00:00
dockerd 100% 69MB 127.4MB/s 00:00
docker-init 100% 746KB 105.5MB/s 00:00
docker-proxy 100% 2810KB 119.4MB/s 00:00
runc 100% 9403KB 123.9MB/s 00:00
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
[root@kubernetes-01 work]#
创建和分发 systemd unit 文件
cd /opt/k8s/work
cat > docker.service <<"EOF"
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
[Service]
WorkingDirectory=##DOCKER_DIR##
Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/opt/k8s/bin/dockerd DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUPMAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
- EOF 前后有双引号，这样 bash 不会替换文档中的变量，如 $DOCKER_NETWORK_OPTIONS (这些环境变量是 systemd 负责替换的。)；
- dockerd 运行时会调用其它 docker 命令，如 docker-proxy，所以需要将 docker 命令所在的目录加到 PATH 环境变量中；
- flanneld 启动时将网络配置写入 /run/flannel/docker 文件中，dockerd 启动前读取该文件中的环境变量 DOCKER_NETWORK_OPTIONS ，然后设置 docker0 网桥网段； 如果指定了多个 EnvironmentFile 选项，则必须将 /run/flannel/docker 放在最后(确保 docker0 使用 flanneld 生成的 bip 参数)；
- docker 需要以 root 用于运行；
- docker 从 1.13 版本开始，可能将 iptables FORWARD chain的默认策略设置为DROP，从而导致 ping 其它 Node 上的 Pod IP 失败，遇到这种情况时，需要手动设置策略为 ACCEPT：sudo iptables -P FORWARD ACCEPT；并且把以下命令写入 /etc/rc.local 文件中，防止节点重启iptables FORWARD chain的默认策略又还原为DROP：/sbin/iptables -P FORWARD ACCEPT
分发 systemd unit 文件到所有 worker 机器:
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
sed -i -e "s|##DOCKER_DIR##|{DOCKER_DIR}|" docker.service
for node_ip in{NODE_IPS[@]}
do
echo ">>> {node_ip}"
scp docker.service root@{node_ip}:/etc/systemd/system/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# sed -i -e "s|##DOCKER_DIR##|{DOCKER_DIR}|" docker.service
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> scp docker.service root@{node_ip}:/etc/systemd/system/
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
docker.service 100% 487 675.4KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
docker.service 100% 487 512.2KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
docker.service 100% 487 485.1KB/s 00:00
[root@kubernetes-01 work]#
配置和分发 docker 配置文件
使用国内的仓库镜像服务器以加快 pull image 的速度，同时增加下载的并发数 (需要重启 dockerd 生效)：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > docker-daemon.json <<EOF
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
"insecure-registries": ["docker02:35000"],
"max-concurrent-downloads": 20,
"live-restore": true,
"max-concurrent-uploads": 10,
"debug": true,
"data-root": "{DOCKER_DIR}/data",
"exec-root": "{DOCKER_DIR}/exec",
"log-opts": {
"max-size": "100m",
"max-file": "5"
}
}
EOF
分发 docker 配置文件到所有 worker 节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p /etc/docker/{DOCKER_DIR}/{data,exec}"
scp docker-daemon.json root@{node_ip}:/etc/docker/daemon.json
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "mkdir -p /etc/docker/ {DOCKER_DIR}/{data,exec}"
> scp docker-daemon.json root@{node_ip}:/etc/docker/daemon.json
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
docker-daemon.json 100% 417 272.7KB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
docker-daemon.json 100% 417 305.8KB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
docker-daemon.json 100% 417 226.0KB/s 00:00
启动 docker 服务
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable docker && systemctl restart docker"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable docker && systemctl restart docker"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /etc/systemd/system/docker.service.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /etc/systemd/system/docker.service.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /etc/systemd/system/docker.service.
检查服务运行状态
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status docker|grep Active"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
for node_ip in{NODE_IPS[@]}
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@${node_ip} "systemctl status docker|grep Active"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:29:30 CST; 4s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:29:30 CST; 4s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 14:29:30 CST; 4s ago
[root@kubernetes-01 work]#
确保状态为 active (running)，否则可以通过 journalctl -u docker 查看日志，确认原因。
查看 docker 的状态信息
[root@kubernetes-01 work]# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.4.108-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.82GiB
Name: kubernetes-01
ID: 4FKH:ERXJ:64V7:WMHA:PBUP:G27H:L3BI:ADXT:2RAZ:2TC7:UEAX:ZX4N
Docker Root Dir: /data/k8s/docker/data
Debug Mode: true
File Descriptors: 22
Goroutines: 40
System Time: 2021-03-31T14:31:05.770114236+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
docker02:35000
127.0.0.0/8
Registry Mirrors:
https://docker.mirrors.ustc.edu.cn/
https://hub-mirror.c.163.com/
Live Restore Enabled: true
Product License: Community Engine
[root@kubernetes-01 work]#
7.3、Kubelet
kubelet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，执行交互式命令，如 exec、run、logs 等。kubelet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况。为确保安全，部署时关闭了 kubelet 的非安全 http 端口，对请求进行认证和授权，拒绝未授权的访问(如 apiserver、heapster 的请求)。
注：如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行。
创建 kubelet bootstrap kubeconfig 文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in {NODE_NAMES[@]}
do
echo ">>>{node_name}"
# 创建 token
export BOOTSTRAP_TOKEN=(kubeadm token create \
--description kubelet-bootstrap-token \
--groups system:bootstrappers:{node_name} \
--kubeconfig ~/.kube/config)
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/cert/ca.pem \
--embed-certs=true \
--server={KUBE_APISERVER} \
--kubeconfig=kubelet-bootstrap-{node_name}.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token={BOOTSTRAP_TOKEN} \
--kubeconfig=kubelet-bootstrap-{node_name}.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=kubelet-bootstrap-{node_name}.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=kubelet-bootstrap-{node_name}.kubeconfig
done
- 向 kubeconfig 写入的是 token，bootstrap 结束后 kube-controller-manager 为 kubelet 创建 client 和 server 证书；
查看 kubeadm 为各节点创建的 token：
[root@kubernetes-01 work]# kubeadm token list --kubeconfig ~/.kube/config
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
5bgx89.2yhcvzr6eh443q85 23h 2021-04-01T14:55:30+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:kubernetes-02
qkrmag.qhzu0kfzevmyaz44 23h 2021-04-01T14:55:29+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:kubernetes-01
sgjlb2.vm7s1pv7mdzho6ze 23h 2021-04-01T14:55:30+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:kubernetes-03
[root@kubernetes-01 work]#
- token 有效期为 1 天，超期后将不能再被用来 boostrap kubelet，且会被 kube-controller-manager 的 tokencleaner 清理；
- kube-apiserver 接收 kubelet 的 bootstrap token 后，将请求的 user 设置为 system:bootstrap:
，group 设置为 system:bootstrappers，后续将为这个 group 设置 ClusterRoleBinding；
分发 bootstrap kubeconfig 文件到所有 worker 节点
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in {NODE_NAMES[@]}
do
echo ">>>{node_name}"
scp kubelet-bootstrap-{node_name}.kubeconfig root@{node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_name in {NODE_NAMES[@]}
> do
> echo ">>>{node_name}"
> scp kubelet-bootstrap-{node_name}.kubeconfig root@{node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
> done
>>> kubernetes-01
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
kubelet-bootstrap-kubernetes-01.kubeconfig 100% 2106 2.1MB/s 00:00
>>> kubernetes-02
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
kubelet-bootstrap-kubernetes-02.kubeconfig 100% 2106 1.6MB/s 00:00
>>> kubernetes-03
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
kubelet-bootstrap-kubernetes-03.kubeconfig 100% 2106 865.2KB/s 00:00
[root@kubernetes-01 work]#
创建和分发 kubelet 参数配置文件
从 v1.10 开始，部分 kubelet 参数需在配置文件中配置，kubelet –help 会提示：
DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag
创建 kubelet 参数配置文件模板（可配置项参考代码中注释）：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubelet-config.yaml.template <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "##NODE_IP##"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
authentication:
anonymous:
enabled: false
webhook:
enabled: true
x509:
clientCAFile: "/etc/kubernetes/cert/ca.pem"
authorization:
mode: Webhook
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "##NODE_IP##"
clusterDomain: "{CLUSTER_DNS_DOMAIN}"
clusterDNS:
- "{CLUSTER_DNS_SVC_IP}"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "${CLUSTER_CIDR}"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF
- address：kubelet 安全端口（https，10250）监听的地址，不能为 127.0.0.1，否则 kube-apiserver、heapster 等不能调用 kubelet 的 API；
- readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定；
- authentication.anonymous.enabled：设置为 false，不允许匿名访问 10250 端口；
- authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证书认证；
- authentication.webhook.enabled=true：开启 HTTPs bearer token 认证； 对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将被拒绝，提示 Unauthorized；
- authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)；
- featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自动 rotate 证书，证书的有效期取决于 kube-controller-manager 的 –experimental-cluster-signing-duration 参数；
- 需要 root 账户运行；
为各节点创建和分发 kubelet 配置文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do echo ">>>{node_ip}"
sed -e "s/##NODE_IP##/{node_ip}/" kubelet-config.yaml.template>kubelet-config-{node_ip}.yaml.template
scp kubelet-config-{node_ip}.yaml.template root@{node_ip}:/etc/kubernetes/kubelet-config.yaml
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> sed -e "s/##NODE_IP##/{node_ip}/" kubelet-config.yaml.template>kubelet-config-{node_ip}.yaml.template
> scp kubelet-config-{node_ip}.yaml.template root@{node_ip}:/etc/kubernetes/kubelet-config.yaml
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
kubelet-config-172.20.0.20.yaml.template 100% 1534 2.2MB/s 00:00
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
kubelet-config-172.20.0.21.yaml.template 100% 1534 1.9MB/s 00:00
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
kubelet-config-172.20.0.22.yaml.template 100% 1534 1.3MB/s 00:00
[root@kubernetes-01 work]#
创建 kubelet systemd unit 文件模板：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubelet.service.template <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
# 这里的服务变更为docker了、原教程里面是containerd
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory={K8S_DIR}/kubelet
ExecStart=/opt/k8s/bin/kubelet \\
--fail-swap-on=false \\
--cgroup-driver=cgroupfs \\
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
--cert-dir=/etc/kubernetes/cert \\
# 配置calico服务、必须添加cni网络插件
--network-plugin=cni \\
--root-dir={K8S_DIR}/kubelet \\
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
--config=/etc/kubernetes/kubelet-config.yaml \\
--hostname-override=##NODE_NAME## \\
--image-pull-progress-deadline=15m \\
--volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\
--logtostderr=true \\
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF
- 如果设置了 –hostname-override 选项，则 kube-proxy 也需要设置该选项，否则会出现找不到 Node 的情况；
- –bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求；
- K8S approve kubelet 的 csr 请求后，在 –cert-dir 目录创建证书和私钥文件，然后写入 –kubeconfig 文件；
- –pod-infra-container-image 不使用 redhat 的 pod-infrastructure:latest 镜像，它不能回收容器的僵尸；
为各节点创建和分发 kubelet systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in {NODE_NAMES[@]}
do echo ">>>{node_name}"
sed -e "s/##NODE_NAME##/{node_name}/" kubelet.service.template>kubelet-{node_name}.service
scp kubelet-{node_name}.service root@{node_name}:/etc/systemd/system/kubelet.service
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_name in {NODE_NAMES[@]}
> do
> echo ">>>{node_name}"
> sed -e "s/##NODE_NAME##/{node_name}/" kubelet.service.template>kubelet-{node_name}.service
> scp kubelet-{node_name}.service root@{node_name}:/etc/systemd/system/kubelet.service
> done
>>> kubernetes-01
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
kubelet-kubernetes-01.service 100% 825 987.6KB/s 00:00
>>> kubernetes-02
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
kubelet-kubernetes-02.service 100% 825 747.1KB/s 00:00
>>> kubernetes-03
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
kubelet-kubernetes-03.service 100% 825 669.0KB/s 00:00
[root@kubernetes-01 work]#
授予 kube-apiserver 访问 kubelet API 的权限
在执行 kubectl exec、run、logs 等命令时，apiserver 会将请求转发到 kubelet 的 https 端口。这里定义 RBAC 规则，授权 apiserver 使用的证书（kubernetes.pem）用户名（CN：kuberntes-master）访问 kubelet API 的权限：
[root@kubernetes-01 work]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes-master
clusterrolebinding.rbac.authorization.k8s.io/kube-apiserver:kubelet-apis created
[root@kubernetes-01 work]#
Bootstrap Token Auth 和授予权限
kubelet 启动时查找 –kubeletconfig 参数对应的文件是否存在，如果不存在则使用 –bootstrap-kubeconfig 指定的 kubeconfig 文件向 kube-apiserver 发送证书签名请求 (CSR)。
kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证，认证通过后将请求的 user 设置为 system:bootstrap:
原文这里说：默认情况下，这个 user 和 group 没有创建 CSR 的权限，kubelet 启动失败，错误日志如下：
sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests'
我这里并没有遇到这个问题、但还是建议把下面的命令执行以下：解决办法是：创建一个 clusterrolebinding，将 group system:bootstrappers 和 clusterrole system:node-bootstrapper 绑定：
[root@kubernetes-01 work]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
[root@kubernetes-01 work]#
自动 approve CSR 请求，生成 kubelet client 证书
kubelet 创建 CSR 请求后，下一步需要创建被 approve，有两种方式：
- kube-controller-manager 自动 aprrove；
- 手动使用命令 kubectl certificate approve；
CSR 被 approve 后，kubelet 向 kube-controller-manager 请求创建 client 证书，kube-controller-manager 中的 csrapproving controller 使用 SubjectAccessReview API 来检查 kubelet 请求（对应的 group 是 system:bootstrappers）是否具有相应的权限。
创建三个 ClusterRoleBinding，分别授予 group system:bootstrappers 和 group system:nodes 进行 approve client、renew client、renew server 证书的权限（server csr 是手动 approve 的，见后）：
cd /opt/k8s/work
cat > csr-crb.yaml <<EOF
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-client-cert-renewal
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-server-cert-renewal
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: approve-node-server-renewal-csr
apiGroup: rbac.authorization.k8s.io
EOF
[root@kubernetes-01 work]# kubectl apply -f csr-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/auto-approve-csrs-for-group created
clusterrolebinding.rbac.authorization.k8s.io/node-client-cert-renewal created
clusterrole.rbac.authorization.k8s.io/approve-node-server-renewal-csr created
clusterrolebinding.rbac.authorization.k8s.io/node-server-cert-renewal created
[root@kubernetes-01 work]#
- auto-approve-csrs-for-group：自动 approve node 的第一次 CSR； 注意第一次 CSR 时，请求的 Group 为 system:bootstrappers；
- node-client-cert-renewal：自动 approve node 后续过期的 client 证书，自动生成的证书 Group 为 system:nodes;
- node-server-cert-renewal：自动 approve node 后续过期的 server 证书，自动生成的证书 Group 为 system:nodes;
启动 kubelet 服务
source /opt/k8s/bin/environment.sh
for node_name in {NODE_NAMES[@]}
do
echo ">>>{node_name}"
ssh root@{node_name} "mkdir -p{K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"
ssh root@{node_name} "/usr/sbin/swapoff -a"
ssh root@{node_name} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_name in {NODE_NAMES[@]}
> do
> echo ">>>{node_name}"
> ssh root@{node_name} "mkdir -p{K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"
> ssh root@{node_name} "/usr/sbin/swapoff -a"
> ssh root@{node_name} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
> done
>>> kubernetes-01
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
>>> kubernetes-02
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
>>> kubernetes-03
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
[root@kubernetes-01 work]#
- 启动服务前必须先创建工作目录；
- 关闭 swap 分区，否则 kubelet 会启动失败；
kubelet 启动后使用 –bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个 CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 –kubeletconfig 文件。
注：kube-controller-manager 需要配置 –cluster-signing-cert-file 和 –cluster-signing-key-file 参数，才会为 TLS Bootstrap 创建证书和私钥。
查看 kubelet 情况
稍等一会，三个节点的 CSR 都被自动 approved：
[root@kubernetes-01 work]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-29nmm 25s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:sgjlb2 Approved,Issued
csr-2xgp7 25s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:5bgx89 Approved,Issued
csr-99shf 18s kubernetes.io/kubelet-serving system:node:kubernetes-02 Pending
csr-fxvmw 17s kubernetes.io/kubelet-serving system:node:kubernetes-03 Pending
csr-ntjdd 25s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:qkrmag Approved,Issued
csr-v42ns 12s kubernetes.io/kubelet-serving system:node:kubernetes-01 Pending
[root@kubernetes-01 work]#
- Pending 的 CSR 用于创建 kubelet server 证书，需要手动 approve，参考后文。
所有节点均注册（NotReady 状态是预期的，后续安装了网络插件后就好）：
[root@kubernetes-01 work]# kubectl get node
NAME STATUS ROLES AGE VERSION
kubernetes-01 NotReady <none> 12s v1.19.8
kubernetes-02 NotReady <none> 17s v1.19.8
kubernetes-03 NotReady <none> 17s v1.19.8
[root@kubernetes-01 work]#
kube-controller-manager 为各 node 生成了 kubeconfig 文件和公私钥：
[root@kubernetes-01 work]# ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2246 Mar 31 14:59 /etc/kubernetes/kubelet.kubeconfig
[root@kubernetes-01 work]#
- 没有自动生成 kubelet server 证书；
手动 approve server cert csr
基于安全性考虑，CSR approving controllers 不会自动 approve kubelet server 证书签名请求，需要手动 approve：
kubectl get csr
# 手动 approve
kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
# 自动生成了 server 证书
[root@kubernetes-01 work]# ls -l /etc/kubernetes/cert/kubelet-*
-rw------- 1 root root 1232 Mar 31 14:59 /etc/kubernetes/cert/kubelet-client-2021-03-31-14-59-43.pem
lrwxrwxrwx 1 root root 59 Mar 31 14:59 /etc/kubernetes/cert/kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2021-03-31-14-59-43.pem
-rw------- 1 root root 1277 Mar 31 15:00 /etc/kubernetes/cert/kubelet-server-2021-03-31-15-00-41.pem
lrwxrwxrwx 1 root root 59 Mar 31 15:00 /etc/kubernetes/cert/kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-2021-03-31-15-00-41.pem
[root@kubernetes-01 work]#
kubelet api 认证和授权
kubelet 配置了如下认证参数：
- authentication.anonymous.enabled：设置为 false，不允许匿名访问 10250 端口；
- authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTPs 证书认证；
- authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；
同时配置了如下授权参数：
- authroization.mode=Webhook：开启 RBAC 授权；
kubelet 收到请求后，使用 clientCAFile 对证书签名进行认证，或者查询 bearer token 是否有效。如果两者都没通过，则拒绝请求，提示 Unauthorized：
curl -s --cacert /etc/kubernetes/cert/ca.pem https://172.20.0.20:10250/metrics
curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer 123456" https://172.20.0.20:10250/metrics
通过认证后，kubelet 使用 SubjectAccessReview API 向 kube-apiserver 发送请求，查询证书或 token 对应的 user、group 是否有操作资源的权限(RBAC)；
证书认证和授权
# 权限不足的证书；
[root@kubernetes-01 work]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://172.20.0.20:10250/metrics
Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)
# 使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书；
[root@kubernetes-01 work]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.20.0.20:10250/metrics|head
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
[root@kubernetes-01 work]#
- –cacert、–cert、–key 的参数值必须是文件路径，如上面的 ./admin.pem 不能省略 ./，否则返回 401 Unauthorized；
bear token 认证和授权
创建一个 ServiceAccount，将它和 ClusterRole system:kubelet-api-admin 绑定，从而具有调用 kubelet API 的权限：
kubectl create sa kubelet-api-test
kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://172.20.0.20:10250/metrics | head
cadvisor 和 metrics
cadvisor 是内嵌在 kubelet 二进制中的，统计所在节点各容器的资源(CPU、内存、磁盘、网卡)使用情况的服务。
浏览器访问 https://172.20.0.20:10250/metrics 和 https://172.20.0.20:10250/metrics/cadvisor 分别返回 kubelet 和 cadvisor 的 metrics。
注意：
- kubelet.config.json 设置 authentication.anonymous.enabled 为 false，不允许匿名证书访问 10250 的 https 服务；
- 参考浏览器访问kube-apiserver安全端口，创建和导入相关证书，然后访问上面的 10250 端口；
参考
- kubelet 认证和授权：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/
7.4、Kube-Proxy
kube-proxy 运行在所有 worker 节点上，它监听 apiserver 中 service 和 endpoint 的变化情况，创建路由规则以提供服务 IP 和负载均衡功能。本小节讲解部署 ipvs 模式的 kube-proxy 过程。
注意：如果没有特殊指明，本小节的所有操作均在 Kubernetes-01 节点上执行，然后远程分发文件和执行命令。
创建 kube-proxy 证书
创建证书签名请求：
cd /opt/k8s/work
cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "z0ukun"
}
]
}
EOF
- CN：指定该证书的 User 为 system:kube-proxy；
- 预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限； 该证书只会被 kube-proxy 当做 client 证书使用，所以 hosts 字段为空；
生成证书和私钥：
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ls kube-proxy*
[root@kubernetes-01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
> -ca-key=/opt/k8s/work/ca-key.pem \
> -config=/opt/k8s/work/ca-config.json \
> -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2021/03/31 15:04:33 [INFO] generate received request
2021/03/31 15:04:33 [INFO] received CSR
2021/03/31 15:04:33 [INFO] generating key: rsa-2048
2021/03/31 15:04:33 [INFO] encoded CSR
2021/03/31 15:04:33 [INFO] signed certificate with serial number 616010196370462377273421189169206860799678306037
2021/03/31 15:04:33 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@kubernetes-01 work]# ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
[root@kubernetes-01 work]#
创建和分发 kubeconfig 文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=kube-proxy.pem \
--client-key=kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
分发 kubeconfig 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in {NODE_NAMES[@]}
do
echo ">>>{node_name}"
scp kube-proxy.kubeconfig root@{node_name}:/etc/kubernetes/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_name in{NODE_NAMES[@]}
> do
> echo ">>> {node_name}"
> scp kube-proxy.kubeconfig root@{node_name}:/etc/kubernetes/
> done
>>> kubernetes-01
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
kube-proxy.kubeconfig 100% 6225 7.3MB/s 00:00
>>> kubernetes-02
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
kube-proxy.kubeconfig 100% 6225 4.7MB/s 00:00
>>> kubernetes-03
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
kube-proxy.kubeconfig 100% 6225 4.8MB/s 00:00
[root@kubernetes-01 work]#
创建 kube-proxy 配置文件
从 v1.10 开始，kube-proxy 部分参数可以配置文件中配置。可以使用 –write-config-to 选项生成该配置文件，或者参考 源代码的注释。
创建 kube-proxy config 文件模板：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-proxy-config.yaml.template <<EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
burst: 200
kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
qps: 100
bindAddress: ##NODE_IP##
healthzBindAddress: ##NODE_IP##:10256
metricsBindAddress: ##NODE_IP##:10249
enableProfiling: true
clusterCIDR: ${CLUSTER_CIDR}
hostnameOverride: ##NODE_NAME##
mode: "ipvs"
portRange: ""
iptables:
masqueradeAll: false
ipvs:
scheduler: rr
excludeCIDRs: []
EOF
- bindAddress: 监听地址； clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
- clusterCIDR: kube-proxy 根据 –cluster-cidr 判断集群内部和外部流量，指定 –cluster-cidr 或 –masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
- hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
- mode: 使用 ipvs 模式；
为各节点创建和分发 kube-proxy 配置文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
echo ">>> {NODE_NAMES[i]}"
sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-proxy-config.yaml.template>kube-proxy-config-{NODE_NAMES[i]}.yaml.template
scp kube-proxy-config-{NODE_NAMES[i]}.yaml.template root@{NODE_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yaml
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for (( i=0; i < 3; i++ ))
> do
> echo ">>> {NODE_NAMES[i]}"
> sed -e "s/##NODE_NAME##/{NODE_NAMES[i]}/" -e "s/##NODE_IP##/{NODE_IPS[i]}/" kube-proxy-config.yaml.template>kube-proxy-config-{NODE_NAMES[i]}.yaml.template
> scp kube-proxy-config-{NODE_NAMES[i]}.yaml.template root@{NODE_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yaml
> done
>>> kubernetes-01
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
kube-proxy-config-kubernetes-01.yaml.template 100% 453 561.0KB/s 00:00
>>> kubernetes-02
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
kube-proxy-config-kubernetes-02.yaml.template 100% 453 420.7KB/s 00:00
>>> kubernetes-03
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
kube-proxy-config-kubernetes-03.yaml.template 100% 453 494.7KB/s 00:00
[root@kubernetes-01 work]#
创建和分发 kube-proxy systemd unit 文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy \\
--config=/etc/kubernetes/kube-proxy-config.yaml \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
分发 kube-proxy systemd unit 文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in {NODE_NAMES[@]}
do echo ">>>{node_name}"
scp kube-proxy.service root@{node_name}:/etc/systemd/system/
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_name in{NODE_NAMES[@]}
> do
> echo ">>> {node_name}"
> scp kube-proxy.service root@{node_name}:/etc/systemd/system/
> done
>>> kubernetes-01
Warning: Permanently added 'kubernetes-01,172.20.0.20' (ECDSA) to the list of known hosts.
kube-proxy.service 100% 393 368.6KB/s 00:00
>>> kubernetes-02
Warning: Permanently added 'kubernetes-02,172.20.0.21' (ECDSA) to the list of known hosts.
kube-proxy.service 100% 393 426.3KB/s 00:00
>>> kubernetes-03
Warning: Permanently added 'kubernetes-03,172.20.0.22' (ECDSA) to the list of known hosts.
kube-proxy.service 100% 393 322.4KB/s 00:00
[root@kubernetes-01 work]#
启动 kube-proxy 服务
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "mkdir -p{K8S_DIR}/kube-proxy"
ssh root@{node_ip} "modprobe ip_vs_rr"
ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh root@{node_ip} "mkdir -p{K8S_DIR}/kube-proxy"
> ssh root@{node_ip} "modprobe ip_vs_rr"
> ssh root@{node_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /etc/systemd/system/kube-proxy.service.
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /etc/systemd/system/kube-proxy.service.
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /etc/systemd/system/kube-proxy.service.
[root@kubernetes-01 work]#
检查启动结果
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "systemctl status kube-proxy|grep Active"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "systemctl status kube-proxy|grep Active"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 15:06:00 CST; 13s ago
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 15:06:01 CST; 12s ago
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
Active: active (running) since Wed 2021-03-31 15:06:01 CST; 12s ago
[root@kubernetes-01 work]#
确保状态为 active (running)，否则可以通过 journalctl -u kube-proxy 查看日志，确认原因。
查看监听端口
[root@kubernetes-01 work]# sudo netstat -lnpt|grep kube-prox
tcp 0 0 172.20.0.20:10256 0.0.0.0:* LISTEN 9366/kube-proxy
tcp 0 0 172.20.0.20:10249 0.0.0.0:* LISTEN 9366/kube-proxy
[root@kubernetes-01 work]#
- 10249：http prometheus metrics port;
- 10256：http healthz port;
查看 ipvs 路由规则
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh root@{node_ip} "/usr/sbin/ipvsadm -ln"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh root@{node_ip} "/usr/sbin/ipvsadm -ln"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr
-> 172.20.0.20:6443 Masq 1 0 0
-> 172.20.0.21:6443 Masq 1 0 0
-> 172.20.0.22:6443 Masq 1 0 0
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr
-> 172.20.0.20:6443 Masq 1 0 0
-> 172.20.0.21:6443 Masq 1 0 0
-> 172.20.0.22:6443 Masq 1 0 0
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr
-> 172.20.0.20:6443 Masq 1 0 0
-> 172.20.0.21:6443 Masq 1 0 0
-> 172.20.0.22:6443 Masq 1 0 0
[root@kubernetes-01 work]#
可见所有通过 https 访问 K8S SVC kubernetes 的请求都转发到 kube-apiserver 节点的 6443 端口。
8、Calico网络配置
这里我们采用 Yaml 资源文件的方式部署Calico ；我们先去后面的链接 https://docs.projectcalico.org/manifests/calico.yaml 下载我们需要的Calico资源文件。然后我们需要对Yaml文件进行修改：
[root@kubernetes-01 work]# curl https://docs.projectcalico.org/manifests/calico.yaml -O
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 184k 100 184k 0 0 132k 0 0:00:01 0:00:01 --:--:-- 133k
[root@kubernetes-01 work]# cp calico.yaml calico.yaml.orig
[root@kubernetes-01 work]# vi calico.yaml
[root@kubernetes-01 work]# diff calico.yaml.orig calico.yaml
3673a3674,3677
> - name: CALICO_IPV4POOL_CIDR
# 定义calico cidr 网络
> value: "172.30.0.0/16"
> - name: IP_AUTODETECTION_METHOD
# 定义本地网络接口
> value: "interface=eth.*"
[root@kubernetes-01 work]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
[root@kubernetes-01 work]# mkdir -p /opt/cni/bin && cp /opt/k8s/bin/* /opt/cni/bin/
- 将 Pod 网段地址修改为 172.30.0.0/16；
- calico 自动探查互联网卡，如果有多快网卡，则可以配置用于互联的网络接口命名正则表达式，如上面的 ens160 (根据自己服务器的网络接口名修改)；
然后我们直接创建Calico资源，Calico会以 Daemonset 方式运行在所有的 K8S 节点上。
[root@kubernetes-01 work]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-69496d8b75-2mvdw 1/1 Running 0 21m 172.30.31.65 kubernetes-01 <none> <none>
calico-node-mbtf8 1/1 Running 0 21m 172.20.0.20 kubernetes-01 <none> <none>
calico-node-r442b 1/1 Running 0 21m 172.20.0.22 kubernetes-03 <none> <none>
calico-node-zv6ms 1/1 Running 0 21m 172.20.0.21 kubernetes-02 <none> <none>
[root@kubernetes-01 work]#
[root@kubernetes-01 work]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 52:54:dc:33:23:15 brd ff:ff:ff:ff:ff:ff
inet 172.20.0.20/24 brd 172.20.0.255 scope global noprefixroute dynamic eth0
valid_lft 85351sec preferred_lft 85351sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:60:98:a9:d4 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 16:3a:4e:b9:26:1c brd ff:ff:ff:ff:ff:ff
5: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
link/ether 2a:23:9d:48:03:e5 brd ff:ff:ff:ff:ff:ff
inet 10.254.0.1/32 scope global kube-ipvs0
valid_lft forever preferred_lft forever
6: cali4daa57f953d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 172.30.31.64/32 scope global tunl0
valid_lft forever preferred_lft forever
[root@kubernetes-01 work]#
注：之前参考的 https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/06-6.calico.md 文档中有一个bug、下面这个目录不需要修改；因为 Kubelet 会定时汇报当前节点的状态给APIServer，以供调度的时候使用。而我们修改了目录之后 Calico 组件默认还是会部署到 /opt/cni/bin 目录下面会导致Calico启动失败、这样Kubelet就无法获取节点状态。
错误提示如下：Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) had taints that the pod didn’t tolerate
如果你修改了这个目录之后可以，你可以手动mkdir新建 /opt/cni/bin 目录，然后把 /opt/k8s/bin 的文件拷贝到 /opt/cni/bin 目录下面。稍等片刻、Calico就可以自动启动了。
9、集群验证
本小节验证 K8S 集群是否工作正常。
注：如果没有特殊指明，本文档的所有操作均在 Kubernetes-01 节点上执行，然后远程分发文件和执行命令。
检查节点状态
[root@kubernetes-01 work]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetes-01 Ready <none> 32m v1.19.8
kubernetes-02 Ready <none> 32m v1.19.8
kubernetes-03 Ready <none> 32m v1.19.8
[root@kubernetes-01 work]#
都为 Ready 且版本为 v1.19.8 时正常。
创建测试文件
cd /opt/k8s/work
cat > nginx-ds.yml <<EOF
apiVersion: v1
kind: Service
metadata:
name: nginx-ds
labels:
app: nginx-ds
spec:
type: NodePort
selector:
app: nginx-ds
ports:
- name: http
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ds
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
app: nginx-ds
template:
metadata:
labels:
app: nginx-ds
spec:
containers:
- name: my-nginx
image: nginx:1.7.9
ports:
- containerPort: 80
EOF
执行测试
[root@kubernetes-01 work]# kubectl create -f nginx-ds.yml
service/nginx-ds created
daemonset.apps/nginx-ds created
[root@kubernetes-01 work]#
检查各节点的 Pod IP 连通性
[root@kubernetes-01 work]# kubectl get pods -o wide -l app=nginx-ds
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ds-hhkgm 1/1 Running 0 4m27s 172.30.31.66 kubernetes-01 <none> <none>
nginx-ds-knnwf 1/1 Running 0 4m27s 172.30.218.129 kubernetes-02 <none> <none>
nginx-ds-v7nqb 1/1 Running 0 4m27s 172.30.11.1 kubernetes-03 <none> <none>
[root@kubernetes-01 work]#
在所有 Node 上分别 ping 上面三个 Pod IP，看是否连通：
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh {node_ip} "ping -c 1 172.30.31.66"
ssh{node_ip} "ping -c 1 172.30.218.129"
ssh {node_ip} "ping -c 1 172.30.11.1"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh{node_ip} "ping -c 1 172.30.31.66"
> ssh {node_ip} "ping -c 1 172.30.218.129"
> ssh{node_ip} "ping -c 1 172.30.11.1"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
PING 172.30.31.66 (172.30.31.66) 56(84) bytes of data.
64 bytes from 172.30.31.66: icmp_seq=1 ttl=64 time=0.124 ms
--- 172.30.31.66 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.124/0.124/0.124/0.000 ms
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
PING 172.30.218.129 (172.30.218.129) 56(84) bytes of data.
64 bytes from 172.30.218.129: icmp_seq=1 ttl=63 time=0.544 ms
--- 172.30.218.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.544/0.544/0.544/0.000 ms
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
PING 172.30.11.1 (172.30.11.1) 56(84) bytes of data.
64 bytes from 172.30.11.1: icmp_seq=1 ttl=63 time=0.520 ms
--- 172.30.11.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.520/0.520/0.520/0.000 ms
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
PING 172.30.31.66 (172.30.31.66) 56(84) bytes of data.
64 bytes from 172.30.31.66: icmp_seq=1 ttl=63 time=0.338 ms
--- 172.30.31.66 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.338/0.338/0.338/0.000 ms
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
PING 172.30.218.129 (172.30.218.129) 56(84) bytes of data.
64 bytes from 172.30.218.129: icmp_seq=1 ttl=64 time=0.071 ms
--- 172.30.218.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.071/0.071/0.071/0.000 ms
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
PING 172.30.11.1 (172.30.11.1) 56(84) bytes of data.
64 bytes from 172.30.11.1: icmp_seq=1 ttl=63 time=0.451 ms
--- 172.30.11.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.451/0.451/0.451/0.000 ms
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
PING 172.30.31.66 (172.30.31.66) 56(84) bytes of data.
64 bytes from 172.30.31.66: icmp_seq=1 ttl=63 time=0.500 ms
--- 172.30.31.66 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.500/0.500/0.500/0.000 ms
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
PING 172.30.218.129 (172.30.218.129) 56(84) bytes of data.
64 bytes from 172.30.218.129: icmp_seq=1 ttl=63 time=0.415 ms
--- 172.30.218.129 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.415/0.415/0.415/0.000 ms
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
PING 172.30.11.1 (172.30.11.1) 56(84) bytes of data.
64 bytes from 172.30.11.1: icmp_seq=1 ttl=64 time=0.060 ms
--- 172.30.11.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms
[root@kubernetes-01 work]#
检查服务 IP 和端口可达性
[root@kubernetes-01 work]# kubectl get svc -l app=nginx-ds
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ds NodePort 10.254.35.136 <none> 80:32311/TCP 5m31s
[root@kubernetes-01 work]#
可见：
- Service Cluster IP：10.254.35.136
- 服务端口：80
- NodePort 端口：32311
在所有 Node 上 curl Service IP：
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh {node_ip} "curl -s 10.254.35.136"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in{NODE_IPS[@]}
> do
> echo ">>> {node_ip}"
> ssh{node_ip} "curl -s 10.254.35.136"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@kubernetes-01 work]#
预期输出 nginx 欢迎页面内容。
检查服务的 NodePort 可达性
在所有 Node 上执行：
source /opt/k8s/bin/environment.sh
for node_ip in {NODE_IPS[@]}
do
echo ">>>{node_ip}"
ssh {node_ip} "curl -s{node_ip}:32311"
done
[root@kubernetes-01 work]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 work]# for node_ip in {NODE_IPS[@]}
> do
> echo ">>>{node_ip}"
> ssh {node_ip} "curl -s{node_ip}:32311"
> done
>>> 172.20.0.20
Warning: Permanently added '172.20.0.20' (ECDSA) to the list of known hosts.
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
>>> 172.20.0.21
Warning: Permanently added '172.20.0.21' (ECDSA) to the list of known hosts.
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
>>> 172.20.0.22
Warning: Permanently added '172.20.0.22' (ECDSA) to the list of known hosts.
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@kubernetes-01 work]#
预期输出 nginx 欢迎页面内容。
10、集群插件部署-CoreDNS
注：如果没有特殊指明，本小节所有操作均在 Kubernetes-01 节点上执行。
下载CoreDNS
CoreDNS下载地址：https://github.com/coredns/deployment
CoreDNS官方网站：https://coredns.io/
cd /opt/k8s/work
git clone https://github.com/coredns/deployment.git
mv deployment coredns-deployment
[root@kubernetes-01 work]# git clone https://github.com/coredns/deployment.git
Cloning into 'deployment'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 870 (delta 2), reused 1 (delta 0), pack-reused 859
Receiving objects: 100% (870/870), 244.72 KiB | 101.00 KiB/s, done.
Resolving deltas: 100% (470/470), done.
[root@kubernetes-01 work]# mv deployment coredns-deployment
创建CoreDNS
cd /opt/k8s/work/coredns-deployment/kubernetes
source /opt/k8s/bin/environment.sh
./deploy.sh -i {CLUSTER_DNS_SVC_IP} -d{CLUSTER_DNS_DOMAIN} | kubectl apply -f -
[root@kubernetes-01 work]# cd /opt/k8s/work/coredns-deployment/kubernetes
[root@kubernetes-01 kubernetes]# source /opt/k8s/bin/environment.sh
[root@kubernetes-01 kubernetes]# ./deploy.sh -i {CLUSTER_DNS_SVC_IP} -d{CLUSTER_DNS_DOMAIN} | kubectl apply -f -
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
[root@kubernetes-01 kubernetes]#
检查CoreDNS功能
[root@kubernetes-01 kubernetes]# kubectl get all -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
pod/coredns-867bfd96bd-fscvd 0/1 ContainerCreating 0 50s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP,9153/TCP 50s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 0/1 1 0 50s
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-867bfd96bd 1 1 0 50s
[root@kubernetes-01 kubernetes]#
新建一个 Deployment：
cd /opt/k8s/work
cat > my-nginx.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
selector:
matchLabels:
run: my-nginx
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx:1.7.9
ports:
- containerPort: 80
EOF
[root@kubernetes-01 kubernetes]# kubectl create -f my-nginx.yaml
deployment.apps/my-nginx created
[root@kubernetes-01 kubernetes]#
export 该 Deployment, 生成 my-nginx 服务：
[root@kubernetes-01 kubernetes]# kubectl expose deploy my-nginx
service/my-nginx exposed
[root@kubernetes-01 kubernetes]# kubectl get services my-nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
my-nginx ClusterIP 10.254.149.187 <none> 80/TCP 30s run=my-nginx
[root@kubernetes-01 kubernetes]#
创建另一个 Pod，查看 /etc/resolv.conf 是否包含 kubelet 配置的 –cluster-dns 和 –cluster-domain，是否能够将服务 my-nginx 解析到上面显示的 Cluster IP 10.254.149.187
cd /opt/k8s/work
cat > dnsutils-ds.yml <<EOF
apiVersion: v1
kind: Service
metadata:
name: dnsutils-ds
labels:
app: dnsutils-ds
spec:
type: NodePort
selector:
app: dnsutils-ds
ports:
- name: http
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dnsutils-ds
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
app: dnsutils-ds
template:
metadata:
labels:
app: dnsutils-ds
spec:
containers:
- name: my-dnsutils
image: tutum/dnsutils:latest
command:
- sleep
- "3600"
ports:
- containerPort: 80
EOF
[root@kubernetes-01 kubernetes]# kubectl create -f dnsutils-ds.yml
service/dnsutils-ds created
daemonset.apps/dnsutils-ds created
[root@kubernetes-01 kubernetes]# kubectl get pods -lapp=dnsutils-ds -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dnsutils-ds-994bl 1/1 Running 0 3m45s 172.30.218.131 kubernetes-02 <none> <none>
dnsutils-ds-drjdb 1/1 Running 0 3m45s 172.30.11.4 kubernetes-03 <none> <none>
dnsutils-ds-hw7ff 1/1 Running 0 3m45s 172.30.31.67 kubernetes-01 <none> <none>
[root@kubernetes-01 kubernetes]# kubectl -it exec dnsutils-ds-994bl cat /etc/resolv.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
nameserver 10.254.0.2
search default.svc.cluster.local svc.cluster.local cluster.local pekdemo.pekdemo.com
options ndots:5
[root@kubernetes-01 kubernetes]# kubectl -it exec dnsutils-ds-994bl cat /etc/resolv.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
nameserver 10.254.0.2
search default.svc.cluster.local svc.cluster.local cluster.local pekdemo.pekdemo.com
options ndots:5
[root@kubernetes-01 kubernetes]# kubectl -it exec dnsutils-ds-994bl nslookup www.baidu.com
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Server: 10.254.0.2
Address: 10.254.0.2#53
Non-authoritative answer:
www.baidu.com canonical name = www.a.shifen.com.
Name: www.a.shifen.com
Address: 180.101.49.11
Name: www.a.shifen.com
Address: 180.101.49.12
[root@kubernetes-01 kubernetes]#
[root@kubernetes-01 kubernetes]# kubectl -it exec dnsutils-ds-drjdb nslookup www.baidu.com
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Server: 10.254.0.2
Address: 10.254.0.2#53
www.baidu.com canonical name = www.a.shifen.com.
Name: www.a.shifen.com
Address: 180.101.49.12
Name: www.a.shifen.com
Address: 180.101.49.11
[root@kubernetes-01 kubernetes]#
[root@kubernetes-01 kubernetes]# kubectl -it exec dnsutils-ds-hw7ff nslookup www.baidu.com
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Server: 10.254.0.2
Address: 10.254.0.2#53
Non-authoritative answer:
www.baidu.com canonical name = www.a.shifen.com.
Name: www.a.shifen.com
Address: 180.101.49.11
Name: www.a.shifen.com
Address: 180.101.49.12
[root@kubernetes-01 kubernetes]#
