使用二进制包部署kubernetes V1.20集群

概述

Kubernetes，又称为 k8s（首字母为 k、首字母与尾字母之间有 8 个字符、尾字母为 s，所以简称 k8s）或者简称为 “kube” ，是一种可自动实施 Linux 容器操作的开源平台。它可以帮助用户省去应用容器化过程的许多手动部署和扩展操作。也就是说，您可以将运行 Linux 容器的多组主机聚集在一起，由 Kubernetes 帮助您轻松高效地管理这些集群。而且，这些集群可跨公共云、私有云或混合云部署主机。

一、环境准备

1.1 部署k8s方式

(1) 方式一：kubeadm部署
Kubeadm是一个K8s部署工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。
(2) 方式二：二进制包
从Github下载kubernetes发行二进制包，手动部署每个组件，组成kubernetes集群。

注意：服务器最低要求2核CPU与2G内存，磁盘空间20G以上。

1.2 软件版本

软件	版本
Linux	Centos 7.9_x64
Docker	Docker CE 19.03
Kubernetes	Kubernetes v1.20

1.3 服务器整体规划

角色	IP	组件
k8s-master-01	192.168.20.21	kube-apiserver, kube-controller-manager, kube-scheduler, etcd
k8s-master-02	192.168.20.22	kube-apiserver, kube-controller-manager, kube-scheduler, etcd
k8s-master-03	192.168.20.23	Nginx, keepalived
k8s-worker-01	192.168.20.24	kubelet, kube-proxy, docker, etcd
k8s-worker-02	192.168.20.25	kubelet, kube-proxy, docker, etcd
k8s-worker-03	192.168.20.26	Nginx, keepalived
Vip	192.168.20.27	/

这套集群我分三部分实施

一是先部署一套单Master架构(3台)

二是扩容为双或多Master架构(4台或者6台)

三是配置前端Web负载均衡

单Master架构图

单Master集群规划

角色	IP	组件
k8s-master-01	192.168.20.21	kube-apiserver kube-controller-manager kube-scheduler etcd
k8s-worker-01	192.168.20.24	kubelet kube-proxy docker etcd
k8s-worker-02	192.168.20.25	kubelet kube-proxy docker etcd

1.4 操作系统初始化

systemctl stop firewalld \
&& systemctl disable firewalld \
&& sed -i 's/enforcing/disabled/' /etc/selinux/config \
&& sed -ri 's/.*swap.*/#&/' /etc/fstab \
&& echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/k8s.conf \
&& echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/k8s.conf \
&& sysctl --system \
&& yum install ntpdate -y \
&& timedatectl set-timezone Asia/Shanghai \
&& ntpdate ntp.aliyun.com \
&& date

cat >> /etc/hosts << EOF
192.168.20.21  k8s-master-01   #后续扩容的IP也可以添加进来
192.168.20.24  k8s-worker-01
192.168.20.25  k8s-worker-02
EOF

二、部署Etcd集群

Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为解决Etcd单点故障，应采用集群方式部署，这里使用3台组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。

节点名称	IP
etcd-1	192.168.20.21
etcd-2	192.168.20.24
etcd-3	192.168.20.25

这里为了节省资源与k8s节点机器复用，etcd也可以在k8s集群之外部署，只要kube-apiserver能正常连接到就行

2.1 整备cfssl证书生成工具

CFSSL是CloudFlare开源的一款PKI/TLS工具。 CFSSL 包含一个命令行工具和一个用于签名，验证并且捆绑TLS证书的 HTTP API 服务。使用Go语言编写。

[k8s-master-01]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[k8s-master-01]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[k8s-master-01]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[k8s-master-01]# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
[k8s-master-01]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[k8s-master-01]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
[k8s-master-01]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.2 生成Etcd CA证书

创建工作目录
[k8s-master-01]# mkdir -pv ~/TLS/{etcd,k8s} && cd ~/TLS/etcd
自签CA

[k8s-master-01]# cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

[k8s-master-01]# cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成证书

1
2
3

[k8s-master-01]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[k8s-master-01]# ls *pem
ca-key.pem  ca.pem

2.3 使用自签CA签发Etcd https证书

[k8s-master-01]# cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.20.21",
    "192.168.20.24",
    "192.168.20.25"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

上述IP为Etcd集群内部通信IP，一个不能少，也不能错！为了方便后期扩容可以多写几个预留的IP。

生成证书

1
2
3

[k8s-master-01]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
[k8s-master-01]# ls server*pem
server-key.pem  server.pem

2.4 配置Etcd集群

下载Etcd二进制文件
地址https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz

创建工作目录并解压二进制包

1
2
3

[k8s-master-01]# mkdir -pv /opt/etcd/{bin,cfg,ssl}
[k8s-master-01]# tar -xvf etcd-v3.5.4-linux-amd64.tar.gz
[k8s-master-01]# mv etcd-v3.5.4-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

创建Etcd配置文件

[k8s-master-01]# cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.20.21:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.21:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.21:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.21:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.21:2380,etcd-2=https://192.168.20.24:2380,etcd-3=https://192.168.20.25:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

配置文件参数说明

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIALCLUSTER_TOKEN：集群Token

ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

添加systemd管理etcd

[k8s-master-01]# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

将Etcd生成的CA证书拷贝到指定位置

1	`[k8s-master-01]# cp ~/TLS/etcd/capem ~/TLS/etcd/serverpem /opt/etcd/ssl/`

启动并设置开机自启

1
2
3

[k8s-master-01]# systemctl daemon-reload
[k8s-master-01]# systemctl start etcd
[k8s-master-01]# systemctl enable etcd

此时Etcd服务是不能正常启动的，需要另外两台Etcd服务同时启动才行

将Master节点生成的Etcd文件拷贝到另外两台机器上

[k8s-master-01]# scp -r /opt/etcd/ root@192.168.20.24:/opt/ 
[k8s-master-01]# scp -r /opt/etcd/ root@192.168.20.25:/opt/
[k8s-master-01]# scp /usr/lib/systemd/system/etcd.service root@192.168.20.24:/usr/lib/systemd/system/
[k8s-master-01]# scp /usr/lib/systemd/system/etcd.service root@192.168.20.24:/usr/lib/systemd/system/

分别修改节点2与节点3etcd.conf文件配置

vim /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd-1"   # 修改此处，节点2改为etcd-2，节点3改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.20.21:2380"   # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.21:2379" # 修改此处为当前服务器IP

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.21:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.21:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.20.21:2380,etcd-2=https://192.168.20.24:2380,etcd-3=https://192.168.20.25:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

重复上面启动并设置开机自启操作

查看集群状态(在任何Etcd节点下操作亦可)
执行以下操作

ETCD_API=3 /opt/etcd/bin/etcdctl \
--write-out=table \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--endpoints=https://192.168.20.21:2379,https://192.168.20.24:2379,https://192.168.20.25:2379 endpoint health

输出

+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.20.25:2379 |   true | 18.033533ms |       |
| https://192.168.20.24:2379 |   true | 18.334299ms |       |
| https://192.168.20.21:2379 |   true | 19.755788ms |       |
+----------------------------+--------+-------------+-------+

三、安装Docker

这里使用Docker作为容器引擎，也可以更换别的，比如CRI-O和Containerd，因为Kubernetes在v1.24版本正式移除 Dockershim相关代码。

在所有Worker节点安装，安装方式随便，用yum或者二进制都行，这里使用二进制安装
下载地址https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

3.1 解压二进制包

1
2
3

[k8s-worker-01]# tar -xvf docker-19.03.9.tgz
[k8s-worker-01]# mv docker/* /usr/local/bin/
[k8s-worker-01]# docker version

3.2 配置docker镜像加速器

[k8s-worker-01]# mkdir /etc/docker
[k8s-worker-01]#cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

3.3 添加systemd管理docker

[k8s-worker-01]# cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
EOF

3.4 启动并设置开机自启

1
2
3

[k8s-worker-01]# systemctl daemon-reload
[k8s-worker-01]# systemctl daemon-reload
[k8s-worker-01]# systemctl daemon-reload

四、部署Master Node

4.1 部署kube-apiserver

(1).自签证书签发机构(CA)

[k8s-master-01]# cd ~/TLS/k8s
[k8s-master-01]# cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

[k8s-master-01]# cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成证书：生成ca.pem和ca-key.pem文件

1
2
3

[k8s-master-01]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[k8s-master-01]# ls *pem
[k8s-master-01]# ca-key.pem  ca.pem

(2).使用自签CA签发kube-apiserver HTTPS证书

[k8s-master-01]# cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.20.21",
      "192.168.20.22",
      "192.168.20.23",
      "192.168.20.24",
      "192.168.20.25",
      "192.168.20.26",
      "192.168.20.27",
      "192.168.20.28",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

上述文件hosts字段中IP为所有Master/LB/VIP IP，一个都不能少，一个不能错，为了方便后期扩容可以多写几个预留的IP.

生成证书，生成server.pem和server-key.pem

1
2
3

[k8s-master-01]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
[k8s-master-01]# ls server*pem
server-key.pem  server.pem

4.2 安装kube-apiserver

下载Kubernetes二进制包
下载地址https://dl.k8s.io/v1.20.1/kubernetes-server-linux-amd64.tar.gz

根据不同CPU架构下载对应的二进制包，打开链接会看到 Source Code、 Client binaries、 Server binaries、 Node binaries四个下载包，但只下载一个Server包就够了，里面已经包含Master和Worker Node二进制文件。

(1).创建目录并解压二进制包

[k8s-master-01]# mkdir -pv /opt/kubernetes/{bin,cfg,ssl,logs}
[k8s-master-01]# tar -xvf kubernetes-server-linux-amd64.tar.gz
[k8s-master-01]# cd kubernetes/server/bin
[k8s-master-01]# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
[k8s-master-01]# cp kubectl /usr/local/bin/

(2).创建kube-apiserver配置文件

[k8s-master-01]# cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.20.21:2379,https://192.168.20.224:2379,https://192.168.20.25:2379 \\
--bind-address=192.168.20.21 \\
--secure-port=6443 \\
--advertise-address=192.168.20.21 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=api \\
--service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

上面两个\ \ 第一个是转义符，第二个是换行符，使用转义符是为了使用EOF保留换行符

配置文件参数说明

–logtostderr：启用日志

–v：日志等级

–log-dir：日志目录

–etcd-servers：etcd集群地址

–bind-address：监听地址

–secure-port：https安全端口

–advertise-address：集群通告地址

–allow-privileged：启用授权

–service-cluster-ip-range：Service虚拟IP地址段

–enable-admission-plugins：准入控制模块

–authorization-mode：认证授权，启用RBAC授权和节点自管理

–enable-bootstrap-token-auth：启用TLS bootstrap机制

–token-auth-file：bootstrap token文件

–service-node-port-range：Service nodeport类型默认分配端口范围

–kubelet-client-xxx：apiserver访问kubelet客户端证书

–tls-xxx-file：apiserver https证书

1.20版本必须加的参数：–service-account-issuer，–service-account-signing-key-file

–etcd-xxxfile：连接Etcd集群证书

–audit-log-xxx：审计日志

启动聚合层相关配置：–requestheader-client-ca-file，–proxy-client-cert-file，–proxy-client-key-file，–requestheader-allowed-names，–requestheader-extra-headers-prefix，–requestheader-group-headers，–requestheader-username-headers，–enable-aggregator-routing

(3).拷贝生成的证书

1 2	`# 把kube-apiserver生成的证书文件拷贝到kubernetes工作目录 [k8s-master-01]# cp ~/TLS/k8s/capem ~/TLS/k8s/serverpem /opt/kubernetes/ssl/`

(4).启用TLS Bootstrapping 机制

TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

(5).创建token文件

# 生成token
[k8s-master-01]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
dbd2c718063399d5e20f7df6a8f56ca5

# 格式：token，用户名，UID，用户组
[k8s-master-01]# cat > /opt/kubernetes/cfg/token.csv << EOF
dbd2c718063399d5e20f7df6a8f56ca5,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

(6).systemd管理kube-apiserver

[k8s-master-01]# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

(7).启动并设置开机启动

1
2
3

[k8s-master-01]# systemctl daemon-reload
[k8s-master-01]# systemctl start kube-apiserver 
[k8s-master-01]# systemctl enable kube-apiserver

4.2 部署kube-controller-manager

(1).创建配置文件

[k8s-master-01]# cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

配置参数说明

–kubeconfig：连接apiserver配置文件

–leader-elect：当该组件启动多个时，自动选举（HA）

–cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

(2).生成kube-controller-manager证书

# 切换工作目录
[k8s-master-01]# cd ~/TLS/k8s

# 创建证书请求文件
[k8s-master-01]# cat > kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing", 
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
[k8s-master-01]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

(3).生成kubeconfig文件（以下是shell命令，直接在终端执行）

# Master 节点执行

KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://192.168.20.21:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-credentials kube-controller-manager \
  --client-certificate=./kube-controller-manager.pem \
  --client-key=./kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=${KUBE_CONFIG}

kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

(4).systemd管理controller-manager

[k8s-master-01]# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

(5).启动并设置开机启动

1
2
3

[k8s-master-01]# systemctl daemon-reload
[k8s-master-01]# systemctl start kube-controller-manager
[k8s-master-01]# systemctl enable kube-controller-manager

4.3 部署kube-scheduler

(1).添加配置文件

[k8s-master-01]# cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF

参数说明

–kubeconfig：连接apiserver配置文件

–leader-elect：当该组件启动多个时，自动选举（HA）

(2).生成kube-scheduler证书

# 切换工作目录
[k8s-master-01]# cd ~/TLS/k8s

# 创建证书请求文件
[k8s-master-01]# cat > kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

生成证书
[k8s-master-01]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

(3).生成kubeconfig文件（以下是shell命令，直接在终端执行）

# Master 节点执行

KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://192.168.20.21:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-credentials kube-scheduler \
  --client-certificate=./kube-scheduler.pem \
  --client-key=./kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-scheduler \
  --kubeconfig=${KUBE_CONFIG}

kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

(4).systemd管理scheduler

[k8s-master-01]# cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

(5).启动并设置开机启动

1
2
3

[k8s-master-01]# systemctl daemon-reload
[k8s-master-01]# systemctl start kube-scheduler
[k8s-master-01]# systemctl enable kube-scheduler

4.4 查看集群状态

(1).生成kubectl连接集群的证书

[k8s-master-01]# cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
[k8s-master-01]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

(2).生成kubeconfig文件

mkdir -pv /root/.kube

KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://192.168.20.21:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-credentials cluster-admin \
  --client-certificate=./admin.pem \
  --client-key=./admin-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=${KUBE_CONFIG}

kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

(3).通过kubectl工具查看当前集群组件状态

[k8s-master-01]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}

(3).授权kubelet-bootstrap用户允许请求证书


# Master 节点执行

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

五、部署Worker Node

5.1 创建Worker运行环境

[k8s-worker-01]# mkdir -pv /opt/kubernetes/{bin,cfg,ssl,logs}

# 将Master节点解压的kubernetes-server-linux-amd64目录或者指定文件拷贝到其他节点
[k8s-worker-01]# cp kubernetes-server-linux-amd64/server/bin/{kubelet,kube-proxy} /opt/kubernetes/bin

# 把Master节点生成的ca.pem文件拷贝到当前Worker节点指定目录
[k8s-master-01]# scp ~/TLS/k8s/ca.pem root@192.168.20.24:/opt/kubernetes/ssl/

5.2 部署kubelet

(1).创建配置文件

[k8s-worker-01]# cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=k8s-worker-01 \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF

要把–hostname-override=k8s-worker-01 参数后的值改为当前主机名

参数说明

–hostname-override：显示名称，集群中唯一

–network-plugin：启用CNI

–kubeconfig：空路径，会自动生成，后面用于连接apiserver

–bootstrap-kubeconfig：首次启动向apiserver申请证书

–config：配置参数文件

–cert-dir：kubelet证书生成目录

–pod-infra-container-image：管理Pod网络容器的镜像

(2).配置参数文件

[k8s-worker-01]# cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

(3).生成kubelet初次加入集群引导kubeconfig文件

KUBE_CONFIG="/opt/kubernetes/cfg/bootstrap.kubeconfig"
KUBE_APISERVER="https://192.168.20.21:6443" # kube-apiserver IP:PORT
TOKEN="dbd2c718063399d5e20f7df6a8f56ca5" # 与token.csv里保持一致

# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=${KUBE_CONFIG}

kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

(4).systemd管理kubelet

[k8s-worker-01]# cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

(5).启动并设置开机启动

1
2
3

[k8s-worker-01]# systemctl daemon-reload
[k8s-worker-01]# systemctl start kubelet
[k8s-worker-01]# systemctl enable kubelet

(6).批准kubelet证书申请并加入集群

# 查看kubelet证书请求
[k8s-master-01]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-kIJfIKIKfKG-KK3D4DKkd__nidkeFDI34dFD_dDGde2   57s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 批准申请
[k8s-master-01]# kubectl certificate approve node-csr-kIJfIKIKfKG-KK3D4DKkd__nidkeFDI34dFD_dDGde2

# 查看节点（由于网络插件还没有部署，节点会没有准备就绪 NotReady）
[k8s-master-01]# kubectl get node
NAME          STATUS     ROLES    AGE     VERSION
k8s-worker-01 NotReady   <none>   5s      v1.20.15

5.3 部署kube-proxy

(1).创建配置文件

[k8s-worker-01]# cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

(2).配置参数文件

[k8s-worker-01]# cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-worker-01
clusterCIDR: 10.0.0.0/24
EOF

要把hostnameOverride: k8s-worker-01 参数后的值改为当前主机名

(3).生成kube-proxy.kubeconfig文件

# 切换工作目录,生成kube-proxy证书.
[k8s-master-01]# cd ~/TLS/k8s

# 创建证书请求文件
[k8s-master-01]# cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

# 生成证书
[k8s-master-01]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
# 查看生成的证书文件
[k8s-master-01]# ls kube-proxy*.pem
kube-proxy-key.pem  kube-proxy.pem

(4).生成kubeconfig文件

# 切换到k8s-worker-01 节点执行

KUBE_CONFIG="/opt/kubernetes/cfg/kube-proxy.kubeconfig"
KUBE_APISERVER="https://192.168.20.21:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=${KUBE_CONFIG}

kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

(5).systemd管理kube-proxy

[k8s-worker-01]# cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

(6).启动并设置开机启动

1
2
3

[k8s-worker-01]# systemctl daemon-reload
[k8s-worker-01]# systemctl start kube-proxy
[k8s-worker-01]# systemctl enable kube-proxy

5.4 新增Worker Node

(1).拷贝已部署好的Node相关文件到新节点

1
2
3

# 把k8s-Worker-01节点涉及文件拷贝到新增Worker节点上
[k8s-worker-01]# scp -r /opt/kubernetes root@192.168.20.25:/opt/
[k8s-worker-01]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.20.25:/usr/lib/systemd/system

(2).删除kubelet证书和kubeconfig文件

1 2	`[k8s-worker-02]# rm -f /opt/kubernetes/cfg/kubelet.kubeconfig [k8s-worker-02]# rm -f /opt/kubernetes/ssl/kubelet*`

这几个文件是证书申请审批后自动生成的，每个Node不同，必须删除

(3).修改配置文件中的主机名

[k8s-worker-02]# vim /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-worker-02
[k8s-worker-02]# vim /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-worker-02

(4) 启动并设置开机启动

1
2
3

[k8s-worker-02]# systemctl daemon-reload
[k8s-worker-02]# systemctl start kubelet kube-proxy
[k8s-worker-02]# systemctl enable kubelet kube-proxy

(5).在Master上批准新Node kubelet证书申请

# 查看证书请求
[k8s-master-01]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-5kdKRFKjurf-dfskk2dgn__krfKGFDkhhkGF_GNFDfd   67s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 同意授权请求
kubectl certificate approve node-csr-5kdKRFKjurf-dfskk2dgn__krfKGFDkhhkGF_GNFDfd

(6).查看Node状态

[k8s-master-01]# kubectl get nodes
NAME          STATUS   ROLES    AGE    VERSION
k8s-worker-01 Ready    <none>   50m    v1.20.15
k8s-worker-02 Ready    <none>   12s    v1.20.15

STATUS显示Ready是已经部署Calico成功运行后的结果

六、网络与授权

6.1 部署网络组件Calico

Calico是一个纯三层的数据中心网络方案，是目前Kubernetes主流的网络方案
下载地址https://projectcalico.docs.tigera.io/manifests/calico.yaml

# 部署Calico(切换Master节点执行)
# 检查apiserver版本是否与calico.yaml文件里配置是否一致
[k8s-master-01]# kubectl api-versions | grep polic
policy/v1beta1

[k8s-master-01]# kubectl apply -f calico.yaml
[k8s-master-01]# get pods -n kube-system

NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-97769f7c7-q9thh   1/1     Running   0          5m25s
calico-node-pjxtr                         1/1     Running   0          5m25s

# 等Calico Pod都Running，节点也会准备就绪
[k8s-master-01]# kubectl get node
NAME          STATUS     ROLES    AGE     VERSION
k8s-worker-01 Ready      <none>   5s      v1.20.15

6.2 授权apiserver访问kubelet

应用场景：例如kubectl logs

[k8s-master-01]# cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f apiserver-to-kubelet-rbac.yaml

七、部署Dashboard和CoreDNS

7.1 部署Dashboard

Dashboard 是基于网页的 Kubernetes 用户界面

yaml文件下载地址https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml

# 执行安装Dashboard
[k8s-master-01]# kubectl apply -f recommended.yaml

# 查看Pod名称
[k8s-master-01]# kubectl get pods --namespace=kubernetes-dashboard -o wide
NAME                                         READY   STATUS    RESTARTS   AGE   IP               NODE          NOMINATED NODE   READINESS GATES
dashboard-metrics-scraper-5b8896d7fc-4wsfd   1/1     Running   0          51s   172.16.101.197   k8s-work-01   <none>           <none>
kubernetes-dashboard-cb988587b-mfqjh         1/1     Running   0          51s   172.16.101.196   k8s-work-01   <none>           <none>

修改Dashboard访问类型，默认是通过API Server访问，且URL字符串太长比较麻烦，所以改成NodePort访问(IP:Port)

1
2
3

[k8s-master-01]# kubectl --namespace=kubernetes-dashboard get service kubernetes-dashboard     # 查看当前访问类型
NAME                   TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   ClusterIP  10.0.0.96    <none>        443/TCP         15m

执行编辑操作

[k8s-master-01]# kubectl --namespace=kubernetes-dashboard edit service kubernetes-dashboard

spec:
  clusterIP: 10.0.0.96
  clusterIPs:
  - 10.0.0.96
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31730
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort   # 这个字段修改NodePort
status:
  loadBalancer: {}

重新查看访问类型

1
2
3

[k8s-master-01]# kubectl --namespace=kubernetes-dashboard get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.0.0.96    <none>        443:31730/TCP   17m

创建service account并绑定默认cluster-admin管理员集群角色

1
2
3

[k8s-master-01]# kubectl create serviceaccount dashboard-admin -n kube-system
[k8s-master-01]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
[k8s-master-01]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

访问地址https://NodeIP:Port

把刚才创建管理员输出token字符串输入登录输入框

Dashboard显示的面板

7.2 部署CoreDNS

Kubernetes包含一个DNS服务器Kube-DNS，用于服务发现。此DNS服务器利用SkyDNS中的库为Kubernetes pod 和服务提供DNS请求。

yaml文件下载https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/coredns/coredns.yaml.base
配置coredns.yaml

*******************************  修改一  **************************************
spec:
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  replicas: 2    # 这行参数需要新增
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns

*******************************  修改二  **************************************
- name: coredns
        #image: registry.k8s.io/coredns/coredns:v1.8.6   # 这是默认镜像地址
        image: coredns/coredns:latest                    # 修改镜像地址，后面tag自行选择
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            #memory: __DNS__MEMORY__LIMIT__
            memory: 70Mi        # 默认这处内存需要设置
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]

*******************************  修改三  **************************************
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.0.0.2   # 添加一个虚拟网段IP地址
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp

需要执行如下iptables策略，不然内网DNS无法正常解析
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F

如果执行完CoreDNS无法正常解析DNS，尝试重启解析的Pod与service服务

创建CoreDNS并解析测试

# 创建CoreDNS
[k8s-master-01]# kubectl apply -f coredns.yaml

# 新建busybox.yaml
[k8s-master-01]# touch busybox.yaml && vim busybox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox1
  labels:
    app: busybox1
spec:
  containers:
  - image: busybox:1.28.4
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

# 创建Pod
[k8s-master-01]# kubectl apply -f busybox.yaml

# 查看运行状态
[k8s-master-01]# kubectl get pods busybox1
NAME       READY   STATUS    RESTARTS   AGE
busybox1   1/1     Running   5          37s

# 测试解析
[k8s-master-01]# kubectl exec -ti busybox1 -- nslookup kubernetes.default
Server:    10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local

DNS解析没问题，至此单Master kubernetes集群搭建完成。

八、扩容多Master(高可用架构)

Kubernetes作为容器集群系统，通过健康检查+重启策略实现了Pod故障自我修复能力，通过调度算法实现将Pod分布式部署，并保持预期副本数，根据Node失效状态自动在其他Node拉起Pod，实现了应用层的高可用性。

针对Kubernetes集群，高可用性还应包含以下两个层面的考虑：Etcd数据库的高可用性和Kubernetes Master组件的高可用性。而Etcd我们已经采用3个节点组建集群实现高可用，本节将对Master节点高可用进行说明和实施。

Master节点扮演着总控中心的角色，通过不断与工作节点上的Kubelet和kube-proxy进行通信来维护整个集群的健康工作状态。如果Master节点故障，将无法使用kubectl工具或者API做任何集群管理。

Master节点主要有三个服务kube-apiserver、kube-controller-manager和kube-scheduler，其中kube-controller-manager和kube-scheduler组件自身通过选择机制已经实现了高可用，所以Master高可用主要针对kube-apiserver组件，而该组件是以HTTP API提供服务，因此对他高可用与Web服务器类似，增加负载均衡器对其负载均衡即可，并且可水平扩容。

多Master架构图

8.1 拷贝k8s-Master-01节点文件

# 先创建所需目录
[k8s-master-02]# mkdir -pv /opt/etcd/ssl
[k8s-master-02]# mkdir -pv ~/.kube/

# 切换k8s-Master-01节点
[k8s-master-01]# scp -r /opt/kubernetes root@192.168.20.22:/opt
[k8s-master-01]# scp -r /opt/etcd/ssl root@192.168.20.22:/opt/etcd
[k8s-master-01]# scp /usr/lib/systemd/system/kube* root@192.168.20.22:/usr/lib/systemd/system
[k8s-master-01]# scp /usr/local/bin/kubectl root@192.168.20.22:/usr/local/bin/
[k8s-master-01]# scp ~/.kube/config root@192.168.20.22:~/.kube/

8.2 删除证书文件及IP

[k8s-master-02]# rm -f /opt/kubernetes/ssl/kubelet*

[k8s-master-02]# sed -i 's/--bind-address=192.168.20.21/--bind-address=192.168.20.22/g' /opt/kubernetes/cfg/kube-apiserver.conf
[k8s-master-02]# sed -i 's/--advertise-address=192.168.20.21/--advertise-address=192.168.20.22/g' /opt/kubernetes/cfg/kube-apiserver.conf

[k8s-master-02]# sed -i 's/https\:\/\/192.168.20.21/https\:\/\/192.168.20.22/g' ~/.kube/config

8.3 启动并设置开机启动

1
2
3

[k8s-master-02]# systemctl daemon-reload
[k8s-master-02]# systemctl start kube-apiserver.service && systemctl start kube-controller-manager.service && systemctl start scheduler.service
[k8s-master-02]# systemctl enable kube-apiserver.service && systemctl enable kube-controller-manager.service && systemctl enable scheduler.service

8.4 查看集群状态

[k8s-master-02]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}

9、部署Nginx+Keepalived高可用负载均衡器

Nginx是一个主流Web服务和反向代理服务器，这里用四层实现对apiserver实现负载均衡。
Keepalived是一个主流高可用软件，基于VIP绑定实现服务器双机热备，在上述拓扑中，Keepalived主要根据Nginx运行状态判断是否需要故障转移（漂移VIP），例如当Nginx主节点挂掉，VIP会自动绑定在Nginx备节点，从而保证VIP一直可用，实现Nginx高可用。

如果你是在公有云上，一般都不支持keepalived，那么你可以直接用它们的负载均衡器产品，直接负载均衡多台Master kube-apiserver，架构与上面一样。

9.1 软件安装

yum install -y epel-release && yum install -y nginx keepalived

Nginx也可以编译安装，灵活添加所需模块，为了进行四层转发，需要在编译时添加以下两个参数。

1 2	`--with-stream --with-stream_ssl_module`

9.2 Nginx.conf配置(主备一致)

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.20.21:6443;   # Master1 APISERVER IP:PORT
       server 192.168.20.22:6443;   # Master2 APISERVER IP:PORT
    }
    
    server {
       listen 0.0.0.0:6443; # 与其他节点复用需要注意改端口号
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}

9.3 keepalived.conf(Nginx Master)

global_defs { 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER   #主备名称别一样
    interface eno16777736  # 注意网卡名，Centos 7网卡名有的是ens33
    virtual_router_id 51   # VRRP 路由 ID实例，每个实例是唯一的
    priority 100           # 优先级，备服务器设置 90
    advert_int 1           # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.20.27/24
    } 
    track_script {
        check_nginx
    } 
}

添加上面文件里的Nginx运行状态脚本，注意所属路径(主备节点都添加)

#!/bin/bash

RUN=$(ps aux | grep -w nginx | grep -v grep | wc -l)
if [ $RUN -eq 0 ];then
  systemctl stop keepalived.service
  exit 1
fi

别忘了添加执行权限chmod +x /etc/keepalived/check_nginx.sh

9.4 keepalived.conf(Nginx Backup)

global_defs {
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
    state BACKUP
    interface eno16777736
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 虚拟IP
    virtual_ipaddress {
        192.168.20.27/24
    }
    track_script {
        check_nginx
    }
}

9.5 启动服务并设置开机自启

1
2
3

systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived

为了防止keepalived主备VIP出现“脑裂”情况，需要注意以下两点
(1)、确保本机SElinux是disabled状态
(2)、确保本地防火墙放行vrrp协议，如下操作：
firewall-cmd –direct –permanent –add-rule ipv4 filter INPUT 0 –in-interface eno16777736 –destination 224.0.0.18 –protocol vrrp -j ACCEPT
firewall-cmd –reload

9.6 Nginx+Keepalived高可用测试

显示Master节点VIP状态

显示Backup节点VIP状态

在Master节点关闭Nginx服务

观察Backup节点VIP状态

Master节点恢复Nginx服务

在k8s集群任意一个节点，使用curl查看K8s版本测试，使用VIP访问

curl -k https://192.168.20.27:6443/version
{
  "major": "1",
  "minor": "20",
  "gitVersion": "v1.20.15",
  "gitCommit": "8f1e5bf0b9729a899b8df86249b56e2c74aebc55",
  "gitTreeState": "clean",
  "buildDate": "2022-01-19T17:23:01Z",
  "goVersion": "go1.15.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}

查看Nginx日志

1 2	`192.168.20.24 192.168.20.21:6443 - [09/Jun/2022:05:23:57 +0800] 200 1660 192.168.20.24 192.168.20.22:6443 - [09/Jun/2022:05:25:04 +0800] 200 1935`

9.7 修改所有Worker Node连接LB VIP

试想下，虽然我们增加了Master02 Node和负载均衡器，但是我们是从单Master架构扩容的，也就是说目前所有的Worker Node组件连接都还是Master01 Node，如果不改为连接VIP走负载均衡器，那么Master还是单点故障。
因此接下来就是要改所有Worker Node（kubectl get node命令查看到的节点）组件配置文件，由原来192.168.20.21修改为192.168.20.27（VIP）
在所有Worker Node执行

1
2
3

sed -i 's/https\:\/\/192.168.20.21/https\:\/\/192.168.20.27/g' /opt/kubernetes/cfg/bootstrap.kubeconfig
sed -i 's/https\:\/\/192.168.20.21/https\:\/\/192.168.20.27/g' /opt/kubernetes/cfg/kubelet.kubeconfig
sed -i 's/https\:\/\/192.168.20.21/https\:\/\/192.168.20.27/g' /opt/kubernetes/cfg/kube-proxy.kubeconfig

重启所有Worker Node节点kubelet与kube-proxy服务

检查节点状态

[k8s-master-01]# kubectl get node
NAME          STATUS   ROLES    AGE     VERSION
k8s-work-01   Ready    <none>   5d14h   v1.20.15
k8s-work-02   Ready    <none>   5d13h   v1.20.15

至此，一套完整的 Kubernetes 高可用集群就部署完成了!

折腾

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议，转载请注明出处！

单机部署Mysql多实例上一篇

Centos 7 GCC编译安装最新版本下一篇

使用二进制包部署kubernetes V1.20集群

概述

一、环境准备

1.1 部署k8s方式

1.2 软件版本

1.3 服务器整体规划

1.4 操作系统初始化

二、部署Etcd集群

2.1 整备cfssl证书生成工具

2.2 生成Etcd CA证书

2.3 使用自签CA签发Etcd https证书

2.4 配置Etcd集群

三、安装Docker

3.1 解压二进制包

3.2 配置docker镜像加速器

3.3 添加systemd管理docker

3.4 启动并设置开机自启

四、部署Master Node

4.1 部署kube-apiserver

4.2 安装kube-apiserver

4.2 部署kube-controller-manager

4.3 部署kube-scheduler

4.4 查看集群状态

五、部署Worker Node

5.1 创建Worker运行环境

5.2 部署kubelet

5.3 部署kube-proxy

5.4 新增Worker Node

六、 网络与授权

6.1 部署网络组件Calico

6.2 授权apiserver访问kubelet

七、部署Dashboard和CoreDNS

7.1 部署Dashboard

7.2 部署CoreDNS

八、扩容多Master(高可用架构)

8.1 拷贝k8s-Master-01节点文件

8.2 删除证书文件及IP

8.3 启动并设置开机启动

8.4 查看集群状态

9、部署Nginx+Keepalived高可用负载均衡器

9.1 软件安装

9.2 Nginx.conf配置(主备一致)

9.3 keepalived.conf(Nginx Master)

9.4 keepalived.conf(Nginx Backup)

9.5 启动服务并设置开机自启

9.6 Nginx+Keepalived高可用测试

9.7 修改所有Worker Node连接LB VIP

六、网络与授权