kubeadm部署k8s


架构图

K8S架构图

K8S架构图

Kubernetes常用资源对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Kubernetes常用资源对象
① Node – 集群节点
② Pod – 集群中运行的最小单位
③ Namespace – 命名空间
④ Service - 服务
⑤ Deployment – 无状态pod集合
⑥ StatefulSet – 有状态pod集合
⑦ DaemonSet – 守护进程pod集合(每个node上都会运行pod)
⑧ Volume – 持久化存储卷
⑨ PersistentVolume – 持久化存储卷声明
⑩ Ingress – 访问权(端口转发&负载均衡)

与资源对象交互方式:
kubectl get pods -o wide --all-namespaces
kubectl get svc -o wide --all-namespaces
kubectl get pods -n default
kubectl apply -f nginx.yml

定义对象:YAML文件,yaml中常用的项:
apiVersion
kind
metadata
spec

更多YAML资源定义示例参考:
https://kubernetes.io/docs/concepts/

准备工作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 统一时区和时间信息
timedatectl set-local-rtc 1
timedatectl set-timezone Asia/Shanghai
timedatectl status

# 修改主机名
# 找到preserve_hostname: false 改为 preserve_hostname: true
sudo vim /etc/cloud/cloud.cfg

#修改主机名(永久)
sudo vim /etc/hostname # 修改主机名
sudo vim /etc/hosts # 定义hosts文件

# 关闭防火墙
ufw disable

# 关闭selinux
sudo setenforce 0 # 临时设置, 0代表permissive 1代表enforcing

# 永久禁用
cat /etc/selinux/config <<EOF
SELINUX=permissive
EOF

Kubernetes集群部署[单主集群]

手工部署Kubernetes是一个很艰巨的活,Kubernetes包含众多组件,你需要了解网络配置、Docker的安装与使用、镜像仓库的构建、角色证书的创建、Kubernetes的基本原理和构成、Kubernetes应用程序的yaml文件编写等。

Kubernetes提供了一种自动化部署的工具: kubeadm。 通过该工具能够通过几条简单的命令和配置完成环境的搭建。
Google将部署K8S所需的组件都封装成Docker镜像,kubeadm通过集成的YAML文件服务化运行这些镜像从而完成组件间的部署与连接。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
# ⑴ 系统环境准备(关闭防火墙、关闭SWAP需在每台机器上执行)
Master Ubuntu 18.04 192.168.31.111
Node01 Ubuntu 18.04 192.168.31.112
Node02 Ubuntu 18.04 192.168.31.113

# 关闭防火墙
ufw disable

# 关闭SWAP
swapoff -a ; sed -i '/ swap / s/^/#/' /etc/fstab
vim /etc/fstab fstab # 注释带有”swap”字样的行并重新启动

# ⑵ 安装Docker(需在每台机器上执行)
FYI: https://docs.docker.com/install/linux/docker-ce/ubuntu/

# docker-io
apt install -y docker docker.io
# 或者
# docker-ce
① sudo apt-get update
② sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
③ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
④ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
⑤ sudo apt-get update
⑥ sudo apt-get install docker-ce
⑦ 验证docker是否安装成功: docker version

# ⑶ 拉取K8S组件镜像并重新Tag(需在每台机器上执行)
方式一: kubeadm使用默认image-repository初始化
docker pull mirrorgooglecontainers/kube-apiserver:v1.14.1
docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.1
docker pull mirrorgooglecontainers/kube-scheduler:v1.14.1
docker pull mirrorgooglecontainers/kube-proxy:v1.14.1
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.3.10
docker pull coredns/coredns:1.3.1
docker pull rancher/coreos-flannel:v0.10.0-amd64
docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5


docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.14.1 k8s.gcr.io/kube-proxy:v1.14.1
docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.14.1 k8s.gcr.io/kube-scheduler:v1.14.1
docker tag docker.io/mirrorgooglecontainers/kube-apiserver:v1.14.1 k8s.gcr.io/kube-apiserver:v1.14.1
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.14.1 k8s.gcr.io/kube-controller-manager:v1.14.1
docker tag docker.io/mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
docker tag rancher/coreos-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
docker tag docker.io/mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
docker tag docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5

docker rmi mirrorgooglecontainers/kube-apiserver:v1.14.1
docker rmi mirrorgooglecontainers/kube-controller-manager:v1.14.1
docker rmi mirrorgooglecontainers/kube-scheduler:v1.14.1
docker rmi mirrorgooglecontainers/kube-proxy:v1.14.1
docker rmi mirrorgooglecontainers/pause:3.1
docker rmi mirrorgooglecontainers/etcd:3.3.10
docker rmi coredns/coredns:1.3.1
docker rmi rancher/coreos-flannel:v0.10.0-amd64
docker rmi mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker rmi mirrorgooglecontainers/defaultbackend-amd64:1.5

方式二: kubeadm使用mirrorgooglecontainers仓库(--image-repository=mirrorgooglecontainers)初始化
docker pull coredns/coredns:1.3.1
docker pull rancher/coreos-flannel:v0.10.0-amd64
docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5

docker tag docker.io/coredns/coredns:1.3.1 mirrorgooglecontainers/coredns:1.3.1
docker tag rancher/coreos-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64

docker rmi coredns/coredns:1.3.1
docker rmi rancher/coreos-flannel:v0.10.0-amd64

# ⑷ 安装kubeadm, kubelet and kubectl (需在每台机器上执行)
FYI: https://kubernetes.io/docs/setup/independent/install-kubeadm/

① apt-get update && apt-get install -y apt-transport-https curl
② curl -x "http://127.0.0.1:8123" -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
③ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
④ apt-get update
⑤ apt-get install -y kubelet kubeadm kubectl
⑥ apt-mark hold kubelet kubeadm kubectl
⑦ 验证是否安装成功,输出如下结果则表明安装成功
kubectl version
kubeadm version


# ⑸ 初始化集群、网络插件部署、节点加入集群
FYI: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

# <Master节点执行> (使用polipo代理会出现问题)
① kubeadm init --pod-network-cidr=173.18.0.0/16 --kubernetes-version=v1.14.1 --apiserver-advertise-address=192.168.31.111 --image-repository=mirrorgooglecontainers
# 出现kubelet报错(cni相关的错误)执行更新
apt-get update & apt-get upgrade

# 保存该条命令执行后输出的结果,用于Node节点加入集群
kubeadm join 192.168.31.111:6443 --token 8mkpzy.rvymuw1brst7t7gp \
--discovery-token-ca-cert-hash sha256:3172e30a39357a38f1f9a5ec87c8f36e6b758864b2ea2779ebe935546e780e33

# 注:如果忘记了,重新获取的命令
# kubeadm token create --print-join-command --ttl 0

# 重新生成CA授权码
# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

② kubectl 配置
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

③ 安装flannel
# mkdir -p /data/k8s/ && cd /data/k8s/ && curl https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml -O kube-flannel.yml

# 最新版kube-flannel.yml
curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml -O kube-flannel.yml

# 修改Network为173.18.0.0/16 与 pod-network-cidr=173.18.0.0/16一致
vim kube-flannel.yml
kubectl apply -f kube-flannel.yml

# <每个Node节点执行>
④ kubeadm join 192.168.31.111:6443 --token 8mkpzy.rvymuw1brst7t7gp \
--discovery-token-ca-cert-hash sha256:3172e30a39357a38f1f9a5ec87c8f36e6b758864b2ea2779ebe935546e780e33

# 至此K8S集群已经搭建完成。

# 默认情况下POD不会创建在Master节点上,如果出于资源利用的考虑,可以通过以下命令使POD能够创建在Master节点上:
kubectl taint nodes --all node-role.kubernetes.io/master-

# 测试示例
kubectl run nginx --image=nginx --replicas=3
kubectl get pod
kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort
kubectl get svc nginx

配置dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# ⑹ 部署Kubernetes DashBoard
FYI: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

① curl https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml -O kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard.yaml


② kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='^*$' &
③ 访问 http://MASTERIP::/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

# 说明: 也可以NodePort来访问
# Google默认提供的dashboard YAML配置中服务(Service)没有使用NodePort这种类型,为了便于记忆访问URL,将YAML配置修改如下:
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 32222
selector:
k8s-app: kubernetes-dashboard

kubectl apply -f kubernetes-dashboard.yaml

访问 http://MASTERIP:32222/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

部署Nginx Ingress(服务发现与负载均衡)

1
2
3
4
5
6
7
8
9
10
11
12
13
⑺ 部署Nginx Ingress(服务发现与负载均衡)
fyi: https://kubernetes.github.io/ingress-nginx/deploy/

① curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml -O mandatory.yaml
kubectl apply -f mandatory.yaml

② curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml -O service-nodeport.yaml
kubectl apply -f service-nodeport.yaml

③ 验证Ingress
创建右边Ingress对象后,
即可通过http://edtech.h3c.com/sso 来访问

Ingress的yaml修改

查看日志

1
journalctl -f -u kubelet

问题解决

1
2
3
4
5
6
7
8
9
# 问题:
# kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

# 解决:
执行脚本
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 问题:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu-m NotReady master 55m v1.14.1

kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-f5d4745f9-nlnzz 0/1 Pending 0 58m
kube-system coredns-f5d4745f9-qk24m 0/1 Pending 0 58m

#解决:
# 可以看到节点还没有Ready,dns的两个pod也没不正常,还需要安装网络配置。
# flannel下载地址 https://github.com/coreos/flannel/releases
sysctl net.bridge.bridge-nf-call-iptables=1
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml

#flannel 默认会使用主机的第一张网卡,如果你有多张网卡,需要通过配置单独指定。修改 kube-flannel.yml 中的以下部分
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=enp0s3

dashboard登录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
# 建立账号与授权
# 参考: https://github.com/kubernetes/dashboard/wiki/Creating-sample-user
# 建立dashboard-rbac.yaml文件, 内容如下:
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system

# 执行安装(所建立的账号为admin-user):
kubectl create -f dashboard-rbac.yaml

# 查看账号是否成功建立:
kubectl get sa --all-namespaces | grep admin-user

# 获得登录授权码(获得登录授权的Token)
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
# token内容拷贝下来,然后进入Web界面的token,复制到Enter Token,点击"登录"


# 若要删除,需要将sa和clusterrolebindings里的账号同时删除。
kubectl delete -f dashboard-rbac.yaml
# 或执行下面指令
kubectl delete sa/admin-user --namespace kube-system
kubectl delete clusterrolebindings/admin-user --namespace kube-system

# 若显示跳过按钮
# 在kubernetes-dashboard.yaml的Deployment中添加enable-skip-login=true, 如下
args:
- --auto-generate-certificates
- --enable-skip-login=true
- --disable-settings-authorizer=true
# 参数参考:https://github.com/kubernetes/dashboard/wiki/Dashboard-arguments

# 报错namespaces is forbidden:xxx
# 需要创建kubernetes-dashboard账号, yaml文件如下:
apiVersion: v1
kind: ServiceAccount
metadata:
name: kubernetes-dashboard
namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system

kubernete 1.5.2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apt-get install -y kubelet=1.5.2 kubeadm kubectl=1.5.2

docker pull mirrorgooglecontainers/kube-apiserver:v1.15.2
docker pull mirrorgooglecontainers/kube-controller-manager:v1.15.2
docker pull mirrorgooglecontainers/kube-scheduler:v1.15.2
docker pull mirrorgooglecontainers/kube-proxy:v1.15.2
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.3.17
docker pull coredns/coredns:1.6.4
docker pull rancher/coreos-flannel:v0.11.0
docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5


docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.15.2 k8s.gcr.io/kube-proxy:v1.15.2
docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.15.2 k8s.gcr.io/kube-scheduler:v1.15.2
docker tag docker.io/mirrorgooglecontainers/kube-apiserver:v1.15.2 k8s.gcr.io/kube-apiserver:v1.15.2
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.15.2 k8s.gcr.io/kube-controller-manager:v1.15.2
docker tag docker.io/mirrorgooglecontainers/etcd:3.3.17 k8s.gcr.io/etcd:3.3.17
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag docker.io/coredns/coredns:1.6.4 k8s.gcr.io/coredns:1.6.4
docker tag rancher/coreos-flannel:v0.11.0 quay.io/coreos/flannel:v0.11.0
docker tag docker.io/mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
docker tag docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5

docker rmi mirrorgooglecontainers/kube-apiserver:v1.15.2
docker rmi mirrorgooglecontainers/kube-controller-manager:v1.15.2
docker rmi mirrorgooglecontainers/kube-scheduler:v1.15.2
docker rmi mirrorgooglecontainers/kube-proxy:v1.15.2
docker rmi mirrorgooglecontainers/pause:3.1
docker rmi mirrorgooglecontainers/etcd:3.3.17
docker rmi coredns/coredns:1.6.4
docker rmi rancher/coreos-flannel:v0.11.0
docker rmi mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker rmi mirrorgooglecontainers/defaultbackend-amd64:1.5


kubernete v1.6.0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
apt-get install -y kubelet=1.6.0 kubeadm kubectl=1.6.0

docker pull mirrorgooglecontainers/kube-apiserver:v1.16.0-rc.2
docker pull mirrorgooglecontainers/kube-controller-manager:v1.16.0-beta.0
docker pull mirrorgooglecontainers/kube-scheduler:v1.16.0-beta.1
docker pull mirrorgooglecontainers/kube-proxy:v1.16.0-rc.2
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.3.17
docker pull coredns/coredns:1.6.4
docker pull rancher/coreos-flannel:v0.11.0
docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5


docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.16.0-rc.2 k8s.gcr.io/kube-proxy:v1.16.0-rc.2
docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.16.0-beta.1 k8s.gcr.io/kube-scheduler:v1.16.0-beta.1
docker tag docker.io/mirrorgooglecontainers/kube-apiserver:v1.16.0-rc.2 k8s.gcr.io/kube-apiserver:v1.16.0-rc.2
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.16.0-beta.0 k8s.gcr.io/kube-controller-manager:v1.16.0-beta.0
docker tag docker.io/mirrorgooglecontainers/etcd:3.3.17 k8s.gcr.io/etcd:3.3.17
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag docker.io/coredns/coredns:1.6.4 k8s.gcr.io/coredns:1.6.4
docker tag rancher/coreos-flannel:v0.11.0 quay.io/coreos/flannel:v0.11.0
docker tag docker.io/mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
docker tag docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5

docker rmi mirrorgooglecontainers/kube-apiserver:v1.16.0-rc.2
docker rmi mirrorgooglecontainers/kube-controller-manager:v1.16.0-beta.0
docker rmi mirrorgooglecontainers/kube-scheduler:v1.16.0-beta.1
docker rmi mirrorgooglecontainers/kube-proxy:v1.16.0-rc.2
docker rmi mirrorgooglecontainers/pause:3.1
docker rmi mirrorgooglecontainers/etcd:3.3.17
docker rmi coredns/coredns:1.6.4
docker rmi rancher/coreos-flannel:v0.11.0
docker rmi mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.1
docker rmi mirrorgooglecontainers/defaultbackend-amd64:1.5


文章作者: Murray
版权声明: 本博客所有文章除特別声明外,均采用 CC BY 4.0 许可协议。转载请注明来源 Murray !
  目录