rancher rke部署k8s

一、rke搭建

(1)先安装关闭防火墙selinux

]# sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
]# systemctl stop firewalld && systemctl disable firewalld

(2)关闭swap交换分区

]# swapoff -a
//如果swap在/etc/fstab里面挂载了则需要"#"掉他。

(3)查看主机是否是打开ipv4路由转发，没有则加上

]# tee /etc/sysctl.d/rke.conf <<EOF
net.ipv4.ip_forward=1
EOF
]# sysctl -p /etc/sysctl.d/rke.conf
net.ipv4.ip_forward=1

(4)创建普通用户，必须用普通用户来创建集群

]# useradd -s /usr/bin/bash ops
//更改普通用户拥有权限
]# echo "ops ALL=(ALL) ALL" >> /etc/sudoers
//将用户添加到wheel组里面
]# usermod -aG wheel ops
]# id ops
uid=1001(ops) gid=1001(ops) groups=1001(ops),10(wheel)
]# echo "Xzm7_GTD.com" |passwd --stdin ops

(5)生成ssh密钥

]# ssh-keygen 
]# ls ~/.ssh/
authorized_keys  id_rsa  id_rsa.pub  known_hosts
//免密登录copy
]# ssh-copy-id -i ~/.ssh/id_rsa.pub ops@ip

(6)安装docker

]# cd /etc/yum.repos.d/
]# wget https://download.docker.com/linux/centos/docker-ce.repo
]# yum install docker-ce docker-ce-cli containerd.io
]# systemctl start docker 
]# systemctl enable docker 
//添加docker加速地址
]# tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://ba276e4c.mirror.aliyuncs.com"],
  "data-root": "/data/docker"
}
EOF
]# systemctl daemon-reload
]# systemctl restart docker

(7)将用户添加到docker组

]# usermod -aG docker ops
]# id ops
uid=1001(ops) gid=1001(ops) groups=1001(ops),10(wheel),993(docker)

(8)下载rke_linux-adm64

]# wget https://github.com/rancher/rke/releases/download/v1.2.9/rke_linux-amd64
]# mv rke_linux-adm64 /usr/local/bin/rke && chmod +x /usr/local/bin/rke
]# rke config 
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: /home/ops/.ssh/id_rsa
[+] Number of Hosts [1]: 1
[+] SSH Address of host (1) [none]: 192.168.122.100
[+] SSH Port of host (1) [22]: 22
[+] SSH Private Key Path of host (192.168.122.100) [none]: 
[-] You have entered empty SSH key path, trying fetch from SSH key parameter
[+] SSH Private Key of host (192.168.122.100) [none]: 
[-] You have entered empty SSH key, defaulting to cluster level SSH key: ~/.ssh/id_rsa
[+] SSH User of host (192.168.122.100) [ubuntu]: ops
[+] Is host (192.168.122.100) a Control Plane host (y/n)? [y]: y
[+] Is host (192.168.122.100) a Worker host (y/n)? [n]: y
[+] Is host (192.168.122.100) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (192.168.122.100) [none]: 
[+] Internal IP of host (192.168.122.100) [none]: 
[+] Docker socket path on host (192.168.122.100) [/var/run/docker.sock]: /var/run/docker.sock
[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: calico
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.20.8-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]: 

]# rke up
//如报错
Failed to get job complete status for job rke-network-plugin-deploy-job in namespace kube-system
]# rke up 再执行一次

注意主机配额必须2c2g最低配置，否则集群无法启动

rke生成空白cluster.yml文件命令

rke config --empty --name cluster.yml     #生成空白cluster.yml文件
rke config --print                        #打印填写cluster.yml文件

########################yaml################################
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: "192.168.30.110"
  port: "22"
  internal_address: ""
  role: [controlplane,etcd,worker]
  hostname_override: "node1"
  user: admin
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: /home/ops/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
- address: "192.168.30.129"
  port: "22"
  internal_address: ""
  role: [controlplane,etcd,worker]
  hostname_override: "node2"
  user: admin
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
- address: "192.168.30.133"
  port: "22"
  internal_address: ""
  role: [controlplane,etcd,worker]
  hostname_override: "node3"
  user: admin
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    uid: 0
    gid: 0
    snapshot: null
    retention: ""
    creation: ""
    backup_config: null
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
    always_pull_images: false
    secrets_encryption_config: null
    audit_log: null
    admission_configuration: null
    event_rate_limit: null
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
    generate_serving_certificate: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: flannel
  options: {}
  mtu: 0
  node_selector: {}
  update_strategy: null
authentication:
  strategy: x509
  sans: []
  webhook: null
addons: ""
addons_include: []
system_images:
  etcd: rancher/coreos-etcd:v3.4.3-rancher1
  alpine: rancher/rke-tools:v0.1.58
  nginx_proxy: rancher/rke-tools:v0.1.58
  cert_downloader: rancher/rke-tools:v0.1.58
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.58
  kubedns: rancher/k8s-dns-kube-dns:1.15.2
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.2
  kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.2
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  coredns: rancher/coredns-coredns:1.6.9
  coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  nodelocal: rancher/k8s-dns-node-cache:1.15.7
  kubernetes: rancher/hyperkube:v1.18.3-rancher2
  flannel: rancher/coreos-flannel:v0.12.0
  flannel_cni: rancher/flannel-cni:v0.3.0-rancher6
  calico_node: rancher/calico-node:v3.13.4
  calico_cni: rancher/calico-cni:v3.13.4
  calico_controllers: rancher/calico-kube-controllers:v3.13.4
  calico_ctl: rancher/calico-ctl:v3.13.4
  calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.4
  canal_node: rancher/calico-node:v3.13.4
  canal_cni: rancher/calico-cni:v3.13.4
  canal_flannel: rancher/coreos-flannel:v0.12.0
  canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.4
  weave_node: weaveworks/weave-kube:2.6.4
  weave_cni: weaveworks/weave-npc:2.6.4
  pod_infra_container: rancher/pause:3.1
  ingress: rancher/nginx-ingress-controller:nginx-0.32.0-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
  metrics_server: rancher/metrics-server:v0.3.6
  windows_pod_infra_container: rancher/kubelet-pause:v0.1.4
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: null
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
  dns_policy: ""
  extra_envs: []
  extra_volumes: []
  extra_volume_mounts: []
  update_strategy: null
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
  ssh_cert: ""
  ssh_cert_path: ""
monitoring:
  provider: ""
  options: {}
  node_selector: {}
  update_strategy: null
  replicas: null
restore:
  restore: false
  snapshot_name: ""
dns: null

cluster.yaml

(9)启动rke搭建kubernetes集群

1、安装kubectl命令

]# rke up
]# tee /etc/yum.repos.d/kubectl.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF
]# yum -y install kubectl.x86_64
//cd 到执行rkeconfig目录
]# cd ~/
]# vim .bash_profile
export KUBECONFIG=/home/ops/kube_config_cluster.yml
]# source .bash_profile
]# kubectl get nodes
NAME            STATUS   ROLES                      AGE   VERSION
10.60.161.223   Ready    controlplane,etcd,worker   79m   v1.20.8

2、通过RKE安装k8s集群成功,启动的时候有些节点启动的比较慢。需要稍微等待一段时间

• (10)可以先找一台网络好一点的，pull全部镜像，再保存到本地，cp到所有主机

]# docker save -o images.tgz `docker images|awk 'NR>1 {print $1":"$2}'`

二、RKE的环境清理

(1)所有节点分别执行

]# mkdir rancher
] cat > clear.sh << EOF
for i in $(df -h|grep kubelet |awk -F % '{print $2}')
do
 umount -l ${i}
done

rm /var/lib/kubelet/* -rf
rm /etc/kubernetes/* -rf
rm /var/lib/rancher/* -rf
rm /var/lib/etcd/* -rf
rm /var/lib/cni/* -rf

rm -rf /var/run/calico 

iptables -F && iptables -t nat -F

ip link del flannel.1

docker ps -a|awk '{print $1}'|xargs docker rm -f
docker volume ls|awk '{print $2}'|xargs docker volume rm

rm -rf /var/etcd/
rm -rf /run/kubernetes/
for i in $(docker ps -aq)
do
  docker rm -fv $(i)
done

for i in $(docker volume ls|awk '{print $2}')
do
  docker volume rm ${i}
done

rm -rf /etc/cni
rm -rf /opt/cni

systemctl restart docker
EOF
]# sh rancher/clear.sh
清理脚本

(2)清理残留目录结束。如果还有问题可能需要卸载所有节点上的docker

首先查看Docker版本
]# yum list installed | grep docker
docker-ce.x86_64  18.05.0.ce-3.el7.centos @docker-ce-edge
 
执行卸载
]# yum -y remove docker-ce.x86_64
 
删除存储目录
]# rm -rf /etc/docker
]# rm -rf /run/docker
]# rm -rf /var/lib/dockershim
]# rm -rf /var/lib/docker


如果发现删除不掉，需要先 umount，如
]# umount /var/lib/docker/devicemapper

卸载docker

三、扩容节点，缩容节点

(1)添加节点

修改cluster.yal 将需要添加的节点配置，然后运行

more cluster.yml
nodes:
  - address: 172.20.101.103
    user: ptmind
    role: [controlplane,worker,etcd]
  - address: 172.20.101.104
    user: ptmind
    role: [controlplane,worker,etcd]
  - address: 172.20.101.105
    user: ptmind
    role: [controlplane,worker,etcd]

  - address: 172.20.101.106
    user: ptmind
    role: [worker]
    labels: {traefik: traefik-outer}

(2)在新节点创建账户

useradd -s /usr/bin/bash ops -G docker       #创建账户添加到docker组里
将rke节点的ops账户执行ssh-copy-id命令
ssh-copy-id ops@10.10.10.10                  #rke部署节点添加新节点密钥登录

(3)执行添加节点操作

]# rke up --update-only

(3)rke 删除节点

修改cluster.yal 将需要删除的节点配置删除，然后运行

more cluster.yml
nodes:
  - address: 172.20.101.103
    user: ptmind
    role: [controlplane,worker,etcd]
  - address: 172.20.101.104
    user: ptmind
    role: [controlplane,worker,etcd]
  - address: 172.20.101.105
    user: ptmind
    role: [controlplane,worker,etcd]

删除#  - address: 172.20.101.106
删除#    user: ptmind
删除#    role: [worker]
删除#    labels: {traefik: traefik-outer}

• (4)执行删除节点操作

]# rke up --update-only

问题：当node节点处于NotReady状态下，对节点不可做操作，比如做了删除节点操作，会报错，删除不了节点。

解决办法：

1、手动删除节点上的组件。

2、通过命令移除节点的角色

]# kubectl label node prod-129 node-role.kubernetes.io/controlplane-

问题：k8s 集群的节点处于 SchedulingDisabled

解决方法：

]# kubectl patch node NodeName -p "{\"spec\":{\"unschedulable\":false}}"

或者：

设置不可调度
]# kubectl cordon node07-ingress
 
取消节点不可调度
]# kubectl uncordon node07-ingress

驱逐节点的pod
]# kubectl drain --ignore-daemonsets --delete-local-data node07-ingress

删除节点
]# kubectl delete node node07-ingress

yum 安装kubectl

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0

yum clean all
yum makecache
yum -y install kubectl