rancher rke部署k8s 一、rke搭建(1)先安装关闭防火墙selinux]# sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config ]# systemctl stop firewalld && systemctl disable firewalld(2)关闭swap交换分区]# swapoff -a //如果swap在/etc/fstab里面挂载了则需要"#"掉他。(3)查看主机是否是打开ipv4路由转发，没有则加上]# tee /etc/sysctl.d/rke.conf <<EOF net.ipv4.ip_forward=1 EOF ]# sysctl -p /etc/sysctl.d/rke.conf net.ipv4.ip_forward=1(4)创建普通用户，必须用普通用户来创建集群]# useradd -s /usr/bin/bash ops //更改普通用户拥有权限 ]# echo "ops ALL=(ALL) ALL" >> /etc/sudoers //将用户添加到wheel组里面 ]# usermod -aG wheel ops ]# id ops uid=1001(ops) gid=1001(ops) groups=1001(ops),10(wheel) ]# echo "Xzm7_GTD.com" |passwd --stdin ops(5)生成ssh密钥]# ssh-keygen ]# ls ~/.ssh/ authorized_keys id_rsa id_rsa.pub known_hosts //免密登录copy ]# ssh-copy-id -i ~/.ssh/id_rsa.pub ops@ip(6)安装docker]# cd /etc/yum.repos.d/ ]# wget https://download.docker.com/linux/centos/docker-ce.repo ]# yum install docker-ce docker-ce-cli containerd.io ]# systemctl start docker ]# systemctl enable docker //添加docker加速地址 ]# tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://ba276e4c.mirror.aliyuncs.com"], "data-root": "/data/docker" } EOF ]# systemctl daemon-reload ]# systemctl restart docker(7)将用户添加到docker组]# usermod -aG docker ops ]# id ops uid=1001(ops) gid=1001(ops) groups=1001(ops),10(wheel),993(docker)(8)下载rke_linux-adm64]# wget https://github.com/rancher/rke/releases/download/v1.2.9/rke_linux-amd64 ]# mv rke_linux-adm64 /usr/local/bin/rke && chmod +x /usr/local/bin/rke ]# rke config [+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: /home/ops/.ssh/id_rsa [+] Number of Hosts [1]: 1 [+] SSH Address of host (1) [none]: 192.168.122.100 [+] SSH Port of host (1) [22]: 22 [+] SSH Private Key Path of host (192.168.122.100) [none]: [-] You have entered empty SSH key path, trying fetch from SSH key parameter [+] SSH Private Key of host (192.168.122.100) [none]: [-] You have entered empty SSH key, defaulting to cluster level SSH key: ~/.ssh/id_rsa [+] SSH User of host (192.168.122.100) [ubuntu]: ops [+] Is host (192.168.122.100) a Control Plane host (y/n)? [y]: y [+] Is host (192.168.122.100) a Worker host (y/n)? [n]: y [+] Is host (192.168.122.100) an etcd host (y/n)? [n]: y [+] Override Hostname of host (192.168.122.100) [none]: [+] Internal IP of host (192.168.122.100) [none]: [+] Docker socket path on host (192.168.122.100) [/var/run/docker.sock]: /var/run/docker.sock [+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: calico [+] Authentication Strategy [x509]: [+] Authorization Mode (rbac, none) [rbac]: [+] Kubernetes Docker image [rancher/hyperkube:v1.20.8-rancher1]: [+] Cluster domain [cluster.local]: [+] Service Cluster IP Range [10.43.0.0/16]: [+] Enable PodSecurityPolicy [n]: [+] Cluster Network CIDR [10.42.0.0/16]: [+] Cluster DNS Service IP [10.43.0.10]: [+] Add addon manifest URLs or YAML files [no]: ]# rke up //如报错 Failed to get job complete status for job rke-network-plugin-deploy-job in namespace kube-system ]# rke up 再执行一次注意主机配额必须2c2g最低配置，否则集群无法启动rke生成空白cluster.yml文件命令rke config --empty --name cluster.yml #生成空白cluster.yml文件 rke config --print #打印填写cluster.yml文件########################yaml################################ # If you intened to deploy Kubernetes in an air-gapped environment, # please consult the documentation on how to configure custom RKE images. nodes: - address: "192.168.30.110" port: "22" internal_address: "" role: [controlplane,etcd,worker] hostname_override: "node1" user: admin docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: /home/ops/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] - address: "192.168.30.129" port: "22" internal_address: "" role: [controlplane,etcd,worker] hostname_override: "node2" user: admin docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] - address: "192.168.30.133" port: "22" internal_address: "" role: [controlplane,etcd,worker] hostname_override: "node3" user: admin docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] services: etcd: image: "" extra_args: {} extra_binds: [] extra_env: [] external_urls: [] ca_cert: "" cert: "" key: "" path: "" uid: 0 gid: 0 snapshot: null retention: "" creation: "" backup_config: null kube-api: image: "" extra_args: {} extra_binds: [] extra_env: [] service_cluster_ip_range: 10.43.0.0/16 service_node_port_range: "" pod_security_policy: false always_pull_images: false secrets_encryption_config: null audit_log: null admission_configuration: null event_rate_limit: null kube-controller: image: "" extra_args: {} extra_binds: [] extra_env: [] cluster_cidr: 10.42.0.0/16 service_cluster_ip_range: 10.43.0.0/16 scheduler: image: "" extra_args: {} extra_binds: [] extra_env: [] kubelet: image: "" extra_args: {} extra_binds: [] extra_env: [] cluster_domain: cluster.local infra_container_image: "" cluster_dns_server: 10.43.0.10 fail_swap_on: false generate_serving_certificate: false kubeproxy: image: "" extra_args: {} extra_binds: [] extra_env: [] network: plugin: flannel options: {} mtu: 0 node_selector: {} update_strategy: null authentication: strategy: x509 sans: [] webhook: null addons: "" addons_include: [] system_images: etcd: rancher/coreos-etcd:v3.4.3-rancher1 alpine: rancher/rke-tools:v0.1.58 nginx_proxy: rancher/rke-tools:v0.1.58 cert_downloader: rancher/rke-tools:v0.1.58 kubernetes_services_sidecar: rancher/rke-tools:v0.1.58 kubedns: rancher/k8s-dns-kube-dns:1.15.2 dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.2 kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.2 kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1 coredns: rancher/coredns-coredns:1.6.9 coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1 nodelocal: rancher/k8s-dns-node-cache:1.15.7 kubernetes: rancher/hyperkube:v1.18.3-rancher2 flannel: rancher/coreos-flannel:v0.12.0 flannel_cni: rancher/flannel-cni:v0.3.0-rancher6 calico_node: rancher/calico-node:v3.13.4 calico_cni: rancher/calico-cni:v3.13.4 calico_controllers: rancher/calico-kube-controllers:v3.13.4 calico_ctl: rancher/calico-ctl:v3.13.4 calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.4 canal_node: rancher/calico-node:v3.13.4 canal_cni: rancher/calico-cni:v3.13.4 canal_flannel: rancher/coreos-flannel:v0.12.0 canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.4 weave_node: weaveworks/weave-kube:2.6.4 weave_cni: weaveworks/weave-npc:2.6.4 pod_infra_container: rancher/pause:3.1 ingress: rancher/nginx-ingress-controller:nginx-0.32.0-rancher1 ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1 metrics_server: rancher/metrics-server:v0.3.6 windows_pod_infra_container: rancher/kubelet-pause:v0.1.4 ssh_key_path: ~/.ssh/id_rsa ssh_cert_path: "" ssh_agent_auth: false authorization: mode: rbac options: {} ignore_docker_version: null kubernetes_version: "" private_registries: [] ingress: provider: "" options: {} node_selector: {} extra_args: {} dns_policy: "" extra_envs: [] extra_volumes: [] extra_volume_mounts: [] update_strategy: null cluster_name: "" cloud_provider: name: "" prefix_path: "" addon_job_timeout: 0 bastion_host: address: "" port: "" user: "" ssh_key: "" ssh_key_path: "" ssh_cert: "" ssh_cert_path: "" monitoring: provider: "" options: {} node_selector: {} update_strategy: null replicas: null restore: restore: false snapshot_name: "" dns: null cluster.yaml(9)启动rke搭建kubernetes集群1、安装kubectl命令]# rke up ]# tee /etc/yum.repos.d/kubectl.repo <<EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 EOF ]# yum -y install kubectl.x86_64 //cd 到执行rkeconfig目录 ]# cd ~/ ]# vim .bash_profile export KUBECONFIG=/home/ops/kube_config_cluster.yml ]# source .bash_profile ]# kubectl get nodes NAME STATUS ROLES AGE VERSION 10.60.161.223 Ready controlplane,etcd,worker 79m v1.20.82、通过RKE安装k8s集群成功,启动的时候有些节点启动的比较慢。需要稍微等待一段时间(10)可以先找一台网络好一点的，pull全部镜像，再保存到本地，cp到所有主机]# docker save -o images.tgz `docker images|awk 'NR>1 {print $1":"$2}'`二、RKE的环境清理(1)所有节点分别执行]# mkdir rancher ] cat > clear.sh << EOF for i in $(df -h|grep kubelet |awk -F % '{print $2}') do umount -l ${i} done rm /var/lib/kubelet/* -rf rm /etc/kubernetes/* -rf rm /var/lib/rancher/* -rf rm /var/lib/etcd/* -rf rm /var/lib/cni/* -rf rm -rf /var/run/calico iptables -F && iptables -t nat -F ip link del flannel.1 docker ps -a|awk '{print $1}'|xargs docker rm -f docker volume ls|awk '{print $2}'|xargs docker volume rm rm -rf /var/etcd/ rm -rf /run/kubernetes/ for i in $(docker ps -aq) do docker rm -fv $(i) done for i in $(docker volume ls|awk '{print $2}') do docker volume rm ${i} done rm -rf /etc/cni rm -rf /opt/cni systemctl restart docker EOF ]# sh rancher/clear.sh 清理脚本(2)清理残留目录结束。如果还有问题可能需要卸载所有节点上的docker首先查看Docker版本 ]# yum list installed | grep docker docker-ce.x86_64 18.05.0.ce-3.el7.centos @docker-ce-edge 执行卸载 ]# yum -y remove docker-ce.x86_64 删除存储目录 ]# rm -rf /etc/docker ]# rm -rf /run/docker ]# rm -rf /var/lib/dockershim ]# rm -rf /var/lib/docker 如果发现删除不掉，需要先 umount，如 ]# umount /var/lib/docker/devicemapper 卸载docker三、扩容节点，缩容节点(1)添加节点修改cluster.yal 将需要添加的节点配置，然后运行more cluster.yml nodes: - address: 172.20.101.103 user: ptmind role: [controlplane,worker,etcd] - address: 172.20.101.104 user: ptmind role: [controlplane,worker,etcd] - address: 172.20.101.105 user: ptmind role: [controlplane,worker,etcd] - address: 172.20.101.106 user: ptmind role: [worker] labels: {traefik: traefik-outer}(2)在新节点创建账户useradd -s /usr/bin/bash ops -G docker #创建账户添加到docker组里 将rke节点的ops账户执行ssh-copy-id命令 ssh-copy-id ops@10.10.10.10 #rke部署节点添加新节点密钥登录(3)执行添加节点操作]# rke up --update-only(3)rke 删除节点修改cluster.yal 将需要删除的节点配置删除，然后运行more cluster.yml nodes: - address: 172.20.101.103 user: ptmind role: [controlplane,worker,etcd] - address: 172.20.101.104 user: ptmind role: [controlplane,worker,etcd] - address: 172.20.101.105 user: ptmind role: [controlplane,worker,etcd] 删除# - address: 172.20.101.106 删除# user: ptmind 删除# role: [worker] 删除# labels: {traefik: traefik-outer}(4)执行删除节点操作]# rke up --update-only问题：当node节点处于NotReady状态下，对节点不可做操作，比如做了删除节点操作，会报错，删除不了节点。解决办法：1、手动删除节点上的组件。2、通过命令移除节点的角色]# kubectl label node prod-129 node-role.kubernetes.io/controlplane-问题：k8s 集群的节点处于 SchedulingDisabled解决方法：]# kubectl patch node NodeName -p "{\"spec\":{\"unschedulable\":false}}"或者：设置不可调度 ]# kubectl cordon node07-ingress 取消节点不可调度 ]# kubectl uncordon node07-ingress 驱逐节点的pod ]# kubectl drain --ignore-daemonsets --delete-local-data node07-ingress 删除节点 ]# kubectl delete node node07-ingressyum 安装kubectl[kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 yum clean all yum makecache yum -y install kubectl

kubeadm部署k8s 关闭防火墙systemctl stop firewalld ; systemctl disable firewalld setenforce 0 sed -i.bak 's@SELINUX=.*@SELINUX=disabled@g' /etc/selinux/config swapoff -a && sudo sed -i '/swap/ s/^/#/' /etc/fstab && mount -a 内核参数修改echo "modprobe br_netfilter" >> /etc/profile cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl -p /etc/sysctl.d/k8s.conf 配置阿里云 Kubernetes yum 源cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF安装基础软件包yum -y install epel-release yum-utilsyum -y install device-mapper-persistent-data lvm2 wget net-tools nfs-utils iptables-services lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo libaio-devel wget vim ncurses-devel autoconf automake zlib-devel openssh-server socat ipvsadm conntrack telnet iptables-services nginx-mod-stream开启ipvs开启 ipvs 不开启 ipvs 将会使用 iptables，但是效率低，所以官网推荐需要开通 ipvs 内核 tee ipvs.modules <<-'EOF' #!/bin/bash ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack" for kernel_module in ${ipvs_modules}; do /sbin/modinfo -F filename ${kernel_module} > /dev/null 2>&1 if [ 0 -eq 0 ]; then /sbin/modprobe ${kernel_module} fi done EOF chmod 755 ipvs.modules && bash ipvs.modules && lsmod | grep ip_vs 安装dockeryum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install docker-ce-19.03.5 docker-ce-cli-19.03.5 containerd.io -y启动dockersystemctl start docker && systemctl enable docker tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://hxn0tro1.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF systemctl daemon-reload && systemctl restart docker && docker info |egrep " Cgroup Driver:" && systemctl status docker --no-pager安装kubeadm# 查看版本 yum search kubelet --showduplicates | sort -r yum -y install kubelet-1.23.0 kubeadm-1.23.0 kubectl-1.23.0 systemctl enable kubelet下载镜像tee image.txt <<-'EOF' docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.20.6 docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.6 docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.6 docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.6 docker pull calico/pod2daemon-flexvol:v3.18.0 docker pull calico/node:v3.18.0 docker pull calico/cni:v3.18.0 docker pull calico/kube-controllers:v3.18.0 docker pull registry.aliyuncs.com/google_containers/etcd:3.4.13-0 docker pull registry.aliyuncs.com/google_containers/coredns:1.7.0 docker pull registry.aliyuncs.com/google_containers/pause:3.2 EOF bash image.txt使用kubeadm初始化集群kubeadm init --kubernetes-version=1.23.0 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.4.6 --service-cidr=10.10.0.0/16 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all --v=5注：--image-repository registry.aliyuncs.com/google_containers 为保证拉取镜像不到国外站点拉取，手动指定仓库地址为 registry.aliyuncs.com/google_containers。kubeadm 默认从k8ss.grc.io 拉取镜像。 本地有导入到的离线镜像，所以会优先使用本地的镜像节点注册命令保存下来，以备稍后使用 --discovery-token-ca-cert-hash sha256:cbc4154daaba6b6a1355a7c54ec22ecdafab566464fd7ed257097dafedeb7065 配置 kubectl 的配置文件，保存一个证书，这样 kubectl 命令可以使用这个证书对 k8s 集群进行管理 # mkdir -p $HOME/.kube # cp -i /etc/kubernetes/admin.conf $HOME/.kube/config # chown $(id -u):$(id -g) $HOME/.kube/config # kubectl get nodes安装网络插件calico# 使用 Calico 网络插件 $ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yamlFlannel# 使用 Flannel 网络插件 $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 或者访问 https://gitee.com/kexinzz/helm/blob/master/kube-flannel.ymlnode污点这里我们会发现node节点污点，删除不掉于是，我们就更改pod yaml接受容忍#找到yaml文件中 tolerations: 字段添加如下内容 tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoSchedule #完成后我们部署pod插件，就可以看到pod成功部署了验证k8s加入工作节点使用这些信息来生成新的join命令kubeadm token create --print-join-command在每个工作节点上，运行之前保存的加入命令，例如# 在工作节点上运行加入命令 $ sudo kubeadm join <Master节点IP>:<Master节点端口> --token <Token> --discovery-token-ca-cert-hash <CA证书哈希值> $ kubeadm join 192.168.4.6:6443 --token 6n60gw.od78mw7dtoyuz8dn \ --discovery-token-ca-cert-hash sha256:e53b20f5eeb50b5f503da46af84e0c74fe736245d32321176d9a0c0f1a119c9f \ --node-name=192.168.4.7 --v=5请注意，上述命令中的 、<Master节点IP>、<Master节点端口>、 和 <CA证书哈希值> 都需要根据你的实际情况进行替换。添加master节点生成 Join Token在现有的某个 master 节点上，获取一个用于加入的 token 和 etcd 参数 ：kubeadm init phase upload-certs --upload-certs这会返回一个证书和一个启动的使用 token。使用这些信息来生成新的 join 命令kubeadm token create --print-join-command加入新的 Master 节点kubeadm join <master-ip>:<master-port> --token <token> \ --discovery-token-ca-cert-hash sha256:<hash> \ --control-plane --certificate-key <certificate-key> \ --node-name=192.168.4.7 --v=5验证集群状态一旦kubeadm join命令成功完成，在新节点上检查kubelet和kubeadmkubectl get nodes

kk部署k8s集群 查看支持的k8s版本./kk version --show-supported-k8s安装 KubeKey下载完成后当前目录下将生成 KubeKey 二进制文件 kkcurl -sSL https://get-kk.kubesphere.io | sh - # 国内 wget https://packages.emal.cc/package/KubeKey/kubekey-v3.1.10-linux-amd64.tar.gz wget https://packages.emal.cc/package/KubeKey/kk创建 manifest 文件注意若只需要使用 kk 打包 KubeSphere 镜像至离线环境中，可直接使用邮件中收到的 manifest 文件构建离线包。无需另外创建或编辑 manifest 文件。若需要使用 kk 部署 Kubernetes 以及镜像仓库，可参考以下步骤如果您访问 DockerHub 受限，执行以下命令将 Kubernetes 镜像地址替换为阿里云仓库export KKZONE=cn创建 manifest 文件。# 如需使用 kk 离线部署镜像仓库，添加 --with-registry 打包镜像仓库的安装文件 ./kk create manifest --with-kubernetes v1.26.12 --with-registry --filename manifest-sample-v1.26.12.yaml # 该命令将创建一个 manifest-sample.yaml 文件编辑 manifest 文件若需要使用 kk 部署 Kubernetes 以及镜像仓库，将从邮件获取到的 KubeSphere 镜像列表添加到新创建的 manifest 文件中即可打开 manifest 文件vi manifest-sample.yaml复制 kk-manifest.yaml 或 kk-manifest-mirror.yaml（若访问 DockerHub 受限） 中的镜像列表，添加到新创建的 manifest-sample.yaml 文件中注意以下 manifest 文件中的镜像列表仅为示例，建议通过 https://get-images.kubesphere.io/ 获取最新的镜像列表。apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Manifest metadata: name: sample spec: arches: - amd64 operatingSystems: [] kubernetesDistributions: - type: kubernetes version: v1.26.12 components: helm: version: v3.14.3 cni: version: v1.2.0 etcd: version: v3.5.13 containerRuntimes: - type: docker version: 24.0.9 - type: containerd version: 1.7.13 calicoctl: version: v3.27.4 crictl: version: v1.29.0 docker-registry: version: "2" harbor: version: v2.10.1 docker-compose: version: v2.26.1 images: - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9 - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.26.12 - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.26.12 - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.26.12 - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.26.12 - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3 - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20 - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4 - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4 - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4 - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.4 - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.27.4 - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.21.3 - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel-cni-plugin:v1.1.2 - registry.cn-beijing.aliyuncs.com/kubesphereio/cilium:v1.15.3 - registry.cn-beijing.aliyuncs.com/kubesphereio/operator-generic:v1.15.3 - registry.cn-beijing.aliyuncs.com/kubesphereio/hybridnet:v0.8.6 - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-ovn:v1.10.10 - registry.cn-beijing.aliyuncs.com/kubesphereio/multus-cni:v3.8 - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0 - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0 - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.9.6-alpine - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-vip:v0.7.2 - registry.cn-beijing.aliyuncs.com/kubesphereio/kata-deploy:stable - registry.cn-beijing.aliyuncs.com/kubesphereio/node-feature-discovery:v0.10.0 registry: auths: {}制作命令执行以下命令构建包含 ks-core 及各扩展组件镜像的离线安装包./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz执行成功后，将显示如下信息：Pipeline[ArtifactExportPipeline] execute successfully下载 KubeSphere Core Helm Chart安装 helmcurl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash下载 KubeSphere Core Helm Chart此处为示例版本，请访问 https://get-images.kubesphere.io 或 KubeSphere GitHub 仓库查看最新 chart 版本。VERSION=1.1.3 # Chart 版本 helm fetch https://charts.kubesphere.io/main/ks-core-${VERSION}.tgz离线部署1、准备工作将联网主机 node1 上的三个文件同步至离线环境的 master 节点kkkubesphere.tar.gzks-core-1.1.3.tgz2、创建配置文件创建离线集群配置文件./kk create config --with-kubernetes v1.26.12修改配置文件vi config-sample.yaml说明按照离线环境的实际配置修改节点信息。指定 registry 仓库的部署节点，用于 KubeKey 部署自建 Harbor 仓库。registry 里可以指定 type 类型为 harbor，否则默认安装 docker registry。对于 Kubernetes v1.24+，建议将 containerManager 设置为 containerd。apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata: name: sample spec: hosts: - {name: master, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: "<REPLACE_WITH_YOUR_ACTUAL_PASSWORD>"} - {name: node2, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: "<REPLACE_WITH_YOUR_ACTUAL_PASSWORD>"} roleGroups: etcd: - master control-plane: - master worker: - node2 # 如需使用 kk 自动部署镜像仓库，请设置该主机组 （建议仓库与集群分离部署，减少相互影响） # 如果需要部署 harbor 并且 containerManager 为 containerd 时，由于部署 harbor 依赖 docker，建议单独节点部署 harbor registry: - node2 controlPlaneEndpoint: ## Internal loadbalancer for apiservers # internalLoadbalancer: haproxy domain: lb.kubesphere.local address: "" port: 6443 kubernetes: version: v1.26.12 containerManager: containerd network: plugin: calico kubePodsCIDR: 10.233.64.0/18 kubeServiceCIDR: 10.233.0.0/18 ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni multusCNI: enabled: false registry: # 如需使用 kk 部署 harbor, 可将该参数设置为 harbor，不设置该参数且需使用 kk 创建容器镜像仓库，将默认使用 docker registry。 # type: harbor # 如使用 kk 部署的 harbor 或其他需要登录的仓库，需设置对应仓库的 auths，如使用 kk 创建默认的 docker registry 仓库，则无需配置 auths 参数。 auths: "dockerhub.kubekey.local": # 部署 harbor 时需指定 harbor 帐号密码 # username: admin # password: Harbor12345 skipTLSVerify: true # 设置集群部署时使用的私有仓库地址。如果您已有可用的镜像仓库，请替换为您的实际镜像仓库地址。 # 如果离线包中为原始 dockerhub 镜像（即 manifest 文件中的镜像地址为 docker.io/***），可以将该参数设置为 dockerhub.kubekey.local/kubesphereio, 表示将镜像全部推送至名为 kubesphereio 的 harbor 项目中。 privateRegistry: "dockerhub.kubekey.local" # 如果构建离线包时 Kubernetes 镜像使用的是阿里云仓库镜像，需配置该参数。如果使用 dockerhub 镜像，则无需配置此参数。 namespaceOverride: "kubesphereio" registryMirrors: [] insecureRegistries: [] addons: []安装harbor./kk init registry -f config-sample.yaml -a kubesphere.tar.gz创建harbor项目tee create_project_harbor.sh <<-'EOF' #!/usr/bin/env bash # Copyright 2018 The KubeSphere Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. url="http://dockerhub.kubekey.local" # 或修改为实际镜像仓库地址 user="admin" passwd="Harbor12345" harbor_projects=( ks kubesphere kubesphereio coredns calico flannel cilium hybridnetdev kubeovn openebs library plndr jenkins argoproj dexidp openpolicyagent curlimages grafana kubeedge nginxinc prom kiwigrid minio opensearchproject istio jaegertracing timberio prometheus-operator jimmidyson elastic thanosio brancz prometheus ) for project in "${harbor_projects[@]}"; do echo "creating $project" curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k # 注意在 curl 命令末尾加上 -k done EOFchmod +x create_project_harbor.sh ./create_project_harbor.sh安装kubernetes集群安装必要软件包yum -y install socat ipvsadm conntrack chrony./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --with-local-storage指定 --with-local-storage 参数会默认部署 openebs localpv，如需对接其他存储，可在 Kubernetes 集群部署完成后自行安装。如果显示如下信息，则表明 Kubernetes 集群创建成功。Pipeline[CreateclusterPipeline] execute successfully Installation is complete.删除集群./kk delete cluster -f config-sample.yaml官方文档地址：kubekey

kk 部署k8s 部署参考如下连接kubekey部署kubernetes集群: https://packages.emal.cc/package/softinstall/produce/KubeKey/v2.26.12/%e9%83%a8%e7%bd%b2%e8%af%b4%e6%98%8e.htmlkubekey官网