Kubespray

Kubespray 是一个基于 Ansible 的 Kubernetes 集群部署工具。在 Kubernetes 官方文档中推荐使用

下载仓库

kubespray的不同版本以不同的tag发布，以v2.27.0为例，下载仓库：

~ git clone -b v2.27.0 https://github.com/kubernetes-sigs/kubespray.git

从download.yaml文件中可以看到当前支持的版本，也可以从release页面查看

...
pod_infra_supported_versions:
  v1.31: "3.10"
  v1.30: "3.9"
  v1.29: "3.9"
...

修改部署集群配置

首先根据示例的inventory复制一份，并进行修改

~ cp -rfp inventory/sample inventory/cluster01

修改集群主机信息

默认ansible以inventory.ini格式管理集群主机，可以修改为yaml格式，以单master+单node为例

inventory/cluster01/hosts.yaml

all:
  hosts:
    master1:
      ansible_host: 10.92.162.228
    worker1:
      ansible_host: 10.92.162.229
  children:
    kube_control_plane:
      hosts:
        master1:
    kube_node:
      hosts:
        worker1:
    etcd:
      hosts:
        master1:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:

修改集群配置

~ vim inventory/cluster01/group_vars/k8s_cluster/k8s-cluster.yml
## 修改k8s版本，默认是v1.31.4（可以修改成当前版本支持范围内的其他版本）
kube_version: v1.30.6
## 修改网络插件，默认是calico，可以修改为其他网络插件
kube_network_plugin: cilium
## 修改service和pod CIDR
kube_service_addresses: 172.16.0.0/12
kube_pods_subnet: 192.168.64.0/18
## 修改kubelet最大pod数和网络节点前缀
kubelet_max_pods: 110
kube_network_node_prefix: 25
## 修改集群名称
cluster_name: cluster01
## 启用证书自动续期
auto_renew_certificates: true
## 修改证书自动续期时间，默认是每月1号03:00
auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:{{ groups['kube_control_plane'].index(inventory_hostname) }}0:00"

修改集群addons配置

~ vim inventory/cluster01/group_vars/k8s_cluster/addons.yml
## 启用metrics-server，其他按需启用
metrics_server_enabled: true
helm_enabled: false
registry_enabled: false
local_path_provisioner_enabled: false
local_volume_provisioner_enabled: false
cephfs_provisioner_enabled: false
rbd_provisioner_enabled: false
gateway_api_enabled: false
ingress_nginx_enabled: false
ingress_publish_status_address: ""
ingress_alb_enabled: false
cert_manager_enabled: false
metallb_enabled: false
metallb_speaker_enabled: "{{ metallb_enabled }}"
metallb_namespace: "metallb-system"
argocd_enabled: false
krew_enabled: false
krew_root_dir: "/usr/local/krew"
kube_vip_enabled: false
node_feature_discovery_enabled: false

制作镜像

ansible依赖python环境，配置起来比较麻烦，这里为了方便，使用仓库自带的Dockerfile制作成镜像，每次部署时直接使用镜像，避免需要配置复杂的python环境

为了方便制作镜像，可以使用docker-compose.yml文件来制作镜像，在项目根目录创建docker-compose.yml文件

~ vim docker-compose.yml
services:
  ansible:
    network_mode: host
    build:
      context: .
      dockerfile: Dockerfile
      network: host
    volumes: # 将ecs.pem文件挂载到容器中，用于ssh免密登录
      - ${HOME}/.ssh/ecs.pem:/root/.ssh/id_rsa

使用下面命令制作镜像，并直接进入容器

~ docker compose run --build ansible

部署集群

在容器中执行下面命令部署集群，可以在命令最后添加-vvv参数来查看详细日志

~ ansible-playbook -i inventory/cluster01/hosts.yaml --become --become-user=root cluster.yml

扩容节点

在hosts.yaml中新增节点信息（角色和ip地址），然后执行扩容指令

~ ansible-playbook -i inventory/cluster01/hosts.yaml --become --become-user=root scale.yml

节点下线

节点下线时，使用kubespray提供的remove-node.yml快速下线节点，也可以使用kubectl命令下线节点，最后将节点退订

~ ansible-playbook -i inventory/cluster01/hosts.yaml --extra-vars "node=node1,node2" remove-node.yml

国内部署（离线部署）

部署时，kubespray均从官方地址下载二进制及一些镜像，国内环境因为某些原因均无法正常获取，需要将二进制文件和安装包下载缓存到文件服务、私有镜像仓库，并配置kubespray离线部署

kubespray内置了快速生成文件列表和镜像列表的脚本，请注意在合适的环境下执行，避免生成错误的文件列表和镜像列表，例如M芯片的mac下执行生成的不适用于x86的机器

~ cd contrib/offline
~ bash generate_list.sh
~ cat temp/files.list # 生成的文件列表
~ cat temp/images.list # 生成的镜像列表

缓存文件

下载文件，执行以下命令将依赖的静态文件全部下载到 temp/files 目录下，文件下载时会按照文件的原有url递归创建子目录

~ wget -x -P temp/files -i temp/files.list

离线文件可以存储到nginx服务站点目录下，kubespray还贴心的提供了nginx配置文件，文件位于contrib/offline/nginx.conf

这里以阿里云oss为例，创建一个bucket，使用ossutil工具上传文件到bucket中

~ ossutil cp -r /root/kubespray/contrib/offline/temp/files/ oss://ssgeek-file-server/

缓存镜像

使用skopeo工具将镜像缓存到私有镜像仓库

~ for image in $(cat temp/images.list); do skopeo copy docker://${image} docker://hub.ssgeek.com/dockerbub/${image#*/} --override-arch=amd64 --override-os=linux; done

配置kubespray离线部署

修改离线部署变量配置文件offline.yml，将文件列表和镜像列表配置为私有地址，注意需要修改一些文件的地址，补充上原域名作为一级path

inventory/cluster01/group_vars/all/offline.yml

registry_host: "hub.ssgeek.com/dockerbub"
files_repo: "https://ssgeek-file-server.oss-cn-beijing-internal.aliyuncs.com"

kubelet_download_url: "{{ files_repo }}/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
kubectl_download_url: "{{ files_repo }}/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
kubeadm_download_url: "{{ files_repo }}/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubeadm"
etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz"
ciliumcli_download_url: "{{ files_repo }}/github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_version }}/cilium-linux-{{ image_arch }}.tar.gz"
crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
crio_download_url: "{{ files_repo }}/storage.googleapis.com/cri-o/artifacts/cri-o.{{ image_arch }}.{{ crio_version }}.tar.gz"
helm_download_url: "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz"
runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}"
crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"
youki_download_url: "{{ files_repo }}/github.com/containers/youki/releases/download/v{{ youki_version }}/youki-{{ youki_version }}-{{ ansible_architecture }}-musl.tar.gz"
kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz"
# gVisor only supports amd64 and uses x86_64 to in the download link
gvisor_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/runsc"
gvisor_containerd_shim_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/containerd-shim-runsc-v1"
nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
krew_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/krew/releases/download/{{ krew_version }}/krew-{{ host_os }}_{{ image_arch }}.tar.gz"
containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/v{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}.{{ image_arch }}.tgz"
skopeo_download_url: "{{ files_repo }}/github.com/lework/skopeo-binary/releases/download/{{ skopeo_version }}/skopeo-linux-{{ image_arch }}"
yq_download_url: "{{ files_repo }}/github.com/mikefarah/yq/releases/download/{{ yq_version }}/yq_linux_{{ image_arch }}"
## Container Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"

配置完成后，执行部署命令，又可以愉快的快速部署集群了