kubeadm源碼分析（内含kubernetes離線包，三步安裝）kubeadm源碼分析

k8s離線安裝包

三步安裝，簡單到難以置信

kubeadm源碼分析

說句實在話，kubeadm的代碼寫的真心一般，品質不是很高。

幾個關鍵點來先說一下kubeadm幹的幾個核心的事：

kubeadm 生成證書在/etc/kubernetes/pki目錄下
kubeadm 生成static pod yaml配置，全部在/etc/kubernetes/manifasts下
kubeadm 生成kubelet配置，kubectl配置等在/etc/kubernetes下
kubeadm 通過client go去啟動dns

kubeadm init

代碼入口 cmd/kubeadm/app/cmd/init.go 建議大家去看看cobra

找到Run函數來分析下主要流程：

如果證書不存在，就建立證書，是以如果我們有自己的證書可以把它放在/etc/kubernetes/pki下即可, 下文細看如果生成證書

if res, _ := certsphase.UsingExternalCA(i.cfg); !res {
if err := certsphase.CreatePKIAssets(i.cfg); err != nil {
return err
}

建立kubeconfig檔案

if err := kubeconfigphase.CreateInitKubeConfigFiles(kubeConfigDir, i.cfg); err != nil {

建立manifest檔案，etcd apiserver manager scheduler都在這裡建立, 可以看到如果你的配置檔案裡已經寫了etcd的位址了，就不建立了，這我們就可以自己裝etcd叢集，而不用預設單點的etcd，很有用

controlplanephase.CreateInitStaticPodManifestFiles(manifestDir, i.cfg);
if len(i.cfg.Etcd.Endpoints) == 0 {
if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(manifestDir, i.cfg); err != nil {
return fmt.Errorf(“error creating local etcd static pod manifest file: %v”, err)

等待APIserver和kubelet啟動成功，這裡就會遇到我們經常遇到的鏡像拉不下來的錯誤，其實有時kubelet因為别的原因也會報這個錯，讓人誤以為是鏡像弄不下來

if err := waitForAPIAndKubelet(waiter); err != nil {
ctx := map[string]string{
“Error”: fmt.Sprintf(“%v”, err),
“APIServerImage”: images.GetCoreImage(kubeadmconstants.KubeAPIServer, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage),
“ControllerManagerImage”: images.GetCoreImage(kubeadmconstants.KubeControllerManager, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage),
“SchedulerImage”: images.GetCoreImage(kubeadmconstants.KubeScheduler, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage),
kubeletFailTempl.Execute(out, ctx)
return fmt.Errorf(“couldn’t initialize a Kubernetes cluster”)

給master加标簽，加污點, 是以想要pod排程到master上可以把污點清除了

if err := markmasterphase.MarkMaster(client, i.cfg.NodeName); err != nil {
return fmt.Errorf(“error marking master: %v”, err)

生成tocken

if err := nodebootstraptokenphase.UpdateOrCreateToken(client, i.cfg.Token, false, i.cfg.TokenTTL.Duration, kubeadmconstants.DefaultTokenUsages, []string{kubeadmconstants.NodeBootstrapTokenAuthGroup}, tokenDescription); err != nil {
return fmt.Errorf(“error updating or creating token: %v”, err)

調用clientgo建立dns和kube-proxy

if err := dnsaddonphase.EnsureDNSAddon(i.cfg, client); err != nil {
return fmt.Errorf(“error ensuring dns addon: %v”, err)
if err := proxyaddonphase.EnsureProxyAddon(i.cfg, client); err != nil {
return fmt.Errorf(“error ensuring proxy addon: %v”, err)

筆者批判代碼無腦式的一個流程到底，要是筆者操刀定抽象成接口 RenderConf Save Run Clean等，DNS kube-porxy以及其它元件去實作，然後問題就是沒把dns和kubeproxy的配置渲染出來，可能是它們不是static pod的原因, 然後就是join時的bug下文提到

證書生成

循環的調用了這一坨函數，我們隻需要看其中一兩個即可，其它的都差不多

certActions := []func(cfg *kubeadmapi.MasterConfiguration) error{
CreateCACertAndKeyfiles,
CreateAPIServerCertAndKeyFiles,
CreateAPIServerKubeletClientCertAndKeyFiles,
CreateServiceAccountKeyAndPublicKeyFiles,
CreateFrontProxyCACertAndKeyFiles,
CreateFrontProxyClientCertAndKeyFiles,

根證書生成：

//傳回了根證書的公鑰和私鑰
func NewCACertAndKey() (*x509.Certificate, *rsa.PrivateKey, error) {
caCert, caKey, err := pkiutil.NewCertificateAuthority()
if err != nil {
return nil, nil, fmt.Errorf(“failure while generating CA certificate and key: %v”, err)
return caCert, caKey, nil

k8s.io/client-go/util/cert 這個庫裡面有兩個函數，一個生成key的一個生成cert的：

key, err := certutil.NewPrivateKey()
config := certutil.Config{
CommonName: “kubernetes”,
cert, err := certutil.NewSelfSignedCACert(config, key)

config裡面我們也可以填充一些别的證書資訊：

type Config struct {
CommonName string
Organization []string
AltNames AltNames
Usages []x509.ExtKeyUsage

私鑰就是封裝了rsa庫裡面的函數：

“crypto/rsa”
“crypto/x509”
func NewPrivateKey() (*rsa.PrivateKey, error) {
return rsa.GenerateKey(cryptorand.Reader, rsaKeySize)

自簽證書,是以根證書裡隻有CommonName資訊，Organization相當于沒設定：

func NewSelfSignedCACert(cfg Config, key *rsa.PrivateKey) (*x509.Certificate, error) {
now := time.Now()
tmpl := x509.Certificate{
SerialNumber: new(big.Int).SetInt64(0),
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
NotBefore: now.UTC(),
NotAfter: now.Add(duration365d * 10).UTC(),
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,
IsCA: true,
certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
return nil, err
return x509.ParseCertificate(certDERBytes)

生成好之後把之寫入檔案：

pkiutil.WriteCertAndKey(pkiDir, baseName, cert, key);
certutil.WriteCert(certificatePath, certutil.EncodeCertPEM(cert))

這裡調用了pem庫進行了編碼

encoding/pem
func EncodeCertPEM(cert *x509.Certificate) []byte {
block := pem.Block{
Type: CertificateBlockType,
Bytes: cert.Raw,
return pem.EncodeToMemory(&block)

然後我們看apiserver的證書生成：

caCert, caKey, err := loadCertificateAuthorithy(cfg.CertificatesDir, kubeadmconstants.CACertAndKeyBaseName)
//從根證書生成apiserver證書
apiCert, apiKey, err := NewAPIServerCertAndKey(cfg, caCert, caKey)

這時需要關注AltNames了比較重要，所有需要通路master的位址域名都得加進去，對應配置檔案中apiServerCertSANs字段,其它東西與根證書無差别

CommonName: kubeadmconstants.APIServerCertCommonName,
AltNames: *altNames,
Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},

建立k8s配置檔案

可以看到建立了這些檔案

return createKubeConfigFiles(
outDir,
cfg,
kubeadmconstants.AdminKubeConfigFileName,
kubeadmconstants.KubeletKubeConfigFileName,
kubeadmconstants.ControllerManagerKubeConfigFileName,
kubeadmconstants.SchedulerKubeConfigFileName,
)

k8s封裝了兩個渲染配置的函數：差別是你的kubeconfig檔案裡會不會産生token，比如你進入dashboard需要一個token，或者你調用api需要一個token那麼請生成帶token的配置生成的conf檔案基本一直隻是比如ClientName這些東西不同，是以加密後的證書也不同，ClientName會被加密到證書裡，然後k8s取出來當使用者使用

是以重點來了，我們做多租戶時也要這樣去生成。然後給該租戶綁定角色。

return kubeconfigutil.CreateWithToken(
spec.APIServer,
“kubernetes”,
spec.ClientName,
certutil.EncodeCertPEM(spec.CACert),
spec.TokenAuth.Token,
), nil
return kubeconfigutil.CreateWithCerts(
certutil.EncodePrivateKeyPEM(clientKey),
certutil.EncodeCertPEM(clientCert),

然後就是填充Config結構體喽, 最後寫到檔案裡，略

“k8s.io/client-go/tools/clientcmd/api
return &clientcmdapi.Config{
Clusters: map[string]*clientcmdapi.Cluster{
clusterName: {
Server: serverURL,
CertificateAuthorityData: caCert,
Contexts: map[string]*clientcmdapi.Context{
contextName: {
Cluster: clusterName,
AuthInfo: userName,
AuthInfos: map[string]*clientcmdapi.AuthInfo{},
CurrentContext: contextName,

建立static pod yaml檔案

這裡傳回了apiserver manager scheduler的pod結構體,

specs := GetStaticPodSpecs(cfg, k8sVersion)
staticPodSpecs := map[string]v1.Pod{
kubeadmconstants.KubeAPIServer: staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.KubeAPIServer,
Image: images.GetCoreImage(kubeadmconstants.KubeAPIServer, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage),
Command: getAPIServerCommand(cfg, k8sVersion),
VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeAPIServer)),
LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeAPIServer, int(cfg.API.BindPort), “/healthz”, v1.URISchemeHTTPS),
Resources: staticpodutil.ComponentResources(“250m”),
Env: getProxyEnvVars(),
}, mounts.GetVolumes(kubeadmconstants.KubeAPIServer)),
kubeadmconstants.KubeControllerManager: staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.KubeControllerManager,
Image: images.GetCoreImage(kubeadmconstants.KubeControllerManager, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage),
Command: getControllerManagerCommand(cfg, k8sVersion),
VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeControllerManager)),
LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeControllerManager, 10252, “/healthz”, v1.URISchemeHTTP),
Resources: staticpodutil.ComponentResources(“200m”),
}, mounts.GetVolumes(kubeadmconstants.KubeControllerManager)),
kubeadmconstants.KubeScheduler: staticpodutil.ComponentPod(v1.Container{
Name: kubeadmconstants.KubeScheduler,
Image: images.GetCoreImage(kubeadmconstants.KubeScheduler, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage),
Command: getSchedulerCommand(cfg),
VolumeMounts: staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeScheduler)),
LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeScheduler, 10251, “/healthz”, v1.URISchemeHTTP),
Resources: staticpodutil.ComponentResources(“100m”),
}, mounts.GetVolumes(kubeadmconstants.KubeScheduler)),
//擷取特定版本的鏡像
func GetCoreImage(image, repoPrefix, k8sVersion, overrideImage string) string {
if overrideImage != “” {
return overrideImage
kubernetesImageTag := kubeadmutil.KubernetesVersionToImageTag(k8sVersion)
etcdImageTag := constants.DefaultEtcdVersion
etcdImageVersion, err := constants.EtcdSupportedVersion(k8sVersion)
if err == nil {
etcdImageTag = etcdImageVersion.String()
return map[string]string{
constants.Etcd: fmt.Sprintf(“%s/%s-%s:%s”, repoPrefix, “etcd”, runtime.GOARCH, etcdImageTag),
constants.KubeAPIServer: fmt.Sprintf(“%s/%s-%s:%s”, repoPrefix, “kube-apiserver”, runtime.GOARCH, kubernetesImageTag),
constants.KubeControllerManager: fmt.Sprintf(“%s/%s-%s:%s”, repoPrefix, “kube-controller-manager”, runtime.GOARCH, kubernetesImageTag),
constants.KubeScheduler: fmt.Sprintf(“%s/%s-%s:%s”, repoPrefix, “kube-scheduler”, runtime.GOARCH, kubernetesImageTag),
}[image]
//然後就把這個pod寫到檔案裡了，比較簡單
staticpodutil.WriteStaticPodToDisk(componentName, manifestDir, spec);

建立etcd的一樣，不多廢話

等待kubelet啟動成功

這個錯誤非常容易遇到，看到這個基本就是kubelet沒起來，我們需要檢查：selinux swap 和Cgroup driver是不是一緻 setenforce 0 && swapoff -a && systemctl restart kubelet如果不行請保證 kubelet的Cgroup driver與docker一緻，docker info|grep Cg

go func(errC chan error, waiter apiclient.Waiter) {
// This goroutine can only make kubeadm init fail. If this check succeeds, it won’t do anything special
if err := waiter.WaitForHealthyKubelet(40*time.Second, “http://localhost:10255/healthz”); err != nil {
errC <- err
}(errorChan, waiter)
if err := waiter.WaitForHealthyKubelet(60*time.Second, “http://localhost:10255/healthz/syncloop”); err != nil {

建立DNS和kubeproxy

我就是在此發現coreDNS的

if features.Enabled(cfg.FeatureGates, features.CoreDNS) {
return coreDNSAddon(cfg, client, k8sVersion)
return kubeDNSAddon(cfg, client, k8sVersion)

然後coreDNS的yaml配置模闆直接是寫在代碼裡的： /app/phases/addons/dns/manifests.go

CoreDNSDeployment = `
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
spec:
replicas: 1
selector:
matchLabels:
template:
serviceAccountName: coredns
tolerations:
– key: CriticalAddonsOnly
operator: Exists
– key: {{ .MasterTaintKey }}
…

然後渲染模闆，最後調用k8sapi建立,這種建立方式可以學習一下，雖然有點拙劣，這地方寫的遠不如kubectl好

coreDNSConfigMap := &v1.ConfigMap{}
if err := kuberuntime.DecodeInto(legacyscheme.Codecs.UniversalDecoder(), configBytes, coreDNSConfigMap); err != nil {
return fmt.Errorf(“unable to decode CoreDNS configmap %v”, err)
// Create the ConfigMap for CoreDNS or update it in case it already exists
if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil {
coreDNSClusterRoles := &rbac.ClusterRole{}
if err := kuberuntime.DecodeInto(legacyscheme.Codecs.UniversalDecoder(), []byte(CoreDNSClusterRole), coreDNSClusterRoles); err != nil {
return fmt.Errorf(“unable to decode CoreDNS clusterroles %v”, err)

這裡值得一提的是kubeproxy的configmap真應該把apiserver位址傳入進來，允許自定義，因為做高可用時需要指定虛拟ip，得修改，很麻煩 kubeproxy大差不差，不說了,想改的話改： app/phases/addons/proxy/manifests.go

kubeadm join

kubeadm join比較簡單，一句話就可以說清楚，擷取cluster info, 建立kubeconfig，怎麼建立的kubeinit裡面已經說了。帶上token讓kubeadm有權限可以拉取

return https.RetrieveValidatedClusterInfo(cfg.DiscoveryFile)
cluster info内容
type Cluster struct {
// LocationOfOrigin indicates where this object came from. It is used for round tripping config post-merge, but never serialized.
LocationOfOrigin string
// Server is the address of the kubernetes cluster (https://hostname:port).
Server string `json:”server”`
// InsecureSkipTLSVerify skips the validity check for the server’s certificate. This will make your HTTPS connections insecure.
// +optional
InsecureSkipTLSVerify bool `json:”insecure-skip-tls-verify,omitempty”`
// CertificateAuthority is the path to a cert file for the certificate authority.
CertificateAuthority string `json:”certificate-authority,omitempty”`
// CertificateAuthorityData contains PEM-encoded certificate authority certificates. Overrides CertificateAuthority
CertificateAuthorityData []byte `json:”certificate-authority-data,omitempty”`
// Extensions holds additional information. This is useful for extenders so that reads and writes don’t clobber unknown fields
Extensions map[string]runtime.Object `json:”extensions,omitempty”`
clusterinfo.Server,
TokenUser,
clusterinfo.CertificateAuthorityData,
cfg.TLSBootstrapToken,

CreateWithToken上文提到了不再贅述，這樣就能去生成kubelet配置檔案了，然後把kubelet啟動起來即可

kubeadm join的問題就是渲染配置時沒有使用指令行傳入的apiserver位址，而用clusterinfo裡的位址，這不利于我們做高可用，可能我們傳入一個虛拟ip，但是配置裡還是apiser的位址 +++ author = “fanux” date = “2014-07-11T10:54:24+02:00” draft = false title = “kubeadm源碼分析” tags = [“event”,”dotScale”,”sketchnote”] image = “images/2014/Jul/titledotscale.png” comments = true # set false to hide Disqus comments share = true # set false to share buttons menu = “” # set “main” to add this content to the main menu +++

kubeadm join的問題就是渲染配置時沒有使用指令行傳入的apiserver位址，而用clusterinfo裡的位址，這不利于我們做高可用，可能我們傳入一個虛拟ip，但是配置裡還是apiser的位址

本文轉自kubernetes中文社群-

kubeadm源碼分析（内含kubernetes離線包，三步安裝）

kubeadm源碼分析（内含kubernetes離線包，三步安裝）kubeadm源碼分析