1.搭建Kubernetes环境

搭建Kubernetes环境

容器编排

有了docker的经验，我们可以轻松的使用容器实现应用的打包、分发。

但是想要将容器技术应用到大规模生产环境，还有更多的问题。容器解决了打包、分发。但是对于生产环境而言，我们还需要服务发现、负载均衡、扩容缩容等更高级的特性。

这些容器的管理、调度工作，就是容器编排Container Orchestration。

Kubernetes

Kubernetes就是一个生产级别的容器编排平台和集群管理系统。

谷歌很早期（2002年前后）就开始了容器化的相关工作。从一开始谷歌就没有引入虚拟机，因为利用虚拟机隔离程序会造成资源的浪费，谷歌认为操作系统本身就应该提供程序隔离的能力。

谷歌自己有一套容器标准叫做lmctfy（Let Me Contain That For You）。2006年谷歌把一部分技术加入了Linux, 就是我们现在看到的cgroups控制群组（Control Groups，目前常用的简写为 cgroups），它与名称空间一样，都是直接由Linux内核提供的功能，用于隔离或者说分配并限制某个进程组能够使用的资源配额。

cgroups 项目最早是由Google的工程师在2006年发起的，当时取的名字就叫做“进程容器”（Process Containers），不过“容器”（Container）这个名词的定义在那时候还没有今天那么清晰，不同场景中常有不同的指向。到2007年这个项目才被重新命名为cgroups，在2008年1月合并到了2.6.24版的内核后正式对外发布。2008年8月Linux推出了自己的container LXC = namespace + cgroups。早期的docker就是依赖于lxc技术

Google自己的容器编排系统叫做Borg。2008年，Google Cloud发展初期，google曾经想将容器技术推向市场，发布了Google App Engine，支持自动监控、serverless、自动扩容/缩容等云原生特性。但是市场并不接受。

直到2015年，业内普遍接受了docker等容器工具，Google才推出了开源版的Borg - Kubernetes系统。仅仅两年就打败了Docker Swarm，成为了容器编排领域的事实标准。

kubeadm

安装


cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF


cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

sudo apt install apt-transport-https ca-certificates curl

sudo curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main
EOF

# install Kubernetes
sudo apt install kubeadm kubelet kubectl kubernetes-cni
# kubeadm is already the newest version (1.26.1-00).
# kubectl is already the newest version (1.26.1-00).
# kubelet is already the newest version (1.26.1-00).

kubeadm init

# kubeadm init
kubeadm init --apiserver-advertise-address $(hostname -i) --pod-network-cidr 10.5.0.0/16

Initializing machine ID from random generator.
I0201 07:03:44.227509    4579 version.go:251] remote version is much newer: v1.26.1; falling back to: stable-1.20
[init] Using Kubernetes version: v1.20.15
[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-210-generic
DOCKER_VERSION: 20.10.1
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03
        [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "", err: exit status 1
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node3] and IPs [10.96.0.1 192.168.0.16]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node3] and IPs [192.168.0.16 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node3] and IPs [192.168.0.16 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 58.522724 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node3 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node node3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: xki0su.skku36yc82040guc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.16:6443 --token xki0su.skku36yc82040guc \
    --discovery-token-ca-cert-hash sha256:1a91dc6db669ceaed8e077855aeb9e2c06662a206ffb64000a38cb822246563c 
Waiting for api server to startup
Warning: resource daemonsets/kube-proxy is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
daemonset.apps/kube-proxy configured
pod "kube-proxy-clpn2" deleted
pod "kube-proxy-g2ssm" deleted

1. preflight

执行 kubeadm init指令后，kubeadm 首先要做的Preflight Checks检查，确定这台机器可以用来部署 Kubernetes:

Linux内核的版本必须是否是 3.10 以上？
Docker版本
Linux Cgroups模块启用
kubeadm和kubelet的版本
Kubernetes安装 / 工作端口
…

[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-210-generic
DOCKER_VERSION: 20.10.1
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03
        [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "", err: exit status 1
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

2. certs

通过Preflight Checks之后，kubeadm将生成 Kubernetes 对外提供服务所需的证书和对应的目录

[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node3] and IPs [10.96.0.1 192.168.0.16]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node3] and IPs [192.168.0.16 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node3] and IPs [192.168.0.16 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key

3. kubeconfig

kubeadm 接下来会为其他组件生成访问 kube-apiserver 所需的配置文件,文件路径为：/etc/kubernetes/xxx.conf。记录了当前节点的服务器地址、监听端口、证书目录等信息。

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

4. kubelet-start

启动kubelet

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet

5. control-plane

kubeadm为control-plane组件生成Pod配置文件，以Static Pod的方式部署kube-apiserver、kube-controller-manager、kube-scheduler三个组件

[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"

6. etcd

kubeadm 生成Etcd的Pod YAML 文件，以Static Pod的方式启动Etcd

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

7. check

kubeadm检查localhost:6443/healthz，等待control-plane组件完全运行

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 58.522724 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node3 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node node3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

8. bootstrap-token

kubeadm为集群生成bootstrap token。任何持有token的节点都可以通过 kubeadm join 加入集群（需要安装了kubelet和kubadm）。

[bootstrap-token] Using token: xki0su.skku36yc82040guc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

在token生成之后，kubeadm在etcd中创建一个cluster-info的ConfigMap，保存ca.crt证书等信息

9. addons

安装默认插件。Kubernetes默认kube-proxy和DNS这两个插件是必须安装的，分别提供整个集群的服务发现和DNS功能。

[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

10.initialized successfully

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.16:6443 --token xki0su.skku36yc82040guc \
    --discovery-token-ca-cert-hash sha256:1a91dc6db669ceaed8e077855aeb9e2c06662a206ffb64000a38cb822246563c 
Waiting for api server to startup
Warning: resource daemonsets/kube-proxy is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
daemonset.apps/kube-proxy configured
pod "kube-proxy-clpn2" deleted
pod "kube-proxy-g2ssm" deleted

kubeadm join

kubeadm init 生成 bootstrap token 之后，在任意安装了 kubelet 和 kubeadm 的机器上执行 kubeadm join

kubeadm join 192.168.0.16:6443 --token xki0su.skku36yc82040guc \
    --discovery-token-ca-cert-hash sha256:1a91dc6db669ceaed8e077855aeb9e2c06662a206ffb64000a38cb822246563c

Initializing machine ID from random generator.
[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.4.0-210-generic
DOCKER_VERSION: 20.10.1
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03
        [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "", err: exit status 1
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

kubeadm首次访问kube-apiserver没有加密(signing request)。
为了拿到保存在ConfigMap中的 cluster-info需要bootstrap token作为验证。
拿到cluster-info里的kube-apiserver的地址、端口、证书之后，kubelet以secure模式连接到apiserver，节点加入集群。

minkube

完整的kubernetes一般都运行在大规模的计算集群上。Google推荐的学习kubernetes的方法是通过minukube。

minikube是一个迷你版的Kubernetes，集成了Kubernetes的绝大多数功能特性，适合初学者。

下载安装minikube

访问 https://minikube.sigs.k8s.io/docs/start/，下载安装对应的版本

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

minikube version
# minikube version: v1.29.0
# commit: ddac20b4b34a9c8c857fc602203b6ba2679794d3

minikube kubectl
# kubectl controls the Kubernetes cluster manager.

minikube start
# Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

minikube status
# minikube
# type: Control Plane
# host: Running
# kubelet: Running
# apiserver: Running
# kubeconfig: Configured

minikube node list
# minikube        192.168.49.2

kubectl

alias kubectl="minikube kubectl --"
source <(kubectl completion bash)

echo 'alias kubectl="minikube kubectl --"' >> ~/.bashrc
echo 'source <(kubectl completion bash)' >> ~/.bashrc

minikube用于管理本地Kubernetes集群
kubectl操作Kubernetes

kubectl version --short 
# Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
# Client Version: v1.26.1
# Kustomize Version: v4.5.7
# Server Version: v1.26.1

kubectl get nodes
# NAME       STATUS   ROLES           AGE     VERSION
# minikube   Ready    control-plane   8m29s   v1.26.1

当前只有一个控制平面节点(Master节点)

kubernetes 架构

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

控制平面节点里有4个组件，分别是apiserver、etcd、scheduler、controller-manager (cloud-controller-manager / kube-controller-manager)。

apiserver是Kubernetes系统的唯一入口，它对外公开了一系列的 RESTful API，负责处理接受请求的工作。
etcd 是一个高可用的分布式键值存储数据库，用来持久化存储系统里的各种资源对象和状态。
scheduler负责容器的编排工作，检查节点的资源状态，把 Pod 调度到最适合的节点上运行：监视新创建的、未指定运行节点（node）的 Pods，并选择节点来让Pod在上面运行
controller-manager 负责维护容器和节点等资源的状态。

查看组件状态，通过-n 指定名字空间

kubectl get pod -n kube-system
# NAME                               READY   STATUS    RESTARTS      AGE
# coredns-787d4945fb-cb6bs           1/1     Running   0             61m
# etcd-minikube                      1/1     Running   0             62m
# kube-apiserver-minikube            1/1     Running   0             62m
# kube-controller-manager-minikube   1/1     Running   0             62m
# kube-proxy-tfzx2                   1/1     Running   0             61m
# kube-scheduler-minikube            1/1     Running   0             62m
# storage-provisioner                1/1     Running   1 (61m ago)   62m

Worker Node里有3个组件：kubelet、kube-proxy、container-runtime

kubelet 代理，负责Node相关的操作。会在集群中每个节点（node）上运行。它保证容器（containers）都运行在 Pod 中。
kube-proxy网络代理，负责管理容器的网络通信。
container-runtime容器运行环境。

查看组件状态

minikube ssh
docker ps | grep kube-proxy

ps -ef | grep kubelet

除了组件之外，kubernetes还支持插件Addons：

DNS：几乎所有 Kubernetes 集群都应该有集群DNS。集群DNS是一个DNS服务器，和环境中的其他DNS服务器一起工作，它为Kubernetes服务提供DNS记录。
Dashboard：基于Web的用户界面。它使用户可以管理集群中运行的应用程序以及集群本身，并进行故障排除。
Container Resource Monitoring 容器资源监控：将关于容器的一些常见的时间序列度量值保存到一个集中的数据库中，并提供浏览这些数据的界面。
Cluster-level Logging 集群日志：将容器的日志数据保存到一个集中的日志存储中

minikube addons list

#|-----------------------------|----------|--------------|--------------------------------|
#| ADDON NAME                    | PROFILE    | STATUS         | MAINTAINER                       |
#| ----------------------------- | ---------- | -------------- | -------------------------------- |
#| ambassador                    | minikube   | disabled       | 3rd party (Ambassador)           |
#| auto-pause                    | minikube   | disabled       | Google                           |
#| cloud-spanner                 | minikube   | disabled       | Google                           |
#| csi-hostpath-driver           | minikube   | disabled       | Kubernetes                       |
#| dashboard                     | minikube   | enabled ✅      | Kubernetes                       |
#| default-storageclass          | minikube   | enabled ✅      | Kubernetes                       |
#| efk                           | minikube   | disabled       | 3rd party (Elastic)              |
#| freshpod                      | minikube   | disabled       | Google                           |
#| gcp-auth                      | minikube   | disabled       | Google                           |
#| gvisor                        | minikube   | disabled       | Google                           |
#| headlamp                      | minikube   | disabled       | 3rd party (kinvolk.io)           |
#| helm-tiller                   | minikube   | disabled       | 3rd party (Helm)                 |
#| inaccel                       | minikube   | disabled       | 3rd party (InAccel               |
#|                               |            |                | [info@inaccel.com])              |
#| ingress                       | minikube   | disabled       | Kubernetes                       |
#| ingress-dns                   | minikube   | disabled       | Google                           |
#| istio                         | minikube   | disabled       | 3rd party (Istio)                |
#| istio-provisioner             | minikube   | disabled       | 3rd party (Istio)                |
#| kong                          | minikube   | disabled       | 3rd party (Kong HQ)              |
#| kubevirt                      | minikube   | disabled       | 3rd party (KubeVirt)             |
#| logviewer                     | minikube   | disabled       | 3rd party (unknown)              |
#| metallb                       | minikube   | disabled       | 3rd party (MetalLB)              |
#| metrics-server                | minikube   | disabled       | Kubernetes                       |
#| nvidia-driver-installer       | minikube   | disabled       | Google                           |
#| nvidia-gpu-device-plugin      | minikube   | disabled       | 3rd party (Nvidia)               |
#| olm                           | minikube   | disabled       | 3rd party (Operator Framework)   |
#| pod-security-policy           | minikube   | disabled       | 3rd party (unknown)              |
#| portainer                     | minikube   | disabled       | 3rd party (Portainer.io)         |
#| registry                      | minikube   | disabled       | Google                           |
#| registry-aliases              | minikube   | disabled       | 3rd party (unknown)              |
#| registry-creds                | minikube   | disabled       | 3rd party (UPMC Enterprises)     |
#| storage-provisioner           | minikube   | enabled ✅      | Google                           |
#| storage-provisioner-gluster   | minikube   | disabled       | 3rd party (Gluster)              |
#| volumesnapshots               | minikube   | disabled       | Kubernetes                       |
#| ----------------------------- | ---------- | -------------- | -------------------------------- |