LJTian Blog
LJTian Blog/
🛻
云原生/
🎑
minikube bug fix
Search
minikube bug fix
🎑

minikube bug fix

问题

原文

Minikube delete can quietly leave most data on the host on linux when the driver is docker #[15222](https://github.com/kubernetes/minikube/issues/15222)

内容：

What Happened?
发生了什么？
Observations
观察
Without carefully matched file permissions minikube delete quietly leaves most data on the host on linux when the driver is docker.
当驱动程序是 docker 时，如果没有仔细匹配文件权限，minikube delete 会悄悄地将大部分数据留在 linux 主机上。
 
When running as a non root user on linux with the driver set to docker minikube start is successful and most data is stored under the folder /var/lib/docker/volumes/
当在 linux 上以非 root 用户身份运行且驱动程序设置为 docker minikube 时，启动成功并且大部分数据存储在文件夹 /var/lib/docker/volumes/ 下
 
When minikube delete is called however there appears to be a permissions issue that can result in the silent failure to delete most data.
然而，当调用 minikube delete 时，似乎存在权限问题，可能导致无法删除大部分数据。
When this happens all this data is left intact (full data of containerized host - so all images/volumes etc).
当这种情况发生时，所有这些数据都保持不变（容器化主机的完整数据 - 所以所有图像/卷等）。
 
I believe the issue occurs when there is a mismatch between the user that originally called start (and thus created the /var/lib/docker/volumes/minikube folder) and the user that calls delete
我相信当最初调用 start 的用户（并因此创建了 /var/lib/docker/volumes/minikube 文件夹）和调用 delete 的用户之间存在不匹配时会出现问题
 
Running with sudo does delete the remaining data but this has the side effect that purge then looks in the wrong home folder (root) rather than the current user for config data to clean
使用 sudo 运行确实会删除剩余的数据，但这会产生副作用，即清除然后会在错误的主文件夹（root）而不是当前用户中查找要清除的配置数据
 
Note that in the successful case the user is additionally reminded that (the relatively small) "Kicbase images have not been deleted".
请注意，在成功的情况下，还会额外提醒用户（相对较小的）“Kicbase 图像未被删除”。
 
Expectations
期望
Expect either for data to be deleted or for information to be displayed informing the user that most data could not be deleted.
期望要么删除数据，要么显示信息通知用户大多数数据无法删除。
To me the biggest issue is this silent failure, this can lead to a system's hard drive filling up and the system becoming unresponsive.
对我来说，最大的问题是这种静默故障，这会导致系统的硬盘驱动器被填满并且系统变得无响应。
 
Expect not to have to call delete twice (once with and once without sudo) to delete and purge all data.
预计不必调用 delete 两次（一次使用 sudo，一次不使用 sudo）来删除和清除所有数据。
 
This was seen on a CENTOS 7 system.
这是在 CENTOS 7 系统上看到的。
 
Some anonymised logs - first we list the data, then try minikube delete. Notice that some 30GB of data remains under /var/lib/docker/volumes/ and yet no problems are reported. Then I re-run using sudo. This time 30GB is removed. Note that a different home folder is then purged.
一些匿名日志——首先我们列出数据，然后尝试删除 minikube。请注意，大约 30GB 的数据仍保留在 /var/lib/docker/volumes/ 下，但未报告任何问题。然后我使用 sudo 重新运行。这次删除了 30GB。请注意，随后会清除一个不同的主文件夹。
 
[myuseraccount@myserver myfolder]$ sudo du -h / | sort -h -r | head -n 20 53G / 37G /var 34G /var/lib/docker 34G /var/lib 29G /var/lib/docker/volumes/minikube/_data/lib 29G /var/lib/docker/volumes/minikube/_data 29G /var/lib/docker/volumes/minikube 29G /var/lib/docker/volumes 28G /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2 28G /var/lib/docker/volumes/minikube/_data/lib/docker 10G /data/jenkins 10G /data 9.9G /data/jenkins/workspace 4.7G /var/lib/docker/overlay2 2.8G /usr 2.0G /home ...
[myuseraccount@myserver myfolder]$ ./minikube delete --all --purge fire Successfully deleted all profiles skull Successfully purged minikube directory located at - [/home/myuseraccount@MYDOMAIN/.minikube]
[myuseraccount@myserver myfolder]$ sudo du -h / | sort -h -r | head -n 20 53G / 37G /var 34G /var/lib/docker 34G /var/lib 29G /var/lib/docker/volumes/minikube/_data/lib 29G /var/lib/docker/volumes/minikube/_data 29G /var/lib/docker/volumes/minikube 29G /var/lib/docker/volumes 28G /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2 28G /var/lib/docker/volumes/minikube/_data/lib/docker 10G /data/jenkins 10G /data 9.9G /data/jenkins/workspace 4.7G /var/lib/docker/overlay2 2.8G /usr 2.0G /home ...
[myuseraccount@myserver myfolder]$ sudo ./minikube delete --all --purge fire Successfully deleted all profiles skull Successfully purged minikube directory located at - [/root/.minikube] pushpin Kicbase images have not been deleted. To delete images run: black_small_square docker rmi gcr.io/k8s-minikube/kicbase:v0.0.34
[myuseraccount@myserver myfolder]$ sudo du -h / | sort -h -r | head -n 20 23G / 10G /data/jenkins 10G /data 9.9G /data/jenkins/workspace 6.8G /var 3.9G /var/lib 3.7G /var/lib/docker 3.6G /var/lib/docker/overlay2 2.8G /usr 2.0G /home ...
Attach the log file nologfile.txt
Operating System No response
Driver No response
 

问题复现

复现过程

目录

  • 1、安装 centos 7 系统
  • 2、安装 docker-ce 容器驱动
  • 3、 创建 user1、user2、user3
  • 4、 给所有用户安装 minikube 执行程序
  • 5、 user1 创建 minikube 集群，并查看生成的资源文件及所属者, user1 执行 delete 操作，查看结果。
  • 6、 user1 创建 minikube 集群，并查看生成的资源文件及所属者, user2 执行 delete 操作，查看结果。(user2 在 docker 组中)
  • 7、user1 创建 minikube 集群，并查看生成的资源文件及所属者, user3 执行 delete 操作，查看结果。(user3 不在 docker 组中)
  • 8、上接7，user3 使用 sudo 执行 delete 操作，查看结果。

过程

1、安装 centos 7 系统
|→使用 livrt-manager 安装的 centos7 系统(最小安装)
[root@localhost user2]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
2、安装 docker-ce 容器驱动
|→采用 aliyun 的 yum 源进行的安装
[root@localhost user2]# cat /etc/yum.repos.d/aliyun.repo 
[aliyum.reop]
name=aliyum.repo
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/
gpgcheck=0
enabled=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[root@localhost user2]# yum install -y docker-ce-20.10.9-3.el7
[root@localhost user2]# systemctl start docker
[root@localhost user2]# systemctl enable docker

[root@localhost user2]# docker version
Client: Docker Engine - Community
 Version:           23.0.4
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        f480fb1
 Built:             Fri Apr 14 10:36:38 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.4
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       cbce331
  Built:            Fri Apr 14 10:34:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
3、 创建 user1、user2、user3
|→ 创建用户并添加 sudo 权限
[root@localhost root]# useradd user1 -d /home/user1
[root@localhost root]# useradd user2 -d /home/user2
[root@localhost root]# useradd user3 -d /home/user3
[root@localhost root]# passwd user1
[root@localhost root]# passwd user2
[root@localhost root]# passwd user3
[root@localhost root]# vi /etc/sudoers
...
## Allow root to run any commands anywhere 
root    ALL=(ALL)       ALL
user1   ALL=(ALL)       ALL
user2   ALL=(ALL)       ALL
user3   ALL=(ALL)       ALL
...
4、 给所有用户安装 minikube 执行程序
|→ 注意下面的用户切换
[root@localhost ~]$curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
[root@localhost ~]$install minikube-linux-amd64 /usr/local/bin/minikube
[user1@localhost ~]$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0
5、 user1 创建 minikube 集群，并查看生成的资源文件及所属者, user1 执行 delete 操作，查看结果。
|→ 感觉 iss 描述的不是这种情况
[user1@localhost ~]$sudo usermod -aG docker $USER && newgrp docker
[user1@localhost ~]$ minikube start
ð  Centos 7.9.2009 (kvm/amd64) 上的 minikube v1.30.1
✨  自动选择 docker 驱动。其他选项：none, ssh
ð  Using Docker driver with root privileges
ð  Starting control plane node minikube in cluster minikube
ð  Pulling base image ...
ð¾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  122.02 MiB / 397.02 MiB  30.73% 1.26 MiB❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.39, but successfully downloaded docker.io/kicbase/stable:v0.0.39 as a fallback image
    > preloaded-images-k8s-v18-v1...:  397.02 MiB / 397.02 MiB  100.00% 1.40 Mi
ð¥  Creating docker container (CPUs=2, Memory=2200MB) ...
ð³  正在 Docker 23.0.2 中准备 Kubernetes v1.26.3…
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ 配置 RBAC 规则 ...
ð  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
ð  Verifying Kubernetes components...
ð  Enabled addons: default-storageclass, storage-provisioner
ð¡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
ð  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[user1@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
7.0G    /
3.5G    /var
3.2G    /var/lib
3.1G    /var/lib/docker
2.0G    /var/lib/docker/overlay2
1.7G    /usr
1.7G    /home
1.1G    /var/lib/docker/volumes/minikube/_data/lib
1.1G    /var/lib/docker/volumes/minikube/_data
1.1G    /var/lib/docker/volumes/minikube
1.1G    /var/lib/docker/volumes
1.1G    /var/lib/docker/overlay2/10e984276cb8535a4842eeb597dd696c1e328821d8faec37e36a3ec4a809c959
1022M   /var/lib/docker/overlay2/10e984276cb8535a4842eeb597dd696c1e328821d8faec37e36a3ec4a809c959/merged
960M    /var/lib/docker/overlay2/10e984276cb8535a4842eeb597dd696c1e328821d8faec37e36a3ec4a809c959/merged/usr
730M    /var/lib/docker/volumes/minikube/_data/lib/docker
728M    /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2
695M    /usr/lib
612M    /var/lib/docker/overlay2/10e984276cb8535a4842eeb597dd696c1e328821d8faec37e36a3ec4a809c959/merged/usr/bin
587M    /home/user1

[user1@localhost ~]$ minikube delete --all --purge
ð¥  正在删除 docker 中的“minikube”…
ð¥  正在移除 /home/user1/.minikube/machines/minikube…
ð  Removed all traces of the "minikube" cluster.
ð¥  成功删除所有配置文件
ð  成功清理 [/home/user1/.minikube] 下的 minukube 目录
ð  Kicbase images have not been deleted. To delete images run:
    ▪ docker rmi docker.io/kicbase/stable:v0.0.39

[user1@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
4.4G    /
1.7G    /usr
1.5G    /var
1.2G    /home
1.1G    /var/lib/docker
1.1G    /var/lib
1023M   /var/lib/docker/overlay2
695M    /usr/lib
558M    /home/minikube
512M    /usr/lib/firmware
结论：可以释放相关资源，符合期望。
6、 user1 创建 minikube 集群，并查看生成的资源文件及所属者, user2 执行 delete 操作，查看结果。(user2 在 docker 组中)
|→ 注意下面的用户切换，并执行了”$sudo usermod -aG docker $USER && newgrp docker”
[user1@localhost ~]$ minikube start
ð  Centos 7.9.2009 (kvm/amd64) 上的 minikube v1.30.1
✨  自动选择 docker 驱动
ð  Using Docker driver with root privileges
ð  Starting control plane node minikube in cluster minikube
ð  Pulling base image ...
ð¾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  95.18 MiB / 397.02 MiB  23.97% 1.20 MiB ❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.39, but successfully downloaded docker.io/kicbase/stable:v0.0.39 as a fallback image
    > preloaded-images-k8s-v18-v1...:  397.02 MiB / 397.02 MiB  100.00% 1.19 Mi
ð¥  Creating docker container (CPUs=2, Memory=2200MB) ...
ð³  正在 Docker 23.0.2 中准备 Kubernetes v1.26.3…
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ 配置 RBAC 规则 ...
ð  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
ð  Verifying Kubernetes components...
ð  Enabled addons: default-storageclass, storage-provisioner
ð¡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
ð  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[user1@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
6.9G    /
3.5G    /var
3.2G    /var/lib
3.1G    /var/lib/docker
2.0G    /var/lib/docker/overlay2
1.7G    /usr
1.6G    /home
1.1G    /var/lib/docker/volumes/minikube/_data/lib
1.1G    /var/lib/docker/volumes/minikube/_data
1.1G    /var/lib/docker/volumes/minikube
1.1G    /var/lib/docker/volumes
1.1G    /var/lib/docker/overlay2/fcb52a95991042935ac7c940b87cbb523c8766b6e2deb6eeb7f9aaa1b29112ca
1022M   /var/lib/docker/overlay2/fcb52a95991042935ac7c940b87cbb523c8766b6e2deb6eeb7f9aaa1b29112ca/merged
960M    /var/lib/docker/overlay2/fcb52a95991042935ac7c940b87cbb523c8766b6e2deb6eeb7f9aaa1b29112ca/merged/usr
730M    /var/lib/docker/volumes/minikube/_data/lib/docker
728M    /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2
695M    /usr/lib
612M    /var/lib/docker/overlay2/fcb52a95991042935ac7c940b87cbb523c8766b6e2deb6eeb7f9aaa1b29112ca/merged/usr/bin
558M    /home/minikube
512M    /usr/lib/firmware
[user2@localhost ~]$sudo usermod -aG docker $USER && newgrp docker
[user2@localhost ~]$ minikube delete --all --purge
ð¥  成功删除所有配置文件
ð  成功清理 [/home/user2/.minikube] 下的 minukube 目录
ð  Kicbase images have not been deleted. To delete images run:
    ▪ docker rmi docker.io/kicbase/stable:v0.0.39
[user2@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
4.8G    /
1.7G    /usr
1.6G    /home
1.5G    /var
1.1G    /var/lib/docker
1.1G    /var/lib
1023M   /var/lib/docker/overlay2
695M    /usr/lib
558M    /home/minikube
512M    /usr/lib/firmware
478M    /home/user1
结论：可以释放相关资源，符合期望。
7、 user1 创建 minikube 集群，并查看生成的资源文件及所属者, user3 执行 delete 操作，查看结果。(user3 不在 docker 组中)
|→ 注意下面的用户切换，未执行了”$sudo usermod -aG docker $USER && newgrp docker”,感觉 iss 描述的是这种情况。
[user1@localhost ~]$ minikube start
ð  Centos 7.9.2009 (kvm/amd64) 上的 minikube v1.30.1
✨  根据现有的配置文件使用 docker 驱动程序
ð  Starting control plane node minikube in cluster minikube
ð  Pulling base image ...
𤷠 docker "minikube" container is missing, will recreate.
ð¥  Creating docker container (CPUs=2, Memory=2200MB) ...
ð³  正在 Docker 23.0.2 中准备 Kubernetes v1.26.3…
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ 配置 RBAC 规则 ...
ð  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
ð  Verifying Kubernetes components...
ð  Enabled addons: storage-provisioner, default-storageclass
ð¡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
ð  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[user1@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
6.9G    /
3.5G    /var
3.2G    /var/lib
3.1G    /var/lib/docker
2.0G    /var/lib/docker/overlay2
1.7G    /usr
1.6G    /home
1.1G    /var/lib/docker/volumes/minikube/_data/lib
1.1G    /var/lib/docker/volumes/minikube/_data
1.1G    /var/lib/docker/volumes/minikube
1.1G    /var/lib/docker/volumes
1.1G    /var/lib/docker/overlay2/dd63b7c843a5182b9c1cc01f09adfbdb025f0a23126e988c93d339b6746a0cd3
1022M   /var/lib/docker/overlay2/dd63b7c843a5182b9c1cc01f09adfbdb025f0a23126e988c93d339b6746a0cd3/merged
960M    /var/lib/docker/overlay2/dd63b7c843a5182b9c1cc01f09adfbdb025f0a23126e988c93d339b6746a0cd3/merged/usr
730M    /var/lib/docker/volumes/minikube/_data/lib/docker
728M    /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2
695M    /usr/lib
612M    /var/lib/docker/overlay2/dd63b7c843a5182b9c1cc01f09adfbdb025f0a23126e988c93d339b6746a0cd3/merged/usr/bin
558M    /home/minikube
512M    /usr/lib/firmware
[user3@localhost ~]$ minikube delete --all --purge
ð¥  成功删除所有配置文件
ð  成功清理 [/home/user3/.minikube] 下的 minukube 目录
[user3@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
6.5G    /
3.5G    /var
3.2G    /var/lib
3.1G    /var/lib/docker
2.0G    /var/lib/docker/overlay2
1.7G    /usr
1.2G    /home
1.1G    /var/lib/docker/volumes/minikube/_data/lib
1.1G    /var/lib/docker/volumes/minikube/_data
1.1G    /var/lib/docker/volumes/minikube
1.1G    /var/lib/docker/volumes
1.1G    /var/lib/docker/overlay2/cfa2e453a6a7a8940b3da8c7880776f55bac43435e2ed66664be4a6dfe1247f1
1022M   /var/lib/docker/overlay2/cfa2e453a6a7a8940b3da8c7880776f55bac43435e2ed66664be4a6dfe1247f1/merged
960M    /var/lib/docker/overlay2/cfa2e453a6a7a8940b3da8c7880776f55bac43435e2ed66664be4a6dfe1247f1/merged/usr
730M    /var/lib/docker/volumes/minikube/_data/lib/docker
728M    /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2
695M    /usr/lib
612M    /var/lib/docker/overlay2/cfa2e453a6a7a8940b3da8c7880776f55bac43435e2ed66664be4a6dfe1247f1/merged/usr/bin
512M    /usr/lib/firmware
478M    /home/user1
结论：不可以释放资源，user3 没有 docker 运行权限，无法执行资源释放，与 iss 描述的内容相似，需要考虑是否合理，可以在执行前判断 docker 运行权限进行警示或者中断。
8、上接7，user3 使用 sudo 执行 delete 操作，查看结果。
|→ 注意下面的用户切换
[user3@localhost ~]$ cp /usr/local/bin/minikube .
[user3@localhost ~]$ sudo ./minikube delete --all --purge
ð¥  成功删除所有配置文件
ð  成功清理 [/root/.minikube] 下的 minukube 目录
ð  Kicbase images have not been deleted. To delete images run:
    ▪ docker rmi docker.io/kicbase/stable:v0.0.39
[user3@localhost ~]$ sudo du -h / | sort -h -r | head -n 20
4.5G    /
1.7G    /usr
1.5G    /var
1.3G    /home
1.1G    /var/lib/docker
1.1G    /var/lib
1023M   /var/lib/docker/overlay2
695M    /usr/lib
512M    /usr/lib/firmware
478M    /home/user1
结论: 实验 7 和 8 符合 iss 问题描述

结论

有docker权限的用户可以释放minikube集群资源，没有docker权限的用户会出现iss描述的情况。 可以考虑加入docker进行权限验证，对后续操作进行拦截或提示。
Users with docker permission can release minikube cluster resources, and users without docker permission will have the situation described by iss. You can consider adding docker to perform permission verification, and intercept or prompt for subsequent operations.
 
Copyright 2022 Travis Fischer