8000 rkt image gc will remove images for *running* containers · Issue #3385 · rkt/rkt · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on Feb 24, 2020. It is now read-only.

rkt image gc will remove images for *running* containers #3385

Closed
dansteen opened this issue Nov 17, 2016 · 4 comments
Closed

rkt image gc will remove images for *running* containers #3385

dansteen opened this issue Nov 17, 2016 · 4 comments
Labels
component/stage0 help wanted kind/bug
Milestone
v1.24.0

Comments

@dansteen
Copy link
dansteen commented Nov 17, 2016 •
edited
Loading

Environment

rkt Version: 1.19.0
appc Version: 0.8.8
Go Version: go1.7.3
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL
--
Linux 4.7.0-0.bpo.1-amd64 x86_64
--
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
--
systemd 230
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

What did you do?

$ sudo rkt image list
ID                      NAME                                                                                    SIZE    IMPORT TIME     LAST USED
sha512-893b2f49db6b     <host>/api:9118b14222f3973806cb762bde7ddde9171cf956   68MiB   43 seconds ago  43 seconds ago
sha512-b7d5fcc132bd     <host>/java:4.0.6                                              407MiB  13 seconds ago  13 seconds ago
sha512-5d072546239b     <host>/base:1.0.11                                    46MiB   5 seconds ago   5
seconds ago
sha512-2222d0a86708     quay.io/coreos/alpine-sh:latest                                                         5.4MiB  3 seconds ago   3
seconds ago

$ sudo rkt run <host>/api:9118b14222f3973806cb762bde7ddde9171cf956 --exec /bin/sleep -- 100

let that run, and in a different shell


$ sudo rkt list
UUID            APP             IMAGE NAME                                                                              STATE   CREATED  STARTED          NETWORKS
f241329e        api    <host>/api:9118b14222f3973806cb762bde7ddde9171cf956   running 5 seconds ago     5 seconds ago   default:ip4=172.16.28.3

$ sudo rkt image gc --grace-period=1s
gc: removed treestore "deps-sha512-d2a299acfa52b59a9968ef52220c5ba72c3921f1d728867624caf41699796449"
successfully removed aci for image: "sha512-893b2f49db6bb3b5e55667375c0f212ed4d4c9d72ba8878ac4820e230e90005e"
successfully removed aci for image: "sha512-b7d5fcc132bdb005766e96a92fb778957aa48840662779c77a4a08c48f9ab38d"
successfully removed aci for image: "sha512-5d072546239b5bb960866e25b2e392fea23c91f6de1cae5ed6f206fbcae4ec11"
successfully removed aci for image: "sha512-2222d0a867084bc1c6f47f86982867dcc013a1bcfab15128a1065108808fc998"
successfully removed aci for image: "sha512-6807fc094d72cb9c60ca0f9b0108366a101a1b8c73236b9898fc8bb9aa0a3554"
gc: 5 image(s) successfully removed

What did you expect to see?

I expected rkt to consider images with still running containers to still be "in use", and to not clean them up.

What did you see instead?

rkt bases its "in use" metric off the last time a container was started off that image and deletes images with still running containers.

As a note, The --grace-period=1s in the commands above is just for testing purposes.
@lucab
Copy link
Member
lucab commented Nov 18, 2016

@dansteen thanks for the report. You are right, the name is definitely misleading as the timestamp is taken from the DB and it is updated at pod starting time. As there is no daemon doing realtime book-keeping for us, image gc/list don't know about currently running pods.

This is not a problem for the treestore, as it knows about running apps from their treestoreid (to be checked: what about non-overlayfs pods?). For the imagestore however the GC only checks the timestamp in DB, which effectively is the last starting time. I think we should do two things here:

  • rename the field as "last started" to reflect the fact that the DB is unaware of running images
  • augment gcStore to take into account images involved in running pods and don't GC them. This seems to be possible via getAppsHashes, but needs to be double-checked.

@lucab lucab added this to the v1.20.0 milestone Nov 18, 2016
This was referenced Nov 18, 2016
Open
Open
@s-urbaniak s-urbaniak 8000 modified the milestones: v1.21.0, v1.20.0 Nov 24, 2016
@squeed squeed modified the milestones: v1.22.0, v1.21.0 Dec 8, 2016
@lucab lucab modified the milestones: v1.23.0, v1.22.0 Jan 5, 2017
@s-urbaniak s-urbaniak modified the milestones: v1.24.0, v1.23.0 Jan 19, 2017
@s-urbaniak
Copy link
Contributor

@dgonyeo do you see a chance of tackling this?

@cgonyeo
Copy link
Member
cgonyeo commented Jan 19, 2017

@s-urbaniak sure.

Some clarification first though:

Does this mean that if I started a container a week ago, and it ran continuously up until it was killed 30 seconds ago, and I run rkt image gc --grace-period=1d, should the image for the container be removed? It was in some form "in use" very recently, but wasn't used to start a container for some time.

Although telling when a container for an image stopped running would be very hard with our architecture, so maybe this is fine.

@lucab
Copy link
Member
lucab commented Jan 19, 2017

I think we should just make explicit that "used" here really means started (see the other two bugs referenced here). The most pressing point here however is adding a check to exclude still-running images from GC at all.

@cgonyeo cgonyeo mentioned this issue Jan 19, 2017
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
component/stage0 help wanted kind/bug
Projects
None yet
Milestone
v1.24.0
Development

No branches or pull requests
5 participants
@dansteen @lucab @s-urbaniak @squeed @cgonyeo
0