Multi stage build leaves "<none>" images behind #34151
Comments
|
This is not a bug; when using multi-stage builds, each stage produces an image. That image is stored in the local image cache, and will be used on subsequent builds (as part of the caching mechanism). You can run each build-stage (and/or tag the stage, if desired). In addition, if you want to tag an intermediate stage as final image, set the If these intermediate images would be purged/pruned automatically, the build cache would be gone with each build, therefore forcing you to rebuild the entire image each time. I'm closing this issue because this is not a bug, but feel free to continue the conversation or if more information is needed. |
|
Do you think its a good idea that give a switch to us that we can decide whether leave these intermediate images? |
|
@thaJeztah |
|
@lukastosic @wrfly |
|
So there is no way to use prune and do not delete those intermediate images after a |
|
@albertom95 can you elaborate? Do you mean: run |
|
@thaJeztah Yes, that prune:
Cause I have a script that runs every 1h or so, and that script does |
|
I'd love to see the switch added to auto-clean the intermediate build artifacts. We've got a bunch of automated builds that occur (always from source for unimportant reasons). We use multi-stage to build from source and then create a much smaller runtime image. The build image is huge in comparison to the runtime. So my top level build scripts always run this command after my
Cheers! |
|
@satchm0h @thaJeztah @albertorm95 After having several node (Angular) projects I fully appreciate keeping intermediate images :) For example we do multi-stage build for angular like this: Resulting final Now the intermediate image "saves the day" because usually for long stretches (2-3 sprints) we don't update It will repeat this intermediate image only when we change So, very useful indeed, but very annoying because we can't control when we want it and when we don't. Also after several sprints we will have several "old" intermediate images that will never be used again, and then we either remove them manually, or we do |
|
@thaJeztah, thanks for hinting I'm thinking about new |
|
Hi. The problem with having intermediate build images as dangling images is that whenever someone runs e.g. If would be nice to be able to tell docker to either REMOVE the intermediate images or TAG them. Let's say we have a following We could use different flags provided to docker with following results:
|
|
When using the new buildkit-based builder (currently opt-in, but can be either enabled as a daemon configuration, or using Buildkit also features (configurable) garbage collecting for the build-cache, which allows for a much more flexible handling of the cache (see #37846). I don't think intermediate stages should automatically be tagged, as in many cases those intermediate stages are not used as actual images, however, having a way to build multiple targets in a single build may be something to think about. Something like but perhaps alternative syntaxes could be thought of, also in light of buildkit supporting different output-formats than just docker images |
|
The following workaround can help removing just temporary stages:
e.g. and |
|
@dluc Thanks for the workaround. It is very disappointing that there is not something builtin to the build command. Multistage builds are great and help with quite a few use cases. One of those use cases is dealing with security where files need to be omitted from the final image. Another is the Angular case discussed (above) where intermediates need to stay around. Both of these cases seem quite valid, and I happen to use both. I don't understand the reluctance to add CLI that lets the user control the lifecycle of the intermediate image. From stackoverflow and several other threads, its obvious that people are having to create workaround like @dluc posted. Why not solve the problem properly? |
|
What annoys me is |
|
This won't be fixed for the classic builder; I'd recommend trying the next generation builder (BuildKit), which is still opt-in (tracking issue for making it the default is #40379) The easiest way to enable buildkit is to set the BuildKit build-cache that's separate from the image store (and can be cleaned up separately using |
However, there is a new question. If I change the process of building intermediate images or some dependent files had changed, many new "none" images will be generated at this time. These redundant none images need to be cleaned up. |
|
|
|
the problem I'm having is every time I run am I missing something? edit: what I want is being able to delete all old and unused intermediate images, from previous builds but keeping the most recent one for caching. |
|
@HeCorr no, there's no "one step" solution to only keeping the "last" intermediate steps when using the classic builder if possible, I would recommend building with buildlkit enabled ( |
|
@thaJeztah okay, I'll give that a try. thanks ;) I actually came up with a dirty trick that works sometimes, which is putting |
|
@thaJeztah |
|
actually.... upon further testing I concluded that buildkit presents the same space inefficiency.. here I used |
|
@HeCorr did you run |
I did but it, well.. cleans the cache, which is not what I want. I wanted it to only keep necessary stuff, so there is a cache, but it doesn't grow like 2GBs on each build.. |
|
You can try the "--keep-storage" option to keep a certain amount of cache around; https://docs.docker.com/engine/reference/commandline/builder_prune/
… On 1 Dec 2020, at 20:08, NinoM4ster ***@***.***> wrote:
@HeCorr did you run docker builder prune to cleanup the build-cache?
I did but it, well.. cleans the cache, which is not what I want.
I wanted it to only keep necessary stuff, so there is a cache, but it doesn't grow like 2GBs on each build..
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
oops.. wrong account. anyway, running I'm honestly starting to reconsider using Docker on my projects... |
|
@HeCorr What's the output of |
|
@tonistiigi I updated First build
Second build
Third build
|
|
I think I see the confusion here. So there's multiple reasons why a You initially mentioned that each FROM alpine
RUN apk add --no-cache make git
WORKDIR /src
COPY ./some/files/. /src
RUN cat one two three > fourIf you build the above Dockerfile with the classic builder, then each step is commited to a new image, without giving it a name DOCKER_BUILDKIT=0 docker build -t myname/myimage:latest .
Sending build context to Docker daemon 4.608kB
Step 1/5 : FROM alpine
---> d6e46aa2470d
Step 2/5 : RUN apk add --no-cache make git
---> Running in d2e8219c765e
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/7) Installing ca-certificates (20191127-r4)
(2/7) Installing nghttp2-libs (1.41.0-r0)
(3/7) Installing libcurl (7.69.1-r1)
(4/7) Installing expat (2.2.9-r1)
(5/7) Installing pcre2 (10.35-r0)
(6/7) Installing git (2.26.2-r0)
(7/7) Installing make (4.3-r0)
Executing busybox-1.31.1-r19.trigger
Executing ca-certificates-20191127-r4.trigger
OK: 22 MiB in 21 packages
Removing intermediate container d2e8219c765e
---> 2796b4bea033
Step 3/5 : WORKDIR /src
---> Running in 0ed1061614b9
Removing intermediate container 0ed1061614b9
---> 2338ec598826
Step 4/5 : COPY ./some/files/. /src
---> a156df6331da
Step 5/5 : RUN cat one two three > four
---> Running in 57bb16203d19
Removing intermediate container 57bb16203d19
---> ebf3ee95c2b6
Successfully built ebf3ee95c2b6
Successfully tagged myname/myimage:latestIn the above output, 5 images are created; The first one is the base image ( docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
myname/myimage latest ebf3ee95c2b6 5 minutes ago 22.3MB
<none> <none> a156df6331da 5 minutes ago 22.3MB
<none> <none> 2338ec598826 5 minutes ago 22.3MB
<none> <none> 2796b4bea033 5 minutes ago 22.3MB
alpine latest d6e46aa2470d 6 weeks ago 5.57MBFor the classic builder, those intermediate images double act as "build-cache"; when repeating the build, the builder checks if the cache can be used for each step, and if so, skips the build for that step, and uses the existing image instead. When using BuildKit, on the other hand, those intermediate images are not created; BuildKit keeps a build-cache separate from the image store. Here's the same when building the same Dockerfile with BuildKit (after first removing docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
myname/myimage latest 5c80efd49bd6 About a minute ago 22.3MB
alpine latest d6e46aa2470d 6 weeks ago 5.57MBAs you can see, the However, there is a second reason the If I would change the Dockerfile, and add a new RUN echo "something changed" > foobarThen rebuild the image: DOCKER_BUILDKIT=1 docker build -t myname/myimage:latest .
[+] Building 1.5s (11/11) FINISHED
=> [internal] load .dockerignore 0.2s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.3s
=> => transferring dockerfile: 194B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [internal] load build context 0.4s
=> => transferring context: 151B 0.0s
=> [1/6] FROM docker.io/library/alpine 0.0s
=> CACHED [2/6] RUN apk add --no-cache make git 0.0s
=> CACHED [3/6] WORKDIR /src 0.0s
=> CACHED [4/6] COPY ./some/files/. /src 0.0s
=> CACHED [5/6] RUN cat one two three > four 0.0s
=> [6/6] RUN echo "something changed" > foobar 0.5s
=> exporting to image 0.3s
=> => exporting layers 0.1s
=> => writing image sha256:116eecf6c6d69a6e7763e7ec85e13ec42cf04064b39e17b50924132c15838c64 0.0s
=> => naming to docker.io/myname/myimage:latest 0.0sYou'll see that all steps (except for the last one) are cached. The final image produced has digest Now, looking at the images present (including "untagged" images); docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
myname/myimage latest 116eecf6c6d6 2 minutes ago 22.3MB
<none> <none> 5c80efd49bd6 4 minutes ago 22.3MB
alpine latest d6e46aa2470d 6 weeks ago 5.57MBYou can see that BuildKit doesn't need that image for its cache, so you can run docker image prune
WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:5c80efd49bd638c2ed78a9799a39e80f1a879b0edaf9b0b56fa18de19f96a113
Total reclaimed space: 0B(in this case, no space was reclaimed, because the last image added a new Note that if you're cleaning up between builds, to remove untagged images ( |
|
I ran it right after same goes for |
@tonistiigi @tiborvass ^^ you may be more familiar how it checks for "dangling" |
|
would it be possible to reuse the same intermediate image for all builds? that would also serve as a cache, since everything is already downloaded, and it would also speedup the I noticed that the |
|
It should already use the cache, unless your dockerfile "busts" an early stage, which causes all following steps to not use the cache. This may happen if (e.g.) you use |
"dangling" is a term used for an unnamed image. Usually, an image that has lost a name because another image with same name has been created. Builder cache is not dangling.
Yes. If you want to clean up dangling images you run |
then why does ok, this is kinda confusing but I found the magic number that seems to work for me... I guess this answers my questions, although either way, I'll stop bothering you guys with this now. |
Most likely just because |
|
heh. I had a feeling that was the case but had forgotten the command for checking it. thanks and sorry. |
^ @tonistiigi @tiborvass we should fix that message indeed if it does not check for cache related to "current" images (I expect that message was just copied from the other prune commands) |
|
The solution I'm using to remove the untagged images is to save the final image, remove the image which removes all the associated untagged images, and then load the final image back in. This then only leaves the single tagged image. |
At first it sounded wrong, because with the classical builder each step produces an image. But I see now why you said that. |
Description
When performing multi stage builds, last image is properly created, but also intermediate images are left behind as images.
We have project with JAVA Maven module (image) and standard PHP+JS website as separate image. Both of these are generating images when in multistage build
Reproduce with JAVA/Maven
Have JAVA/Maven application and do the same steps as described in multi stage build blog post:
This is dockerfile - 1 build step and 1 final image
Execute command:
docker build -t testjava .Result from JAVA/Maven
After build is complete you will see that there is final image testjava but also
Reproduce with simple website image
In the example below, we have source code of the website that has:
Dockerfile that is created is just a "mockup" of multistage build -> it has "useless" 3 build steps and final step that is actually meaningful
Dockerfile
Result of website image
Result of this is one final image and 3 images
Expected result:
Only final image is created
Additional information you deem important (e.g. issue happens only occasionally):
After
docker system prunethose images are gone (as expected)Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
VMWare virtual machine (internal environment) running CentOS 7
The text was updated successfully, but these errors were encountered: