Description
Description
By default conf.yml has the following configuration set, which keeps blob descriptors in memory:
storage:
cache:
blobdescriptor: inmemoryImage tag deletion process deletes blobs and layers from disk, but does not delete in memory descriptors.
Due to in memory descriptors, it's not possible to re-push the same image because registry uses cache layers but such layers are not there actually.
Reproduce
reproducable in both latest V3 (`3.0.0-rc.2) and V2; I'll use V3 in this example
When a image is pushed into the registry, deleted and garbage collect is executed, the blobs are fully removed from disk:
# start a registry
docker run -d -e REGISTRY_STORAGE_DELETE_ENABLED=true -p 5000:5000 --rm --name registry-in-memory-blobs registry:3.0.0-rc.2
# creating a random tag and push it
docker tag registry:3.0.0-rc.2 localhost:5000/registry:3.0.0-rc.2
docker push localhost:5000/registry:3.0.0-rc.2
# then delete the tag and run garbage collect
curl -X DELETE http://localhost:5000/v2/registry/manifests/sha256:776203b7933004f7b6cada731e171469e3e267a1e77912fd8e0fc26c16f896f2
docker exec registry-in-memory-blobs /bin/registry garbage-collect /etc/distribution/config.ymlPrevious commands successfully deleted blobs from disk, but their descriptions are still in memory.
When the same push takes place again, registry says blobs/layers are already there and does not really accept the push, but uses the cache layers instead of pushing:
docker push localhost:5000/registry:3.0.0-rc.2
# 2b07afdb0755: Layer already exists
# 55659e28f40e: Layer already exists
# 6323af7fd13e: Layer already exists
# d73b26c48fce: Layer already exists
# a0904247e36a: Layer already exists
# 3.0.0-rc.2: digest: sha256:776203b7933004f7b6cada731e171469e3e267a1e77912fd8e0fc26c16f896f2 size: 1364But no new layers were created on disk, and if we try to pull it will not work:
docker pull localhost:5000/registry:3.0.0-rc.2
# Error response from daemon: manifest for localhost:5000/registry:3.0.0-rc.2 not found: manifest unknown: manifest unknownSo, the HTTP DELETE request and garbage collect successfully deleted the blobs and layers from disk, but haven't deleted the in memory blob descriptors. It makes inconsistency between what is in memory and what is in disk.
If the blobdescriptor: inmemory is not set, them this behavir does not happen
Expected behavior
registry allows to push the same image after deleting it
registry version
3.0.0-rc.2
Additional Info
No response