Description
Among the container's lifecycle, pulling image is one of the biggest performance bottlenecks on container startup processes. One research shows that time for pulling accounts for 76% of container startup time[FAST '16].
I know there is discussion on #2943, but it seems nobody got started the implementation? To make this progress, I implemented a patch based on the discussion with an example implementation of a remote snapshotter. Through the implementation, I found challenges around metadata snapshotter and namespaces so I fixed the design to achieve it. Could anyone give comments on it?
- Remote Snapshotter patch: remote-snapshotter branch
- Example implementation of remote snapshotter: /snapshotter.go
The overview of the design and implementation
The whole picture is described here.
To make containerd work with remote snapshotter, it needs to:
- Skip downloading layers which can be prepared without downloading contents by a remote snapshotter(I call it remote layers here).
- Make remote snapshots without "Unpack" operation.
- Make the snapshot work with metadata snapshotter so that containerd can bind these snapshots to namespaces
I think we can achieve it by introducing additional filter(Fig 1) during the fetching process(in the Client.fetch method). Currently, there are already some filters in the method, so adding a new filter seems relatively easily.
- The filter takes lists of download candidates(blobs) and checks if each one and the lower layers are remote layers.
- If so, it filters out the layer from the download candidates and makes the snapshot as a remote snapshot.
- Containerd doesn't unpack layers if the layers already exist as snapshots, so we can avoid unpacking them.
The filter talks with remote snapshotters to know whether a layer is a remote layer(Fig 2).
- The filter Stat() the layer with ChainID of the layer.
- If the snapshot doesn't exist, the filter attempts to Prepare() the layer as a remote snapshot.
- If possible, the remote snapshotter prepare an active snapshot with applying a label "RemoteSnapshotLabel" automatically.
- The filter Stat() the active snapshot to check if it is a remote snapshot by seeing the label.
- Only if the snapshot has the label, the filter Commit() it immediately and filter out this layer from the download candidates.
- If the snapshot is a remote snapshot, the snapshotter mounts the remote unpacked layer on the snapshot and commits it.
Question
I'm really keen to make the remote snapshotter real. The capability to boot containers without pulling is great.
- Cloud I get your comments on the design and implementation? Is it acceptable for containerd?
- If not, are there any considerations? Or are there possible alternative designs?