8000 Proof of concept for 'dematerialized' environments by chrisburr · Pull Request #1162 · conda/rattler · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Proof of concept for 'dematerialized' environments #1162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Proof of concept for 'dematerialized' environments #1162

wants to merge 2 commits into from

Conversation

chrisburr
Copy link
Contributor
@chrisburr chrisburr commented Mar 14, 2025 •
edited
Loading

Some context

I'm often working in HPC-like environments where networked file systems are used. The size and number of files involved in making a conda environment can put a lot of stress on the file systems and end up being slow for end users.

In high energy physics the solution we have to distributing software is a read-only distributed filesystem named CMVFS. I'm aware of it also being used by some other fields. Additionally, I'm aware of similar systems being used in other areas.

This idea started from the fact that it would be fairly easy to mount CVMFS "repository" which contains a cache directory of all conda-forge packages pre-extracted.

Proof of concept

The basic idea is to use Linux's overlayfs to build a conda environment by combining the rattler package cache directly as read-only underlays. This also avoids any limitations around hard linking across filesystems. Overlayfs is Linux only feature but I think the problems of limited filesystems are most apparent in Linux environments.

Building a conda environment like this does however have three limitations:

  1. Some files need prefix replacement
  2. .pyc files for noarch python packages
  3. post-link scripts

(2) and (3) could be ignored and still have a fairly usable environment but (1) is absolutely essential to solve. This pull request adds an is_overlay flag which only writes files to the target prefix if prefix replacement is being applied. This directory can then be applied as the highest priority underlay to override the files in the cache.

(2) and (3) could be solved by adding a writable overlay (already included in the example activation script below).

You could even think about going one step further by making a FUSE based file system which generates the prefix-replaced files on the fly.

Example

I've done my testing with the following:

$ cd rattler/py-rattler
$ pixi run build-release
$ pixi run ./create_example.py --base-dir ~/example 'python 3.12.*' pandas numpy ipython pytorch pip
$ pixi run ./start_shell_in_dematerialized.py ~/example/env-dematerialized --without-writable-overlay --overlay-method=overlayfs-fuse

I'm using overlayfs-fuse here as it seems like the AFS filesystem that hosts my home directory deadlocks if I use mount -t overlayfs. In other contexts I've been using --overlay-method=native.

Install time

I didn't rigorously check the timing but these runs are representative from what I've seen. The "dematerialized" case can probably be further optimised (e.g. it currently makes more directories than it needs to).

Filesystem Standard Dematerialized
NVMe + xfs 1s 433ms 181ms
Ceph + ext4 2s 315ms 1s 113ms
OpenAFS 1m 27s 222ms 5s 593ms

Resource usage

Metric Standard Dematerialized
Size on disk 1.5G 91M
Number of files 31997 514
Number of directories 2454 40

@baszalmstra
Copy link
Collaborator

This looks very promising! Would it be possible to automate this process completely from rattler? e.g. also building the underlay and setting up a complete environment (the things you are doing in the python example now)?

@chrisburr chrisburr mentioned this pull request Mar 24, 2025
Draft
6 tasks
@chrisburr
Copy link
Contributor Author

I've put an alternative approach in #1182

While I could think of technical justifications for supporting both, my first instinct is that the FUSE method is the only one worth having.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@chrisburr @baszalmstra
0