8000 Add fsspec support to index remote filesystems, separate package / cache, database / output directories into 3 places. · Issue #143 · conda/conda-index · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add fsspec support to index remote filesystems, separate package / cache, database / output directories into 3 places. #143

New issue

Have a question about this project? Sign up for a free GitHub account 8000 to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dholth opened this issue Mar 29, 2024 · 2 comments · Fixed by #130
Closed

Add fsspec support to index remote filesystems, separate package / cache, database / output directories into 3 places. #143

dholth opened this issue Mar 29, 2024 · 2 comments · Fixed by #130
Assignees
@dholth
Labels
locked [bot] locked due to inactivity type::feature request for a new feature or capability

Comments

@dholth
Copy link
Contributor
dholth commented Mar 29, 2024 •
edited
Loading

What is the idea?

Allow an optional fsspec dependency to index remote packages through the API. When fsspec is not installed local packages can still be indexed as usual.

Separate input, cache (sqlite databases), and output into three directories. By default these can be the same folder, but with fsspec the input will typically be read-only and not usable for the sqlite cache database.

Why is this needed?

This will be used to improve the CDN process.
@dholth dholth added the type::feature request for a new feature or capability label Mar 29, 2024
@dholth dholth self-assigned this Mar 29, 2024
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Mar 29, 2024
@dholth dholth mentioned this issue Apr 1, 2024
Merged
3 tasks
@dholth dholth linked a pull request Apr 5, 2024 that will close this issue
Index remote or local files via fsspec #130
Merged
3 tasks
@dholth dholth closed this as completed Apr 8, 2024
@github-project-automation github-project-automation bot moved this from 🆕 New to 🏁 Done in 🧭 Planning Apr 8, 2024
@jtroe
Copy link
jtroe commented Apr 29, 2024

These changes look like they will help me; I'm currently working on creating an index in s3 and CloudFront. Are there plans to add channel_url to the CLI?

@dholth
Copy link
Contributor Author
dholth commented Apr 29, 2024

That's great! It would be interesting to add channel_url to the CLI. One challenge is that it would not be completely general; for example if the remote filesystem is HTTP then conda_index might call HEAD on each individual remote file to get the equivalent of mtime; those details might need to be stored out-of-band by overriding save_fs_state or by relaxing the comparison in changed_packages to only consider metadata that we could efficiently gather from upstream. On the other hand a s3 list would be reasonably efficient for small or medium sized channels.

@github-actions github-actions bot added the locked [bot] locked due to inactivity label Oct 27, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dholth dholth

Labels
locked [bot] locked due to inactivity type::feature request for a new feature or capability
Projects
🧭 Planning
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Index remote or local files via fsspec conda/conda-index
2 participants
@dholth @jtroe
0