Ideas for machine learning datasets on datahub #1234

rufuspollock · 2024-07-11T00:45:39Z

rufuspollock
Jul 11, 2024
Maintainer

Ideas for machine learning datasets on datahub https://archive.ics.uci.edu

I think we processed some of these back in the day (6y ago) for datahub v2 ... need to locate that ...

davidgasquez · 2024-07-16T15:16:09Z

This made me think.... is there a way for Datahub to wrap HuggingFace datasets somewhat (technically is possible but not sure if legally is)?

Could be an easy way to reduce friction to add a README / website to some of those HuggingFace datasets.

2 replies

@davidgasquez nice idea ... i think it could be kind of simple to create something like this.

tell me a bit more about the idea 🙂

Could work in a couple of ways.

Since Datahub also runs on server mode, it could be to have a path /@hf where you could request any dataset. E.g: https://datahub.io/@hf/OpenCo7/UpVoteWeb would render https://huggingface.co/datasets/OpenCo7/UpVoteWeb. Same "connector" approach could work with GitHub, e.g: https://datahub.io/@gh/datasets/s-and-p-500-companies renders https://github.com/datasets/s-and-p-500-companies.
Another option that adds more fricion would be to have a way to "import" datasets from other places at creation time. This import will create a datapackage.yaml file with the proper resources. In this case, it could use fsspec like hf://OpenCo7/UpVoteWeb/data.parquet

There might be other ways! The coldstart problem is always tricky and these might not work but happy to continue brainstorming! 😅