How to load dataset from MinIO? #7140
Unanswered
sasakiyori
asked this question in
Q&A
Replies: 1 comment 1 reply
-
|
Your issue could be related to dataset path or bucket permissions. is your data_files could be |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
After I upgrade def load_from_s3():
storage_options = {
"key": "minioadmin",
"secret": "minioadmin",
"endpoint_url": "http://localhost:9000",
"use_ssl": False,
}
data = load_dataset(
"json",
data_files="s3://mybucket/mydataset/train/myfile.json",
storage_options=storage_options,
)But how can I achieve the result like below? I want to load whole dataset without specific data file paths. # load from local dir
dataset = load_dataset("/path/to/mydataset", split="train")
# or
dataset = load_dataset("/path/to/mydataset/train") |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I have no idea how to specific the endpoint for MinIO.
If endpoint is not configured,
PermissionError: Forbiddenis returned.But if I set endpoint_url like below, it returns
FileNotFoundError: Unable to find 's3://datasets/mybucket/mydataset', it seems go to other protocols but not s3.Beta Was this translation helpful? Give feedback.
All reactions