8000 CLI: 'ais ls --help' · NVIDIA/aistore@a58881a · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Commit a58881a

Browse files
alex-aizmanalex-aizman
committed
CLI: 'ais ls --help'
Signed-off-by: Alex Aizman <alex.aizman@gmail.com>
1 parent c300a38 commit a58881a

File tree

2 files changed

+83
-56
lines changed

2 files changed

+83
-56
lines changed

‎cmd/cli/cli/bucket_hdlr.go

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -55,34 +55,40 @@ const copyBucketUsage = "Copy entire bucket or selected objects (to select, use
5555
indent1 + "\t- 'ais cp gs://webdataset-coco ais:/dst --prefix d-tokens/ --progress --all'\t- show progress while copying virtual subdirectory 'd-tokens';\n" +
5656
indent1 + "\t- 'ais cp gs://webdataset-coco/d-tokens/ ais:/dst --progress --all'\t- same as above."
5757

58-
// ais ls
59-
var listAnyUsage = "List buckets, objects in buckets, and files in " + archExts + "-formatted objects,\n" +
58+
// ais ls (note duplicated `archExts` constant)
59+
const listAnyUsage = "List buckets, objects in buckets, and files in (.tar, .tgz, .tar.gz, .zip, .tar.lz4)-formatted objects,\n" +
6060
indent1 + "e.g.:\n" +
6161
indent1 + "\t* ais ls \t- list all buckets in a cluster (all providers);\n" +
62-
indent1 + "\t* ais ls ais://abc -props name,size,copies,location \t- list all objects from a given bucket, include only the (4) specified properties;\n" +
63-
indent1 + "\t* ais ls ais://abc -props all \t- same as above but include all properties;\n" +
64-
indent1 + "\t* ais ls ais://abc --page 10000 -size 20 --refresh 3s \t- list a very large bucket (20 items in each page), report progress every 3s;\n" +
62+
indent1 + "\t* ais ls ais://abc -props name,size,copies,location \t- list objects with only these specific properties;\n" +
63+
indent1 + "\t* ais ls ais://abc -props all \t- list objects with all available properties;\n" +
64+
indent1 + "\t* ais ls ais://abc --page-size 20 --refresh 3s \t- list large bucket (20 items per page), progress every 3s;\n" +
6565
indent1 + "\t* ais ls ais://abc --page-size 20 --refresh 3 \t- same as above;\n" +
6666
indent1 + "\t* ais ls ais \t- list all ais buckets;\n" +
67-
indent1 + "\t* ais ls s3 \t- list all s3 buckets that are present in the cluster;\n" +
68-
indent1 + "\t* ais ls s3 --all \t- list all s3 buckets, both in-cluster and remote.\n" +
67+
indent1 + "\t* ais ls s3 \t- list all s3 buckets present in the cluster;\n" +
68+
indent1 + "\t* ais ls s3 --all \t- list all s3 buckets (both in-cluster and remote).\n" +
69+
indent1 + "list archive contents:\n" +
70+
indent1 + "\t* ais ls ais://abc/sample.tar --archive \t- list files inside a tar archive;\n" +
71+
indent1 + "list in pages (continues until '--max-pages', '--limit', Ctrl-C, or end of bucket):\n" +
72+
indent1 + "\t* ais ls s3://abc --paged --limit 1234000 \t- limited paged output (1234 pages), with default properties;\n" +
73+
indent1 + "\t* ais ls s3://abc --paged --limit 1234000 --nr \t- same as above, non-recursively (skips nested directories);\n" +
6974
indent1 + "with template, regex, and/or prefix:\n" +
7075
indent1 + "\t* ais ls gs: --regex \"^abc\" --all \t- list all accessible GCP buckets with names starting with \"abc\";\n" +
71-
indent1 + "\t* ais ls ais://abc --regex \".md\" --props size,checksum \t- list *.md objects with their respective sizes and checksums;\n" +
72-
indent1 + "\t* ais ls gs://abc --template images/\t- list all objects from the virtual subdirectory called \"images\";\n" +
73-
indent1 + "\t* ais ls gs://abc --prefix images/\t- same as above (for more examples, see '--template' below);\n" +
74-
indent1 + "\t* ais ls gs://abc/images/\t- same as above.\n" +
76+
indent1 + "\t* ais ls ais://abc --regex \"\\.md$\" --props size,checksum \t- list markdown files with size and checksum;\n" +
77+
indent1 + "\t* ais ls gs://abc --template images/ \t- list all objects from virtual subdirectory \"images\";\n" +
78+
indent1 + "\t* ais ls gs://abc --prefix images/ \t- same as above (for more examples, see '--template' below);\n" +
79+
indent1 + "\t* ais ls gs://abc/images/ \t- same as above.\n" +
7580
indent1 + "with in-cluster vs remote content comparison (diff):\n" +
76-
indent1 + "\t* ais ls s3://abc --check-versions \t- for each remote object in s3://abc: check whether it has identical in-cluster copy\n" +
77-
indent1 + "\t \t and show missing objects;\n" +
78-
indent1 + "\t* ais ls s3://abc --check-versions --cached\t- for each in-cluster object in s3://abc: check whether it has identical remote copy\n" +
79-
indent1 + "\t \t and show deleted objects.\n" +
81+
indent1 + "\t* ais ls s3://abc --check-versions \t- for each remote object: check for identical in-cluster copy\n" +
82+
indent1 + "\t → \t and show missing objects;\n" +
83+
indent1 + "\t* ais ls s3://abc --check-versions --cached \t- for each in-cluster object: check for identical remote copy\n" +
84+
indent1 + "\t → \t and show deleted objects.\n" +
8085
indent1 + "with summary (bucket sizes and numbers of objects):\n" +
81-
indent1 + "\t* ais ls ais://nnn --summary --prefix=aaa/bbb' \t- summarize objects that match a given prefix;\n" +
82-
indent1 + "\t* ais ls ais://nnn/aaa/bbb --summary' \t- same as above;\n" +
83-
indent1 + "\t* ais ls s3 --summary \t- for each s3 bucket: print number of objects and total size (bytes);\n" +
84-
indent1 + "\t* ais ls s3 --summary --all \t- generate summary report for all s3 buckets; include remote objects and buckets that are _not present_;\n" +
85-
indent1 + "\t* ais ls s3 --summary --all --dont-add\t- same as above but without adding _non-present_ remote buckets to the cluster's BMD."
86+
indent1 + "\t* ais ls ais://nnn --summary --prefix=aaa/bbb \t- summarize objects matching the given prefix;\n" +
87+
indent1 + "\t* ais ls ais://nnn/aaa/bbb --summary \t- same as above;\n" +
88+
indent1 + "\t* ais ls az://azure-bucket --count-only \t- fastest way to count objects in a bucket;\n" +
89+
indent1 + "\t* ais ls s3 --summary \t- for each s3 bucket: print object count and total size;\n" +
90+
indent1 + "\t* ais ls s3 --summary --all \t- summary report for all s3 buckets including remote/non-present;\n" +
91+
indent1 + "\t* ais ls s3 --summary --all --dont-add \t- same, without adding non-present buckets to cluster metadata."
8692

8793
// ais bucket ... props
8894
const setBpropsUsage = "Update bucket properties; the command accepts both JSON-formatted input and plain Name=Value pairs,\n" +

‎docs/cli/bucket.md

Lines changed: 57 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -174,27 +174,42 @@ Operation "destroy-bck" is not supported by "aws://bucket_name"
174174

175175
```console
176176
$ ais ls --help
177-
178177
NAME:
179-
ais ls - (alias for "bucket ls") list buckets, objects in buckets, and files in (.tar, .tgz or .tar.gz, .zip, .tar.lz4)-formatted objects,
178+
ais ls - (alias for "bucket ls") List buckets, objects in buckets, and files in (.tar, .tgz, .tar.gz, .zip, .tar.lz4)-formatted objects,
180179
e.g.:
181180
* ais ls - list all buckets in a cluster (all providers);
182-
* ais ls ais://abc -props name,size,copies,location - list all objects from a given bucket, include only the (4) specified properties;
183-
* ais ls ais://abc -props all - same as above but include all properties;
184-
* ais ls ais://abc --page-size 20 --refresh 3s - list a very large bucket (20 items in each page), report progress every 3s;
181+
* ais ls ais://abc -props name,size,copies,location - list objects with only these specific properties;
182+
* ais ls ais://abc -props all - list objects with all available properties;
183+
* ais ls ais://abc --page-size 20 --refresh 3s - list large bucket (20 items per page), progress every 3s;
184+
* ais ls ais://abc --page-size 20 --refresh 3 - same as above;
185185
* ais ls ais - list all ais buckets;
186-
* ais ls s3 - list all s3 buckets that are present in the cluster;
187-
* ais ls s3 --all - list all s3 buckets, both present and remote;
186+
* ais ls s3 - list all s3 buckets present in the cluster;
187+
* ais ls s3 --all - list all s3 buckets (both in-cluster and remote).
188+
list archive contents:
189+
* ais ls ais://abc/sample.tar --archive - list files inside a tar archive;
190+
list in pages (continues until '--max-pages', '--limit', Ctrl-C, or end of bucket):
191+
* ais ls s3://abc --paged --limit 1234000 - limited paged output (1234 pages), with default properties;
192+
* ais ls s3://abc --paged --limit 1234000 --nr - same as above, non-recursively (skips nested directories);
188193
with template, regex, and/or prefix:
189-
* ais ls gs: --regex "^abc" --all - list all accessible GCP buckets with names starting with "abc";
190-
* ais ls ais://abc --regex ".md" --props size,checksum - list *.md objects with their respective sizes and checksums;
191-
* ais ls gs://abc --template images/ - list all objects from the virtual subdirectory called "images";
192-
* ais ls gs://abc --prefix images/ - same as above (for more examples, see '--template' below);
193-
* ais ls gs://abc/images/ - same as above.
194-
and summary (stats):
195-
* ais ls s3 --summary - for each s3 bucket in the cluster: print object numbers and total size(s);
196-
* ais ls s3 --summary --all - generate summary report for all s3 buckets; include remote objects and buckets that are _not present_;
197-
* ais ls s3 --summary --all --dont-add - same as above but without adding _non-present_ remote buckets to cluster's BMD.
194+
* ais ls gs: --regex "^abc" --all - list all accessible GCP buckets with names starting with "abc";
195+
* ais ls ais://abc --regex "\.md$" --props size,checksum - list markdown files with size and checksum;
196+
* ais ls gs://abc --template images/ - list all objects from virtual subdirectory "images";
197+
* ais ls gs://abc --prefix images/ - same as above (for more examples, see '--template' below);
198+
* ais ls gs://abc/images/ - same as above.
199+
with in-cluster vs remote content comparison (diff):
200+
* ais ls s3://abc --check-versions - for each remote object: check for identical in-cluster copy
201+
→ and show missing objects;
202+
* ais ls s3://abc --check-versions --cached - for each in-cluster object: check for identical remote copy
203+
→ and show deleted objects.
204+
with summary (bucket sizes and numbers of objects):
205+
* ais ls ais://nnn --summary --prefix=aaa/bbb - summarize objects matching the given prefix;
206+
* ais ls ais://nnn/aaa/bbb --summary - same as above;
207+
* ais ls az://azure-bucket --count-only - fastest way to count objects in a bucket;
208+
* ais ls s3 - F438 -summary - for each s3 bucket: print object count and total size;
209+
* ais ls s3 --summary --all - summary report for all s3 buckets including remote/non-present;
210+
* ais ls s3 --summary --all --dont-add - same, without adding non-present buckets to cluster metadata.
211+
...
212+
...
198213
```
199214

200215
## Assorted options
@@ -270,33 +285,39 @@ The command's inline help is also quite extensive, with (inline) examples follow
270285
```console
271286
$ ais ls --help
272287
NAME:
273-
ais ls - (alias for "bucket ls") List buckets, objects in buckets, and files in (.tar, .tgz or .tar.gz, .zip, .tar.lz4)-formatted objects,
288+
ais ls - (alias for "bucket ls") List buckets, objects in buckets, and files in (.tar, .tgz, .tar.gz, .zip, .tar.lz4)-formatted objects,
274289
e.g.:
275290
* ais ls - list all buckets in a cluster (all providers);
276-
* ais ls ais://abc -props name,size,copies,location - list all objects from a given bucket, include only the (4) specified properties;
277-
* ais ls ais://abc -props all - same as above but include all properties;
278-
* ais ls ais://abc --page-size 20 --refresh 3s - list a very large bucket (20 items in each page), report progress every 3s;
291+
* ais ls ais://abc -props name,size,copies,location - list objects with only these specific properties;
292+
* ais ls ais://abc -props all - list objects with all available properties;
293+
* ais ls ais://abc --page-size 20 --refresh 3s - list large bucket (20 items per page), progress every 3s;
279294
* ais ls ais://abc --page-size 20 --refresh 3 - same as above;
280295
* ais ls ais - list all ais buckets;
281-
* ais ls s3 - list all s3 buckets that are present in the cluster;
282-
* ais ls s3 --all - list all s3 buckets, both in-cluster and remote.
296+
* ais ls s3 - list all s3 buckets present in the cluster;
297+
* ais ls s3 --all - list all s3 buckets (both in-cluster and remote).
298+
list archive contents:
299+
* ais ls ais://abc/sample.tar --archive - list files inside a tar archive;
300+
list in pages (continues until '--max-pages', '--limit', Ctrl-C, or end of bucket):
301+
* ais ls s3://abc --paged --limit 1234000 - limited paged output (1234 pages), with default properties;
302+
* ais ls s3://abc --paged --limit 1234000 --nr - same as above, non-recursively (skips nested directories);
283303
with template, regex, and/or prefix:
284-
* ais ls gs: --regex "^abc" --all - list all accessible GCP buckets with names starting with "abc";
285-
* ais ls ais://abc --regex ".md" --props size,checksum - list *.md objects with their respective sizes and checksums;
286-
* ais ls gs://abc --template images/ - list all objects from the virtual subdirectory called "images";
287-
* ais ls gs://abc --prefix images/ - same as above (for more examples, see '--template' below);
288-
* ais ls gs://abc/images/ - same as above.
304+
* ais ls gs: --regex "^abc" --all - list all accessible GCP buckets with names starting with "abc";
305+
* ais ls ais://abc --regex "\.md$" --props size,checksum - list markdown files with size and checksum;
306+
* ais ls gs://abc --template images/ - list all objects from virtual subdirectory "images";
307+
* ais ls gs://abc --prefix images/ - same as above (for more examples, see '--template' below);
308+
* ais ls gs://abc/images/ - same as above.
289309
with in-cluster vs remote content comparison (diff):
290-
* ais ls s3://abc --check-versions - for each remote object in s3://abc: check whether it has identical in-cluster copy
291-
and show missing objects;
292-
* ais ls s3://abc --check-versions --cached - for each in-cluster object in s3://abc: check whether it has identical remote copy
293-
and show deleted objects.
310+
* ais ls s3://abc --check-versions - for each remote object: check for identical in-cluster copy
311+
→ and show missing objects;
312+
* ais ls s3://abc --check-versions --cached - for each in-cluster object: check for identical remote copy
313+
→ and show deleted objects.
294314
with summary (bucket sizes and numbers of objects):
295-
* ais ls ais://nnn --summary --prefix=aaa/bbb' - summarize objects that match a given prefix;
296-
* ais ls ais://nnn/aaa/bbb --summary' - same as above;
297-
* ais ls s3 --summary - for each s3 bucket: print number of objects and total size (bytes);
298-
* ais ls s3 --summary --all - generate summary report for all s3 buckets; include remote objects and buckets that are _not present_;
299-
* ais ls s3 --summary --all --dont-add - same as above but without adding _non-present_ remote buckets to the cluster's BMD.
315+
* ais ls ais://nnn --summary --prefix=aaa/bbb - summarize objects matching the given prefix;
316+
* ais ls ais://nnn/aaa/bbb --summary - same as above;
317+
* ais ls az://azure-bucket --count-only - fastest way to count objects in a bucket;
318+
* ais ls s3 --summary - for each s3 bucket: print object count and total size;
319+
* ais ls s3 --summary --all - summary report for all s3 buckets including remote/non-present;
320+
* ais ls s3 --summary --all --dont-add - same, without adding non-present buckets to cluster metadata.
300321

301322
USAGE:
302323
ais ls [BUCKET[/PREFIX]] [PROVIDER] [command options]

0 commit comments

Comments
 (0)
0