Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Goofys allows you to mount an S3 bucket as a filey system.
It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have a on disk data cache (checkout catfs), and consistency model is close-to-open.
-
On Linux, install via pre-built binaries. You may also need to install fuse-utils first.
-
On Mac OS X, install via Homebrew:
$ brew cask install osxfuse
$ brew install goofys- Or build from source:
$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>
$ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefixUsers can also configure credentials via the
AWS CLI
or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
To mount an S3 bucket on startup, make sure the credential is
configured for root, and can add this to /etc/fstab:
goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666 0 0
Got more questions? Check out questions other people asked
Using --stat-cache-ttl 1s --type-cache-ttl 1s for goofys
-ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the
benchmark can be found in
bench.sh. Raw data
is available as well. Test was run on an EC2 m4.16xlarge in us-west-2a
connecting to a bucket in us-west-2. Units are seconds.
(†) riofs does not wait for HTTP response before returning from release(), so the create files benchmarks do not measure the right thing for it
Enabling --cache has little impact on write speed (since catfs
implements a write-through cache) but read has a large variance. Time
to first byte is competitive with s3fs which suggests layering fuse
filesystems can be a viable approach.
$ cat > ~/.passwd-riofs
export AWS_ACCESS_KEY_ID=AKID1234567890
export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v ~/.passwd-riofs:/root/.passwd-riofs -v /tmp/cache:/tmp/cache kahing/goofys-bench
# result will be written to $TESTBUCKETif CACHE is set to true, the read benchmarks ('Read 1GB' and 'Time to 1st byte') will be cached read.
Copyright (C) 2015 - 2017 Ka-Hing Cheung
Licensed under the Apache License, Version 2.0
goofys has been tested under Linux and OS X.
List of non-POSIX behaviors/limitations:
- only sequential writes supported
- does not store file mode/owner/group
- use
--(dir|file)-modeor--(uid|gid)options
- use
- does not support symlink or hardlink
ctime,atimeis always the same asmtime- cannot rename non-empty directories
unlinkreturns success even if file is not presentfsyncis ignored, files are only flushed onclose
In addition to the items above, the following supportable but not yet implemented:
- creating files larger than 1TB
goofys has been tested with the following non-AWS providers:
- Amplidata
- DreamObjects (Ceph)
- EMC Atmos
- Google Cloud Storage
- OpenStack Swift
- S3Proxy
- Minio (limited)
- Data is stored on Amazon S3
- Amazon SDK for Go
- Other related fuse filesystems
- catfs: caching layer that can be used with goofys
- s3fs: another popular filesystem for S3
- gcsfuse: filesystem for Google Cloud Storage. Goofys borrowed some skeleton code from this project.
- S3Proxy is used for
go test - fuse binding, also used by
gcsfuse