Description
What version of Go are you using (go version)?
$ go version go1.11.5 darwin/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go envGOARCH="amd64"
GOBIN=""
GOCACHE="/Users/alexberry/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/alexbarry/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.5/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/cd/2sl0rxtn7y9bvdgyj8z7w6c80000gp/T/go-build437162119=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do?
I ran go build on a go module that has a dependency on a separate go module which uses git-lfs for large files. This build had been working as of go1.11.
What did you expect to see?
Go mod should fetch the dependency and go build should succeed.
What did you see instead?
The go build errored with: "unknown import path... : module source tree too big."
For some background, I had previously added several files to git-lfs for the sole purpose of getting around the codehost.MaxZipFile limit of 500MB so I could use go modules back when 1.11 was released. Before doing this, I was also hit with "module source tree too big." The large files are not used at all when this repo is used as a dependency here - only one small package from that repo is imported.
After a decent dive into go get back in 1.11, I figured out I also had to add "export-ignore" to my .gitattributes in this repo to exclude the lfs file patterns from the zip when git archive is called. This finally got my builds working and let me use go modules. Everything had been working great after that.
I just upgraded to 1.11.5 (after another developer did and committed some go.sum files that broke our checksums due to the symlink issue) and cleaned my modcache. Then this unrelated build stopped working.
I believe the change was first introduced in go1.11.2 by this commit to src/cmd/go/internal/modfetch/codehost/git.go:
05b2b9b
I understand the reasoning is to guarantee the checksums are consistent, but I really think the git-lfs case needs to be considered. I have to imagine others using git-lfs within go modules would prefer their modules are importable and deal with breaking checksums on their own terms. The original issue there was more focused around export-subst than export-ignore anyway.
If we can't bring back export-ignore, another option might be to allow the MaxZipSize to be passed in, or enable some other method of specifying packages in a go.mod to be considered or excluded for this size requirement?
If there's an alternate workaround here that doesn't keep me on 1.11 or force me to split apart my repo, I'd be happy to give that a shot too. I've been a bit focused on export-ignore, but maybe there's a better flow to getting git-lfs and go mod to play nice?
There are a few smaller items here that might be worth mentioning:
-
Behind the scenes, as far as I can tell
go getactually is downloading the entire package (lfs files now included), before deciding it is too big. Not only do you have to wait, but the work was already done, so it seems strange to error after all this. I guess we save the extraction? -
The archive itself is ~300 MB as downloaded, it seems that the uncompressed size is what triggers the error. Does it need to be this way?
-
go mod tidy reports that it is downloading the dependency, and finishes with no errors. After it's done, it has not only failed to retrieve the module, but it has silently removed the require line altogether from the go.mod file.
-
go mod vendor also fails silently. This means when you're using
go build -mod=vendor, you'll later get a 'cannot find package "." in (path)' error. It's not clear at all this was related to your module being too large.
Thank you for all the help! Just want to add that in general, I've had great success with go modules and love using them.