Description
What version of Go are you using (go version)?
go version go1.18 linux/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/dtrudg/.cache/go-build" GOENV="/home/dtrudg/.config/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/dtrudg/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/dtrudg/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/opt/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/opt/go/pkg/tool/linux_amd64" GOVCS="" GOVERSION="go1.18" GCCGO="gccgo" GOAMD64="v1" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/home/dtrudg/Git_sylabs/singularity/go.mod" GOWORK="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build864842809=/tmp/go-build -gno-record-gcc-switches"
What did you do?
We have a plugin (https://github.com/sylabs/singularity/blob/master/e2e/plugin/testdata/runtime_singularity/callback.go) that indirectly imports a package containing some CGO code (https://github.com/sylabs/singularity/tree/master/internal/pkg/util/user).
I have tried to extract a simpler reproducer, but haven't managed to hit the same issue yet... so what follows is, unfortunately, a terse description of a rather complicated build...
Our main binary is compiled with the following go build and GOFLAGS contains -trimpath
go build
-mod=readonly
-buildmode=pie
-tags "containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper sylog singularity_engine fakeroot_engine apparmor selinux seccomp"
-gcflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0"
-asmflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0"
-o ./singularity
/home/dtrudg/Git_sylabs/singularity/cmd/singularity
The plugin is built by copying the source into a temporary directory, and then using the following go build:
go build
-a
-o /tmp/plugin-1291001354/src/plugin.so
-mod=readonly
-ldflags -X main.PluginRootDir=/usr/local/libexec/singularity/plugin
-trimpath
-buildmode=plugin
-tags containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper sylog singularity_engine fakeroot_engine apparmor selinux seccomp
.
What did you expect to see?
With Go 1.17.8 or prior, the compiled plugin can be loaded by the main binary without issue.
What did you see instead?
With Go 1.18, the plugin is compiled without issue, but when the main binary attempts to load it, it fails with an error:
plugin.Open("/tmp/plugin-1836960432/src/plugin"): plugin was built with a different version of package github.com/sylabs/singularity/internal/pkg/util/user
Examining the pkghashbytes for the "github.com/sylabs/singularity/pkg/image" package, I can see a difference between the main binary and plugin:
main binary:
go.link.pkghashbytes.github.com/sylabs/singularity/internal/pkg/util/user:
[233 225 141 122 49 203 202 104]
plugin:
go.link.pkghashbytes.github.com/sylabs/singularity/internal/pkg/util/user:
[189 134 9 108 2 209 91 245]
I can see that there is a difference in the strings section exposed in cmd/compile/internal/typecheck/iexport.go:
main binary compilation
# github.com/sylabs/singularity/internal/pkg/util/user
export: hdr strings 533, data 950, index 1039
plugin compilation
# github.com/sylabs/singularity/internal/pkg/util/user
export: hdr strings 504, data 950, index 1039
Grabbing these and converting to text we have:
strings section in the main binary build…
Lgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/identity_unix.goNameUIDGIDGecosDirShelluidUser~r0~r1$autotmp
lookupUnixUidNgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/cgo_lookup_unix.goname
lookupUserusernamegidGroup
lookupUnixGid�lookupGroup groupnamecurrentIgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/membership.golistesc:,/tmp/go-build2510098997/b050/_cgo_gotypes.gouser .inittaskCurrentCurrentOriginalGetGrGIDGetGrNamGetPwNamGetPwUID
UIDInAnyGroup UIDInList
strings section in the plugin build…
Lgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/identity_unix.goNameUIDGIDGecosDirShelluidUser~r0~r1$autotmp
lookupUnixUidNgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/cgo_lookup_unix.goname
lookupUserusernamegidGroup
lookupUnixGid�lookupGroup groupnamecurrentIgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/membership.golistesc:_cgo_gotypes.gouser .inittaskCurrentCurrentOriginalGetGrGIDGetGrNamGetPwNamGetPwUID
UIDInAnyGroup UIDInList
The difference here being an extra ,/tmp/go-build2510098997/b050/ prior to _cgo_gotypes.gouser in the build of the main binary.
I'm not sure what might have changed from 1.17.x -> 1.18 to cause this?
A quick strings $(which singularity) (main binary) for different 1.18 and 1.17.8 builds shows at least some kind of difference without the context:
Go 1.18
strings $(which singularity) | grep "_cgo_gotypes.go"
_cgo_gotypes.go
/tmp/go-build2709203403/b271/_cgo_gotypes.go
/tmp/go-build2709203403/b666/_cgo_gotypes.go
Go 1.17.8
strings $(which singularity) | grep "_cgo_gotypes.go"
_cgo_gotypes.go
/tmp/go-build1536705585/b262/_cgo_gotypes.go
Any pointers on where to go here would be appreciated!