8000 depends: build LTO support into Apple's ld64 by theuni · Pull Request #19530 · bitcoin/bitcoin · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

depends: build LTO support into Apple's ld64 #19530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 28, 2020
Merged

Conversation

theuni
Copy link
Member
@theuni theuni commented Jul 15, 2020

This didn't work for a few reasons (various toolchain compatibility issues) the last time I tested it, but after the last round of bumps it works with no apparent issues.

Note that this does not enable LTO by default in any way, only hooks up the machinery for -flto to work correctly when specified.

Lines were split for an easier rebase after #17919 is merged.

@fanquake
Copy link
Member

Concept ACK 🚀

theuni added 2 commits July 16, 2020 19:04
Note that this does not _enable_ lto by default in any way, only hooks up the
machinery for -flto to work correctly.

enable-lto-support is explicitly used for pinned-clang because we know it
works. It is neither enabled nor disabled in the external clang case so that
it can be auto-detected.
tpoechtrager/cctools-port#85 was merged upstream, which
fixes lto detection for external clang with some Linux Distro's including
Ubuntu.
@theuni
Copy link
Member Author
theuni commented Jul 16, 2020

Rebased after #17919 .

Also added another commit after tpoechtrager/cctools-port#85 was merged upstream to fix lto detection with external clang.

@fanquake
Copy link
Member

Checked out this branch at 5962522 and did a make -C depends/ -j6 HOST=x86_64-apple-darwin16. I'm seeing:

[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/endian.o
[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha1.o
[ 97%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha256.o
[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/sha512.o
[ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/checksum.o
[100%] Linking C executable genisoimage
/usr/bin/ld: CMakeFiles/genisoimage.dir/apple.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/desktop.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_file.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_reader.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/eltorito.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/files.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/hash.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/ifo_read.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/joliet.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/mac_label.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/multi.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/name.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/rock.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/scsi.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/stream.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/tree.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/udf.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/volume.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/write.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-alpha.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-hppa.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-mips.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/jte.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/rsync.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/boot-mipsel.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/genisoimage.dir/checksum.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status

Have not investigated further.

@theuni
Copy link
Member Author
theuni commented Jul 17, 2020

@fanquake Strange... this is fine locally, and our c-i didn't have any trouble with it either.

Is the problem locally reproducible?

@theuni
Copy link
Member Author
theuni commented Jul 17, 2020

For me, builds of bitcoind and bitcoin-qt even succeed with lto enabled for dependencies as well with the following hack:

diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
index 6099fd4c71..c4180ac2f8 100644
--- a/depends/hosts/darwin.mk
+++ b/depends/hosts/darwin.mk
@@ -28,8 +28,8 @@ OSX_SDK=$(SDK_PATH)/Xcode-$(XCODE_VERSION)-$(XCODE_BUILD_ID)-extracted-SDK-with-
 #         https://reviews.llvm.org/D64089, we should use that instead. Read the
 #         differential summary there for more details.
 #
-darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin
-darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -nostdinc++ -isystem $(OSX_SDK)/usr/include/c++/v1
+darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -flto=thin
+darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION) -B$(build_prefix)/bin -nostdinc++ -isystem $(OSX_SDK)/usr/include/c++/v1 -flto=thin

 darwin_CFLAGS=-pipe
 darwin_CXXFLAGS=$(darwin_CFLAGS)

It's almost suspicious how well/easily that worked. I'll dig further to make sure it's actually doing full lto under the hood.

@DrahtBot
Copy link
Contributor

Gitian builds

File commit fd59670
(master)
commit 3fa897a
(master and this pull)
bitcoin-core-linux-0.21-res.yml 505c6e4b3d5cb53e... fed919dada1dd203...
bitcoin-core-osx-0.21-res.yml 59e47553324c4d43... a61dd55ddf53cd1f...
bitcoin-core-win-0.21-res.yml 8b4d3e18c2e7e44c... 0c74f587c29bdb8d...
*-aarch64-linux-gnu-debug.tar.gz 8e7308cd154a8b01... 78d04b4e71c541b2...
*-aarch64-linux-gnu.tar.gz 7c3675372886dbf7... 122ff48d3bfeda45...
*-arm-linux-gnueabihf-debug.tar.gz 01b2ece9ca424ef4... ba155e8de4be8cf5...
*-arm-linux-gnueabihf.tar.gz 45215523beb58f6d... c19fca4bd6375f94...
*-osx-unsigned.dmg 314f2723be4b2f70... 918ee034b90ff53a...
*-osx64.tar.gz c6bfc447a2ef1334... 8ad996a5c35c3d09...
*-riscv64-linux-gnu-debug.tar.gz 124dfbac07ea1e64... 6e916f614aefa5d3...
*-riscv64-linux-gnu.tar.gz 22bfb0fc4218d285... c8e0d75cb99273eb...
*-win64-debug.zip 75f1b9e8e0aab430... 25f869a98d19b23b...
*-win64-setup-unsigned.exe 3fb66c88b0126fb3... f0ec2840dbd17c09...
*-win64.zip de89cf52803db359... 49e5fb3efd3b15b5...
*-x86_64-linux-gnu-debug.tar.gz a587a7333cb2823b... 1e4cfb67952e009d...
*-x86_64-linux-gnu.tar.gz 73dd22b3be507039... d121aa1d85662008...
*.tar.gz 005ff019045bb0a4... 990f86ded9a6aff3...
linux-build.log 6201e9df0f29f327... 13477b89192ddeb0...
osx-build.log 83cd2a61f8a16ecb... 090d2b71ffbd54f0...
win-build.log 49cc23d322018d48... 49d1ce2ed45e5506...
bitcoin-core-linux-0.21-res.yml.diff 9d3d37d52750777d...
bitcoin-core-osx-0.21-res.yml.diff 56adcd0fba885953...
bitcoin-core-win-0.21-res.yml.diff d8d9b7eac8b6c98b...
linux-build.log.diff 34eea87cd725d194...
osx-build.log.diff 57bf2ca7d604d02e...
win-build.log.diff d2b07e2dda38edf6...

< 8000 a class="d-inline-block" data-hovercard-type="user" data-hovercard-url="/users/fanquake/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="/fanquake">@fanquake
Copy link
Member

@fanquake Strange... this is fine locally, and our c-i didn't have any trouble with it either.
Is the problem locally reproducible?

Yes. Turns out the issue is actually a change in defaults in GCC 10.1. See #19553. In the mean time, I'll retest here with an earlier GCC.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Jul 22, 2020
While testing bitcoin#19530 I noticed that we couldn't call dsymutil after LTO:
```bash
../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
```

This updates configure to call `AC_PATH_TOOL` so that we end up with the
full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
`/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-otool`.
laanwj added a commit that referenced this pull request Jul 22, 2020
2a701a1 build: pass -fcommon when building genisoimage (fanquake)

Pull request description:

  Starting with the 10.1 release, GCC [defaults to -fno-common](https://gcc.gnu.org/gcc-10/porting_to.html). This causes
  linking issues when building genisoimage:
  ```bash
  [ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/checksum.o
  [100%] Linking C executable genisoimage
  /usr/bin/ld: CMakeFiles/genisoimage.dir/apple.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  /usr/bin/ld: CMakeFiles/genisoimage.dir/boot.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  /usr/bin/ld: CMakeFiles/genisoimage.dir/desktop.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  /usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_file.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  ```

  Rather than patching genisoimage further, just pass -fcommon to preserve
  the legacy GCC behaviour.

  Noticed [while testing #19530](#19530 (comment)).

ACKs for top commit:
  laanwj:
    ACK 2a701a1
  hebasto:
    ACK 2a701a1, tested on Fedora 32 (x86_64, GCC 10.1.1):

Tree-SHA512: 0e6dd1b4c7c3f5b6997616da27a7ea54f757c81677766ee6813c776356527787a32b959f834e699fee9798553b13bcb142469f442c5e7a04f810f2c06e3a9505
@laanwj
Copy link
Member
laanwj commented Jul 22, 2020

Concept ACK, even though we don't use LTO for builds currently, if it isn't unnecessary to disable it explicitly in the toolchain, we shouldn't.

@theuni
Copy link
Member Author
theuni commented Jul 23, 2020

Continuing the conversation from #19565:

The solution seemed to be to pass -object_path_lto through to ld64. Doing that I end up with a properly generated .dSYM:

I think I understand what's happening here. From the clang docs

  The default for :option:`-flto` is "full", in which the
  LLVM bitcode is suitable for monolithic Link Time Optimization (LTO), where
  the linker merges all such modules into a single combined module for
  optimization. With "thin", :doc:`ThinLTO <../ThinLTO>`
  compilation is invoked instead.

  .. note::

     On Darwin, when using :option:`-flto` along with :option:`-g` and
     compiling and linking in separate steps, you also need to pass
     ``-Wl,-object_path_lto,<lto-filename>.o`` at the linking step to instruct the
     ld64 linker not to delete the temporary object file generated during Link
     Time Optimization (this flag is automatically passed to the linker by Clang
     if compilation and linking are done in a single step). This allows debugging
     the executable as well as generating the ``.dSYM`` bundle using :manpage:`dsymutil(1)`.

That's describing us.

The machinery to get clang to do this for you only applies when compiling and linking at the same time. Here's a comment from elsewhere explaining why:

      // Add a 'dsymutil' step if necessary, when debug info is enabled and we
      // have a compile input. We need to run 'dsymutil' ourselves in such cases
      // because the debug info will refer to a temporary object file which
      // will be removed at the end of the compilation process.

I can confirm that is working as intended with a test-case:

$ cat lto1.cpp lto2.cpp

//lto1.cpp
void MyFunc();
int main(){
MyFunc();
}
//lto2.cpp
#include <stdio.h>
void MyFunc()
{
    printf("hello world\n");
}

$ clang++ -target x86_64-apple-darwin16 -mmacosx-version-min=10.12 --sysroot /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers -stdlib=libc++ -mlinker-version=530 -B/home/cory/dev/bitcoin2/depends/x86_64-apple-darwin16/native/bin -nostdinc++ -isystem /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/usr/include/c++/v1 -flto=thin -std=c++11 lto1.cpp lto2.cpp -v

Which produces the following link line and calls dsymutil:

"/home/cory/dev/bitcoin2/depends/x86_64-apple-darwin16/native/bin/x86_64-apple-darwin16-ld" -demangle -object_path_lto /tmp/thinlto-d130c2 -lto_library /home/cory/dev/llvm-project/build/lib/libLTO.so -no_deduplicate -dynamic -arch x86_64 -platform_version macos 10.12.0 0.0.0 -syslibroot /home/cory/dev/bitcoin2/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers -o a.out /tmp/lto1-bd26d0.o /tmp/lto2-439b81.o -lc++ -lSystem
"/home/cory/dev/llvm-project/build/bin/dsymutil" -o a.out.dSYM a.out

So, it sounds as though everything is actually working as intended. We just need to add that link flag if lto and debugging are on. But we're getting ahead of ourselves, as we don't have an option for lto yet :)

fanquake added a commit that referenced this pull request Jul 24, 2020
…mpile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing #19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
naumenkogs pushed a commit to naumenkogs/bitcoin that referenced this pull request Jul 24, 2020
While testing bitcoin#19530 I noticed that we couldn't call dsymutil after LTO:
```bash
../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
```

This updates configure to call `AC_PATH_TOOL` so that we end up with the
full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
`/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-otool`.
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 24, 2020
While testing bitcoin#19530 I noticed that we couldn't call dsymutil after LTO:
```bash
../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
```

This updates configure to call `AC_PATH_TOOL` so that we end up with the
full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
`/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-otool`.
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 24, 2020
2a701a1 build: pass -fcommon when building genisoimage (fanquake)

Pull request description:

  Starting with the 10.1 release, GCC [defaults to -fno-common](https://gcc.gnu.org/gcc-10/porting_to.html). This causes
  linking issues when building genisoimage:
  ```bash
  [ 98%] Building C object genisoimage/CMakeFiles/genisoimage.dir/checksum.o
  [100%] Linking C executable genisoimage
  /usr/bin/ld: CMakeFiles/genisoimage.dir/apple.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  /usr/bin/ld: CMakeFiles/genisoimage.dir/boot.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  /usr/bin/ld: CMakeFiles/genisoimage.dir/desktop.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  /usr/bin/ld: CMakeFiles/genisoimage.dir/dvd_file.o:(.bss+0x0): multiple definition of `outfile'; CMakeFiles/genisoimage.dir/genisoimage.o:(.bss+0x0): first defined here
  ```

  Rather than patching genisoimage further, just pass -fcommon to preserve
  the legacy GCC behaviour.

  Noticed [while testing bitcoin#19530](bitcoin#19530 (comment)).

ACKs for top commit:
  laanwj:
    ACK 2a701a1
  hebasto:
    ACK 2a701a1, tested on Fedora 32 (x86_64, GCC 10.1.1):

Tree-SHA512: 0e6dd1b4c7c3f5b6997616da27a7ea54f757c81677766ee6813c776356527787a32b959f834e699fee9798553b13bcb142469f442c5e7a04f810f2c06e3a9505
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 24, 2020
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
Copy link
Member
@fanquake fanquake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 5962522. The relevant option upstream is here.

@fanquake fanquake merged commit e0ed4cc into bitcoin:master Jul 28, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 28, 2020
5962522 depends: bump native_cctools for fixed lto with external clang (Cory Fields)
00d1ba7 depends: enable lto support for Apple's ld64 (Cory Fields)

Pull request description:

  This didn't work for a few reasons (various toolchain compatibility issues) the last time I tested it, but after the last round of bumps it works with no apparent issues.

  Note that this does not _enable_ LTO by default in any way, only hooks up the machinery for ```-flto``` to work correctly when specified.

  Lines were split for an easier rebase after bitcoin#17919 is merged.

ACKs for top commit:
  fanquake:
    ACK 5962522. The relevant option upstream is [here](https://github.com/tpoechtrager/cctools-port/blob/master/cctools/m4/llvm.m4#L4).

Tree-SHA512: df2775e74e7bc847e6cef94cb8457d503d6c9e2fdea861e51386fa6ed5a7ba688241db3685561ae1a32f66724c1b3801727252025f00c04b90a3bdc8a4f6f93b
zkbot added a commit to zcash/zcash that referenced this pull request Jul 30, 2020
Modernise macOS cross-compilation toolchain

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#13617
  - Excluding the QT GUI changes.
- bitcoin/bitcoin#17550
- bitcoin/bitcoin#16392
  - Excluding the QT GUI changes.
- bitcoin/bitcoin#18589
- bitcoin/bitcoin#19240
- bitcoin/bitcoin#19407
- bitcoin/bitcoin#17919
  - Only the ancillary changes, not the `FORCE_USE_SYSTEM_CLANG` change.
- bitcoin/bitcoin#19530

After these changes, macOS versions earlier than 10.12 are no longer supported.

To cross-compile for macOS:
- Follow the instructions in `contrib/macdeploy/README.md` to generate
  `Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz`
  (requires an Apple Developer Account)
- Extract it into `depends/SDKs` (creating that folder first if it does not exist)
- `HOST=x86_64-apple-darwin16 ./zcutil/build.sh`
Warchant pushed a commit to Warchant/bitcoin that referenced this pull request Aug 6, 2020
While testing bitcoin#19530 I noticed that we couldn't call dsymutil after LTO:
```bash
../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
```

This updates configure to call `AC_PATH_TOOL` so that we end up with the
full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
`/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-otool`.
furszy added a commit to PIVX-Project/PIVX that referenced this pull request May 25, 2021
e1b89ac Fix QPainter non-determinism on macOS (Andrew Chow)
831c317 macOS deploy: use the new plistlib API (Jonas Schnelli)
5857aaf doc: Document ALLOW_HOST_PACKAGES dependency option (skmcontrib)
2329e08 build: Fix behavior when ALLOW_HOST_PACKAGES unset (Hennadii Stepanov)
1768870 depends: native_ds_store 1.3.0 (fanquake)
3f9f3e5 depends: pull upstream libdmg-hfsplus changes (fanquake)
f7606dc depends: latest config.guess & config.sub (fanquake)
cc3ae74 depends: bump native_cctools for fixed lto with external clang (Cory Fields)
b26c648 depends: enable lto support for Apple's ld64 (Cory Fields)
50933d7 depends: Add documentation for FORCE_USE_SYSTEM_CLANG make flag (Carl Dong)
ba3ddf2 depends: Reformat make options as definition list (Carl Dong)
3b855a7 depends: Add justifications for macOS clang flags (Carl Dong)
4104de0 depends: specify libc++ header location for darwin (Cory Fields)
cd4335f depends: force a new host id string if FORCE_USE_SYSTEM_CLANG is in use (Cory Fields)
d30e1af depends: Allow building with system clang (Carl Dong)
234828b depends: Decouple toolchain + binutils (Carl Dong)
1dd3a5a doc: explain why passing -mlinker-version is required (fanquake)
5cc0d0f darwin: pass mlinker-version so that clang enables new features (Cory Fields)
813a552 macos: Bump to xcode 11.3.1 and 10.15 SDK (Cory Fields)
ee7085f depends: bump MacOS toolchain (Cory Fields)
e5b092b contrib: macdeploy: Remove historical extraction notes (Carl Dong)
5893caf contrib: macdeploy: Use apple-sdk-tools instead of xar+pbzx (Carl Dong)
9f2d4ba native_cctools: Don't use libc++ from pinned clang (Carl Dong)
0c8d217 Adapt rest of tooling to new SDK naming scheme (Carl Dong)
bdacfa8 contrib: macdeploy: Correctly generate macOS SDK (Carl Dong)
f7eee2c Fix naming of macOS SDK and clarify version (Andrew Chow)
62f9e23 build: use macOS 10.14 SDK (fanquake)
bc2e1af depends: native_cctools 921, ld64 409.12, libtapi 1000.10.8 (fanquake)
a296d87 depends: clang 6.0.1 (fanquake)
8f6c475 build: Set minimum supported macOS to 10.12 (Fuzzbawls)

Pull request description:

  This backports the following upstream PRs to update the macOS cross-compiling tools:

  bitcoin#17550
  bitcoin#16392
  bitcoin#18589
  bitcoin#19240
  bitcoin#19407
  bitcoin#17919
  bitcoin#19530
  bitcoin#17057
  bitcoin#20333
  bitcoin#18051
  bitcoin#19124
  bitcoin#20298
  bitcoin#20447

  The tools being updated are

  ### Clang
  Upgraded from `3.7.1` to `8.0.0`

  ### cctools

  * cctools `877.8` -> `949.0.1`
  * LD64 `253.9` -> `530`
  * TAPI `1000.10.8`

  ### DSStore
  Upgraded from `1.1.2` to `1.3.0` (this removes the biplist dependency)

  This also effectively bumps our minimum supported macOS version to 10.12 (Sierra).

ACKs for top commit:
  furszy:
    tested ACK e1b89ac
  random-zebra:
    utACK e1b89ac

Tree-SHA512: f5cec8db57e07d8855070646b9e1400d48aac1d01e3c2c3b3e134665c6372d6535f3328888bb9a75087f7b3d5231ecb4b509723bfa51bd40770ffe2810c67f65
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jun 27, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jun 28, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jun 29, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jul 1, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jul 1, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jul 15, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jul 16, 2021
…ross-compile

ef3d4ce build: call AC_PATH_TOOL for dsymutil in macOS cross-compile (fanquake)

Pull request description:

  While testing bitcoin#19530 I noticed that we couldn't call [`dsymutil`](https://www.llvm.org/docs/CommandGuide/dsymutil.html) after LTO:
  ```bash
  ../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
  ```

  This updates configure to call `AC_PATH_TOOL` so that we end up with the
  full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
  `/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-dsymutil`.

ACKs for top commit:
  laanwj:
    Code review ACK ef3d4ce
  theuni:
    ACK ef3d4ce.

Tree-SHA512: e4fa93e7f9f7945289143dfe2a6645ad8ee7f3bee0793412b3509901a30566d6f952e3b39e0e525a54f8dbd0c480f8da70fc6cb80b07800d11b0c6071fbb7466
kwvg pushed a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg pushed a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 25, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 25, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 27, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 30, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 2, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 4, 2021
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0