8000 build: Automatically include both `git`-tracked and bootstrapped files. by dongcarl · Pull Request #18478 · bitcoin/bitcoin · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

build: Automatically include both git-tracked and bootstrapped files. #18478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

dongcarl
Copy link
Contributor
This allows us to:

1. Retain the automatic inclusion of bootstrapped files inside dist
   source tarballs
2. Also automatically include all git-tracked files in EXTRA_DIST

Notes:

1. We already rely on git for `make dist` anyway (see dist-hook).
2. As of automake:77d39959511295f5a30332d5d03f0a6956bd9460 (this is just
   the latest master I have so there's a point of reference for the
   future), we can observe the behaviour of `make dist` in
   `lib/am/distdir.am`. Search for the `distdir-am` target within that
   file. You will see that if a file was already added to the distdir,
   it won't be added again.

I also performed a gitian build, here's a diff of the source tarball and one produced with simply git archive, as expected, the difference is only in the bootstrapped files.

$ diff -rl gitian git
Only in gitian: aclocal.m4
Only in gitian/build-aux: compile
Only in gitian/build-aux: config.guess
Only in gitian/build-aux: config.sub
Only in gitian/build-aux: depcomp
Only in gitian/build-aux: install-sh
Only in gitian/build-aux: ltmain.sh
Only in gitian/build-aux/m4: libtool.m4
Only in gitian/build-aux/m4: lt~obsolete.m4
Only in gitian/build-aux/m4: ltoptions.m4
Only in gitian/build-aux/m4: ltsugar.m4
Only in gitian/build-aux/m4: ltversion.m4
Only in gitian/build-aux: missing
Only in gitian/build-aux: test-driver
Only in gitian: configure
Only in gitian/doc/man: Makefile.in
Only in gitian: Makefile.in
Only in gitian/src/config: bitcoin-config.h.in
Only in gitian/src: Makefile.in
Only in gitian/src/secp256k1: aclocal.m4
Only in gitian/src/secp256k1/build-aux: compile
Only in gitian/src/secp256k1/build-aux: config.guess
Only in gitian/src/secp256k1/build-aux: config.sub
Only in gitian/src/secp256k1/build-aux: depcomp
Only in gitian/src/secp256k1/build-aux: install-sh
Only in gitian/src/secp256k1/build-aux: ltmain.sh
Only in gitian/src/secp256k1/build-aux/m4: libtool.m4
Only in gitian/src/secp256k1/build-aux/m4: lt~obsolete.m4
Only in gitian/src/secp256k1/build-aux/m4: ltoptions.m4
Only in gitian/src/secp256k1/build-aux/m4: ltsugar.m4
Only in gitian/src/secp256k1/build-aux/m4: ltversion.m4
Only in gitian/src/secp256k1/build-aux: missing
Only in gitian/src/secp256k1/build-aux: test-driver
Only in gitian/src/secp256k1: configure
Only in gitian/src/secp256k1: Makefile.in
Only in gitian/src/secp256k1/src: libsecp256k1-config.h.in
Only in gitian/src/univalue: aclocal.m4
Only in gitian/src/univalue/build-aux: compile
Only in gitian/src/univalue/build-aux: config.guess
Only in gitian/src/univalue/build-aux: config.sub
Only in gitian/src/univalue/build-aux: depcomp
Only in gitian/src/univalue/build-aux: install-sh
Only in gitian/src/univalue/build-aux: ltmain.sh
Only in gitian/src/univalue/build-aux/m4: libtool.m4
Only in gitian/src/univalue/build-aux/m4: lt~obsolete.m4
Only in gitian/src/univalue/build-aux/m4: ltoptions.m4
Only in gitian/src/univalue/build-aux/m4: ltsugar.m4
Only in gitian/src/univalue/build-aux/m4: ltversion.m4
Only in gitian/src/univalue/build-aux: missing
Only in gitian/src/univalue/build-aux: test-driver
Only in gitian/src/univalue: configure
Only in gitian/src/univalue: Makefile.in
Only in gitian/src/univalue: univalue-config.h.in

@DrahtBot
Copy link
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@hebasto
Copy link
Member
hebasto commented Mar 31, 2020

Meta question: could we described the "users" of bootstrapped files? What systems they are use? What are their workflows? etc.

I suppose a user of the Bitcoin Core, who is willing to build his binaries from sources, runs a non-unusual Linux distro, and is able to start build with ./autogen.sh. If we assume that non-unusual system has pkg-config installed, why we couldn't make the same assumption about autotools itself?

@dongcarl
Copy link
Contributor Author

In my mind, including bootstrapped files (and effectively pinning the autotools version) is better than not including them. Why would we not also include bootstrapped files if it's easy to do?

0fc

@maflcko
Copy link
Member
maflcko commented Mar 31, 2020

Concept ACK. Travis out-of-tree builds are failing.

@dongcarl dongcarl force-pushed the 2020-03-adventures-in-reconciling-automake-and-git branch from 864bc27 to f2ba8ec Compare March 31, 2020 18:33
hebasto and others added 3 commits March 31, 2020 14:46
This allows us to:

1. Retain the automatic inclusion of bootstrapped files inside dist
   source tarballs
2. Also automatically include all git-tracked files in EXTRA_DIST

Notes:

1. We already rely on git for `make dist` anyway (see dist-hook).
2. As of automake:77d39959511295f5a30332d5d03f0a6956bd9460 (this is just
   the latest master I have so there's a point of reference for the
   future), we can observe the behaviour of `make dist` in
   `lib/am/distdir.am`. Search for the `distdir-am` target within that
   file. You will see that if a file was already added to the distdir,
   it won't be added again.
Some of this file ordering may not be necessary anymore, we can
investigate further and come back to this.
@dongcarl dongcarl force-pushed the 2020-03-adventures-in-reconciling-automake-and-git branch from f2ba8ec to 1b91ade Compare March 31, 2020 18:46
@ryanofsky
Copy link
Contributor

Why would we not also include bootstrapped files if it's easy to do?

I don't care about this very much, but will answer the question.

Advantages of including bootstrapped files:

  • Person building from source no longer has to install autotools and m4, just a c++ compiler, linker, and gnu make.

Disadvantages of including bootstrapped files:

  • Adds to differences between tarball distribution and git checkout. Different instructions for building one vs the other. Different failures that can occur. Possible lack of testing with normal developers running autotools themselves and only less experienced people using the generated scripts

  • Bigger tarball

  • Noisier diffs. Without generated files, someone could reasonably diff two point releases and get a sense of what changed. With generated files, such as the 35,000 line 1.1M top-level configure script, actual changes are easily drowned in sea of noise

  • Aesthetics, simplicity. This is subjective, but personally I like simplicity of source distributions that stick to delivering actual sources and don't mix up sources and intermediate files without clear reason for doing so. Is there a clear reason why it's good to include these intermediate files, and but not include intermediate files like precompiled headers or test datadirs, or maybe even object files or binaries? It's nice to not think about these things and just stick to sources

@DrahtBot
Copy link
Contributor

Gitian builds

File commit 965c0c3
(master)
commit 81df43f
(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz 57aab7e587f3ea0a... 2d249de922ed2266...
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz 5bb0f02f1749f913... 19e71105df4d217a...
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz 3b670b4a99116f5f... 5822a29e3bf6f70e...
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz aad52c43bf0d86ab... 188e7f77118b4c3f...
bitcoin-0.19.99-osx-unsigned.dmg 938cc8fcbcda261c... e8465963adc6bf1c...
bitcoin-0.19.99-osx64.tar.gz e5cd3dd49d87787c... bedec5880b84464b...
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz c5eadbd98b89a453... 11f1e2a05a9d30d3...
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz 585e4150e48a03f7... 8b92f8fb7d723ad0...
bitcoin-0.19.99-win64-debug.zip 46a08a338aa679ce... f2fe736893f394b0...
bitcoin-0.19.99-win64-setup-unsigned.exe 4e794768b43a81bc... f6b50d45230fde1d...
bitcoin-0.19.99-win64.zip a15127e0ed4fd3d6... 8b8509f17e657c49...
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz 7994d9e40a424497... 32be44e4b180200c...
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz c3b2d37b3df511c4... 9b779e630f165842...
bitcoin-0.19.99.tar.gz a39aee426533d6c0... 831780d143ac839d...
bitcoin-core-linux-0.20-res.yml 7cf55f32bec5f8be... 6226c038cdae9425...
bitcoin-core-osx-0.20-res.yml 6b594bf6aad6c167... 2ce1f60873083d02...
bitcoin-core-win-0.20-res.yml e6738ec03bc3e6b6... 28dda787c7ef5167...
linux-build.log 057410ad898e35c5... e3b323907c11b6d3...
osx-build.log 156f4048415dd68c... 3e35a12d69c14ea5...
win-build.log 4c690476ddd8905d... 1277bce228b24178...
bitcoin-core-linux-0.20-res.yml.diff e4e2c39386cf2b77...
bitcoin-core-osx-0.20-res.yml.diff ed8bf1b2681b5f83...
bitcoin-core-win-0.20-res.yml.diff 89e776f97eefcc71...
linux-build.log.diff 203043a1de64b2d4...
osx-build.log.diff cfe1ff3dd568248f...
win-build.log.diff 2c4011876e5c05d3...

@laanwj
Copy link
Member
laanwj commented Apr 1, 2020

I personally prefer using git archive here. It's nice and simple and straightforward.

Calling git ls-files in the makefile is imo a bit strange, given tarballs etc don't build from git.

@dongcarl
Copy link
Contributor Author
dongcarl commented Apr 1, 2020

Noted, closing.

Followups:

  • We might need the no-dist automake option, as I don't think make dist makes sense anymore
  • We should clean up dist related targets in Makefile.am

@dongcarl dongcarl closed this Apr 1, 2020
@luke-jr
Copy link
Member
luke-jr commented Apr 1, 2020

make dist should be used by gitian.

git archive would omit important files (like configure!).

@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants
0