This change unifies Homebrew packages workflow, and does not change
behavior.
Github-Pull: #20527
Rebased-From: c96d1f65a552712f8476269ad64a415717ead50d
0e2a5e448f426219a6464b9aaadcc715534114e6 tests: dumping and minimizing of script assets data (Pieter Wuille)
4567ba034c5ae6e6cc161360f7425c9e844738f0 tests: add generic qa-asset-based script verification unit test (Pieter Wuille)
f06e6d03452cf5e0b1a0863afb08c9e6d3ef452e tests: functional tests for Schnorr/Taproot/Tapscript (Pieter Wuille)
3c226639eb134314a0640d34e4ccb6148dbde22f tests: add BIP340 Schnorr signature support to test framework (Pieter Wuille)
206fb180ec6ee5f916afc6f574000d716daf79b7 --- [TAPROOT] Tests --- (Pieter Wuille)
d7ff237f2996a4c11fdf9399187c2d2b26bf9809 Activate Taproot/Tapscript on regtest (BIP 341, BIP 342) (Pieter Wuille)
e9a021d7e6a454d610a45cb9b3995f0d96a5fbb6 Make Taproot spends standard + policy limits (Pieter Wuille)
865d2c37e2e44678498b7f425b65e01b1e231cde --- [TAPROOT] Regtest activation and policy --- (Pieter Wuille)
72422ce396b8eba7b1a72c171c2f07dae691d1b5 Implement Tapscript script validation rules (BIP 342) (Johnson Lau)
330de894a9a48515d9a473448b6c67adc3d188be Use ScriptExecutionData to pass through annex hash (Pieter Wuille)
8bbed4b7acf4c76eaea8c0e10f3cbf6ba4e53809 Implement Taproot validation (BIP 341) (Pieter Wuille)
0664f5fe1f77f08d235aa3750b59428257b0b91d Support for Schnorr signatures and integration in SignatureCheckers (BIP 340) (Pieter Wuille)
5de246ca8159dcffaa4c136a60c8bfed2028e2ee Implement Taproot signature hashing (BIP 341) (Johnson Lau)
9eb590894f15ff40806039bfd32972fbc260e30d Add TaggedHash function (BIP 340) (Pieter Wuille)
450d2b23710ad296eede81339195376021ab5500 --- [TAPROOT] BIP340/341/342 consensus rules --- (Pieter Wuille)
5d62e3a68b6ea9bb03556ee1fbf5678f20be01a2 refactor: keep spent outputs in PrecomputedTransactionData (Pieter Wuille)
8bd2b4e78452ff69c08c37acf164a6b80e503f13 refactor: rename scriptPubKey in VerifyWitnessProgram to exec_script (Pieter Wuille)
107b57df9fa8b2d625d2b342dc77722282a6ae4c scripted-diff: put ECDSA in name of signature functions (Pieter Wuille)
f8c099e2207c90d758e7a659d6a55fa7ccb7ceaa --- [TAPROOT] Refactors --- (Pieter Wuille)
Pull request description:
This is an implementation of the Schnorr/taproot consensus rules proposed by BIPs [340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), [341](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), and [342](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
See the list of commits [below](https://github.com/bitcoin/bitcoin/pull/19953#issuecomment-691815830). No signing or wallet support of any kind is included, as testing is done entirely through the Python test framework.
This is a successor to https://github.com/bitcoin/bitcoin/pull/17977 (see discussion following [this comment](https://github.com/bitcoin/bitcoin/pull/17977#issuecomment-682285983)), and will have further changes squashed/rebased. The history of this PR can be found in #19997.
ACKs for top commit:
instagibbs:
reACK 0e2a5e448f
benthecarman:
reACK 0e2a5e4
kallewoof:
reACK 0e2a5e448f426219a6464b9aaadcc715534114e6
jonasnick:
ACK 0e2a5e448f426219a6464b9aaadcc715534114e6 almost only looked at bip340/libsecp related code
jonatack:
ACK 0e2a5e448f426219a6464b9aaadcc715534114e6 modulo the last four commits (tests) that I plan to finish reviewing tomorrow
fjahr:
reACK 0e2a5e448f426219a6464b9aaadcc715534114e6
achow101:
ACK 0e2a5e448f426219a6464b9aaadcc715534114e6
Tree-SHA512: 1b00314450a2938a22bccbb4e177230cf08bd365d72055f9d526891f334b364c997e260c10bc19ca78440b6767712c9feea7faad9a1045dd51a5b96f7ca8146e
ba8950ee0134a7958e3e9b041cd54d222feb09a1 build: optionally skip external warnings (Vasil Dimov)
Pull request description:
Add an option to `./configure` to suppress compilation warnings from
external headers. The option is off by default (no change in behavior,
show warnings from external headers).
This option is useful if e.g. Boost or Qt is installed outside of
`/usr/include` (warnings from headers in `/usr/include` are already
suppressed by default) and those warnings stand in the way of compiling
Bitcoin Core with `-Werror[=...]` or they just clutter the build output
too much and make our own warnings hard to spot.
`-isystem /usr/include` bricks GCC's `#include_next`, so we use
`-idirafter` instead. This way we don't have to treat `/usr/include`
specially.
ACKs for top commit:
practicalswift:
ACK ba8950ee0134a7958e3e9b041cd54d222feb09a1: diff looks correct!
hebasto:
ACK ba8950ee0134a7958e3e9b041cd54d222feb09a1, tested on Linux Mint 20 (x86_64).
luke-jr:
utACK ba8950ee0134a7958e3e9b041cd54d222feb09a1
Tree-SHA512: 9b54fae8590be6c79f2688a5aca09e0a9067f481dabecdd49bb278c08a62ac2b0cc704c894fbd53240e77ac84da0c7a237845df0a696cfbdb0359e1c8e2e10c9
This enables the schnorrsig module in libsecp256k1, adds the relevant types
and functions to src/pubkey, as well as in higher-level `SignatureChecker`
classes. The (verification side of the) BIP340 test vectors is also added.
Add an option to `./configure` to suppress compilation warnings from
external headers. The option is off by default (no change in behavior,
show warnings from external headers).
This option is useful if e.g. Boost or Qt is installed outside of
`/usr/include` (warnings from headers in `/usr/include` are already
suppressed by default) and those warnings stand in the way of compiling
Bitcoin Core with `-Werror[=...]` or they just clutter the build output
too much and make our own warnings hard to spot.
c4b85ba704a1b5257dc82786a84f676bacb7b027 Bugfix: Define and use HAVE_FDATASYNC correctly outside LevelDB (Luke Dashjr)
Pull request description:
Fixes a bug introduced in #19614
The LevelDB-specific fdatasync check was only using `AC_SUBST`, which works for Makefiles, but doesn't define anything for C++. Furthermore, the #define is typically 0 or 1, never undefined.
This fixes both issues by defining it and checking its value instead of whether it is merely defined.
Pulled out of #14501 by fanquake's request
ACKs for top commit:
fanquake:
ACK c4b85ba704a1b5257dc82786a84f676bacb7b027 - thanks for catching and fixing my mistake.
laanwj:
Code review ACK c4b85ba704a1b5257dc82786a84f676bacb7b027
Tree-SHA512: 91d5d426ba000b4f3ee7e2315635e24bbb23ceff16269ddf4f65a63d25fc9e9cf94a3b236eed2f8031cc36ddcf78aeb5916efcb244f415943a8a12f907ede8f9
b536813cefc13f5c54a28a7c2fce8c69e89d6624 build: add -fstack-clash-protection to hardening flags (fanquake)
076183b36b76a11438463883ff916f17aef9e001 build: add -fcf-protection=full to hardening options (fanquake)
Pull request description:
Beginning with Ubuntu `19.10`, it's packaged GCC now has some additional hardening options enabled by default (in addition to existing defaults like `-fstack-protector-strong` and reducing the minimum ssp buffer size). The new additions are`-fcf-protection=full` and `-fstack-clash-protection`.
> -fcf-protection=[full|branch|return|none]
> Enable code instrumentation of control-flow transfers to increase program security by checking that target addresses of control-flow transfer instructions (such as indirect function call, function return, indirect jump) are valid. This prevents diverting the flow of control to an unexpected target. This is intended to protect against such threats as Return-oriented Programming (ROP), and similarly call/jmp-oriented programming (COP/JOP).
> -fstack-clash-protection
> Generate code to prevent stack clash style attacks. When this option is enabled, the compiler will only allocate one page of stack space at a time and each page is accessed immediately after allocation. Thus, it prevents allocations from jumping over any stack guard page provided by the operating system.
If your interested you can grab `gcc-9_9.3.0-10ubuntu2.debian.tar.xz` from https://packages.ubuntu.com/focal/g++-9. The relevant changes are part of the `gcc-distro-specs` patches, along with the relevant additions to the gcc manages:
> NOTE: In Ubuntu 19.10 and later versions, -fcf-protection is enabled by default for C, C++, ObjC, ObjC++, if none of -fno-cf-protection nor -fcf-protection=* are found.
> NOTE: In Ubuntu 19.10 and later versions, -fstack-clash-protection is enabled by default for C, C++, ObjC, ObjC++, unless -fno-stack-clash-protection is found.
So, if you're C++ using GCC on Ubuntu 19.10 or later, these options will be active unless you explicitly opt out. This can be observed with a small test:
```c++
int main() { return 0; }
```
```bash
g++ --version
g++ (Ubuntu 9.3.0-10ubuntu2) 9.3.0
g++ test.cpp
objdump -dC a.out
..
0000000000001129 <main>:
1129: f3 0f 1e fa endbr64
112d: 55 push %rbp
112e: 48 89 e5 mov %rsp,%rbp
1131: b8 00 00 00 00 mov $0x0,%eax
1136: 5d pop %rbp
1137: c3 retq
1138: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
113f: 00
# recompile opting out of control flow protection
g++ test.cpp -fcf-protection=none
objdump -dC a.out
...
0000000000001129 <main>:
1129: 55 push %rbp
112a: 48 89 e5 mov %rsp,%rbp
112d: b8 00 00 00 00 mov $0x0,%eax
1132: 5d pop %rbp
1133: c3 retq
1134: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
113b: 00 00 00
113e: 66 90 xchg %ax,%ax
```
Note the insertion of an `endbr64` instruction when compiling and _not_ opting out. This instruction is part of the Intel Control-flow Enforcement Technology [spec](https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf), which the GCC control flow implementation is based on.
If we're still doing gitian builds for the `0.21.0` and `0.22.0` releases, we'd likely update the gitian image to Ubuntu Focal, which would mean that the GCC used for gitian builds would also be using these options by default. So we should decide whether we want to explicitly turn these options on as part of our hardening options (although not just for this reason), or, we should be opting-out.
GCC has supported both options since 8.0.0. Clang has supported `-fcf-protection` from 7.0.0 and will support `-fstack-clash-protection` in it's upcoming [11.0.0 release](https://clang.llvm.org/docs/ReleaseNotes.html#id6).
ACKs for top commit:
jamesob:
ACK b536813cefc13f5c54a28a7c2fce8c69e89d6624 ([`jamesob/ackr/18921.1.fanquake.build_add_stack_clash_an`](https://github.com/jamesob/bitcoin/tree/ackr/18921.1.fanquake.build_add_stack_clash_an))
laanwj:
Code review ACK b536813cefc13f5c54a28a7c2fce8c69e89d6624
Tree-SHA512: abc9adf23cdf1be384f5fb9aa5bfffdda86b9ecd671064298d4cda0440828b509f070f9b19c88c7ce50ead9ff32afff9f14c5e78d75f01241568fbfa077be0b7
1ccb9f30c040daf688f89f0d63e9f5e7b131d193 Move Win32 defines to configure.ac to ensure they are globally defined (Luke Dashjr)
Pull request description:
#9245 no longer needs this, since the main `_WIN32_WINNT` got bumped by something else.
So rather than just lose it, might as well get it merged in independently.
I'm not aware of any practical effects, but it seems safer to use the same API versions everywhere.
ACKs for top commit:
fanquake:
ACK 1ccb9f30c040daf688f89f0d63e9f5e7b131d193 - checked that the binaries produced are the same.
Tree-SHA512: 273e9186579197be01b443b6968e26b9a8031d356fabc5b73aa967fcdb837df195b7ce0fc4e4529c85d9b86da6f2d7ff1bf56a3ff0cbbcd8cee8a9c2bf70a244
4af4672525f698c7b491023fcd36a17a4e982070 build, qt: Add Qt version checking (Hennadii Stepanov)
30e336f785e1b662cade3cfacea91a9785ea1023 build: Drop unused bitcoin_cv_qt58 (Hennadii Stepanov)
Pull request description:
Now `configure` script checks that Qt version is not less then minimum required (currently [5.5.1](https://github.com/bitcoin/bitcoin/pull/15393)).
This PR is an alternative to #15706 (see https://github.com/bitcoin/bitcoin/pull/15706#issuecomment-629076962).
Closes#15688.
The first commit removes dead code (see https://github.com/bitcoin/bitcoin/pull/18297#issuecomment-603252662).
ACKs for top commit:
fanquake:
ACK 4af4672525f698c7b491023fcd36a17a4e982070 - this looks ok. I've tested this with Qt 5.15.0 and Qt 5.7.1 system libs, as well as 5.9.8 from depends.
Tree-SHA512: 8e3b82fa3a98926814923331038185633fabad962c271f31bd158e1ab293dcde52ab1dbf997745540a9ed27e16835cf5b5f3701d405876d877fa561eb03cc619
2f8a4c9a06ace680b3dff7cd7d5e33204fe45909 build: Enable some commonly enabled compiler diagnostics (practicalswift)
Pull request description:
Enable some commonly enabled compiler diagnostics as discussed in #17344.
| Compiler diagnostic | no# of emitted unique GCC warnings in `master` | no# of emitted unique Clang warnings in `master` |
| ------------- | ------------- | ------------- |
| `-Wduplicated-branches`: Warn if `if`/`else` branches have duplicated code | 0 | Not supported |
| `-Wduplicated-cond`: Warn if `if`/`else` chain has duplicated conditions | 0 | Not supported |
| `-Wlogical-op`: Warn about logical operations being used where bitwise were probably wanted | 0 | Not supported |
| `-Woverloaded-virtual`: Warn if you overload (not `override`) a virtual function | 0 | 0 |
| ~~`-Wunused-member-function`: Warn on unused member function~~ | Not supported | 2 |
| ~~`-Wunused-template`: Warn on unused template~~ | Not supported | 1 |
There is a large overlap between this list and [Jason Turner's list of recommended compiler diagnostics in the Collaborative Collection of C++ Best Practices (`cppbestpractices`) project](https://github.com/lefticus/cppbestpractices/blob/master/02-Use_the_Tools_Available.md#gcc--clang). There is also an overlap with the recommendations given in the [C++ Core Guidelines](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines) (with editors Bjarne Stroustrup and Herb Sutter).
Closes#17344.
ACKs for top commit:
jonatack:
ACK 2f8a4c9a06ace6 no warnings for me with these locally on debian 5.7.10-1 (2020-07-26) x86_64 with gcc 10 and clang 12
fanquake:
ACK 2f8a4c9a06ace680b3dff7cd7d5e33204fe45909 - no-longer seeing any obvious issues with doing this.
hebasto:
ACK 2f8a4c9a06ace680b3dff7cd7d5e33204fe45909, no new warnings in Travis jobs.
Tree-SHA512: f669ea22b31263a555f999eff6a9d65750662e95831b188c3192a2cf0127fb7b5136deb762a6b0b7bbdfb0dc6a40caf48251a62b164fffb81dd562bdd15ec3c8
70452a070b3b6d4d650a948b3337bc7b8bb2c3c3 build: set minimum required Boost to 1.58 (fanquake)
Pull request description:
Any systems which only have an older installable Boost can use depends.
1.58.0 retains compatibility with the packages [installable on Ubuntu 16.04](https://packages.ubuntu.com/xenial/libboost-dev).
The projects usage of Boost wont be going away any time soon, if ever (i.e #15382), and our usage of the test framework.
Fixes: #19506
ACKs for top commit:
practicalswift:
ACK 70452a070b3b6d4d650a948b3337bc7b8bb2c3c3 -- patch looks correct
laanwj:
ACK 70452a070b3b6d4d650a948b3337bc7b8bb2c3c3
hebasto:
ACK 70452a070b3b6d4d650a948b3337bc7b8bb2c3c3, tested on Linux Mint 20 (x86_64).
Tree-SHA512: d290415e3c70a394b3d7659c0480a35b4082bdce8d48b1c64a0025f7ad6e21567b4dc85813869513ad246d27f950706930410587c11c1aa3693ae6245084765c
This flag was added to binutils/ld in the 2.30 release,
see commit c11c786f0b45617bb8807ab6a57220d5ff50e414:
> The new "-z separate-code" option will generate separate code LOAD
segment which must be in wholly disjoint pages from any other data.
It was made the default for Linux/x86 targets in the 2.31 release, see commit
f6aec96dce1ddbd8961a3aa8a2925db2021719bb:
> This patch adds --enable-separate-code to ld configure to turn on
-z separate-code by default and enables it by default for Linux/x86.
This avoids mixing code pages with data to improve cache performance
as well as security.
> To reduce x86-64 executable and shared object sizes, the maximum page
size is reduced from 2MB to 4KB when -z separate-code is turned on by
default. Note: -z max-page-size= can be used to set the maximum page
size.
> We compared SPEC CPU 2017 performance before and after this change on
Skylake server. There are no any significant performance changes.
Everything is mostly below +/-1%.
Support was also added to LLVMs lld: https://reviews.llvm.org/D64903, however
there is remains off by default.
There were concerns about an increase in binary size, however in our case, the
increase (1 page worth of bytes) would seem negligible, given we are shipping a
multi-megabyte binary, which then downloads 100's of GBs of data.
Also note that most recent versions of distros are shipping a new enough version
of binutils that this is available and/or on by default (assuming the distro has
not turned it off, I haven't checked everywhere):
CentOS 8: 2.30
Debian Buster 2.31.1
Fedora 29: 2.31.1
FreeBSD: 2.33
GNU Guix: 2.33 / 2.34
Ubuntu 18.04: 2.30
Related threads / discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=1623218
