67bb7be864f38ef5afc731aa427146cb2af500dd tests: Add fuzzing harness for CHash{160,256}, C{HMAC_,}SHA{1,256,512}, CRIPEMD160, CSipHasher, etc. (practicalswift)
Pull request description:
Add fuzzing harness for `CHash{160,256}`, `C{HMAC_,}SHA{1,256,512}`, `CRIPEMD160`, `CSipHasher`, etc.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
Top commit has no ACKs.
Tree-SHA512: 5377b361097211a7d0b90a26ed1c6dadb9ecce11349036d19f8c9ad2818cd98709bbcbf1c2361dd18eae122b8dbce1c71bb5aa2e85660677e235b8974ae33fcc
fa8337fcdbcab8368d5bf111c4b2e5acf25e6e22 clang-format scheduler (MarcoFalke)
fa3d41b5ab8fe62bc2e36c3e1211aace69e6da65 doc: Switch scheduler to doxygen comments (MarcoFalke)
fac43f9889f500bcb62d830c030dec42fe791031 scheduler: Replace stop(true) with StopWhenDrained() (MarcoFalke)
fa9cca0550f3d0ee8c276146f40007f76fbb97c2 doc: Remove unused documentation about unimplemented features (MarcoFalke)
fab2950d703217ec34b27e677e4f33ebbf99ca08 doc: Switch boost::thread to std::thread in scheduler (MarcoFalke)
fa9819695aac260be0ba170eb15ecba8cb519843 test: Remove unused scheduler.h include from the common setup (MarcoFalke)
fa609c4f76f215c19ea4021e78c102dee2b8c3d1 scheduler: Remove unused REVERSE_LOCK (MarcoFalke)
Pull request description:
This accumulates a bunch of cleanup that was long overdue, but I haven't yet gotten around to address. Specifically, but not limited to:
* Remove unused code, documentation and includes
* Upgrade to doxygen documentation
Please refer to the individual commits for more details.
ACKs for top commit:
jnewbery:
utACK fa8337fcdbcab8368d5bf111c4b2e5acf25e6e22
Tree-SHA512: 0c825ad9767e2697a3ef1ec1be13fdc2b18eeb7493ad0be5b65cc9f209391e78b17ee66e35e094c5e171c12b0f1624f287a110f6bddaf3024b708877afa8552e
37ae687f95c82f2d64ed880533d158060d4fc3de Add tests for CPubKey serialization/unserialization (Elichai Turkel)
9b8907faded8e4ec312c0dd4b4b15e1793876acd Check size after Unserializing CPubKey (Elichai Turkel)
Pull request description:
Found by practicalswift, closes#19235
Currently all the public API(except the pointer-like API) in CPubKey that sets/constructs a pubkey goes through `CPubKey::Set` which checks if that the length and size match and if not invalidates the key.
This adds the same check to `CPubKey::Unserialize`, sadly I don't see an easy way to just push this to the existing checks in `CPubKey::Set` but it's only a simple condition.
The problem with not invalidating is that if you write a pubkey like: `{0x02,0x00}` it will think the actual length is 33(because of `size()`) and will access uninitialized memory if you call any of the functions on CPubKey.
ACKs for top commit:
practicalswift:
re-ACK 37ae687f95c82f2d64ed880533d158060d4fc3de
jonatack:
Code review re-ACK 37ae687 per `git diff eab8ee3 37ae687` only change since last review at eab8ee3 is passing the `pubkey` param by reference to const instead of by value in `src/test/key_tests.cpp::CmpSerializationPubkey`
MarcoFalke:
ACK 37ae687f95c82f2d64ed880533d158060d4fc3de
Tree-SHA512: 30173755555dfc76d6263fb6a59f41be36049ffae7b4e1b92b922d668f5e5e2331f7374d5fa10d5d59fc53020d2966156905ffcfa8b8129c1f6d0ca062174ff1
51e9393c1f6c9eaac554f821f5327f63bd09c8cf refactor: s/command/msg_type/ in CNetMsgMaker and CSerializedNetMsg (Sebastian Falbesoner)
Pull request description:
Follow-up PR for #18533 -- another small step towards getting rid of the confusing "command" terminology. Also see PR #18610 which tackled the functional tests.
ACKs for top commit:
MarcoFalke:
ACK 51e9393c1f6c9eaac554f821f5327f63bd09c8cf
Tree-SHA512: bb6f05a7be6823d5c4eab1d05b31fee944e700946827ad9425d59a3957fd879776c88c606319cbe9832d9451b275baedf913b71429ea3e01e4e82bf2d419e819
fa1904e5f0d164fbcf41398f9ebbaafe82c28419 net: Remove dead logging code (MarcoFalke)
fac12ebf4f3b77b05112d2b00f8d3f4669621a4c net: Avoid redundant and confusing FAILED log (MarcoFalke)
Pull request description:
Remove a redundant and confusing "FAILED" log message and gets rid of the unused return type in `ProcessMessage`
ACKs for top commit:
jnewbery:
utACK fa1904e5f0d164fbcf41398f9ebbaafe82c28419
gzhao408:
utACK fa1904e5f0
troygiorshev:
ACK fa1904e5f0d164fbcf41398f9ebbaafe82c28419
naumenkogs:
utACK fa1904e
Tree-SHA512: bfa553d5efa022727ed17877fb7c08c14849d804fe6d6a7ce172d513857beba35de41ea40b27ff1aedf68b81e2cda7b2a948ac985fcaaf1b6cfb96cce4837c90
26acc8dd9b512f220c1facdba2c5de7976d3c258 Add sanity check asserts to span when -DDEBUG (Pieter Wuille)
2676aeadfa0e43dcaaccc4720623cdfe0beed528 Simplify usage of Span in several places (Pieter Wuille)
ab303a16d114b1e94c6cf0e4c5db5389dfa197f6 Add Span constructors for arrays and vectors (Pieter Wuille)
bb3d38fc061d8482e68cd335a45c9cd8bb66a475 Make pointer-based Span construction safer (Pieter Wuille)
1f790a1147ad9a5fe06987d84b6cd71f91cbec4b Make Span size type unsigned (Pieter Wuille)
Pull request description:
This improves our Span class by making it closer to the C++20 `std::span` one:
* ~~Support conversion between compatible Spans (e.g. `Span<char>` to `Span<const char>`).~~ (done in #18591)
* Make the size type `std::size_t` rather than `std::ptrdiff_t` (the C++20 one underwent the same change).
* Support construction of Spans directly from arrays, `std::string`s, `std::array`s, `std::vector`s, `prevector`s, ... (for all but arrays, this only works for const containers to prevent surprises).
And then make use of those improvements in various call sites.
I realize the template magic used looks scary, but it's only needed to make overload resultion make the right choices. Note that the operations done on values are all extremely simple: no casts, explicit conversions, or warning-silencing constructions. That should hopefully make it simpler to review.
ACKs for top commit:
laanwj:
Code review ACK 26acc8dd9b512f220c1facdba2c5de7976d3c258
promag:
Code review ACK 26acc8dd9b512f220c1facdba2c5de7976d3c258.
Tree-SHA512: 5a5bd346a140edf782b5b3b3f04d9160c7b9e9def35159814a07780ab1dd352545b88d3cc491e0f80d161f829c49ebfb952fddc9180f1a56f1257aa51f38788a
- Move the decision whether to translate an error message to where it is
defined. This simplifies call sites: no more `InitError(Untranslated(...))`.
- Make all functions in `util/error.h` consistently return a
`bilingual_str`. We've decided to use this as error message type so
let's roll with it.
This has no functional changes: no messages are changed, no new
translation messages are defined.
b00266fe0cf05fe6044f471105ce2bfed4349626 refactor: replace pointers by references within tx_verify.{h,cpp} (Sebastian Falbesoner)
Pull request description:
This PR gets rid of another unnecessary use of raw pointers, similar to PR #19053 (see also issue #19062 where useful commands for finding potential candidates are listed) but in the tx verification module.
For the functions `CalculateSequenceLocks()` and `SequenceLocks()`, the `prevHeights` vector parameter type is changed to be passed as a reference. Note that there were no checks for null pointers -- if one would pass `nullptr` to one of the functions, the following line would immediately lead to a crash:
dcacea096e/src/consensus/tx_verify.cpp (L32)
ACKs for top commit:
Empact:
Code Review ACK b00266fe0c
Tree-SHA512: 0eb71591467905434082029128bdca4df94988c372af40dca325654f6c002c72a00c73776cb5e72d6de2b2f218649211a5dbf19300a2e01f1841d6034e0f01e0
89f9fef1f71dfeff4baa59bc42bc9049a46d911b refactor: Specify boost/thread/thread.hpp explicitly (Hennadii Stepanov)
fad8c890f5ae6e083e416781b4857a1a53ad5249 txdb: Remove unused boost/thread (MarcoFalke)
faa958bc283023334b9377f5fb2210459ca16d82 txindex: Remove unused boost/thread (MarcoFalke)
Pull request description:
There are predefined interruption points for `boost::thread`: https://www.boost.org/doc/libs/1_71_0/doc/html/thread/thread_management.html#interruption_points
However, non-boost threads such as `std::thread` or the `main()` thread can obviously not be interrupted. So remove all unused boost/thread from methods that are never executed in a `boost::thread`.
Most of them were accompanied by a `ShutdownRequested` anyway. So even if the current thread was a `boost::thread`, the interruption point would be redundant. (We only interrupt threads during shutdown)
ACKs for top commit:
fanquake:
ACK 89f9fef1f71dfeff4baa59bc42bc9049a46d911b
hebasto:
ACK 89f9fef1f71dfeff4baa59bc42bc9049a46d911b, tested on Linux Mint 19.3 (x86_64), verified shutdown in different scenarios.
Tree-SHA512: 17221dadedf2d107e5bda9e4f371cc4f8ffce6ad27cae41aa2b8f1150d8f1adf23d396585ca4a2dd25b1dc6f0d5c81fecd950d8557966ccb45a6d4a85a331d90
8b3136bd307123a255b9166aa42a497a44bcce70 refactor: replace CNode pointers by references within net_processing.{h,cpp} (Sebastian Falbesoner)
Pull request description:
This PR is inspired by a [recent code review comment](https://github.com/bitcoin/bitcoin/pull/19010#discussion_r426954791) on a PR that introduced new functions to the net_processing module. The point of the discussion was basically that whenever we pass something not by value (in the concrete example it was about `CNode*` and `CConnman*`) we should either use
* a pointer (```CType*```) with null pointer check or
* a reference (```CType&```)
To keep things simple, this PR for a first approach
* only tackles `CNode*` pointers
* only within the net_processing module, i.e. no changes that would need adaption in other modules
* keeps the names of the variables as they are
I'm aware that PRs like this are kind of a PITA to review, but I think the code quality would increase if we get rid of pointers without nullptr check -- bloating up the code by adding all the missing checks would be the worse alternative, in my opinion.
Possible follow-up PRs, in case this is received well:
* replace CNode pointers by references for net module
* replace CConnman pointers by references for net_processing module
* ...
ACKs for top commit:
MarcoFalke:
ACK 8b3136bd307123a255b9166aa42a497a44bcce70 🔻
practicalswift:
ACK 8b3136bd307123a255b9166aa42a497a44bcce70
Tree-SHA512: 15b6a569ecdcb39341002b9f4e09b38ed4df077e3a3a50dfb1b72d98bdc9f9769c7c504f106456aa7748af8591af7bb836b72d46086df715ab116e4ac3224b3b
fab860aed4878b831dae463e1ee68029b66210f5 fuzz: Stop nodes in process_message* fuzzers (MarcoFalke)
6666c828e072a5e99ea0c16394ca3e5b9de07409 fuzz: Give CNode ownership to ConnmanTestMsg in process_message fuzz harness (MarcoFalke)
Pull request description:
Background is that I saw an integer overflow in net_processing
```
#30629113 REDUCE cov: 25793 ft: 142917 corp: 3421/2417Kb lim: 4096 exec/s: 89 rss: 614Mb L: 1719/4096 MS: 1 EraseBytes-
net_processing.cpp:977:25: runtime error: signed integer overflow: 2147483624 + 100 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior net_processing.cpp:977:25 in
net_processing.cpp:985:9: runtime error: signed integer overflow: -2147483572 - 100 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior net_processing.cpp:985:9 in
```
Telling from the line numbers, it looks like `nMisbehavior` wrapped around.
Fix that by calling `StopNodes` after each exec, which should clear the node state and thus `nMisbehavior`.
ACKs for top commit:
practicalswift:
ACK fab860aed4878b831dae463e1ee68029b66210f5
Tree-SHA512: 891c081d5843565d891aec028b6c27ef3fa39bc40ae78238e81d8f784b4d4b49cb870998574725a5159dd03aeeb2e0b9bc3d3bb51d57d1231ef42e3394b2d639
f898ef65c947776750e49d050633f830546bbdc6 tests: Add fuzzing harness for functions in script/sign.h (practicalswift)
c91d2f06150cda258a17e78d9b7065b594d34a85 tests: Add fuzzing harness for functions in script/sigcache.h (practicalswift)
d3d8adb79fbe34b15cf29334607f9b76d303aa1a tests: Add fuzzing harness for functions in script/interpreter.h (practicalswift)
fa80117cfdeca7e5d3a2a09b385c0e938bf701e9 tests: Add fuzzing harness for functions in script/descriptor.h (practicalswift)
43fb8f0ca331a7f79f0d287817da7f4b894bdbfa tests: Add fuzzing harness for functions in script/bitcoinconsensus.h (practicalswift)
8de72711c685e638fa54d485694fb1b1af024adc tests: Fill fuzzing coverage gaps for functions in script/script.h, script/script_error.h and script/standard.h (practicalswift)
c571ecb07145b4ce8c17ca80489f8f1497388c4d tests: Add fuzzing helper functions ConsumeDataStream, ConsumeTxDestination and ConsumeUInt160 (practicalswift)
Pull request description:
Add fuzzing harnesses for functions in `script/`:
* Add fuzzing helper functions `ConsumeDataStream` and `ConsumeUInt160`
* Fill fuzzing coverage gaps for functions in `script/script.h`, `script/script_error.h` and `script/standard.h`
* Add fuzzing harness for functions in `script/bitcoinconsensus.h`
* Add fuzzing harness for functions in `script/descriptor.h`
* Add fuzzing harness for functions in `script/interpreter.h`
* Add fuzzing harness for functions in `script/sigcache.h`
* Add fuzzing harness for functions in `script/sign.h`
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
MarcoFalke:
ACK f898ef65c947776750e49d050633f830546bbdc6 🔉
Tree-SHA512: f6e77b34dc79f23de5fa9e38ac06e6554b5b946ec3e9a67e2bd982e60aca37ce844f785457ef427a5e3b45e31c305456bca8587cc9f4a0b50b3852e39726eb04
9a19c9ada53e84d1ee8521b48f656faad48b737a Always define the raii_event_tests test suite (Craig Andrews)
Pull request description:
The test suite must always be defined (even when EVENT_SET_MEM_FUNCTIONS_IMPLEMENTED is not defined) so that the test harness doesn't fail due to not being able to find the raii_event_tests test.
This improves upon 95f97f4 actually fixing https://github.com/bitcoin/bitcoin/issues/9493
ACKs for top commit:
MarcoFalke:
ACK 9a19c9ada53e84d1ee8521b48f656faad48b737a 🎹
Tree-SHA512: 3c42f17a9b5d56c8841f3aa9ac19da91c10aff210026266f31f7eb98a62528740d7c518c121452b68e8f801d6c80ecfb627d137ec6ed533289fa3beb08b4f176
f871f15c9d760b56a6a3b78b3941d3983d60810c scripted-diff: replace gArgs with argsman (glowang)
357f02bf2942f2944a04a1ceaa687f89d5da7d28 Create a local class inherited from BasicTestingSetup with a localized args manager and put it into the getarg_tests namespace (glowang)
Pull request description:
Replaced the global argsManager gArgs with a locally defined one in getarg_tests. This is to avoid confusion in arg settings between the test's ArgsManager and the #18804
ACKs for top commit:
MarcoFalke:
ACK f871f15c9d760b56a6a3b78b3941d3983d60810c
ryanofsky:
Code review ACK f871f15c9d760b56a6a3b78b3941d3983d60810c. Changes look good and thanks for updating. In future would recommend using clang-format-diff and following [coding style](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#coding-style-c) notes, because it's atypical to indent namespace content, or indent protected keywords or put spaces around ::. Also it's fragile to define test setup class in a namespace, but test setup methods outside of the namespace and inside the test fixture instead. Would be simpler to just define the testing setup completely before using it without a namespace like: 8ad5f1c376/src/test/rpc_tests.cpp (L23) and it would have been a slightly smaller change too.
Tree-SHA512: 016594639396d60667fadec8ea80ef7af634fbb2014c704f02406fe3251c5362757c21f1763d8bdb94ca4a3026ab9dc786a92a9a934efc8cd807655d9deee779
5478d6c099e76fe070703cc5383cba7b91468b0f logging: thread safety annotations (Anthony Towns)
e685ca19928eec4e687c66f5edfcfff085a42c27 util/system.cpp: add thread safety annotations for dir_locks (Anthony Towns)
a7887899480db72328784009181d93904e6d479d test/checkqueue_tests: thread safety annotations (Anthony Towns)
479c5846f7477625ec275fbb8a076c7ef157172b rpc/blockchain.cpp: thread safety annotations for latestblock (Anthony Towns)
8b5af3d4c1270267ad85e78f661bf8fab06f3aad net: fMsgProcWake use LOCK instead of lock_guard (Anthony Towns)
de7c5f41aba860751ef7824245e6d9d5088a1200 wallet/wallet.h: Remove mutexScanning which was only protecting a single atomic bool (Anthony Towns)
c3cf2f55013c4ea1c1ef4a878fc7ff8e92f2c42d rpc/blockchain.cpp: Remove g_utxosetscan mutex that is only protecting a single atomic variable (Anthony Towns)
Pull request description:
In a few cases we need to use `std::mutex` rather than the sync.h primitives. But `std::lock_guard<std::mutex>` doesn't include the clang thread safety annotations unless you also use clang's C library, which means you can't indicate when variables should be guarded by `std::mutex` mutexes.
This adds an annotated version of `std::lock_guard<std::mutex>` to threadsafety.h to fix that, and modifies places where `std::mutex` is used to take advantage of the annotations.
It's based on top of #16112, and turns the thread safety comments included there into annotations.
It also changes the RAII classes in wallet/wallet.h and rpc/blockchain.cpp to just use the atomic<bool> flag for synchronisation rather than having a mutex that doesn't actually guard anything as well.
ACKs for top commit:
MarcoFalke:
ACK 5478d6c099e76fe070703cc5383cba7b91468b0f 🗾
hebasto:
re-ACK 5478d6c099e76fe070703cc5383cba7b91468b0f, only renamed s/`MutexGuard`/`LockGuard`/, and dropped the commit "test/util_threadnames_tests: add thread safety annotations" since the [previous](https://github.com/bitcoin/bitcoin/pull/16127#pullrequestreview-414184113) review.
ryanofsky:
Code review ACK 5478d6c099e76fe070703cc5383cba7b91468b0f. Thanks for taking suggestions! Only changes since last review are dropping thread rename test commit d53072ec730d8eec5a5b72f7e65a54b141e62b19 and renaming mutex guard to lock guard
Tree-SHA512: 7b00d31f6f2b5a222ec69431eb810a74abf0542db3a65d1bbad54e354c40df2857ec89c00b4a5e466c81ba223267ca95f3f98d5fbc1a1d052a2c3a7d2209790a
c57f03ce1741b38af448bec7b22ab9f8ac21f067 refactor: Replace const char* to std::string (Calvin Kim)
Pull request description:
Rationale: Addresses #19000
Some functions should be returning std::string instead of const char*.
This commit changes that.
Main benefits/reasoning:
1. The functions never return nullptr, so returning a string makes code at call sites easier to review (reviewers don't have to read the source code to verify that a nullptr is never returned)
2. All call sites convert to string anyway
ACKs for top commit:
MarcoFalke:
re-ACK c57f03ce17 (no changes since previous review) 🚃
Empact:
Fair enough, Code Review ACK c57f03ce17
practicalswift:
ACK c57f03ce1741b38af448bec7b22ab9f8ac21f067 -- patch looks correct
hebasto:
re-ACK c57f03ce1741b38af448bec7b22ab9f8ac21f067
Tree-SHA512: 9ce99bb38fe399b54844315048204cafce0f27fd8f24cae357fa7ac6f5d8094d57bbf5f5c1f5878a65f2d35e4a3f95d527eb17f49250b690c591c0df86ca84fd
71f016c6eb42e1ac2c905e04ba4d20c2009e533f Remove old serialization primitives (Pieter Wuille)
92beff15d3ae2646c00bd78146d7592a7097ce9c Convert LimitedString to formatter (Pieter Wuille)
ef17c03e074b6c3f185afa4eff572ba687c2a171 Convert wallet to new serialization (Pieter Wuille)
65c589e45e8b8914698a0fd25cd5aafdda30869c Convert Qt to new serialization (Pieter Wuille)
Pull request description:
This is the final step 🥳 of the serialization improvements extracted from #10785.
It converts the LimitedString wrapper to a new-style formatter, and updates the wallet and Qt code to use the new serialization framework. Finally all remaining old primitives are removed.
ACKs for top commit:
jonatack:
ACK 71f016c6eb42e1ac2 reviewed diff, builds/tests/re-fuzzed.
laanwj:
Code review ACK 71f016c6eb42e1ac2c905e04ba4d20c2009e533f
Tree-SHA512: d952194bc73259f6510bd4ab1348a1febbbf9862af30f905991812fb0e1f23f15948cdb3fc662be54d648e8f6d95b11060055d2e7a8c2cb5bf008224870b1ea1
fab6b9d18fd48bbbd1939b1173723bc04c5824b5 validation: Mark g_chainman DEPRECATED (MarcoFalke)
fa1d97b25686a5caca623599f6d608fd08616fe8 validation: Make ProcessNewBlock*() members of ChainstateManager (MarcoFalke)
fa24d4909864096934577abc26cfa9be47f634ba validation: Make PruneOneBlockFile() a member of ChainstateManager (MarcoFalke)
fa84b1cd846f6499b741710fd478ec9ad49b5120 validation: Make LoadBlockIndex() a member of ChainstateManager (MarcoFalke)
fa05fdf0f19fa4b557cc5e9ba436e3215b83c4e6 net: Pass chainman into PeerLogicValidation (MarcoFalke)
fa7b626d7a150e5cbd4d163d2dab6f8a55fc2cc4 node: Add chainman alias for g_chainman (MarcoFalke)
Pull request description:
The global `g_chainman` has recently been introduced in #17737. The chainstate manager is primarily needed for the assumeutxo feature, but it can also simplify testing in the future.
The goal of this pull is to make the global chainstate manager internal to validation, so that all external code does not depend on globals and that unit or fuzz tests can pass in their (potentially mocked) chainstate manager.
I suggest reviewing the pull request commit-by-commit. It should be relatively straightforward refactoring that does not change behavior at all.
ACKs for top commit:
ryanofsky:
Code review ACK fab6b9d18fd48bbbd1939b1173723bc04c5824b5. Had to be rebased but still looks good
Tree-SHA512: dcbf114aeef4f8320d466369769f22ce4dd8f46a846870354df176c3de9ff17c64630fbd777e7121d7470d7a8564ed8d37b77168746e8df7489c6877e55d7b4f
