790b440197bde322432a5bab161f1869b667e681 Fix benchmark CSV output (Hennadii Stepanov)
Pull request description:
The `SHA256AutoDetect` return output is used, among other use cases, to name benchmarks. Using a comma breaks the `bench_bitcoin` CSV output.
This PR replaces the comma with a semicolon, which fixes https://github.com/bitcoin/bitcoin/issues/33331.
ACKs for top commit:
Raimo33:
Code Review ACK 790b440197bde322432a5bab161f1869b667e681
l0rinc:
Code review ACK 790b440197bde322432a5bab161f1869b667e681
janb84:
code review ACK 790b440197bde322432a5bab161f1869b667e681
Tree-SHA512: 096bfa29a0639a4d97d510a3e2a15f071f384148c3035e4d0fc525794682e499c45a0d0c95728d5c78010098393b2c486a7fa9c21c1e2fbb600dea7c5638a55f
8b6264768030db1840041abeeaeefd6c227a2644 test: send duplicate blocktxn message in p2p_compactblocks.py (Eugene Siegel)
5e585a0fc4fd68dd7b4982054b34deae2e7aeb89 net: check for empty header before calling FillBlock (Eugene Siegel)
Pull request description:
This avoids an Assume crash if multiple blocktxn messages are received. The first call to `FillBlock` would make the header empty via `SetNull` and the call right before the second `FillBlock` would crash [here](689a321976/src/net_processing.cpp (L3333)) since `LookupBlockIndex` won't find anything. Fix that by checking for an empty header before the Assume.
ACKs for top commit:
instagibbs:
reACK 8b62647680
fjahr:
tACK 8b6264768030db1840041abeeaeefd6c227a2644
achow101:
ACK 8b6264768030db1840041abeeaeefd6c227a2644
mzumsande:
Code Review ACK 8b6264768030db1840041abeeaeefd6c227a2644
Tree-SHA512: d43a6f652161d4f7e6137f207a3e95259fc51509279d20347b1698c91179c39c8fcb75d2668b13a6b220f478a03578573208a415804be1d8843acb057fa1a73a
c76797481155754329ec6a6f58e8402569043944 clang-tidy: Fix critical warnings (Fabian Jahr)
54dc34ec2279378c78fa2d9155668e39e20decda index: Remove unused coinstatsindex recovery code (Fabian Jahr)
37c4fba1f4c18632bceb16f41f5a8f1a61fb9096 index: Check BIP30 blocks when rewinding Coinstatsindex (Fabian Jahr)
51df9de8e5b9c8ecd8339d95b630f312fcb9414e doc: Add release note for 30469 (Fabian Jahr)
bb8d673183294a43c716ff8738da2492f3d7a94b test: Add coinstatsindex compatibility test (Fabian Jahr)
b2e8b64ddc351124ac1390ee906a8fcd2781ca50 index, refactor: Append blocks to coinstatsindex without db read (Fabian Jahr)
431a076ae6e3cc32a8725d4a01483d27c5081a34 index: Fix coinstatsindex overflow issue (Fabian Jahr)
84e813a02bb7b3c735ae413f06c0fc156bfeb7ac index, refactor: DRY coinbase check (Fabian Jahr)
fab842b3248744fb0030486f64d3febe815f8377 index, refactor: Rename ReverseBlock to RevertBlock (Fabian Jahr)
Pull request description:
Closes https://github.com/bitcoin/bitcoin/issues/26362
This continues the work that was started with #26426. It fixes the overflow issue by switching the tracked values that are in danger of overflowing from `CAmount` to `arith_uint256`.
The current approach opts for a simple solution to ensure compatibility with datadirs including the previous version of the index: The new version of the index goes into a separate location in the datadir (`index/coinstatsindex/` rather than `index/coinstats/` before, the new naming is more consistent with the naming of the other indexes). There is no explicit concept of versioning of the index which earlier versions of this PR had. Having the two different versions of the index in separate places allows for downgrading of the node without having to rebuild the index. However, there will be a warning printed in the logs if the new code (v30) detects the old index still being present. A future version could delete a left-over legacy index automatically.
The PR also includes several minor improvements but most notably it lets new entries be calculated and stored without needing to read any DB records.
ACKs for top commit:
achow101:
ACK c76797481155754329ec6a6f58e8402569043944
TheCharlatan:
ACK c76797481155754329ec6a6f58e8402569043944
mzumsande:
Tested / Code Review ACK c76797481155754329ec6a6f58e8402569043944
Tree-SHA512: 3fa4a19dd1a01c1b01390247bc9daa6871eece7c1899eac976e0cc21ede09c79c65f758d14daafc46a43c4ddd7055c85fb28ff03029132d48936b248639c6ab9
Previously in debug builds, this would cause an Assume crash if
FillBlock had been called previously. This could happen when multiple
blocktxn messages were received.
Co-Authored-By: Greg Sanders <gsanders87@gmail.com>
188de70c86414b8b2ad5143f5c607b67686526ea net: Add interrupt to pcp retry loop (TheCharlatan)
Pull request description:
Without this interrupt bitcoind takes a long time to exit if requested to do so after a failed pcp lookup on startup.
ACKs for top commit:
achow101:
ACK 188de70c86414b8b2ad5143f5c607b67686526ea
fjahr:
utACK 188de70c86414b8b2ad5143f5c607b67686526ea
hodlinator:
utACK 188de70c86414b8b2ad5143f5c607b67686526ea
Tree-SHA512: 426dabd10ac0ef5de246c83d281ba70957e4032d251054aa6028b4d7ce4e35cd35ac70e67dc07bd418673bcdd2f4457b76f174ac5e7d0dd3caa05de5da952dac
589b65f06c3376df4cde3fac5c1d39a2d3254920 clang-tidy: Disable `UndefinedBinaryOperatorResult` check in `src/ipc` (Hennadii Stepanov)
Pull request description:
The warnings are false positive and have been fixed upstream. See: https://github.com/capnproto/capnproto/pull/2334.
This PR:
1. Disables the `UndefinedBinaryOperatorResult` clang-tidy check for source files generated by the `mpgen` tool.
2. Is an alternative to the draft https://github.com/bitcoin/bitcoin/pull/33281.
3. Fixes https://github.com/bitcoin/bitcoin/issues/33256.
ACKs for top commit:
Sjors:
ACK 589b65f06c3376df4cde3fac5c1d39a2d3254920
fjahr:
ACK 589b65f06c3376df4cde3fac5c1d39a2d3254920
achow101:
ACK 589b65f06c3376df4cde3fac5c1d39a2d3254920
ryanofsky:
Code review ACK 589b65f06c3376df4cde3fac5c1d39a2d3254920. Thanks for the fix!
Tree-SHA512: 6d376a82641a5b85d4dd1fa164fdcbd8e15f1262e7d4f582f4d9959031d35852e28ff1b8268336e39ba6779fdd10ecdb986af42407d0545f4217f41d64556272
The `SHA256AutoDetect` return output is used, among other use cases, to
name benchmarks. Using a comma breaks the CSV output.
This change replaces the comma with a semicolon, which fixes the issue.
The std::move in coinstatsindex was not necessary since it was passed as a const reference argument.
The other change in the utxo supply fuzz test changes a line that seems to have triggered a false alarm.
The coinstatsindex currently looks for block data at a hash key if the prev block in CustomAppend is different than expected. This is not needed since base index should always prevent us ending up in this scenario since it should rewind the index before calling CustomAppend in this case. But even if we run into this and our belt-and-suspenders code is getting hit, the index could not recover properly from the hash key index data so it can be removed without any real impact.
This is practically irrelevant due to the unlikeliness of a re-org
reaching so deep that it would drop the BIP30 blocks from the chain
(91842 and 91880). However this serves as documentation and ensures that
the functions RevertBlock and CustomAppend are consistent.
The index originally stored cumulative values in a CAmount type but this allowed for
potential overflow issues which were observed on Signet. Fix this by
storing the values that are in danger of overflowing in a arith_uint256.
Also turns an unnecessary copy into a reference in RevertBlock and
CustomAppend and gets
rid of the explicit total unspendable tracking which can be calculated
by adding the four categories of unspendables together.
The warnings are false positive and have been fixed upstream.
See: https://github.com/capnproto/capnproto/pull/2334.
This change disables the `UndefinedBinaryOperatorResult` clang-tidy
check for source files generated by the `mpgen` tool.
When the router doesn't support natpmp and PCP, one'd normally expect
the UDP packet to be ignored, and hit a time out. This logs a warning
that is already in the debug category. However, there's also the case in
which sending an UDP packet causes a ICMP response. This is returned to
user space as "connection refused" (despite UDP having no concept of
connections).
Move the warnings from `Send` and `Recv` to debug level too, to reduce
log spam in that case.
Closes#33301.
88db09bafe9ec363525e5e526c5f6cdd13691447 net: handle multi-part netlink responses (willcl-ark)
42e99ad77396e4e9b02d67daf46349e215e99a0f net: skip non-route netlink responses (willcl-ark)
57ce645f05d18d8ad10711c347a5989076f1f788 net: filter for default routes in netlink responses (willcl-ark)
Pull request description:
...for default route in pcp pinholing.
Currently we only make a single recv call, which trucates results from large routing tables, or in the case the kernel may split the message into multiple responses (which may happen with `NLM_F_DUMP`).
We also do not filter on the default route. For IPv6, this led to selecting the first route with an `RTA_GATEWAY` attribute, often a non-default route instead of the actual default. This caused PCP port mapping failures because the wrong gateway was used.
Fix both issues by adding multi-part handling of responses and filter for the default route.
Limit responses to ~ 1MB to prevent any router-based DoS.
ACKs for top commit:
achow101:
ACK 88db09bafe9ec363525e5e526c5f6cdd13691447
davidgumberg:
Code Review re-ACK 88db09b
Sjors:
re-utACK 88db09bafe9ec363525e5e526c5f6cdd13691447
Tree-SHA512: ea5948edebfad5896a487a61737aa5af99f529fad3cf3da68dced456266948238a7143383847e79a7bb90134e023eb173c25116d8eb80ff57fa4c4a0377ca1ed
Handle multi-part netlink responses to prevent truncated results from
large routing tables.
Previously, we only made a single recv call, which led to incomplete
results when the kernel split the message into multiple responses (which
happens frequently with NLM_F_DUMP).
Also guard against a potential hanging issue where the code would
indefinitely wait for NLMSG_DONE for non-multi-part responses by
detecting the NLM_F_MULTI flag and only continue waiting when necessary.
2885bd0e1c4fc863a7f28ff0fd353f5cffb03442 doc: unify `datacarriersize` warning with release notes (Lőrinc)
Pull request description:
Follow-up to https://github.com/bitcoin/bitcoin/pull/32406
---
The [release notes](a189d63618/doc/release-notes-32406.md (L1)) claim
> [...] marked as deprecated and are expected to be removed in a future release
but the [warning itself](2885bd0e1c/src/init.cpp (L907)) claims
> [...] marked as deprecated. They **will** be removed in a future version.
To be less aggressive (since some have objected against this version online) - and to unify the deprecation warning with the release notes - I have changed the warning to communicate our expectation in a friendlier way.
ACKs for top commit:
cedwies:
ACK 2885bd0
ryanofsky:
Code review ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442. I don't think it is good for the release notes and the runtime warning message to say two different things. I'd also be happy if release notes were updated to match the runtime warning, instead of vice versa. Whatever is more accurate is better.
ajtowns:
ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
kevkevinpal:
ACK [2885bd0](2885bd0e1c)
achow101:
ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
janb84:
ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
Zero-1729:
crACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
jonatack:
ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
hodlinator:
ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
w0xlt:
ACK 2885bd0e1c
optout21:
ACK 2885bd0e1c4fc863a7f28ff0fd353f5cffb03442
Tree-SHA512: a9d2a64ab96b3dd7f3a1a29622930054fd5c56e573bc96330f4ef3327dc024b21b3fbc8a698d17aea7c76f57f0c2ccd6403b2df344ae2f69c645ceb8b6fa54a5
b7b249d3adfbd3c7b0c4d0499a86300f57982972 Revert "[refactor] rewrite vTxHashes as a vector of CTransactionRef" (Anthony Towns)
b9300d8d0a74b15a220a2ce529d5157d109c7ed3 Revert "refactor: Simplify `extra_txn` to be a vec of CTransactionRef instead of a vec of pair<Wtxid, CTransactionRef>" (Anthony Towns)
df5a50e5de204013ba56e1c4967a249ca07ba086 bench/blockencodings: add compact block reconstruction benchmark (Anthony Towns)
Pull request description:
Reconstructing compact blocks is on the hot path for block relay, so revert changes from #28391 and #29752 that made it slower. Also add a benchmark to validate reconstruction performance, and a comment giving some background as to the approach.
ACKs for top commit:
achow101:
ACK b7b249d3adfbd3c7b0c4d0499a86300f57982972
polespinasa:
lgtm code review and tested ACK b7b249d3adfbd3c7b0c4d0499a86300f57982972
cedwies:
code-review ACK b7b249d
davidgumberg:
crACK b7b249d3adfbd3c
instagibbs:
ACK b7b249d3adfbd3c7b0c4d0499a86300f57982972
Tree-SHA512: dc266e7ac08281a5899fb1d8d0ad43eb4085f8ec42606833832800a568f4a43e3931f942d4dc53cf680af620b7e893e80c9fe9220f83894b4609184b1b3b3b42
46ca7712cb5fcf759cfc9f4f32d74215c8c83763 threading: remove unused template instantiations (Cory Fields)
b537a6a6dbd3b2a6725ccfafd57c9cc50cd617b0 threading: remove obsolete critsect macros (Cory Fields)
0d0e0a39b4a58fc48d2f0107e8c08bece58130bc threading: use a reverse lock rather than manual critsect macros (Cory Fields)
3ddd554d318110e8270498760340854929c59405 tests: Add Assertions in reverse_lock tests to exercise thread-safety annotations (Cory Fields)
c88b1cbf57a333261dbb8cf2eae91cf76e056d96 tests: get rid of remaining manual critsect usage (Cory Fields)
Pull request description:
Now that #32467 is merged, the only remaining usage of our old `CRITICAL_SECTION` macros (other than tests) is in `getblocktemplate()` and it can safely be replaced with a `REVERSE_LOCK`.
This PR makes that replacement, replaces the old `CRITICAL_SECTION` macro usage in tests, then deletes the macros themselves.
~While testing this a few weeks ago, I noticed that `REVERSE_LOCK` does not currently work properly with our thread-safety annotations as after the `REVERSE_LOCK` is acquired, clang still believes that the mutex is locked. #32465 fixes this problem. Without that fix, this PR would potentially allow a false-negative if code were added in the future to this chunk of `getblocktemplate` which required `cs_main` to be locked.~
~I added a test for the reverse lock here in the form of a compiler warning in `reverselock_tests.cpp` to simulate that possibility. This PR will therefore cause a new warning (and should fail a warnings-as-errors ci check) until #32465 is merged and this is rebased on top of it.~
Edit: Rebased on top of #32465, so this should now pass tests.
ACKs for top commit:
maflcko:
review ACK 46ca7712cb5fcf759cfc9f4f32d74215c8c83763 📌
fjahr:
Code review ACK 46ca7712cb5fcf759cfc9f4f32d74215c8c83763
TheCharlatan:
ACK 46ca7712cb5fcf759cfc9f4f32d74215c8c83763
furszy:
ACK 46ca7712cb5fcf759cfc9f4f32d74215c8c83763
Tree-SHA512: 5e423c8539ed5ddd784f5c3657bbd63be509d54942c25149f04e3764bcdf897bebf655553338d5af7b8c4f546fc1d4dd4176c2bce6f4683e76ae4bb91ba2ec80
a602f6fb7bf5f9e57299f4d6e246c82379fad8d2 test: index with an unclean restart after a reorg (Martin Zumsande)
01b95ac6f496e24e525b2fc9d69ee8b543da65ff index: don't commit state in BaseIndex::Rewind (Martin Zumsande)
Pull request description:
The committed state of an index should never be ahead of the flushed chainstate.
Otherwise, in the case of an unclean shutdown, the blocks necessary to revert
from the prematurely committed state are not be available, which would corrupt the coinstatsindex in particular.
Instead, the index state will be committed with the next ChainStateFlushed notification.
Fixes#33208
ACKs for top commit:
achow101:
ACK a602f6fb7bf5f9e57299f4d6e246c82379fad8d2
stickies-v:
re-ACK a602f6fb7bf5f9e57299f4d6e246c82379fad8d2
Tree-SHA512: 2559ea3fe066caf746a54ad7daac5031332f3976848e937c3dc8b35fa2ce925674115d8742458bf3703b3916f04f851c26523b6b94aeb1da651ba5a1b167a419
