New code can call the method without having first to retrieve the raw
FILE* pointer via Get().
Also, move implementation to the cpp file. Can be reviewed with:
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
* Add m_ prefix to the std::FILE member variable
* Add std namespace where possible, to avoid confusion with member
functions of the same name.
* Add AutoFile::feof() member function, to be used in place of
std::feof(AutoFile::Get())
* Simplify fclose() in terms of release()
* Fix typo in the error message in the ignore member function.
The shift operators will call the write or read member function, which
already does the check. Also, call sites are free to directly call
::(Un)Serialize(s, obj) to skip this check, so removing it increases
consistency.
89ba8905f5c68ae29412f9c4010314c5a113c234 test: indexes, fix on error infinite loop (furszy)
Pull request description:
Coming from https://github.com/bitcoin/bitcoin/pull/28036#issuecomment-1623813703, I thought that we were going to fix it there but seems that got merged without it for some reason.
As index sync failures trigger a shutdown request without notifying `BaseIndex::BlockUntilSyncedToCurrentChain` in any way, we also need to check whether a shutdown was requested or not inside 'IndexWaitSynced'.
Otherwise, any error inside the index sync process will hang the test forever.
ACKs for top commit:
MarcoFalke:
lgtm ACK 89ba8905f5c68ae29412f9c4010314c5a113c234
jamesob:
ACK 89ba890
ryanofsky:
Code review ACK 89ba8905f5c68ae29412f9c4010314c5a113c234. Just comment update since last review
Tree-SHA512: 1f6daf34e51d3fbc802799bfa4ac0ef0d8f774db5f9e2f5d35df18a77679778475c94efc3da1fb723ebaf3583e4075e4a5cbe4a5104ad0c50e2b32076e247b29
This has the benefit of moving the StartShutdown call out of the
blockstorage file and thus out of the kernel's responsibility. The user
can now decide if he wants to start shutdown / interrupt after a block
import or not.
As index sync failures trigger a shutdown request without notifying
BaseIndex::BlockUntilSyncedToCurrentChain in any way, we also need
to check whether a shutdown was requested or not inside 'IndexWaitSynced'.
Otherwise, any error inside the index sync process will hang the test
forever.
ca91c244ef1ba7eac6643d66a5fc56d3a2a8b550 index: verify blocks data existence only once (furszy)
fcbdaeef4d5a63e3e5b479c6fcad730eb86fb923 init: don't start indexes sync thread prematurely (furszy)
2ec89f1970935d27631bcd17b7923a79cdb1edbb refactor: simplify pruning violation check (furszy)
c82ef91eae38f385d408b095ebbc8a180ce52ebb make GetFirstStoredBlock assert that 'start_block' always has data (furszy)
430e7027a18870a296abb0bbd9332cbe40d8fdc0 refactor: index, decouple 'Init' from 'Start' (furszy)
225e213110602b4fd1d345167f5f92d26557f6c1 refactor: init indexes, decouple 'Start()' from the creation step (furszy)
2ebc7e68cc9d347807b646f601b27940c9590c89 doc: describe 'init load' thread actions (Martin Zumsande)
04575106b2529f495ce8110ddf7ed2247d4bc339 scripted-diff: rename 'loadblk' thread name to 'initload' (furszy)
ed4462cc78afd2065bbf5bd79728852b65b9ad7f init: start indexes sync earlier (furszy)
Pull request description:
Simplifies index startup code, eliminating the `g_indexes_ready_to_sync` variable,
deduplicating code and moving the prune violation check out of the `BaseIndex` class.
Also makes startup more efficient by running the prune violation check once for all indexes
instead of once for each index, and by delaying the prune violation check and moving it off
of the main thread so the node can start up faster and perform the block data availability
verification even when the '-reindex" or the "-reindex-chainstate" flags are enabled (which
hasn't being possible so far).
ACKs for top commit:
ryanofsky:
Code review ACK ca91c244ef1ba7eac6643d66a5fc56d3a2a8b550. Just rebase and suggested changes since last review (Start return check, and code simplification)
TheCharlatan:
re-ACK ca91c244ef1ba7eac6643d66a5fc56d3a2a8b550
Tree-SHA512: e9c98ce89aeb29e8d0f505f17b34aa54fe44efefbf017f4746e3b446ab4de25ade4f707254a0bbe4b99b69731b04a4067ce529eb7aa834ced196784b694cf7ce
At present, during init, we traverse the chain (once per index)
to confirm that all necessary blocks to sync each index up to
the current tip are present.
To make the process more efficient, we can fetch the oldest block
from the indexers and perform the chain data existence check from
that point only once.
This also moves the pruning violation check to the end of the
'loadinit' thread, which is where the reindex, block loading and
chain activation processes happen.
Making the node's startup process faster, allowing us to remove
the global g_indexes_ready_to_sync flag, and enabling the
execution of the pruning violation verification even when the
reindex or reindex-chainstate flags are enabled (which has being
skipped so far).
By moving the 'StartIndexes()' call into the 'initload'
thread, we can remove the threads active wait. Optimizing
the available resources.
The only difference with the current state is that now the
indexes threads will only be started when they can process
work and not before it.
By generalizing 'GetFirstStoredBlock' and implementing
'CheckBlockDataAvailability' we can dedup code and
avoid repeating work when multiple indexes are enabled.
E.g. get the oldest block across all indexes and
perform the pruning violation check from that point
up to the tip only once (this feature is being introduced
in a follow-up commit).
This commit shouldn't change behavior in any way.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
And transfer the responsibility of verifying whether 'start_block'
has data or not to the caller.
This is because the 'GetFirstStoredBlock' function responsibility
is to return the first block containing data. And the current
implementation can return 'start_block' when it has no data!. Which
is misleading at least.
Edge case behavior change:
Previously, if the block tip lacked data but all preceding blocks
contained data, there was no prune violation. And now, such
scenario will result in a prune violation.
No behavior change.
The goal here is to group indexes, so we can perform the same
initialization and verification process equally for all of them.
The checks performed inside `StartIndexes` will be expanded
in the subsequent commits.
The thread does not only load blocks, it loads the mempool and,
in a future commit, will start the indexes as well.
Also, renamed the 'ThreadImport' function to 'ImportBlocks'
And the 'm_load_block' class member to 'm_thread_load'.
-BEGIN VERIFY SCRIPT-
sed -i "s/ThreadImport/ImportBlocks/g" $(git grep -l ThreadImport -- ':!/doc/')
sed -i "s/loadblk/initload/g" $(git grep -l loadblk -- ':!/doc/release-notes/')
sed -i "s/m_load_block/m_thread_load/g" $(git grep -l m_load_block)
-END VERIFY SCRIPT-
The mempool load can take a while, and it is not
needed for the indexes' synchronization.
Also, having the mempool load function call
inside 'blockstorage.cpp' wasn't structurally
correct.
8b5397c00e821d7eaab22f512e9d71a1a0392ebf wallet: bdb: include bdb header from our implementation files only (Cory Fields)
6e010626af7ed51f1748323ece2f46335e145f2f wallet: bdb: don't use bdb define in header (Cory Fields)
004b184b027520a4f9019d1432a816e6ec891fe3 wallet: bdb: move BerkeleyDatabase constructor to cpp file (Cory Fields)
b3582baa3a2f84db7d2fb5a681121a5f2d6de3a1 wallet: bdb: move SafeDbt to cpp file (Cory Fields)
e5e5aa1da261633c8f73b97d5aefe5dc450a7db9 wallet: bdb: move SpanFromDbt to below SafeDbt's implementation (Cory Fields)
4216f69250937b1ca4650dc0c21678a8444c6650 wallet: bdb: move TxnBegin to cpp file since it uses a bdb function (Cory Fields)
43369f37060a1b4c987672707c500d35c9a27c1d wallet: bdb: drop default parameter (Cory Fields)
Pull request description:
Only `#include` upstream bdb headers from our cpp files.
It's generally good practice to avoid including 3rd party deps in headers as otherwise they tend to sneak into new compilation units. IMO this makes for a nice cleanup.
There's a good bit of code movement here, but each commit is small and _should_ be obviously correct.
Note: in the future, the buildsystem can add the bdb include path for `bdb.cpp` and `salvage.cpp` only, rather than all wallet sources.
ACKs for top commit:
achow101:
reACK 8b5397c00e821d7eaab22f512e9d71a1a0392ebf
hebasto:
ACK 8b5397c00e821d7eaab22f512e9d71a1a0392ebf
Tree-SHA512: 0ef6e8a9c4c6e2d1e5d6a3534495f91900e4175143911a5848258c56da54535b85fad67b6d573da5f7b96e7881299b5a8ca2327e708f305b317b9a3e85038d66
7ecc29a0b7a23d8f5d3c1e6a0dad29b3ad839eb9 test: wallet, add coverage for addressbook migration (furszy)
a277f8357ad8b0eb26f33fc36f919d868c06847b wallet: migration bugfix, persist empty labels (furszy)
1b64f6498c394a143df196172a14204fe3b8a744 wallet: migration bugfix, clone 'send' record label to all wallets (furszy)
Pull request description:
Addressing two specific bugs encountered during the wallet migration process, related to the address book, and improves the test coverage for it.
Bug 1: Non-Cloning of External 'Send' Records
The external 'send' records were not being correctly cloned to all wallets.
Bug 2: Persistence of Empty Labels
As address book entries without associated db label records can be treated as change (the `label` field inside the `CAddressBookData` class is optional, `nullopt` labels make `CAddressBookData ::IsChange()` return true), we must persist empty labels during the migration process.
The user might have called `setlabel` with an "" string for an external address and that must be retained during migration.
ACKs for top commit:
achow101:
ACK 7ecc29a0b7a23d8f5d3c1e6a0dad29b3ad839eb9
Tree-SHA512: b8a8483a4178a37c49af11eb7ba8a82ca95e54a6cd799e155e33f9fbe7f37b259e28372c77d6944d46b6765f9eaca6b8ca8d1cdd9d223120a3653e4e41d0b6b7
fac6af16f4a254458b8cb3522317422b40362f8d Allow std::byte serialization (MarcoFalke)
fade43edc4405e7c51cec9325d8502f3786f7438 Allow FastRandomContext::randbytes for all byte types (MarcoFalke)
Pull request description:
I need this for some stuff, but it should also be useful by itself for other developers that need it.
ACKs for top commit:
sipa:
utACK fac6af16f4a254458b8cb3522317422b40362f8d
dergoegge:
Code review ACK fac6af16f4a254458b8cb3522317422b40362f8d
Tree-SHA512: db4b1bbd6bf6ef6503d59b0b4ed1681db8d935d2d10f8d89f071978ea59b49a1d319bccb4e9717c0c88a4908bbeca4fd0cbff6c655d8a443554fd14146fe16de
fabed7eb796637c02e3677ebbe183d90b258ba69 test: Restore unlimited timeout in IndexWaitSynced (MarcoFalke)
Pull request description:
The timeout was unlimited before, so just restore that value for now: https://github.com/bitcoin/bitcoin/pull/27988#issuecomment-1619218007 .
(Strictly speaking, this is a behavior change for the blockfilterindex and txindex tests, because it only restores the coinstatsindex behavior.)
ACKs for top commit:
ajtowns:
utACK fabed7eb796637c02e3677ebbe183d90b258ba69
mzumsande:
ACK fabed7eb796637c02e3677ebbe183d90b258ba69
furszy:
ACK fabed7eb
Tree-SHA512: 66a878be58bbe53ad8e0c23f05569dd42df688be747551fbd202ada22d20a8285714e58fa2a71664deadb070ddf86cfad88c01042ff95ed26f6b40e4a10cec0a
bea9fc2600635020fd28ec7a6613c92a6f349a86 wallet: sqlite: force sqlite3.h to be included by the cpp files (Cory Fields)
Pull request description:
Only `#include` upstream sqlite headers from our cpp files.
Like #28039 but simpler :)
ACKs for top commit:
achow101:
ACK bea9fc2600635020fd28ec7a6613c92a6f349a86
TheCharlatan:
Nice, ACK bea9fc2600635020fd28ec7a6613c92a6f349a86
kristapsk:
utACK bea9fc2600635020fd28ec7a6613c92a6f349a86
hebasto:
ACK bea9fc2600635020fd28ec7a6613c92a6f349a86, I have reviewed the code and it looks OK.
Tree-SHA512: cb83ac51eed7e0740f1c75ee87c7849fa7e535bc4836c499290041eb995ccfd82533e3babfe83a164257b62b180f206112d6a1bae7ea290ad0ec7f55d62432da
6eb33bd0c21b3e075fbab596351cacafdc947472 kernel: Add fatalError method to notifications (TheCharlatan)
7320db96f8d2aeff0bc5dc67d8b7b37f5f808990 kernel: Add flushError method to notifications (TheCharlatan)
3fa9094b92c5d37f486b0f8265062d3456796a50 scripted-diff: Rename FatalError to FatalErrorf (TheCharlatan)
edb55e2777063dfeba0a52bbd0b92af8b4688501 kernel: Pass interrupt reference to chainman (TheCharlatan)
e2d680a32d757de0ef8eb836047a0daa1d82e3c4 util: Add SignalInterrupt class and use in shutdown.cpp (TheCharlatan)
Pull request description:
Get rid of all `ShutdownRequested` calls in validation code by introducing an interrupt object that applications can use to cancel long-running kernel operations.
Replace all `AbortNode` calls in validation code with new fatal error and flush error notifications so kernel applications can be notified about failures and choose how to handle them.
---
This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/27587https://github.com/orgs/bitcoin/projects/3 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel".
The pull request mostly allows dropping the kernel dependency on shutdown.cpp. The only dependency left after this is a `StartShutdown` call which will be removed in followup PR https://github.com/bitcoin/bitcoin/pull/27711. This PR also drops the last reference to the `uiInterface` global in kernel code. The process of moving the `uiInterface` out of the kernel was started in https://github.com/bitcoin/bitcoin/pull/27636.
This pull request contains a subset of patches originally proposed in #27711. It will be part of a series of changes required to make handling of interrupts (or in other words the current shutdown procedure) in the kernel library more transparent and less reliable on global mutable state. The set of patches contained here was originally proposed by @ryanofsky [here](https://github.com/bitcoin/bitcoin/pull/27711#issuecomment-1580779869).
ACKs for top commit:
achow101:
light ACK 6eb33bd0c21b3e075fbab596351cacafdc947472
hebasto:
ACK 6eb33bd0c21b3e075fbab596351cacafdc947472, I have reviewed the code and it looks OK.
ryanofsky:
Code review ACK 6eb33bd0c21b3e075fbab596351cacafdc947472. No changes since last review other than rebase.
Tree-SHA512: 7d2d05fa4805428a09466d43c11ae32946cbb25aa5e741b1eec9cd142e4de4bb311e13ebf1bb125ae490c9d08274f2d56c93314e10f3d69e7fec7445e504987c
8fbb6e99bfc85a1b9003cae402a7335843a86abd wallet: Give deprecation warning when loading a legacy wallet (Andrew Chow)
Pull request description:
Next step in legacy wallet deprecation.
ACKs for top commit:
S3RK:
reACK 8fbb6e99bfc85a1b9003cae402a7335843a86abd
jonatack:
re-ACK 8fbb6e99bfc85a1b9003cae402a7335843a86abd
Tree-SHA512: 902984b09452926cf199f06e5fb56e4985325cdd5e0dcc829992158488f42d5fbc33e9a30a29303feac24c8315193e8d31712022e2a0503abd6b67169a0027f4
99c0eb9701e71f16aa360a420b7e4851d5b92510 Fix RPCConsole wallet selection (John Moffett)
Pull request description:
If a user opens multiple wallets in the GUI from the menu bar, the last one opened is the active one in the main window. However, For the RPC Console window, the _first_ one opened is active. This can be confusing, as wallet RPC commands may be sent to a wallet the user didn't intend.
This PR makes the RPC Console switch to the wallet just opened / restored / created from the menu bar, which is how the main GUI now works.
Similar to https://github.com/bitcoin-core/gui/pull/665 and specifically requested [in a comment](https://github.com/bitcoin-core/gui/pull/665#issuecomment-1270003660).
ACKs for top commit:
luke-jr:
utACK 99c0eb9701e71f16aa360a420b7e4851d5b92510
hebasto:
ACK 99c0eb9701e71f16aa360a420b7e4851d5b92510, tested on Ubuntu 23.04.
Tree-SHA512: d5e5acdaa114130ad4d27fd3f25393bc8d02d92b5001cd39352601d04283cdad3bd62c4da6d369c69764e3b188e9cd3e83152c00b09bd42966082ad09037c328
a582b4141f0756faa3793fb1c772898a984c83e4 gui: send, left alignment for "bytes" and "change" label (furszy)
210ef1e980e0887c5675b66bcd5cbccd00aceec6 qt: remove confusing "Dust" label from coincontrol / sendcoins dialog (Sebastian Falbesoner)
Pull request description:
In contrast to to all other labels on the coin selection dialog, the displayed dust information has nothing to do with the selected coins. All that this label shows is whether at least one of the _outputs_ qualify as dust, but the outputs are set in a different dialog. (Even worse, the dust check is currently simply wrong because it only looks at an output's nValue and just assumes a P2PKH script size.)
As the label clearly doesn't help the user and is, quite the contrary, rather increasing confusion/misguidance, it seems sensible to remove it. The label from the sendcoins dialog is also removed with the same rationale. Additionally, the "bytes" and "change" labels are aligned to the left (second commit).
Closes https://github.com/bitcoin-core/gui/issues/699.
ACKs for top commit:
furszy:
ACK a582b41
hebasto:
Looks good. ACK a582b4141f0756faa3793fb1c772898a984c83e4.
Tree-SHA512: ebc00b68bdeab69f6ab643e4b89301a7e3d04a8a4027b50813314ddddb1387bc97a83313851e375dfbce97751c234686c82af7f4e55fa5ef29f4fed4e8fc11d9
5df988b534df842ddb658ad2a7ff0f12996c8478 test: add coverage for descriptor ID (furszy)
6a9510d2dabde1c9ee6c4226e3d16ca32eb48ac5 wallet: bugfix, always use apostrophe for spkm descriptor ID (furszy)
97a965d98f1582ea1b1377bd258c9088380e1b8b refactor: extract descriptor ID calculation from spkm GetID() (furszy)
1d207e3931cf076f69d4a8335cdd6c8ebb2a963f wallet: do not allow loading descriptor with an invalid ID (furszy)
Pull request description:
Aiming to fix#27915.
As we re-write the descriptor's db record every time that
the wallet is loaded (at `TopUp` time), if the spkm ID differs
from the one in db, the wallet will enter in an unrecoverable
corruption state (due to the storage of a descriptor with an ID
that is not linked to any other descriptor record in DB), and
no soft version will be able to open it anymore.
Because we cannot change the past, to stay compatible between
releases, we need to always use the apostrophe version for the
spkm IDs.
ACKs for top commit:
achow101:
ACK 5df988b534df842ddb658ad2a7ff0f12996c8478
Sjors:
tACK 5df988b534df842ddb658ad2a7ff0f12996c8478
Tree-SHA512: f63fc4aac7d21a4e515657471758d28857575e751865bfa359298f8b89b2568970029ca487a873c1786a5716325f453f06cd417ed193f3366417f6e8c2987332
If a user opens multiple wallets in the GUI from the
menu bar, the last one opened is the active one in
the main window. However, For the RPC Console window,
the _first_ one opened is active. This can be
confusing, as wallet RPC commands may be sent to a
wallet the user didn't intend.
This commit makes the RPC Console switch to the wallet
opened from the menu bar.
In contrast to to all other labels on the coin selection dialog, the
displayed dust information has nothing to do with the selected coins.
All that this label shows is whether at least one of the _outputs_
qualify as dust, but the outputs are set in a different dialog.
(Even worse, the dust check is currently simply wrong because it only
looks at an output's nValue and just assumes a P2PKH script size.)
As the label clearly doesn't help the user and is, quite the contrary,
rather increasing confusion/misguidance, it seems sensible to remove it.
Also, remove the label from the sendcoins dialog with the same rationale.
6c97757a480b6e71a0750330d69ff18ac7cc6127 script: appease spelling linter (Jon Atack)
1316119ce7ba3de4135bbf1e5ac28c9ea26f62e1 script: update ignored-words.txt (Jon Atack)
146c861da2e4236997bee3eed6110a5016a8b86b script: update linter dependencies (Jon Atack)
92408224a4cb2f454465d5ff8445c247f2c4318a test: fix PEP484 no implicit optional argument types errors (Jon Atack)
f86a3014338de6a2204bbdda10795b75ef6654c0 script, test: add missing python type annotations (Jon Atack)
Pull request description:
With these updates, `./test/lint/lint-python.py` and `./test/lint/lint-spelling.py` should be green again for developers using relatively recent Python dependencies, in particular mypy 0.991 (released 11/2022) and later. Please see the commit messages for details.
ACKs for top commit:
fanquake:
ACK 6c97757a480b6e71a0750330d69ff18ac7cc6127
Tree-SHA512: 8a46a4d36d5978affdcecf4f2ace20ca1b52d483e098304911a2169afe60ccb9b042fa90c04b762d94f3ce53d2cafe6f24476ae839867a770c7f31e7e7242d99
fa086248e57b89cc549a090f727c0978082727c0 test: Use same timeout for all index sync (MarcoFalke)
Pull request description:
Seems odd to use different timeouts.
Fix this by using the same timeout for all syncs.
May also fix https://github.com/bitcoin/bitcoin/issues/27355 or at least make it less frequent?
ACKs for top commit:
mzumsande:
code review ACK fa086248e57b89cc549a090f727c0978082727c0
Tree-SHA512: a61619247c97f3a88dd19eb3f200adedd120e6da8c4e4f2cf83621545b8c289dbad77e16f13cf7973a090f7b2c3391cb0297f09b0cc95fe4f55de21ae247670f
3210f224dbeaf0f9de09d4e83c3e6666128a6717 refactor: remove in-code warning suppression (fanquake)
Pull request description:
Should no-longer be needed post #27872. If it is, then suppress-external-warnings should be fixed.
ACKs for top commit:
hebasto:
ACK 3210f224dbeaf0f9de09d4e83c3e6666128a6717
Tree-SHA512: 2405250b7308779d576f13ce9144944abd5b2293499a0c0fe940398dae951cb871246a55c0e644a038ee238f9510b5845c3e39f9658d9f10225a076d8122f078
5fa4055452861ca1700008e1761815e88b29fae7 net: do not `break` when `addr` is not from a distinct network group (brunoerg)
Pull request description:
When the address is from a network group we already caught,
do a `continue` and try to find another address until conditions
are met or we reach the limit (`nTries`).
ACKs for top commit:
amitiuttarwar:
utACK 5fa4055452861ca1700008e1761815e88b29fae7
achow101:
ACK 5fa4055452861ca1700008e1761815e88b29fae7
mzumsande:
utACK 5fa4055452861ca1700008e1761815e88b29fae7
Tree-SHA512: 225bb6df450b46960db934983c583e862d1a17bacfc46d3657a0eb25a0204e106e8cd18de36764e210e0a92489ab4b5773437e4a641c9b455bde74ff8a041787
