On the ::SLOW path we would use OpenSSL as an additional source of
random bytes. This commit removes that functionality. Note that this was
always only an additional source, and that we never checked the return
value
RAND_bytes(): https://www.openssl.org/docs/manmaster/man3/RAND_bytes.html
RAND_bytes() puts num cryptographically strong pseudo-random bytes into buf.
On the ::SLOW or ::SLEEP paths, we would feed our RNG output back into
OpenSSL using RAND_add. This commit removes that functionality.
RAND_add(): https://www.openssl.org/docs/manmaster/man3/RAND_add.html
RAND_add() mixes the num bytes at buf into the internal state of the
random generator. This function will not normally be needed, as
mentioned above. The randomness argument is an estimate of how much
randomness is contained in buf, in bytes, and should be a number
between zero and num.
2f5f7d6b135e4eab368bbafd9e6e979aa72398de GuessVerificationProgress: cap the ratio to 1 (darosior)
Pull request description:
Noticed `getblockchaininfo` would return a `verificationprogress` > 1, especially while generating. This caps the verification progress to `1`.
Tried to append a check to functional tests but this would pass even without the patch, so it seems better to not add a superfluous check (but this can easily be reproduced by trying to generate blocks in the background and `watch`ing `getblockchainfo`).
ACKs for top commit:
laanwj:
ACK 2f5f7d6b135e4eab368bbafd9e6e979aa72398de
promag:
ACK 2f5f7d6b135e4eab368bbafd9e6e979aa72398de.
Tree-SHA512: fa3aca12acab9c14dab3b2cc94351082f548ea6e6c588987cd86e928a00feb023e8112433658a0e85084e294bfd940eaafa33fb46c4add94146a0901bc1c4f80
d1c02775aa74a0610809ac54bb241ddad61d2d8c Report amount of data gathered from environment (Pieter Wuille)
64e1e022cedf6776c5dffd488ca2e766adca5dc3 Use thread-safe atomic in perfmon seeder (Pieter Wuille)
d61f2bb076d8f17840a8e79f1583d7f6e3e6d09a Run background seeding periodically instead of unpredictably (Pieter Wuille)
483b94292e89587e5ab40a30b8a90e2f56e847f3 Add information gathered through getauxval() (Pieter Wuille)
11793ea22e1298fa7d3b44a5b6d20830248d8cf4 Feed CPUID data into RNG (Pieter Wuille)
a81c494b4c9a8c2f1a319a03375826f12863706f Use sysctl for seeding on MacOS/BSD (Pieter Wuille)
2554c1b81bb8c40e1989025c6f18e7935720b156 Gather additional entropy from the environment (Pieter Wuille)
c2a262a78c3bcc4d5e13612ab0214874abe15de0 Seed randomness with process id / thread id / various clocks (Pieter Wuille)
723c79666770b30cce9f962bed5ece8cc7d74580 [MOVEONLY] Move cpuid code from random & sha256 to compat/cpuid (Pieter Wuille)
cea3902015185adc88adbd031d919f91bc844fd7 [MOVEONLY] Move perfmon data gathering to new randomenv module (Pieter Wuille)
b51bae1a5a4fa8ef7825dd1bb09e3f47f96d7a5a doc: minor corrections in random.cpp (fanquake)
Pull request description:
This introduces a new `randomenv` module that queries varies non-cryptographic (and non-RNG) sources of entropy available on the system; things like user IDs, system configuration, time, statistics, CPUID data.
The idea is that these provide a fallback in scenarios where system entropy is somehow broken (note that if system entropy *fails* we will abort regardless; this is only meant to function as a last resort against undetected failure). It includes some data sources OpenSSL currently uses, and more.
The separation between random and randomenv is a bit arbitrary, but I felt that all this "non-essential" functionality deserved to be separated from the core random module.
ACKs for top commit:
TheBlueMatt:
utACK d1c02775aa74a0610809ac54bb241ddad61d2d8c. Certainly no longer measuring the time elapsed between a 1ms sleep (which got removed in the latest change) is a fair tradeoff for adding about 2 million other actually-higher-entropy bits :).
laanwj:
ACK d1c02775aa74a0610809ac54bb241ddad61d2d8c
Tree-SHA512: d290a8db6538a164348118ee02079e4f4c8551749ea78fa44b2aad57f5df2ccbc2a12dc7d80d8f3e916d68cdd8e204faf9e1bcbec15f9054eba6b22f17c66ae3
5e8a56348b5e1026e9ddcae0b2fa2a68faf4439e test: add unit test for non-standard txs with too large scriptSig (Sebastian Falbesoner)
Pull request description:
Approaches the first missing test of issue #17394: Checks that the function `IsStandardTx()` returns rejection reason `"scriptsig-size"` if any one the inputs' scriptSig is larger than 1650 bytes.
ACKs for top commit:
MarcoFalke:
ACK 5e8a56348b5e1026e9ddcae0b2fa2a68faf4439e
instagibbs:
ACK 5e8a56348b
Tree-SHA512: 79977b12ddea9438a37cefdbb48cc551e4ad02a8ccfaa2d2837ced9f3a185e2e07cc366c243b9e3c7736245e90e315d7b4110efc6b440c63dbef7ee2c9d78a73
edb6b768a4185a4aaa6281ee50a6538f7426cb1e fix uninitialized variable nMinerConfirmationWindow (NullFunctor)
Pull request description:
It is used for the computation of `BIP9WarningHeight`, and by that time it isn't initialized.
ACKs for top commit:
jnewbery:
utACK edb6b768a
promag:
ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, commit description could be cleaned up though.
MarcoFalke:
ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, used python3 to do the addition locally 📍
practicalswift:
ACK edb6b768a4185a4aaa6281ee50a6538f7426cb1e, used `clang++ -O2` on the previous version^W^W^W^W^W^W`bc` to verify the addition locally 🏓
Sjors:
Code review ACK edb6b76. Nit: commit description has duplicate text.
Tree-SHA512: 6fa0be0ecfbfd5d537f2c5b4a9333c76530c1f3182f777330cc7939b0496e37b75d8f8810cdaf471a9bd3247b425f2e239578300dfa0d5a87cd14a6ccfafa619
* Instead of calling RandAddSeedSleep anytime the scheduler goes
idle, call its replacement (RandAddSeedPeriodic) just once per
minute. This has better guarantees of actually being run, and
helps limit how frequently the dynamic env data is gathered.
* Since this code runs once per minute regardless now, we no
longer need to keep track of the last time strengthening was
run; just do it always.
* Make strengthening time context dependent (100 ms at startup,
10 ms once per minute afterwards).
e2c03c1156a1a8cb2c04c180f2ddbd3535126a46 doc: Add relase note for db→walletdb rename (Wladimir J. van der Laan)
4c1d263d93988ceed53e8f6b5decaf034b68137e scripted-diff: Change `BCLog::DB` to `BCLog::WALLETDB` (Wladimir J. van der Laan)
6b42b3ba9087225fddb91dd764c42c28d0c42d0f Rename `db` log category to `walletdb` (like `coindb`) (Wladimir J. van der Laan)
Pull request description:
Rename the `db` log category to `walletdb` (in the style of, and to distinguish from `coindb`). Deprecate (but still accept) '-debug=db'.
Second commit is a scripted commit that changes the enum item name.
ACKs for top commit:
hebasto:
ACK e2c03c1156a1a8cb2c04c180f2ddbd3535126a46, tested on Linux Mint 19.2:
Tree-SHA512: a044de6f9a70e735cbb1caa4ed6bf75bc2269b2d5bc3241a25b6a6d69c1fc1d83456e252b431388ae61f4821e4fc06ecc1b634816ceadbe9a3c0e494bee6c11e
083c954b02a4e7d0708349eeaf3bac2b5947fb0e Add settings_tests (Russell Yanofsky)
7f40528cd50fc43ac0bd3e785de24d661adddb7a Deduplicate settings merge code (Russell Yanofsky)
9dcb952fe5f85529ab28e091af7534e72c21c90f Add util::Settings struct and helper functions. (Russell Yanofsky)
e2e37cfe8af088bd8ea884be2f79f0f3cac555d5 Remove includeconf nested scope (Russell Yanofsky)
5a84aa880f6da0bac0e2144733fdef3b8558c761 Rename includeconf variables for clarity (Russell Yanofsky)
dc8e1e75487461ec9bff433144f0db831b682403 Clarify emptyIncludeConf logic (Russell Yanofsky)
Pull request description:
This is a refactoring-only change that makes it easier to add a new settings source.
This PR doesn't change behavior. The [`util_ArgsMerge`](deb2327b43/src/test/util_tests.cpp (L626-L822)) and [`util_ChainMerge`](deb2327b43/src/test/util_tests.cpp (L843-L924)) tests added in #15869 and #15988 were written specifically to confirm that ArgsManager settings are parsed, merged, and returned the same way before and after this change.
This change:
- Makes it easier to add new settings sources that can get merged with existing sources (see 70675c3e4975203ad6222ba2b00c83b4e4213793 from #15935).
- Separates parsing of settings from merging of settings, and deduplicates merging code so it doesn't happen five different places ([GetArg](c459c5f701/src/util/system.cpp (L221-L244)), [GetNetBoolArg](c459c5f701/src/util/system.cpp (L255-L261)), [GetArgs](c459c5f701/src/util/system.cpp (L460-L467)), [IsArgNegated](c459c5f701/src/util/system.cpp (L482-L491)), [GetUnsuitableSectionOnlyArgs](c459c5f701/src/util/system.cpp (L343-L352))) in inconsistent ways.
- Documents and tests current strange merging behaviors, so they be cleaned up in the future if resulting code simplifications and UX improvements warrant loss of backwards compatibility. The newly documented behaviors are: command line [ignored arguments](69d44f3cc7/src/util/system.cpp (L323-L326)) and [more ignored arguments](69d44f3cc7/src/util/settings.cpp (L67-L72)), and config file [reverse precedence](69d44f3cc7/src/util/settings.cpp (L61-L65)), [inconsistently applied top-level settings](69d44f3cc7/src/util/settings.cpp (L55-L59)), and [zombie values](69d44f3cc7/src/util/settings.cpp (L101-L108)).
The original motivation for this change was to make it easy to add a new persistent setting source without introducing more bugs and inconsistencies. Two commits building on top of this to add a persistent `-wallet` setting are pretty straightforward and show how the new code can be extended:
* 70675c3e4975203ad6222ba2b00c83b4e4213793 from #15935 – _Add \<datadir>/settings.json persistent settings storage_
* 04c80c40df9fc6f4734ba238ea7f65607cf88089 from #15937 – _Add loadwallet and createwallet RPC load_on_startup options_
ACKs for top commit:
ariard:
ACK 083c954
jnewbery:
ACK 083c954b02a4e7d0708349eeaf3bac2b5947fb0e
jamesob:
ACK 083c954b02a4e7d0708349eeaf3bac2b5947fb0e
Tree-SHA512: 5d106746a44d64d3963c4ef3f4a2fa668a4bedcc9018d3ea12c86beae2fda48a0b036241665837f68685712366f70f2e1faba84d193fa1f456013503097b7659
fad1de66a29bf6bd348a932150dad7d472feb3d0 wallet: Remove unused boost::this_thread::interruption_point (MarcoFalke)
Pull request description:
`BerkeleyEnvironment::Open` is only called from the main thread (init) or an http rpc thread, neither of which can be interrupted, so remove the useless interruption point.
`BerkeleyEnvironment{}` is only used in tests, which run in a single process/thread, so remove the useless interruption point.
ACKs for top commit:
laanwj:
ACK fad1de66a29bf6bd348a932150dad7d472feb3d0
fanquake:
ACK fad1de66a29bf6bd348a932150dad7d472feb3d0
Tree-SHA512: dacd8398e966e4a6ce5cf7d3ed821c9c267eff40b14c0635085441647cdb72d1642807f89355419f1710f814c7963e35a10d102d0b985c7198261dfc736256f8
0b75a7f0680d16a41043864a897470324917b1e8 wallet: Reuse existing batch in CWallet::SetUsedDestinationState (João Barbosa)
01f45dd00eb032a19d142026e4d019944192da19 wallet: Avoid recursive lock in CWallet::SetUsedDestinationState (João Barbosa)
Pull request description:
This PR makes 2 distinct changes around `CWallet::SetUsedDestinationState`:
- 1st the recursive lock is removed and now it requires the lock to be held;
- 2nd change is to support, in the best case, just a wallet database flush when transaction is added to the wallet.
ACKs for top commit:
achow101:
ACK 0b75a7f0680d16a41043864a897470324917b1e8
MarcoFalke:
ACK 0b75a7f0680d16a41043864a897470324917b1e8
ryanofsky:
Code review ACK 0b75a7f0680d16a41043864a897470324917b1e8. Code changes looks fine but PR description should be updated to say what benefits of the change are. I might have missed something, but I didn't see a place where multiple batches were used previously and a single batch was used now. So the main benefit of this change appears to be removing a recursive lock? And maybe moving toward a consistent convention for passing batch instances?
Tree-SHA512: abcf23a5850d29990668db20d6f624cca3e89629cc9ed003e0d05cde1b58ab2ff365034f156684ad13e55764b54c6c0c2bc7d5f96b8af7dc5e45a3be955d6b15
36b68de5b2938722911db900ca299f7008780d01 Remove getBlockDepth method from Chain::interface (Antoine Riard)
b66c429c56c85fa16c309be0b2bca9c25fdd3e1a Remove locked_chain from GetDepthInMainChain and its callers (Antoine Riard)
0ff03871add000f8b4d8f82aeb168eed2fc9dc5f Use CWallet::m_last_block_processed_height in GetDepthInMainChain (Antoine Riard)
f77b1de16feee097a88e99d2ecdd4d84beb4f915 Only return early from BlockUntilSyncedToCurrentChain if current tip is exact match (Antoine Riard)
769ff05e48fb53d4b62c59060424a0fea71d0aab Refactor some importprunedfunds checks with guard clause (Antoine Riard)
5971d3848e09abf571e5308185275296127efca4 Add block_height field in struct Confirmation (Antoine Riard)
9700fcb47feca9d78e005b8d18b41148c8f6b25f Replace CWalletTx::SetConf by Confirmation initialization list (Antoine Riard)
5aacc3eff15b9b5bdc951f1e274f00d581f63bce Add m_last_block_processed_height field in CWallet (Antoine Riard)
10b4729e33f76092bd8cfa06d1a5e0a066436f76 Pass block height in Chain::BlockConnected/Chain::BlockDisconnected (Antoine Riard)
Pull request description:
Work starter to remove Chain::Lock interface by adding m_last_block_processed_height in CWallet and m_block_height in CMerkleTx to avoid GetDepthInMainChain having to keep a lock . Once this one done, it should ease work to wipe out more cs_main locks from wallet code.
I think it's ready for a first round of review before to get further.
- `BlockUntilSyncedToCurrent` : restrain isPotentialTip to isTip because we want to be sure that wallet see BlockDisconnected callbacks if its height differs from the Chain one. It means during a reorg, an RPC could return before the BlockDisconnected callback had been triggered. This could cause a tx that had been included in the disconnected block to be displayed as confirmed, for example.
~~- `AbandonTransaction` : in case of conflicted tx (nIndex = -1), we set its m_block_height to the one of conflicting blocks, but if this height is superior to CWallet::m_last_block_processed_height, that means tx isn't conflicted anymore so we return 0 as tx is again unconfirmed~~ After #16624, we instead rely on Confirmation.
~~- `AddToWalletIfInvolvingMe`: in case of block disconnected, transactions are added to mempool again, so we need to replace old txn in `mapWallet` with a height set to zero so we remove check on block_hash.IsNull~~ Already done in #16624
ACKs for top commit:
jnewbery:
@jkczyz you've ACKed an intermediate commit (github annoyingly orders commits in date order, not commit order). Did you mean to ACK the final commit in this branch (36b68de5b2938722911db900ca299f7008780d01).
jkczyz:
> @jkczyz you've ACKed an intermediate commit (github annoyingly orders commits in date order, not commit order). Did you mean to ACK the final commit in this branch ([36b68de](36b68de5b2)).
meshcollider:
utACK 36b68de5b2938722911db900ca299f7008780d01
ryanofsky:
Code review ACK 36b68de5b2938722911db900ca299f7008780d01. Changes since last review: new jkczyz refactor importprunedfunds commit, changed BlockUntilSyncedToCurrentChainChanges commit title and description, changed Confirmation struct field order and line-wrapped comment
jnewbery:
utACK 36b68de5b2938722911db900ca299f7008780d01
promag:
Code review ACK 36b68de5b2938722911db900ca299f7008780d01.
Tree-SHA512: 08b89a0bcc39f67c82a6cb6aee195e6a11697770c788ba737b90986b4893f44e90d1ab9ef87239ea3766508b7e24ea882b7199df41173ab27a3d000328c14644
Get rid of settings merging code in util/system.cpp repeated 5 places,
inconsistently:
- ArgsManagerHelper::GetArg
- ArgsManagerHelper::GetNetBoolArg
- ArgsManager::GetArgs
- ArgsManager::IsArgNegated
- ArgsManager::GetUnsuitableSectionOnlyArgs
Having settings merging code separated from parsing simplifies parsing somewhat
(for example negated values can simply be represented as false values instead
of partially cleared or emply placeholder lists).
Having settings merge happen one place instead of 5 makes it easier to add new
settings sources and harder to introduce new inconsistencies in the way
settings are merged.
This commit does not change behavior in any way.
Implement merging of settings from different sources (command line and config
file) separately from parsing code in system.cpp, so it is easier to add new
sources.
Document current inconsistent merging behavior without changing it.
This commit only adds new settings code without using it. The next commit calls
the new code to replace existing code in system.cpp.
Co-authored-by: John Newbery <john@johnnewbery.com>
fae43a97ca947cd0802392e9bb86d9d0572c0fba test: Seed test RNG context for each test case, print seed (MarcoFalke)
Pull request description:
Debugging failing unit tests is hard if the failure is non-deterministic and the seed is not known.
Fix that by printing the seed and making it possible to set the seed from outside.
ACKs for top commit:
davereikher:
Tested ACK fae43a97ca947cd0802392e9bb86d9d0572c0fba
Tree-SHA512: 33d848dd1f4180d3664ecf60e9810c2a93590c05276b2c46b1e4fe6e376b45916a46b90c803bb602750ab666da3a05ce499e550024685a90b8cc38fab6667cb8
5506ecfe7a65d5705616bc048f2f1735b89993fb [refactor] Replace global int nScriptCheckThreads with bool (John Newbery)
d9957623b48a7c3eff0ac750d1245fabfb1843a2 [tests] Don't use TestingSetup in the checkqueue_tests (John Newbery)
Pull request description:
The meaning of this value is confusing. Refactor it and add comments.
ACKs for top commit:
sipa:
ACK 5506ecfe7a65d5705616bc048f2f1735b89993fb
promag:
ACK 5506ecfe7a65d5705616bc048f2f1735b89993fb, only change was addressing my nits.
laanwj:
Code review ACK 5506ecfe7a65d5705616bc048f2f1735b89993fb
MarcoFalke:
ACK 5506ecfe7a65d5705616bc048f2f1735b89993fb 🥐
Tree-SHA512: 78536727c98d2c23f3c0f3f169131474fef9a4486ae65029011caf06eab30f6f70ff73a65b2fb04a5d969fc1150858d1c6ea4767f04d48c1eea6b829316d0e63
fa4c6fa9b1139791f45f1495d662c1c7cd2f7ed6 doc: Add documentation for new test/lib (MarcoFalke)
faec28252cf4f8e754c689a7a44fd421f631db50 scripted-diff: test: Move setup_common to test library (MarcoFalke)
Pull request description:
Sorry for clickbait, this is only a move-only scripted-diff commit and one documentation commit.
Longer term, someone who knows something about build systems can make this an actual library. Motivation for this is that each module gets compiled for each target that includes it. For example, setup_common is compiled 27 times (for the fuzz suite) and another 3 times for the other tests (bench, unit test, gui)
ACKs for top commit:
practicalswift:
ACK fa4c6fa9b1139791f45f1495d662c1c7cd2f7ed6 -- diff looks correct and Travis is happy
jonatack:
ACK fa4c6fa9b1139791f45f1495d662c1c7cd2f7ed6 with the reserve that the commit messages (and PR description) contain the motivation for this change. Built, ran tests, light code review.
ryanofsky:
Code review ACK fa4c6fa9b1139791f45f1495d662c1c7cd2f7ed6. I didn't realize `lib` was actually name of existing directory, not a new name. But in any case this looks good and nice to have one scripted diff instead of two.
Tree-SHA512: 2e176df90c60578276e4a6dc83ff57ff59d8e666ecf30c5ceacb8c326725da91baa4cac3dfa7a2e1605f58122a3e3e27e4938ff33e3a0ce7ea53afffebbf57a4
The global nScriptCheckThreads int is confusing and is only needed for
its int-ness in AppInitMain. Move all `-par` parsing logic there and
replace the int nScriptCheckThreads with a bool
g_parallel_script_checks.
Also tidy up logic and improve comments.
