1538 Commits

Author SHA1 Message Date
MarcoFalke
fa899fb7aa
fuzz: Speed up utxo_snapshot fuzz target
This speeds up the fuzz target, which allows "valid" inputs. It does not
affect the "INVALID" fuzz target.
2024-08-20 07:54:04 +02:00
MarcoFalke
fa386642b4
fuzz: Speed up utxo_snapshot by lazy re-init
The re-init is expensive, so skip it if there is no need.

Also, add an even faster fuzz target utxo_snapshot_invalid, which does
not need any re-init at all.
2024-08-16 08:06:59 +02:00
MarcoFalke
fa645c7a86
fuzz: Remove unused DataStream object 2024-08-15 08:53:04 +02:00
merge-script
24ced52744
Merge bitcoin/bitcoin#28687: C++20 std::views::reverse
2925bd537cbd8c70594e23f6c4298b7101f7f73d refactor: use c++20 std::views::reverse instead of reverse_iterator.h (stickies-v)

Pull request description:

  C++20 introduces [`std::ranges::views::reverse`](https://en.cppreference.com/w/cpp/ranges/reverse_view), which allows us to drop our own `reverse_iterator.h` implementation and also makes it easier to chain views (even though I think we currently don't use this).

ACKs for top commit:
  achow101:
    ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d
  maflcko:
    ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d 🎷

Tree-SHA512: 567666ec44af5d1beb7a271836bcc89c4c577abc77f522fcc18bc6d4de516ae9b0df766d0bfa6dd217569e6878331c2aee1d9815620860375e3510dad7fed476
2024-08-09 09:51:19 +01:00
Ava Chow
27a770b34b
Merge bitcoin/bitcoin#28280: Don't empty dbcache on prune flushes: >30% faster IBD
589db872e116779ab9cae693171ac8a8c02d9923 validation: don't erase coins cache on prune flushes (Andrew Toth)
0e8918755f725b6269ed2be5a0b46f1611233515 Add linked-list test to CCoinsViewCache::SanityCheck (Pieter Wuille)
05cf4e18758618bb493d26014d3a9c89bf28d898 coins: move Sync logic to CoinsViewCacheCursor (Andrew Toth)
7825b8b9aeceb4ff607650cdc9c49e5de9c7719f coins: pass linked list of flagged entries to BatchWrite (Andrew Toth)
a14edada8a051e280af6fedd5130be40247e2d7a test: add cache entry linked list tests (Andrew Toth)
24ce37cb867b95e86d9fd4e50858d64ee8a59abf coins: track flagged cache entries in linked list (Andrew Toth)
58b7ed156d5993b69375bb455b03bd8b17f64fa4 coins: call ClearFlags in CCoinsCacheEntry destructor (Andrew Toth)
8bd3959feaa0e71585bc5e0976f584fb06ee6d14 refactor: require self and sentinel parameters for AddFlags (Andrew Toth)
75f36d241d2a344c5c4ce2c80250bdde91b3295e refactor: add CoinsCachePair alias (Andrew Toth)
f08faeade2f99ae1de6f3c481697541cc16186c7 refactor: move flags to private uint8_t and rename to m_flags (Andrew Toth)
4e4fb4cbabcc10e723534f5f80f3e3cf09f6ca50 refactor: disallow setting flags in CCoinsCacheEntry constructors (Andrew Toth)
8737c0cefa6ec49a4d17d9bef9e5e1a7990af1ac refactor: encapsulate flags setting with AddFlags and ClearFlags (Andrew Toth)
9715d3bf1e4013476539f1523a6b54d2055c32c6 refactor: encapsulate flags get access for all other checks (Andrew Toth)
df34a94e57659c31873c26c86a8de5a032400061 refactor: encapsulate flags access for dirty and fresh checks (Andrew Toth)

Pull request description:

  Since https://github.com/bitcoin/bitcoin/pull/17487 we no longer need to clear the coins cache when syncing to disk. A warm coins cache significantly speeds up block connection, and only needs to be fully flushed when nearing the `dbcache` limit.

  For frequent pruning flushes there's no need to empty the cache and kill connect block speed. However, simply using `Sync` in place of `Flush` actually slows down a pruned full IBD with a high `dbcache` value. This is because as the cache grows, sync takes longer since every coin in the cache is scanned to check if it's dirty. For frequent prune flushes and a large cache this constant scanning starts to really slow IBD down, and just emptying the cache on every prune becomes faster.

  To fix this, we can add two pointers to each cache entry and construct a doubly linked list of dirty entries. We can then only iterate through all dirty entries on each `Sync`, and simply clear the pointers after.

  With this approach a full IBD with `dbcache=16384` and `prune=550` was 32% faster than master. For default `dbcache=450` speedup was ~9%. All benchmarks were run with `stopatheight=800000`.

  |  | prune | dbcache | time | max RSS | speedup |
  |-----------:|----------:|------------:|--------:|-------------:|--------------:|
  | master | 550 | 16384 | 8:52:57 | 2,417,464k | - |
  | branch | 550 | 16384 | 6:01:00 | 16,216,736k | 32% |
  | branch | 550 | 450 | 8:05:08 | 2,818,072k | 8.8% |
  | master | 10000 | 5000 | 8:19:59 | 2,962,752k | - |
  | branch | 10000 | 5000| 5:56:39 | 6,179,764k | 28.8% |
  | master | 0 | 16384 | 4:51:53 | 14,726,408k | - |
  | branch | 0 | 16384 | 4:43:11 | 16,526,348k | 2.7% |
  | master | 0 | 450 | 7:08:07 | 3,005,892k | - |
  | branch | 0 | 450 | 6:57:24 | 3,013,556k |2.6%|

  While the 2 pointers add memory to each cache entry, it did not slow down IBD. For non-pruned IBD results were similar for this branch and master. When I performed the initial IBD, the full UTXO set could be held in memory when using the max `dbcache` value. For non-pruned IBD with max `dbcache` to tip ended up using 12% more memory, but it was also 2.7% faster somehow. For smaller `dbcache` values the `dbcache` limit is respected so does not consume more memory, and the potentially more frequent flushes were not significant enough to cause any slowdown.

  For reviewers, the commits in order do the following:
  First 4 commits encapsulate all accesses to `flags` on cache entries, and then the 5th makes `flags` private.
  Commits `refactor: add CoinsCachePair alias` to `coins: call ClearFlags in CCoinsCacheEntry destructor` create the linked list head nodes and cache entry self references and pass them into `AddFlags`.
  Commit `coins: track flagged cache entries in linked list` actually adds the entries into a linked list when they are flagged DIRTY or FRESH and removes them from the linked list when they are destroyed or the flags are cleared manually. However, the linked list is not yet used anywhere.
  Commit `test: add cache entry linked list tests` adds unit tests for the linked list.
  Commit `coins: pass linked list of flagged entries to BatchWrite` uses the linked list to iterate through DIRTY entries instead of using the entire coins cache.
  Commit `validation: don't erase coins cache on prune flushes` uses `Sync` instead of `Flush` for pruning flushes, so the cache is no longer cleared.

  Inspired by [this comment](https://github.com/bitcoin/bitcoin/pull/15265#issuecomment-457720636).

  Fixes https://github.com/bitcoin/bitcoin/issues/11315.

ACKs for top commit:
  paplorinc:
    ACK 589db872e116779ab9cae693171ac8a8c02d9923
  sipa:
    reACK 589db872e116779ab9cae693171ac8a8c02d9923
  achow101:
    ACK 589db872e116779ab9cae693171ac8a8c02d9923
  mzumsande:
    re-ACK 589db872e116779ab9cae693171ac8a8c02d9923

Tree-SHA512: 23b2bc01c83edacb5b39aa60bb0b766de9a74ce17f0c59bf13b97b4328a7b758ad9aff6581c3ca88e2973f7658380651530d497444f48d6e22ea0bfc51cc921d
2024-08-07 20:06:39 -04:00
Ryan Ofsky
870447fd58
Merge bitcoin/bitcoin#30212: rename TransactionError:ALREADY_IN_CHAIN
e9de0a76b99fd4708295e1178f6c079db92e7f36 doc: release note for 30212 (willcl-ark)
87b188052528e97729a85d9701864eaff1ea5ec6 rpc: clarify ALREADY_IN_CHAIN rpc errors (willcl-ark)

Pull request description:

  Closes: #19363

  Renaming this error improves clarity around the returned error both internally and externally when a transactions' outputs are already found in the utxo set (`TransactionError::ALREADY_IN_CHAIN -> TransactionError::ALREADY_IN_UTXO_SET`)

ACKs for top commit:
  tdb3:
    ACK e9de0a76b99fd4708295e1178f6c079db92e7f36
  ismaelsadeeq:
    ACK e9de0a76b99fd4708295e1178f6c079db92e7f36
  ryanofsky:
    Code review ACK e9de0a76b99fd4708295e1178f6c079db92e7f36.

Tree-SHA512: 7d2617200909790340951fe56a241448f9ce511900777cb2a712e8b9c0778a27d1f912b460f82335844224f1abb4322bc898ca076440959edade55c082a09237
2024-08-06 11:31:03 -04:00
josibake
59c0ece0a7
fuzz: replace hardcoded numbers for bech32 limits
Use bech32::CharLimit::BECH32 and bech32::CHECKSUM_SIZE instead of
hardcoded values. This is a follow-up fix for #34007
(where this file was missed)
2024-08-06 11:03:31 +02:00
Andrew Toth
7825b8b9ae
coins: pass linked list of flagged entries to BatchWrite
BatchWrite now iterates through the linked
list of flagged entries instead of the entire
coinsCache map.

Co-Authored-By: l0rinc <pap.lorinc@gmail.com>
2024-08-05 19:43:56 -04:00
stickies-v
2925bd537c
refactor: use c++20 std::views::reverse instead of reverse_iterator.h
Use std::ranges::views::reverse instead of the implementation in
reverse_iterator.h, and remove it as it is no longer used.
2024-08-06 00:23:38 +01:00
willcl-ark
87b1880525
rpc: clarify ALREADY_IN_CHAIN rpc errors
When using `sendrawtransaction` the ALREADY_IN_CHAIN error help string
may be confusing.

Rename TransactionError::ALREADY_IN_CHAIN to
TransactionError::ALREADY_IN_UTXO_SET and update the rpc help string.

Remove backwards compatibility alias as no longer required.
2024-08-05 15:45:58 +01:00
Hodlinator
c06f2368e2
refactor: Hand-replace some uint256S -> uint256
chainparams.cpp - workaround for MSVC bug triggering C7595 - Calling consteval constructors in initializer lists fails, but works on GCC (13.2.0) & Clang (17.0.6).
2024-08-05 14:51:47 +02:00
glozow
1a19a4d960
Merge bitcoin/bitcoin#29656: chainparams: Change nChainTx type to uint64_t
bf0efb4fc72d3c49a2c498c944e55466dfa046dc scripted-diff: Modernize naming of nChainTx and nTxCount (Fabian Jahr)
72e5d1be1f4491565249d43e836ee42cfd858866 test: Add basic check for nChainTx type (Fabian Jahr)
dc2938e9799d79696d1db2438ef33d90542d984b chainparams: Change nChainTx to uint64_t (Fabian Jahr)

Pull request description:

  This picks up the work from #29331 and closes #29258.

  This simply changes the type and addresses the comments from #29331 by changing the type in all relevant places and removing unnecessary casts. This also adds an extremely simple unit test.

  Additionally this modernizes the name of `nChainTx` which helps reviewers check all use of the symbol and can make silent merge conflicts.

ACKs for top commit:
  maflcko:
    only rebase in scripted-diff, re-ACK bf0efb4fc72d3c49a2c498c944e55466dfa046dc 🔈
  glozow:
    reACK bf0efb4fc72 via range-diff

Tree-SHA512: ee4020926d0800236fe655d0c7b127215ab36b553b04d5f91494f4b7fac6e1cfe7ee298b07c0983db5a3f4786932acaa54f5fd2ccd45f2fcdcfa13427358dc3b
2024-08-05 10:00:25 +01:00
glozow
bba01ba18d
Merge bitcoin/bitcoin#30285: cluster mempool: merging & postprocessing of linearizations
bbcee5a0d67db46526ba29a1a4a7c590d303de03 clusterlin: improve rechunking in LinearizationChunking (optimization) (Pieter Wuille)
04d7a04ea426dd0a69b61e3b887867b0277d84d1 clusterlin: add MergeLinearizations function + fuzz test + benchmark (Pieter Wuille)
4f8958d7563ae2d0d359ec1e6885f8cb5e40a5e0 clusterlin: add PostLinearize + benchmarks + fuzz tests (Pieter Wuille)
0e2812d2938b933debffba5b873637fa1d348b81 clusterlin: add algorithms for connectedness/connected components (Pieter Wuille)
0e52728a2d6ccafcfecfefbb5a0432a9881d8e0d clusterlin: rename Intersect -> IntersectPrefixes (Pieter Wuille)

Pull request description:

  Part of cluster mempool: #30289

  Depends on #30126, and was split off from it. #28676 depends on this.

  This adds the algorithms for merging & postprocessing linearizations.

  The `PostLinearize(depgraph, linearization)` function performs an in-place improvement of `linearization`, using two iterations of the [Linearization post-processing](https://delvingbitcoin.org/t/linearization-post-processing-o-n-2-fancy-chunking/201/8) algorithm. The first running from back to front, the second from front to back.

  The `MergeLinearizations(depgraph, linearization1, linearization2)` function computes a new linearization for the provided cluster, given two existing linearizations for that cluster, which is at least as good as both inputs. The algorithm is described at a high level in [merging incomparable linearizations](https://delvingbitcoin.org/t/merging-incomparable-linearizations/209).

  For background and references, see [Introduction to cluster linearization](https://delvingbitcoin.org/t/introduction-to-cluster-linearization/1032).

ACKs for top commit:
  sdaftuar:
    ACK bbcee5a0d67db46526ba29a1a4a7c590d303de03
  glozow:
    code review ACK bbcee5a0d67
  instagibbs:
    ACK bbcee5a0d6

Tree-SHA512: d2b5a3f132d1ef22ddf9c56421ab8b397efe45b3c4c705548dda56f5b39fe4b8f57a0d2a4c65b338462d80bb5b9b84a9a39efa1b4f390420a8005ce31817774e
2024-08-05 09:42:22 +01:00
Fabian Jahr
bf0efb4fc7
scripted-diff: Modernize naming of nChainTx and nTxCount
-BEGIN VERIFY SCRIPT-
sed -i 's/nChainTx/m_chain_tx_count/g' $(git grep -l 'nChainTx' ./src)
sed -i 's/nTxCount/tx_count/g' $(git grep -l 'nTxCount' ./src)
-END VERIFY SCRIPT-
2024-08-04 14:24:43 +02:00
glozow
2aff9a36c3
Merge bitcoin/bitcoin#30352: policy: Add PayToAnchor(P2A), OP_1 <0x4e73> as a standard output script for spending
75648cea5a9032b3d388cbebacb94d908e08924e test: add P2A ProduceSignature coverage (Greg Sanders)
7998ce6b20fba62c022228355907b612ba6692e1 Add release note for P2A output feature (Greg Sanders)
71c9b02a04742eeecab14aae4697b1a3eb51ff7f test: add P2A coverage for decodescript (Greg Sanders)
1349e9ec1558484f2912a2444c410170fcec8745 test: Add anchor mempool acceptance test (Greg Sanders)
9d892099378b2ad5f52220403bdeae43c61d6955 policy: stop 3rd party wtxid malleability of anchor spend (Greg Sanders)
b60aaf8b239978947d2b0e3f56e7d8a4092d7570 policy: make anchor spend standard (Greg Sanders)
455fca86cfada1823aa28615b5683f9dc73dbb9a policy: Add OP_1 <0x4e73> as a standard output type (Greg Sanders)

Pull request description:

  This is a sub-feature taken out of the original proposal for ephemeral anchors #30239

  This PR makes *spending* of `OP_1 <0x4e73>` (i.e. `bc1pfeessrawgf`) standard. Creation of this output type is already standard.

  Any future witness output types are considered relay-standard to create, but not to spend. This preserves upgrade hooks, such as a completely new output type for a softfork such as BIP341.  It also gives us a bit of room to use a new output type for policy uses.

  This particular sized witness program has no other known use-cases (https://bitcoin.stackexchange.com/a/110664/17078), s it affords insufficient cryptographic security for a secure commitment to data, such as a script or a public key. This makes this type of output "keyless", or unauthenticated.

  As a witness program, the `scriptSig` of the input MUST be blank, by BIP141. This helps ensure txid-stability of the spending transaction, which may be required for smart contracting wallets. If we do not use segwit, a miner can simply insert an `OP_NOP` in the `scriptSig` without effecting the result of program execution.

  An additional relay restriction is to disallow non-empty witness data, which an adversary may use to penalize the "honest" transactor when RBF'ing the transaction due to the incremental fee requirement of RBF rules.

  The intended use-case for this output type is to "anchor" the transaction with a spending child to bring exogenous CPFP fees into the transaction package, encouraging the inclusion of the package in a block. The minimal size of creation and spending of this output makes it an attractive contrast to outputs like `p2sh(OP_TRUE)` and `p2wsh(OP_TRUE)` which
  are significantly larger in vbyte terms.

  Combined with TRUC transactions which limits the size of child transactions significantly, this is an attractive option for presigned transactions that need to be fee-bumped after the fact.

ACKs for top commit:
  sdaftuar:
    utACK 75648cea5a9032b3d388cbebacb94d908e08924e
  theStack:
    re-ACK 75648cea5a9032b3d388cbebacb94d908e08924e
  ismaelsadeeq:
    re-ACK 75648cea5a9032b3d388cbebacb94d908e08924e via [diff](e7ce6dc070..75648cea5a)
  glozow:
    ACK 75648cea5a9032b3d388cbebacb94d908e08924e
  tdb3:
    ACK 75648cea5a9032b3d388cbebacb94d908e08924e

Tree-SHA512: d529de23d20857e6cdb40fa611d0446b49989eaafed06c28280e8fd1897f1ed8d89a4eabbec1bbf8df3d319910066c3dbbba5a70a87ff0b2967d5205db32ad1e
2024-08-02 15:49:44 +01:00
Andrew Toth
8bd3959fea
refactor: require self and sentinel parameters for AddFlags
No behavior change. Prepares for adding the CoinsCachePairs to
a linked list when flagged.
2024-08-02 00:29:42 -04:00
Andrew Toth
f08faeade2
refactor: move flags to private uint8_t and rename to m_flags
No behavior change. This prepares to add CCoinsCacheEntrys
to a doubly linked list when a flag is added.
2024-08-01 23:36:00 -04:00
Andrew Toth
8737c0cefa
refactor: encapsulate flags setting with AddFlags and ClearFlags
No behavior change. This prepares moving the cache entry
flags field to private access.
2024-08-01 23:36:00 -04:00
Andrew Toth
df34a94e57
refactor: encapsulate flags access for dirty and fresh checks
No behavior change. This prepares moving the cache entry flags field
to private access.

Co-Authored-By: l0rinc <pap.lorinc@gmail.com>
2024-08-01 23:36:00 -04:00
Pieter Wuille
04d7a04ea4 clusterlin: add MergeLinearizations function + fuzz test + benchmark 2024-08-01 16:03:34 -04:00
Pieter Wuille
4f8958d756 clusterlin: add PostLinearize + benchmarks + fuzz tests 2024-08-01 16:02:09 -04:00
Pieter Wuille
0e2812d293 clusterlin: add algorithms for connectedness/connected components
Add utility functions to DepGraph for finding connected components.
2024-08-01 15:43:59 -04:00
Pieter Wuille
0e52728a2d clusterlin: rename Intersect -> IntersectPrefixes
This makes it clearer what the function does.
2024-08-01 14:07:54 -04:00
glozow
ebd82fa9fa
Merge bitcoin/bitcoin#30532: refactor: remove deprecated TxidFromString() in favour of transaction_identifier::FromHex()
f553e6d86fe114e90585ea6d4b8e345a209cae5d refactor: remove TxidFromString (stickies-v)
285ab50ace4c04209f331ccaf121152b977cc490 test: replace WtxidFromString with Wtxid::FromHex (stickies-v)
9a0b2a69c4d836d7382f6fe7703de40288fc7421 fuzz: increase FromHex() coverage (stickies-v)
526a87ba6b4f20592ad3c090b82e83ecff2107fc test: add uint256::FromHex unittest coverage (stickies-v)

Pull request description:

  Since fab6ddbee6, `TxidFromString()` has been deprecated because it is less robust than the `transaction_identifier::FromHex()` introduced in [the same PR](https://github.com/bitcoin/bitcoin/pull/30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters.

  In this PR, `TxidFromString` is removed completely to clean up the code and prevent further unsafe usage. Unit and fuzz test coverage on `uint256::FromHex()` and functions that wrap it is increased.

  Note: `TxidFromSring` allowed users to prefix strings with "0x", this is no longer allowed for `transaction_identifier::FromHex()`, so a helper function for input validation may prove helpful in the future _(this overlaps with the `uint256::uint256S()` vs `uint256::FromHex()` future cleanup)_. It is not relevant to this PR, though, besides the fact that this unused (except for in tests) functionality is removed.

  The only users of `TxidFromString` are:
  - `test`, where it is straightforward to drop in the new `FromHex()` methods without much further concern
  - `qt` coincontrol. There is no need for input validation here, but txids are not guaranteed to be 64 characters. This is already handled by the existing code, so again, using `FromHex()` here seems quite straightforward.

  Addresses @maflcko's suggestion: https://github.com/bitcoin/bitcoin/pull/30482#discussion_r1691826934

  Also removes `WtxidFromString()`, which is a test-only helper function.

  ### Testing GUI changes

  To test the GUI coincontrol affected lines, `regtest` is probably the easiest way to quickly get some test coins, you can use e.g.

  ```
  alias cli="./src/bitcoin-cli -regtest"
  cli createwallet "coincontrol"
  # generate 10 spendable outputs on 1 address
  cli generatetoaddress 10 $(cli -rpcwallet=coincontrol getnewaddress)
  # generate 10 spendable outputs on another address
  cli generatetoaddress 10 $(cli -rpcwallet=coincontrol getnewaddress)
  # make previous outputs spendable
  cli generatetoaddress 100 $(cli -rpcwallet=coincontrol getnewaddress)
  ```

ACKs for top commit:
  maflcko:
    re-ACK f553e6d86fe114e90585ea6d4b8e345a209cae5d 🔻
  hodlinator:
    ACK f553e6d86fe114e90585ea6d4b8e345a209cae5d
  paplorinc:
    ACK f553e6d86fe114e90585ea6d4b8e345a209cae5d
  TheCharlatan:
    Nice, ACK f553e6d86fe114e90585ea6d4b8e345a209cae5d

Tree-SHA512: c1c7e6ea4cbf05cf660ba178ffc4f35f0328f7aa6ad81872e2462fb91a6a22e4681ff64b3d0202a5a9abcb650c939561585cd309164a69ab6081c0765ee271ef
2024-08-01 12:02:52 +01:00
stickies-v
9a0b2a69c4
fuzz: increase FromHex() coverage 2024-07-31 16:47:38 +01:00
dergoegge
afd237bb5d [fuzz] Harness for version handshake 2024-07-31 13:25:52 +01:00
Greg Sanders
455fca86cf policy: Add OP_1 <0x4e73> as a standard output type
These outputs are called anchors, and allow
key-less anchor spends which are vsize-minimized
versus keyed anchors which require larger outputs
when creating and inputs when spending.
2024-07-30 14:06:58 -04:00
Ryan Ofsky
123888dcb8
Merge bitcoin/bitcoin#30447: fuzz: Deglobalize signature cache in sigcache test
fae0db0360466aed536f4ce96d357cf579100080 fuzz: Deglobalize signature cache in sigcache test (TheCharlatan)

Pull request description:

  The body of the fuzz test should ideally be a pure function. If data is persisted in the cache over many iterations, and there is a crash, reproducing it from the input might be difficult. Solve this by getting rid of the global state. This is a follow-up from #30425.

ACKs for top commit:
  dergoegge:
    utACK fae0db0360466aed536f4ce96d357cf579100080
  ryanofsky:
    Code review ACK fae0db0360466aed536f4ce96d357cf579100080

Tree-SHA512: 93dcbb9f2497f13856970469042d6870f04de10fe206827a8db1aae7fc8f3ac7fd900bee7945b5fe4c9e33883268dabb15be7e7bc91cf353ffc0d118cd60e97d
2024-07-26 07:41:10 -04:00
Pieter Wuille
28549791b3 clusterlin: permit passing in existing linearization to Linearize
This implements the LIMO algorithm for linearizing by improving an existing
linearization. See
https://delvingbitcoin.org/t/limo-combining-the-best-parts-of-linearization-search-and-merging
for details.
2024-07-25 10:16:40 -04:00
Pieter Wuille
97d98718b0 clusterlin: add LinearizationChunking class
It encapsulates a given linearization in chunked form, permitting arbitrary
subsets of transactions to be removed from the linearization. Its purpose
is adding the Intersect function, which is a crucial operation that will
be used in a further commit to make Linearize improve existing linearizations.
2024-07-25 10:16:40 -04:00
Pieter Wuille
d5918dc3c6 clusterlin: randomize the SearchCandidateFinder search order
To make search non-deterministic, change the BFS logic from always picking
the first queue item to randomly picking the first or second queue item.
2024-07-25 10:16:40 -04:00
Pieter Wuille
46aad9b099 clusterlin: add Linearize function
This adds a first version of the overall linearization interface, which given
a DepGraph constructs a good linearization, by incrementally including good
candidate sets (found using AncestorCandidateFinder and SearchCandidateFinder).
2024-07-25 10:16:37 -04:00
Pieter Wuille
ee0ddfe4f6 clusterlin: add chunking algorithm
A fuzz test is added which verifies various of its expected properties, including
correctness
2024-07-25 10:16:37 -04:00
Pieter Wuille
2a41f151af clusterlin: add SearchCandidateFinder class
Similar to AncestorCandidateFinder, this encapsulates the state needed for
finding good candidate sets using a search algorithm.
2024-07-25 10:16:37 -04:00
Pieter Wuille
4828079db3 clusterlin: add AncestorCandidateFinder class
This is a class that encapsulates precomputed ancestor set feerates, and
presents an interface for getting the best remaining ancestor set.
2024-07-25 10:16:37 -04:00
Pieter Wuille
58f7e01db4 tests: framework for testing DepGraph class
This introduces a bespoke fuzzing-focused serialization format for DepGraphs,
and then tests that this format can represent any graph, roundtrips, and then
uses that to test the correctness of DepGraph itself.

This forms the basis for future fuzz tests that need to work with interesting
graphs.
2024-07-25 10:16:37 -04:00
merge-script
bee23ce9ec
Merge bitcoin/bitcoin#30399: test: Add arguments for creating a slimmer TestingSetup
f46b2202560a76b473e229b77303b8f877c16cac fuzz: Use BasicTestingSetup for coins_view target (TheCharlatan)
9e2a723d5da4fc277a42fed37424f578e348ebf8 test: Add arguments for creating a slimmer setup (TheCharlatan)

Pull request description:

  This adds arguments to some of the testing setup constructors for creating an environment without networking and a validation interface. This is useful for improving the performance of the utxo snapshot fuzz test,  which constructs a new TestingSetup on each iteration.

  Using this slimmed down `TestingSetup` in future might also make the tests a bit faster when run in aggregate.

ACKs for top commit:
  maflcko:
    review ACK f46b2202560a76b473e229b77303b8f877c16cac
  dergoegge:
    utACK f46b2202560a76b473e229b77303b8f877c16cac

Tree-SHA512: 9dc62512b127b781fc9e2d8ef2b5a9b06ebb927a8294b6d872001c553984a7eb1f348e0257b32435b34b5505b5d0323f73bdd572a673da272d3e1e8538ab49d6
2024-07-25 13:53:50 +01:00
MarcoFalke
fa7b57e5f5
refactor: Replace ParseHashStr with FromHex
No need to have two functions with different names that achieve the
exact same thing.
2024-07-24 17:40:18 +02:00
MarcoFalke
fa33a63bd9
fuzz: Speed up PickValue in txorphan
Co-Authored-By: l0rinc <pap.lorinc@gmail.com>
2024-07-23 10:37:58 +02:00
Lőrinc
bccfca0382 Fix lint-spelling warnings
These warnings were often polluting the CI output, e.g. https://github.com/bitcoin/bitcoin/pull/30499/checks?check_run_id=27745036545

> ./test/lint/lint-spelling.py

before the change:
```
doc/design/libraries.md💯 targetted ==> targeted
doc/developer-notes.md:495: dependant ==> dependent
src/bench/sign_transaction.cpp:49: hashIn ==> hashing, hash in
src/bitcoin-chainstate.cpp:213: hashIn ==> hashing, hash in
src/bitcoin-chainstate.cpp:213: hashIn ==> hashing, hash in
src/coins.cpp:24: viewIn ==> viewing, view in
src/coins.cpp:24: viewIn ==> viewing, view in
src/coins.cpp:29: viewIn ==> viewing, view in
src/coins.cpp:29: viewIn ==> viewing, view in
src/coins.h:44: outIn ==> outing, out in
src/coins.h:44: outIn ==> outing, out in
src/coins.h:45: outIn ==> outing, out in
src/coins.h:45: outIn ==> outing, out in
src/coins.h:215: viewIn ==> viewing, view in
src/coins.h:220: viewIn ==> viewing, view in
src/primitives/transaction.h:37: hashIn ==> hashing, hash in
src/primitives/transaction.h:37: hashIn ==> hashing, hash in
src/protocol.cpp:51: hashIn ==> hashing, hash in
src/protocol.cpp:51: hashIn ==> hashing, hash in
src/protocol.h:497: hashIn ==> hashing, hash in
src/qt/forms/optionsdialog.ui:344: incomin ==> incoming
src/qt/optionsdialog.cpp:445: proxys ==> proxies
src/rpc/mining.cpp:987: hashIn ==> hashing, hash in
src/rpc/mining.cpp:987: hashIn ==> hashing, hash in
src/script/interpreter.h:298: amountIn ==> amounting, amount in
src/script/interpreter.h:298: amountIn ==> amounting, amount in
src/script/interpreter.h:299: amountIn ==> amounting, amount in
src/script/interpreter.h:299: amountIn ==> amounting, amount in
src/script/sigcache.h:70: amountIn ==> amounting, amount in
src/script/sigcache.h:70: amountIn ==> amounting, amount in
src/signet.cpp:144: amountIn ==> amounting, amount in
src/test/fuzz/util/net.cpp:386: occured ==> occurred
src/test/fuzz/util/net.cpp:398: occured ==> occurred
src/util/vecdeque.h:79: deques ==> dequeues
src/util/vecdeque.h:160: deques ==> dequeues
src/util/vecdeque.h:184: deques ==> dequeues
src/util/vecdeque.h:194: deques ==> dequeues
src/validation.cpp:2130: re-declared ==> redeclared
src/validation.h:348: outIn ==> outing, out in
src/validation.h:349: outIn ==> outing, out in
test/functional/wallet_bumpfee.py:851: atleast ==> at least
```
2024-07-22 13:59:42 +02:00
TheCharlatan
fae0db0360
fuzz: Deglobalize signature cache in sigcache test
The body of the fuzz test should ideally be a pure function. If data is
persisted in the cache over many iterations, and there is a crash,
reproducing it from the input might be difficult.
2024-07-19 17:17:02 +02:00
MarcoFalke
fa80b16b20
fuzz: Limit parse_univalue input length 2024-07-19 15:39:02 +02:00
TheCharlatan
f46b220256
fuzz: Use BasicTestingSetup for coins_view target 2024-07-19 13:37:35 +02:00
TheCharlatan
9e2a723d5d
test: Add arguments for creating a slimmer setup
Adds more testing options for creating an environment without networking
and a validation interface. This is useful for improving the performance
of the utxo snapshot fuzz test, which constructs a new TestingSetup on
each iteration.
2024-07-19 13:37:31 +02:00
merge-script
1db0be8353
Merge bitcoin/bitcoin#28263: Add fuzz test for FSChaCha20Poly1305, AEADChacha20Poly1305
8607773750e60931e51a33e48cd077a1dedf9db3 Add fuzz test for FSChaCha20Poly1305 (stratospher)
c807f3322897ca8c0da114556e5936e389da5059 Add fuzz test for AEADChacha20Poly1305 (stratospher)

Pull request description:

  This PR adds fuzz tests for `AEADChaCha20Poly1305` and `FSChaCha20Poly1305` introduced in #28008.

  Run using:
  ```
  $ FUZZ=crypto_aeadchacha20poly1305 src/test/fuzz/fuzz
  $ FUZZ=crypto_fschacha20poly1305 src/test/fuzz/fuzz
  ```

ACKs for top commit:
  dergoegge:
    tACK 8607773750e60931e51a33e48cd077a1dedf9db3
  marcofleon:
    Tested ACK 8607773750e60931e51a33e48cd077a1dedf9db3. Ran both targets for ~200 CPU hours. Coverage of intended targets looks good to me. The simulation of damaged keys and checks that follow seem useful as well.

Tree-SHA512: b6b85661d896e653caeed330f941fde665fc2bbd97ecd340808a3f365c469fe9134aa77316569a771dc36d1158cac1a5f76700bcfc45fff12aef07562e48feb9
2024-07-16 12:13:02 +01:00
merge-script
ff827a8f46
Merge bitcoin/bitcoin#30407: test: [refactor] Pass TestOpts
fa690c8e532672f7ab53be6f7a0bb3070858745e test: [refactor] Pass TestOpts (MarcoFalke)

Pull request description:

  Currently optional test context setup settings are passed by adding a new optional argument to the constructors. For example `extra_args`. This is problematic, because:

  * Adding more optional settings in the future requires touching all affected constructors, increasing their verbosity.
  * Setting only a later option requires setting the earlier ones.
  * Clang-tidy named args passed to `std::make_unique` are not checked.

  Fix all issues by adding a new struct `TestOpts`, which holds all options. Notes:

  * The chain type is not an option in the struct for now, because the default values vary.
  * The struct holds all possible test options globally. Not all fields may be used by all constructors. Albeit harmless, it is up to the test author to not set a field that is unused.

ACKs for top commit:
  kevkevinpal:
    utACK [fa690c8](fa690c8e53)
  dergoegge:
    utACK fa690c8e532672f7ab53be6f7a0bb3070858745e
  TheCharlatan:
    Nice, ACK fa690c8e532672f7ab53be6f7a0bb3070858745e

Tree-SHA512: 8db8efa5dff854a73757d3f454f8f902e41bb4358f5f9bae29dbb3e251e20ee93489605de51d0822ba31d97835cd15526a29c075278dd6a8bbde26134feb4f49
2024-07-15 17:21:55 +01:00
merge-script
262260ce1e
Merge bitcoin/bitcoin#30197: fuzz: bound some miniscript operations to avoid fuzz timeouts
bc34bc288824978ef4b98e8802b47cb863c8a8c2 fuzz: limit the number of nested wrappers in descriptors (Antoine Poinsot)
8d7340105f5299e9a45e84f1704b8b4545cb85f0 fuzz: limit the number of sub-fragments per fragment for descriptors (Antoine Poinsot)

Pull request description:

  Some of the logic in the miniscript module is quadratic. It only becomes an issue for very large uninteresting descriptors (like a `thresh` with 130k sub-fragments or a fragment with more than 60k nested `j:` wrappers).

  This PR fixes the two types of fuzz timeouts reported by Marco in https://github.com/bitcoin/bitcoin/issues/28812 by trying to pinpoint the problematic descriptors through a simple analysis of the string, without limiting the size of the string itself. This is the same approach as was adopted for limiting the depth of derivation paths.

ACKs for top commit:
  dergoegge:
    utACK bc34bc288824978ef4b98e8802b47cb863c8a8c2
  stickies-v:
    Light ACK bc34bc288824978ef4b98e8802b47cb863c8a8c2
  marcofleon:
    Code review ACK bc34bc288824978ef4b98e8802b47cb863c8a8c2. The added comments are useful, thanks for those. Tested on the three inputs in https://github.com/bitcoin/bitcoin/issues/28812 that caused the timeouts.

Tree-SHA512: 8811c7b225684c5ecc1eb1256cf39dfa60d4518161e70210086c8a01b38927481ebe747af86aa5f4803187672d43fadabcfdfbf4e3b049738d629a25143f0e77
2024-07-15 14:11:14 +01:00
stratospher
8607773750 Add fuzz test for FSChaCha20Poly1305 2024-07-15 18:26:45 +05:30
stratospher
c807f33228 Add fuzz test for AEADChacha20Poly1305 2024-07-15 18:25:59 +05:30
merge-script
01ed4927f0
Merge bitcoin/bitcoin#30412: MiniMiner: use FeeFrac in AncestorFeerateComparator
09370529fb9f6d06f6d16bdb1fb336f7a265d8ba fuzz: mini_miner_selection fixups. (glozow)
de273d53004f48e4c8c965f7ce0bd375fd8d0d69 MiniMiner: use FeeFrac in AncestorFeerateComparator (glozow)

Pull request description:

  Closes #30284. Closes #30367, see https://github.com/bitcoin/bitcoin/issues/30367#issuecomment-2217459257

  Previously, we were only comparing feerates up to 1/1000 precision, since CFeeRate comparison just looks at their respective nSatoshisPerK. This could lead to MiniMiner selecting packages in the wrong order (i.e. by txid) if their feerates were less than 0.001sat/vB different. Fix this by creating + comparing `FeeFrac`s instead.

  Also, `FeeFrac::Mul` doesn't have the overflow problem.

  Also added a few minor fuzzer fixups that caught my eye while I was debugging this.

ACKs for top commit:
  ismaelsadeeq:
    Tested ACK 09370529fb9f6d06f6d16bdb1fb336f7a265d8ba
  murchandamus:
    ACK 09370529fb9f6d06f6d16bdb1fb336f7a265d8ba with nits
  dergoegge:
    tACK 09370529fb9f6d06f6d16bdb1fb336f7a265d8ba

Tree-SHA512: e5b6d6c3f7289f30cd8280d0a47cd852d0180b83d1b27ff9514f50c97103b0f069484e48cba2ca3a57419beadc1996c1b9dd8d0a0f34bc4f4223d8adaf414ce5
2024-07-15 09:59:44 +01:00