The major changes are:
* Announcements from outbound (and whitelisted) peers are now
always preferred over those from inbound peers. This used to be
the case for the first request (by delaying the first request
from inbound peers), and a bias after. The 2s delay for requests
from inbound peers still exists, but after that, if viable
outbound peers remain for any given transaction, they will
always be tried first.
* No more hard cap of 100 in flight transactions per peer, as
there is less need for it (memory usage is linear in the number
of announcements, but independent from the number in flight,
and CPU usage isn't affected by it). Furthermore, if only one
peer announces a transaction, and it has over 100 in flight and
requestable already, we still want to request it from them. The
cap is replaced with an additional 2s delay (possibly combined
with the existing 2s delays for inbound connections).
Backported from: 242d1647
173a1d2d
Original Author: Pieter Wuille <pieter@wuille.net>
Conflicts:
- replaced GenTx with uint256 because no segwit
- removed additional 2s penalty for non-segwit peers
- used int64_t instead of std::chrono::microseconds per utiltime
- implemented TxRequest as g_txrequest instead of as a member of
PeerManager, which we don't have
- removed the Dogecoin-specific strict max inflight test
- make exceptions for whitelisted nodes as there is no fine-
grained permission system
This makes orphan processing work like handling getdata messages:
After every actual transaction validation attempt, interrupt
processing to deal with messages arriving from other peers.
Cherry-picked from: 866c8058a
Reduces complexity in processing of orphans when receiving tx
messages over p2p. and isolate the functionality in its own
function ProcessOrphanTx
Backported from: 9453018f
6e051f3d
Original Author: Pieter Wuille <pieter.wuille@gmail.com>
Headers-first is the primary method of announcement on the network.
If a node fell back sending blocks by inv, it's probably for a
re-org. The final block hash provided should be the highest, so
send a getheaders and then fetch the blocks we need to catch up.
Backported from: 74673663
Original Author: John Newbery <john@johnnewbery.com>
random_shuffle is deprecated since c++11 and removed in c++17. Use
the shuffle function instead, now that we have a standardized
interface available on FastRandomContext.
This enables c++17 compilation
adds a line when no copyright for Dogecoin Core Developers exists
but the file has been edited by us, to the last year found in git
log, or extends the year range on an existing line when a file
has been modified since the year previously listed.
Excludes subtrees.
On boost 1.81 std::array is no longer included by default and
causes a compile error on net_processing.cpp:
macOS with apple clang 14.0:
CXX libdogecoin_server_a-net_processing.o
net_processing.cpp:1112:47: error: implicit instantiation of undefined template 'std::array<std::pair<unsigned long long, CNode *>, 2>'
std::array<std::pair<uint64_t, CNode*>,2> best{{{0, nullptr}, {0, nullptr}}};
debian:exprimental with gcc 12.1:
net_processing.cpp: In function 'void RelayAddress(const CAddress&, bool, CConnman&)':
net_processing.cpp:1112:47: error: variable 'std::array<std::pair<long unsigned int, CNode*>, 2> best' has initializer but incomplete type
1112 | std::array<std::pair<uint64_t, CNode*>,2> best{{{0, nullptr}, {0, nullptr}}};
|
this commit fixes the problem by explicitly including <array>.
- Lower the timeout until we will request a transaction from an
additional peer on top of the one we received an inv from, from
1 minute to 30 seconds. Waiting a full minute will often mean
that the transaction will skip 2 blocks (one that is currently
templated, and the one that we'd make a minute after that.
This allows the transaction to at least have a chance to be
included in the next block template.
- (Inferred) lower the timeout between requesting and receiving a
transaction from 10 minutes to 5 minutes. We keep the multiplier
from Bitcoin Core of 10x the re-request timeout, to allow faster
cleanup of stale requests when 2 peers have a slow connection.
- Change the frequency of checking timeouts from once every 10
minutes, to once every 90 seconds, on average. We randomize this
check to be performed every 60-120 seconds. This allows the node
to be triggered into cleanup more frequently and with that also
helps towards the goal of faster cleanup of stale requests.
Changes all timing around when we send inv messages to our peers to
be mockable with setmocktime (on regtest.)
This is needed for testing transaction scheduling in time that is
faster-than-real.
- Add an explicit memory bound to m_tx_process_time
Previously there was an implicit bound based on the handling
of m_tx_announced, but that approach is error-prone
(particularly if we start automatically removing things from
that set).
- Remove NOTFOUND transactions from in-flight data structures
This prevents a bug where the in-flight queue for our peers
will not be drained, resulting in not downloading any new
transactions from our peers.
- Expire old entries from the in-flight tx map
If a peer hasn't responded to a getdata request, eventually
time out the request and remove it from the in-flight data
structures. This is to prevent any bugs in our handling of
those in-flight data structures from filling up the
in-flight map and preventing us from requesting more
transactions (such as the NOTFOUND bug mentioned above).
- Fix bug around transaction requests
If a transaction is already in-flight when a peer announces
a new tx to us, we schedule a time in the future to
reconsider whether to download. At that future time, there
was a bug that would prevent transactions from being
rescheduled for potential download again (ie if the
transaction was still in-flight at the time of
reconsideration, such as from some other peer). This fixes
that bug.
- Improve NOTFOUND comment
Cherry-picked from: 218697b6, 23163b75, e32e0840, f635a3ba
and 308b7673
Co-authored-by: Anthony Towns <aj@erisian.com.au>
While limitations on the influence of attackers on addrman already
exist (affected buckets are restricted to a subset based on incoming
IP / network group), there is no reason to permit them to let them
feed us addresses at more than a multiple of the normal network
rate.
This commit introduces a "token bucket" rate limiter for the
processing of addresses in incoming ADDR messages. Every connection
gets an associated token bucket. Processing an address in an ADDR
message from non-whitelisted peers consumes a token from the bucket.
If the bucket is empty, the address is ignored (it is not processed,
stored or forwarded). The token counter increases at a rate of 0.1
tokens per second, and will accrue up to a maximum of 1000 tokens
(the maximum we accept in a single ADDR message).
When a GETADDR is sent to a peer, it immediately gets 1000 additional
tokens, as we actively desire many addresses from such peers (this
may temporarily cause the token count to exceed 1000). This is
similar to allowing bursts.
The rate limit of 0.1 addr/s was copied from Bitcoin Core.
Backported from: 0d64b8f7, 5648138f, f424d601 and d930c7f5
Original authors: Pieter Wuille <pieter@wuille.net>, and
Jon Atack <jon@atack.com>
Qt wallets display information to users about the broadcast state
of transactions, without truly knowing the actual state. The fact
that a peer has requested details for a transaction does not mean
it was accepted to their mempool or relayed to any other peers and
miners, because the only way to test if the transaction can be
accepted is by requesting and processing it.
We remove the "offline" and "maturity warning" statuses, which
saves large wallets memory and processing time.
Backported from: beef7ec4
Original author: Matt Corallo <git@bluematt.me>
While looking at the network code, we realized this code is unused and
largely unmaintained. Given issue #2231, and its removal in 1.21, it's
time to remove it now.
This has the nice benefit of removing a single point of failure/control
(because it depends on a keypair to send alerts).
This code restores the pre-alert behavior before the merge of PR #1470.
Throughout the code, we use mockable time in places where we want
to test things that are subject to timing constraints, but don't
want to wait for great amounts of time when we use the regtest
network to ensure nothing gets broken. Mockable time is a system
time override that is only enabled on chains that have the
fMineBlocksOnDemand parameter set (currently: regtest).
Currently, only time expressed in seconds is able to be mocked, but
we have a couple of places where time is evaluated in microseconds,
one of them being the time between sending out addr messages.
This introduces a mockable time in microseconds and refactors the
time evaluation for addr messages to be mocked, so that we can
more reliably (and faster) test that.
Other protocol features may want to use this too, but currently the
sending of addr messages is the only place we test. Bitcoin Core
nowadays has a much better time system that we inherit in future
versions, but for the current scope I found this not worth the
effort of backporting, as these would impact a much larger part of
the code base.
Inspired by: 1a8f0d5a from Amiti Uttarwar
The largest sensible size for a locator is log in the number of blocks.
But, as noted by Coinr8d on BCT a maximum size message could encode a
hundred thousand locators. If height were used to limit the messages
that could open new attacks where peers on long low diff forks would
get disconnected and end up stuck.
Ideally, nodes first first learn to limit the size of locators they
send before limiting what would be processed, but common implementations
back off with an exponent of 2 and have an implicit limit of 2^32
blocks, so they already cannot produce locators over some size.
This sets the limit to an absurdly high amount of 101 in order to
maximize compatibility with existing software.
Cherry-picked from: e254ff5d
The timeout window for block downloads scales proportionally to the target
spacing for the chain, as set in chainparams.cpp. This causes issues on regtest
because the spacing is set to 1 second, allowing insufficient time for very
large blocks to sync when requested in batch, preventing success of the pruning
qa test.
We fix this by introducing a minimum multiplier (in seconds) that will be used
instead of the target block spacing whenever the latter is lower. With a value
of 10 seconds, pruning tests pass.
Introduces a counter for getheader requests that have been sent to
a peer but are pending response, reducing the number of parallel
requests a node pushes out to its peers when needing to sync large
amounts of headers. All getheader requests are serialized during
initial sync, except when a non-connecting header is received,
allowing the node to resolve issues with peers sending faulty
blocks using the DoS mechanism, and when we get an inv for a block
that we do not know, because it's possible we're only connected to
legacy nodes that do not implement header announcement properly.
Checking scrypt PoW is expensive and needless in this case. All block
headers are already checked when they are accepted, and they will be
checked again on the receiving end.
* Enable full block tests
* Fix invalidblocktest
* Move watch only address funding to immediately before it's used, so node 0 doesn't spend the output before it checks it later.
* Fix `fundrawtransaction` tests and sanitize fee calculation at the same time
* Correct resolution of chain parameters when validating tx inputs, especially from previous coinbase transactions
* Set block versions on full block tests so that the generated blocks are AuxPoW compatible
* Modify chain consensus parameters to be height aware
* Correct implementation of simplified rewards in parameters
* Correct max money
* Use base block version in IsSuperMajority() instead of full version
* Correct mining of blocks in AuxPoW tests
* Add in missing pre-AuxPoW consensus checks
Changes are as below:
Wrap CBlockHeader::nVersion into a new class (CBlockVersion). This allows to take care of interpreting the field into a base version, auxpow flag and the chain ID.
Update getauxblock.py for new 'generate' RPC call.
Add 'auxpow' to block JSON.
Accept auxpow as PoW verification.
Add unit tests for auxpow verification.
Add check for memory-layout of CBlockVersion.
Weaken auxpow chain ID checks for the testnet.
Allow Params() to overrule when to check the auxpow chain ID and for legacy blocks. Use this to disable the checks on testnet.
Introduce CPureBlockHeader.
Split the block header part that is used by auxpow and the "real" block header (that uses auxpow) to resolve the cyclic dependency between the two.
Differentiate between uint256 and arith_uint256.
This change was done upstream, modify the auxpow code.
Add missing lock in auxpow_tests.
Fix REST header check for auxpow headers.
Those can be longer, thus take that into account. Also perform the check actually on an auxpow header.
Correctly set the coinbase for getauxblock results.
Call IncrementExtraNonce in getauxblock so that the coinbase is actually initialised with the stuff it should be. (BIP30 block height and COINBASE_FLAGS.)
Implement getauxblock plus regression test.
Turn auxpow test into FIXTURE test.
This allows using of the Params() calls.
Move CMerkleTx code to auxpow.cpp.
Otherwise we get linker errors when building without wallet.
Fix rebase with BIP66.
Update the code to handle BIP66's nVersion=3.
Enforce that auxpow parent blocks have no auxpow block version.
This is for compatibility with namecoind. See also https://github.com/namecoin/namecoin/pull/199.
Move auxpow-related parameters to Consensus::Params.
d943491 qa: add a test to detect leaky p2p messages (Cory Fields)
8650bbb qa: Expose on-connection to mininode listeners (Matt Corallo)
5b5e4f8 qa: mininode learns when a socket connects, not its first action (Matt Corallo)
cbfc5a6 net: require a verack before responding to anything else (Cory Fields)
8502e7a net: parse reject earlier (Cory Fields)
c45b9fb net: correctly ban before the handshake is complete (Cory Fields)
7a8c251901 made this logic hard to follow. After that change, messages would
not be sent to a peer via SendMessages() before the handshake was complete, but
messages could still be sent as a response to an incoming message.
For example, if a peer had not yet sent a verack, we wouldn't notify it about
new blocks, but we would respond to a PING with a PONG.
This change makes the behavior straightforward: until we've received a verack,
never send any message other than version/verack/reject.
The behavior until a VERACK is received has always been undefined, this change
just tightens our policy.
This also makes testing much easier, because we can now connect but not send
version/verack, and anything sent to us is an error.
Prior to this change, all messages were ignored until a VERSION message was
received, as well as possibly incurring a ban score.
Since REJECT messages can be sent at any time (including as a response to a bad
VERSION message), make sure to always parse them.
Moving this parsing up keeps it from being caught in the
if (pfrom->nVersion == 0) check below.
7a8c251901 made a change to avoid getting into SendMessages() until the
version handshake (VERSION + VERACK) is complete. That was done to avoid
leaking out messages to nodes who could connect, but never bothered sending
us their version/verack.
Unfortunately, the ban tally and possible disconnect are done as part of
SendMessages(). So after 7a8c251901, if a peer managed to do something
bannable before completing the handshake (say send 100 non-version messages
before their version), they wouldn't actually end up getting
disconnected/banned. That's fixed here by checking the banscore as part of
ProcessMessages() in addition to SendMessages().
These are (afaik) all long-standing races or concurrent accesses. Going
forward, we can clean these up so that they're not all individual atomic
accesses.
- Reintroduce cs_vRecv to guard receive-specific vars
- Lock vRecv/vSend for CNodeStats
- Make some vars atomic.
- Only set the connection time in CNode's constructor so that it doesn't change
0729102 Net: pass interruptMsgProc as const where possible (Jorge Timón)
fc7f2ff Net: Make CNetMsgMaker more const (Jorge Timón)
d45955f Net: CConnman: Make some methods const (Jorge Timón)
This avoids having some vars set if the version negotiation fails.
Also copy it all into CNode at the same site. nVersion and
fSuccessfullyConnected are set last, as they are the gates for the other vars.
Make them atomic for that reason.
