Tests 5 scenarios for transaction download scheduling:
1. Whether eventually, after a series of timeouts, all our peers
that announced a transaction are sent a getdata request
2. Whether outbound peers are prioritized over inbound peers when
a getdata request takes longer than optimal
3. That we honor the maximum in-flight capacity, that this is on
a per-peer basis and that it resets itself after timeout
4. That when we have an inflight getdata request when a peer
disconnects, we recover after the initial 30 second timeout
and fetch the transaction from another peer
5. That we recover after a peer sends us a notfound message for
a tx we had an inflight getdata request for.
- Lower the timeout until we will request a transaction from an
additional peer on top of the one we received an inv from, from
1 minute to 30 seconds. Waiting a full minute will often mean
that the transaction will skip 2 blocks (one that is currently
templated, and the one that we'd make a minute after that.
This allows the transaction to at least have a chance to be
included in the next block template.
- (Inferred) lower the timeout between requesting and receiving a
transaction from 10 minutes to 5 minutes. We keep the multiplier
from Bitcoin Core of 10x the re-request timeout, to allow faster
cleanup of stale requests when 2 peers have a slow connection.
- Change the frequency of checking timeouts from once every 10
minutes, to once every 90 seconds, on average. We randomize this
check to be performed every 60-120 seconds. This allows the node
to be triggered into cleanup more frequently and with that also
helps towards the goal of faster cleanup of stale requests.
Changes all timing around when we send inv messages to our peers to
be mockable with setmocktime (on regtest.)
This is needed for testing transaction scheduling in time that is
faster-than-real.
- Add an explicit memory bound to m_tx_process_time
Previously there was an implicit bound based on the handling
of m_tx_announced, but that approach is error-prone
(particularly if we start automatically removing things from
that set).
- Remove NOTFOUND transactions from in-flight data structures
This prevents a bug where the in-flight queue for our peers
will not be drained, resulting in not downloading any new
transactions from our peers.
- Expire old entries from the in-flight tx map
If a peer hasn't responded to a getdata request, eventually
time out the request and remove it from the in-flight data
structures. This is to prevent any bugs in our handling of
those in-flight data structures from filling up the
in-flight map and preventing us from requesting more
transactions (such as the NOTFOUND bug mentioned above).
- Fix bug around transaction requests
If a transaction is already in-flight when a peer announces
a new tx to us, we schedule a time in the future to
reconsider whether to download. At that future time, there
was a bug that would prevent transactions from being
rescheduled for potential download again (ie if the
transaction was still in-flight at the time of
reconsideration, such as from some other peer). This fixes
that bug.
- Improve NOTFOUND comment
Cherry-picked from: 218697b6, 23163b75, e32e0840, f635a3ba
and 308b7673
Co-authored-by: Anthony Towns <aj@erisian.com.au>
While working with other networking code, Patrick and I noticed that we
use signed types and wrongly-sized types for several networking related
values, including data lengths, timeout durations, and port numbers.
This commit corrects several of these types and improves error handling
slightly to account for potentially invalid values.
- Removes hard coded GZIP_ENV because this is now handled properly
by gitian.
- Removes target '.mm.o' because that was needed for XCode <= 4.2
Cherry-picked from: f8c6697
Omitted all unmaintained dialects unless there only is a dialect.
Excluded files:
- bg_BG -> bg
- ca@valencia and ca_ES -> ca
- el_GR -> el
- es_AR, es_CL, es_CO, es_DO, es_ES, es_MX, es_UY and es_VE -> es
- et_EE -> et
- fa_IR -> fa
- fr_CA and fr_FR -> fr
- ro_RO -> ro
- ru_RU -> ru
- tr_TR -> tr
- vi_VN -> vi
- zh and zh_HK -> zh_CN
While limitations on the influence of attackers on addrman already
exist (affected buckets are restricted to a subset based on incoming
IP / network group), there is no reason to permit them to let them
feed us addresses at more than a multiple of the normal network
rate.
This commit introduces a "token bucket" rate limiter for the
processing of addresses in incoming ADDR messages. Every connection
gets an associated token bucket. Processing an address in an ADDR
message from non-whitelisted peers consumes a token from the bucket.
If the bucket is empty, the address is ignored (it is not processed,
stored or forwarded). The token counter increases at a rate of 0.1
tokens per second, and will accrue up to a maximum of 1000 tokens
(the maximum we accept in a single ADDR message).
When a GETADDR is sent to a peer, it immediately gets 1000 additional
tokens, as we actively desire many addresses from such peers (this
may temporarily cause the token count to exceed 1000). This is
similar to allowing bursts.
The rate limit of 0.1 addr/s was copied from Bitcoin Core.
Backported from: 0d64b8f7, 5648138f, f424d601 and d930c7f5
Original authors: Pieter Wuille <pieter@wuille.net>, and
Jon Atack <jon@atack.com>
Qt wallets display information to users about the broadcast state
of transactions, without truly knowing the actual state. The fact
that a peer has requested details for a transaction does not mean
it was accepted to their mempool or relayed to any other peers and
miners, because the only way to test if the transaction can be
accepted is by requesting and processing it.
We remove the "offline" and "maturity warning" statuses, which
saves large wallets memory and processing time.
Backported from: beef7ec4
Original author: Matt Corallo <git@bluematt.me>
the p2p-policy test has a number of issues because it is a real-
time relay test that can at the moment not be mocked and it is
insufficiently hardened against signature length variation.
This makes 2 changes to the test to make it more reliable:
1. Increase the maximum wait time for transactions to be relayed to
2 minutes instead of 30 seconds. This gives us more certainty
that the PoissonNextSend() function doesn't schedule outside of
our window.
2. Whenever we sign a transaction with an unexpected signature
length, retry constructing the transaction (with a different
input). Fees are changed to be 100% exact.
Note that this makes the test potentially take a longer time to
complete, so we move it up in the list of the test runner, to
be triggered early.
