- Jan 8, 2016
- 254
- 357
BUIP022: XInv
Proposer: Peter Tschipper
Submitted on: 07/30/2016
Summary
There are still a few areas where bandwidth savings can be significant and one of the largest consumers of bandwidth in the Bitcoin application happens during the sharing of inventory. Every time a transaction or block is received by a node it has to share that knowledge with every other peer it's connected with. Consider that a node with 125 connections generally has to send out 125 inventory messages (minus any that were received from already) for every transaction, resulting in a considerable use of outgoing bandwidth during each day. By using a similar approach to compressing Xthin's we can also compress inventory messages by 75% and by using other techniques of concatenation, savings can be 90% or higher depending on transaction rates.
From here on we'll refer to inventory compression as XInv.
Design
Potential Issues
Hash Collisions: We know from Xthins that hash collisions in the memory pool using 64 bit transactions hashes just don't happen today and will likely never happen in the future. But what would happen if there were one? In such a case the original, or first transaction, would propagate through the p2p network just as it normally does. The second transaction however would not propagate right away because any node receiving the XInv message would think it already has that transaction and wouldn't issue a GETDATA to retrieve it. However once the first transaction were mined and subsequently removed from the memory pool the second transaction could then propagate through the network. This scenario would be an extremely rare, if ever, event with an end result of a delay of propagation
Attack Scenarios: Could someone attack by causing a hash collision? The attacker would need to get the transaction hash of any new transaction and then attempt to create another transaction with the same first 64 bit hash. But by the time they do so the original transaction would have propagated either entirely through the network or would certainly be far ahead. The attackers transaction would then be in waiting until the original transaction were mined and cleared from the memory pool. There's not much of an attack possible here but it would be good to hear feedback from others in our group as to what they think about this or other potential scenarios.
Backward Compatibility
XInv will be backwardly compatible with older clients in that any client not supporting the XINV service bit will still receive inventory messages in the old format and the XINV client will still be able to process inventory messages form older nodes.
Fall Back
XInv can be entirely and easily turned off with a simple bitcoin.conf setting.
Deployment
This enhancement does not require a hard or soft fork.
Copyright
This document is placed in the public domain.
Proposer: Peter Tschipper
Submitted on: 07/30/2016
Summary
There are still a few areas where bandwidth savings can be significant and one of the largest consumers of bandwidth in the Bitcoin application happens during the sharing of inventory. Every time a transaction or block is received by a node it has to share that knowledge with every other peer it's connected with. Consider that a node with 125 connections generally has to send out 125 inventory messages (minus any that were received from already) for every transaction, resulting in a considerable use of outgoing bandwidth during each day. By using a similar approach to compressing Xthin's we can also compress inventory messages by 75% and by using other techniques of concatenation, savings can be 90% or higher depending on transaction rates.
From here on we'll refer to inventory compression as XInv.
Design
- A simple design of using the first 64 bits of the transaction hash can be used rather than the full transaction hash. We know now by experience from Xthins' that 64 bits is a safe size to use and prevents any mempool hash collisions either now or in the future. We could go all the way to 32 bits however that would be a much more complicated design and problematic is several ways.
- Keeping in mind that an inventory message today is 36 bytes in size. By going to 64 bit hashes the inventory message becomes just 12 bytes. With part of that 12 bytes being taken up by the 4 byte enumerated message type. There really isn't any need for a 4 byte message type; one byte would suffice and give us all the room we need for future growth and by doing so our inventory message becomes 9 bytes or just 25% of the original size.
- Other savings involves the concatenation of inventory. Most inventory messages are sent out one a time. That's an enormous loss of bandwidth when you consider that each message has a 24 byte message header, a 20 byte IP header and requires a TCP ACK of at least 50 bytes. So by concatenating inventory together before sending, we can increase our bandwidth savings significantly. And the higher the transaction rate are the more we can concatenate. (This concatenation does happen to some extent today however it's not consistent and most inventory messages still go out one at a time).
- As with Xthin's, XInv will use a service bit which will easily allow other projects to adopt and incorporate the technology. As a safeguard a service bit also give's us the benefit of being able to completely turn off XInv if needed or wanted.
Potential Issues
Hash Collisions: We know from Xthins that hash collisions in the memory pool using 64 bit transactions hashes just don't happen today and will likely never happen in the future. But what would happen if there were one? In such a case the original, or first transaction, would propagate through the p2p network just as it normally does. The second transaction however would not propagate right away because any node receiving the XInv message would think it already has that transaction and wouldn't issue a GETDATA to retrieve it. However once the first transaction were mined and subsequently removed from the memory pool the second transaction could then propagate through the network. This scenario would be an extremely rare, if ever, event with an end result of a delay of propagation
Attack Scenarios: Could someone attack by causing a hash collision? The attacker would need to get the transaction hash of any new transaction and then attempt to create another transaction with the same first 64 bit hash. But by the time they do so the original transaction would have propagated either entirely through the network or would certainly be far ahead. The attackers transaction would then be in waiting until the original transaction were mined and cleared from the memory pool. There's not much of an attack possible here but it would be good to hear feedback from others in our group as to what they think about this or other potential scenarios.
Backward Compatibility
XInv will be backwardly compatible with older clients in that any client not supporting the XINV service bit will still receive inventory messages in the old format and the XINV client will still be able to process inventory messages form older nodes.
Fall Back
XInv can be entirely and easily turned off with a simple bitcoin.conf setting.
Deployment
This enhancement does not require a hard or soft fork.
Copyright
This document is placed in the public domain.
