eXpedited Discussion

[theZerg]

https://github.com/gandrewstone/BitcoinUnlimited/tree/XpeditedBlocks

 

[theZerg]

@Peter Tschipper, @theZerg,



Thanks for the feedback. I have been looking through to code on github, trying to understand

the details of how Expedited blocks works.



I still have a few questions on exactly how it currently works. For example, when the expedited thin block is constructed, does it include any full transactions?

Yes but which txns to include is an interesting optimization problem

Or does it just send a thin block without any full transactions assuming mempools are synchronized, and rely on a round trip communication if the recipient is missing transactions? Is there somewhere I can look to find more information, documentation, or a thread on the forum or something where this is discussed?

If transactions are missing, it does re-request them. However, this re-request is only currently capable of going a single hop directly to the requester. I would prefer if the re-request was integrated into the new "request manager" so it could request the data from multiple sources.

I have also been reading up on Matt Corallo's Relay Network. The more I research and think about it, the more excited I get. I think that building on the framework you guys have created, there is a good incremental path to building a peer-to-peer block relaying capability with performance better than Corallo's Relay Network.

yes I think so.

The way I envision it working is with miners manually configuring their nodes to connect to the nodes they want to. Most miners would likely connect directly to each other, with perhaps some dedicated relay nodes also. So most of the time blocks would get transmitted between miners in one hop.

Yes, but I also have in my mental "roadmap" a mode where nodes automatically choose other nodes to request expedited service. This would create a fractal overlay minimum-spanning tree-like network on top of the current P2P.

Speeding up block transmission by pre-communicating information before a block is solved can be implemented incrementally in stages:


Stage 1: Assume mempools are synchronized and send expedited thin block without any full transactions. Missing transactions would have to be requested in a round-trip communication. The miner can at least start head-first mining while the missing transactions are fetched.


Stage 2: Use some sort of heuristic to guess at what transactions the recipient has so they can be included along with the thin block. The code would keep track, for each connection, of which transactions it thinks its peers have. For example, it could keep track of INVs like @Dusty suggested:


Stage 3: Nodes explicitly communicate with each other which transactions they have. This could build upon the Bloom filter method from Xthin. The primary difference is that it would do this communication before blocks are found, not after. Then when it sends the thin block, it can include transactions that the other node likely doesn't have.



Stage 4: Nodes pre-transmits transactions that it thinks its peer is missing that are in the block it is working on before the block is found. Then when the block is found, it can transmit a thin block without needing to add any missing transactions.



Stage 5: Nodes communicate a data structure that refers to a set of transactions that it expects to include in a block. This way, when it constructs the thin block, instead of including all the transactions hashes, most of them can be replaced with a single hash that refers to the pre-communicated set of transactions. (If combined with weak blocks, this becomes very conceptually similar to @Peter R's subchains)

Exactly!

A network of automatically chosen expedited forwarding would sort of look like a "sticky pixel" fractal (just look at the pics): https://kar.kent.ac.uk/13627/1/sticky_pixels_evolutionary_roberts.pdf


Look at the newly checked in code... the older checkin was missing a lot.

Look at
void SendExpeditedBlock(CXThinBlock& thinBlock,unsigned char hops,const CNode* skip)

and CRequestManager

 

[Mengerian]

Look at the newly checked in code... the older checkin was missing a lot.

Look at
void SendExpeditedBlock(CXThinBlock& thinBlock,unsigned char hops,const CNode* skip)

and CRequestManager

OK, cool, thanks. Going to check it out.