BUIP010 (passed): Xtreme Thinblocks

jtoomim

Active Member
Jan 2, 2016
130
253
I think we may already be faster or close to what Matt's relay network is doing for the miners.
Very unlikely. First, the relay network should have smaller data packages, since it uses two bytes per cached transaction instead of six. Second, the relay network does not require bitcoind to validate the block in between each hop. Since validating a 1 MB block can take up to 1 second, and since it typically takes 6 "bitcoind hops" to traverse the p2p network, that's kind of a big deal. Third, the relay network has an optimized topology, and will only send each block at most halfway around the world, whereas the topology of the bitcoin p2p network is random and can result in the data traversing the globe several times over the course of the 6 hops.
[doublepost=1456534062][/doublepost]
Thanks for the info...
We've coded it to handle any collisions even though they will likely never happen.
^by mistake.
 

Peter R

Well-Known Member
Aug 28, 2015
1,398
5,595
"Second, the relay network does not require bitcoind to validate the block in between each hop. Since validating a 1 MB block can take up to 1 second"

BU also uses @Peter Tschipper's Xpress validation, so only the transactions that weren't already in the receiving node's mempool are validated. For a block sent at 50x compression, only ~20 kB of transactional data needs to be validated.

Peter, do you have any measurements on block validation time for, say, a block that was sent with a 50x compression ratio?

"since it typically takes 6 "bitcoind hops" to traverse the p2p network, that's kind of a big deal."

Not that I'm disagreeing (6 sounds reasonable), but do you know where this number comes from? I'd be interested to see how it was measured.

"Third, the relay network has an optimized topology, and will only send each block at most halfway around the world, whereas the topology of the bitcoin p2p network is random and can result in the data traversing the globe several times over the course of the 6 hops."

Hmm...this is really interesting. Indeed, the paths the block takes are random, but there are also TONS of potential paths between Node A and Node F. The path that actually delivers the block from A to F will NOT be random but will instead typically be one of the best paths. For example, by the time a block has whizzed around the world 6 times, Node F has probably already received the block in question over a more direct path. This would be an interesting problem to study in more depth.
 
  • Like
Reactions: bluemoon and solex

jtoomim

Active Member
Jan 2, 2016
130
253
> Not that I'm disagreeing (6 sounds reasonable), but do you know where this number comes from? I'd be interested to see how it was measured.

Not measured, unfortunately. Estimated from first principles.

Assume there are 30,000 nodes on the network, of which only 6,000 have open ports. The network is a graph with cycles; redraw the graph as an acyclic tree with only the shortest path to the root node represented. Assume that when you do this, each node has connections to 8 peers below it in the tree. With these assumptions, the 6th level of this tree would have 32768 peers.

It's not a very good estimation, since it doesn't take into account the differences in latency between different nodes and does a lot of hand-waving about the number of non-duplicate connections per node, but since the number of hops is O(log n) versus network size, it's probably not off by more than 2 hops.

(One of the design goals of blocktorrent is increasing the base for that logarithm from 8 to 100 or more in order to decrease the hop count.)
 
  • Like
Reactions: bluemoon and solex

Peter Tschipper

Active Member
Jan 8, 2016
254
357
"Second, the relay network does not require bitcoind to validate the block in between each hop. Since validating a 1 MB block can take up to 1 second"


Peter, do you have any measurements on block validation time for, say, a block that was sent with a 50x compression ratio?

Hmm...this is really interesting. Indeed, the paths the block takes are random, but there are also TONS of potential paths between Node A and Node F. The path that actually delivers the block from A to F will NOT be random but will instead typically be one of the best paths. For example, by the time a block has whizzed around the world 6 times, Node F has probably already received the block in question over a more direct path. This would be an interesting problem to study in more depth.
On those two points....

1) Validation is extremely fast for an xthinblock...on my i7 laptop about .06-08 seconds to validate a full block and update the UTXO. But it varies, one of our issues right now is mempool sync during these spam attacks so we can end up having to validate more tx's than would be normal. More work is being planned to better handle those conditions. But keep in mind, we're not trying to replicate the relay network just yet...this is just for p2p. We have more optimizing to do as we've talked about over the last weeks. More to come.

2) As far as paths are concerned, @jtoomim may not be familiar with the connect-thinblock functionality. It essentially allows someone, if they wanted to, to create their own relay network and have direct pathways so to speak. Miners can even directly connect to each other sharing xthinblocks if they wanted to...we could easily allow miners to have as many outbound connections as they wanted to and just bypass the need for a middle tier relay server. Why go through 6 hops when you can just do it in 1?
 

solex

Moderator
Staff member
Aug 22, 2015
1,558
4,693
@jtoomim
Jonathan, please tell us what you are thinking of next, what your priority is now?

We know the mining take-up of Classic is slower than you expected, however it is happening and we are keen to accelerate the process.

There must be some miners who do not bother with the RN at all. If so, then maybe there are a number using the RN for which something else "near enough" would be an OK alternative, where they are not desperate to squeeze out every last saving in bandwidth.

Would it help if there was a BU relay network using Xtreme thinblocks as per @Peter Tschipper's comment above where miners can run a BU node for block propagation purposes. Would you be prepared to explain this idea to your mining contacts? The benefit is that they will control the block limit between them instead of being constrained by the owner of the RN.
 

jtoomim

Active Member
Jan 2, 2016
130
253
My priority right now is actually developing blocktorrent. I've been making a lot of progress on it over the last week and I'd like to keep that momentum up.
 

theZerg

Moderator
Staff member
Aug 28, 2015
1,012
2,327
Wrt peter r's observation... I am working on allowing nodes to keep careful track of their peers performance and to optimize sending to high performance peers. I hope to ultimately allow a high performance forwarding network emerge from normal operations in the p2p network.
 

Peter R

Well-Known Member
Aug 28, 2015
1,398
5,595
Hey guys,

I was thinking that it would be interesting to set up an Unlimited node in China. Maybe we could measure Xthin block propagation through the GFC.

Here are two VPS options:

http://www.chinavpshosting.com/cloud-server

http://www.sinohosting.net/china-vps.php

Since I wouldn't be running a website, I would NOT need a ICS license from the Chinese gov!

Some tech questions for you guys:

1. Is a particular flavour of Linux preferred (Debian, Ubuntu)?
2. Is there any reason I'd want cPanel access for the VPS (it costs extra)?
3. Can anyone think of a reason why this idea wouldn't work?
 
Last edited:

solex

Moderator
Staff member
Aug 22, 2015
1,558
4,693
@Peter R
Nice initiative. A btc donation address might attract funding to keep this going awhile. I would put some in for this purpose.

Having two, one on each provider, gives better latency statistics (internally, and comparison for traffic through the GFW).

Sorry, can't advise on your 1,2,3.
 
Last edited:
  • Like
Reactions: Peter R

Peter R

Well-Known Member
Aug 28, 2015
1,398
5,595
Thanks everyone for the answers!
@Peter R
Nice initiative. A btc donation address might attract funding to keep this going awhile. I would put some in for this purpose.

Having two, one on each provider, gives better latency statistics (internally, and comparison for traffic through the GFW).
If people agree it would be useful to keep some Chinese nodes running, then a donation address would be great!

I actually have 0.836BTC of public funds, and I've been looking for a "big block" cause to donate to, and I thought setting up a Chinese BU node might be a good idea. If someone else is itching to do this too and just wanted some funding, I'd actually rather donate this money to "Bitcoin Unlimited" for the purposes of experimenting with Xthin propagation through the GFC. Let me know.

@jtoomim: Thanks for the info. I'm leaning towards Sino Hosting, only because I can understand their website. They have data centers in Shanghai, Beijing, Hangzhou and Shenzhen. Any reason you can think of why this wouldn't work? Also, were you paying prices like this? I guess I'd want the "Advanced" setup for $97US/month.

 

sickpig

Active Member
Aug 28, 2015
926
2,541
@Peter R

wrt xthin chinese node: it'd be a pleasure for me to help setting up and keeping it up.

that said it would be ideal to have a group of people from different time zones in such a way of being able to provide assistance for the entire 24hs time span.
 

theZerg

Moderator
Staff member
Aug 28, 2015
1,012
2,327
Lol I just pmed the BU officers suggesting we start a donation drive for this... then see you're already discussing it!

Operationally it will be nice to do this to practice the BU $ handling since it'll only take 1+ BTC. I think that we should open a fund drive tho to see if there are other interested donors. Greater donations could fund a BU "supernode" network running ofc within the normal p2p net.
 

Peter R

Well-Known Member
Aug 28, 2015
1,398
5,595
OK! This sounds great!!

@sickpig: since you're interested in setting up and maintaining the node(s), you're probably a much better person to do this than I am.

@theZerg: I agree that we should do this to practice the BU handling of the funds. If you set up a multisig donation address, I will donate the 0.836 BTC that I referred to earlier.

Here is a first draft at a project plan / announcement.

Improving Block Propagation through the Great Firewall of China with X-technology
A Bitcoin Unlimited Initiative

TL/DR: the goal of the project is to setup Bitcoin Unlimited nodes including within mainland China and exploit Xthin to create a fast block relay through the Great Firewall. If you would like to contribute to the hosting cost of these nodes, please consider donating to the project's wallet at <3xxxxxxxx>.

The release of Xtreme Thin Blocks and Xpress Validation in Bitcoin Unlimited (BU) 0.12 has provided new opportunities to significantly improve the propagation of blocks across Bitcoin's peer-to-peer network. @Peter Tschipper's X-technology leverages the fact that most transactions included in a solved block are already known by most network nodes (common mempools). Rather than requesting the solved block verbatim, the receiving node images its mempool onto a Bloom filter that its sends with its get-data request. The transmitting node sends the block contents by hash for all the transactions imaged onto the Bloom filter, and in full otherwise. Empirical testing shows that this technique reduces the total number of bytes transmitted by a factor of 15X under normal operating conditions.

Nowhere is the efficient propagation of blocks more important than across the Great Firewall of China (GFC). For example, experiments in the fall of 2015 by @jtoomim showed that although 4MB blocks could often be communicated and verified in 7 seconds (per hop), when the block had to penetrate the GFC, this figure increased to between 15 to 150 seconds. Indeed, the difficulty in propagating blocks through the GFC is arguably one of the most significant bottlenecks to on-chain scaling.

The goal of this project is to setup and maintain a small network of BU nodes across the world, with a particular focus on setting up nodes inside and outside of mainland China, and to perform experiments to quantify the improvements that X-technology permits. We suspect that the average amount of time required to communicate each megabyte of block information can be dramatically reduced. Should the results be positive, this small network of BU nodes could grow into an "Xtreme Relay Network," eventually replacing the centralized and less-robust Fast Relay Network, and permitting an order-of-magntitude or greater increase in Bitcoin's transactional throughput.

If you would like to contribute to this initiative, please consider donating to the project's wallet at <3xxxxxx>. All funds are administered by ______, in accordance with Article 3 of Bitcoin Unlimited's Articles of Federation.
 
Last edited:

freetrader

Moderator
Staff member
Dec 16, 2015
2,806
6,088
@Peter R : would you consider changing the title from "Tunnelling" to something like "Improving propagation"?

If someone to do with the GFW gets the wrong impression what this is about, this experiment could meet an unforeseen untimely end...

P.S. technically it's not tunnelling either what we're doing.
 
  • Like
Reactions: satoshis_sockpuppet

theZerg

Moderator
Staff member
Aug 28, 2015
1,012
2,327
compression is 15X.

I think we should say "The goal of this project is to setup and maintain a small network of BU nodes across the world, with a particular focus setting up nodes inside and outside of mainland China, and perform experiments to quantify the improvements that X-technology permits."

That way we can use the money to create a worldwide network, massively reducing block transmission times to everyone. And ofc, we may someday need to pay for outside nodes...
 
  • Like
Reactions: Peter R

solex

Moderator
Staff member
Aug 22, 2015
1,558
4,693
@Peter R
For info. We should have a 2-of-3 address for BU project funds shortly.