Xtreme Thinblocks ready for code review PR#10

[theZerg]

I don't understand. I can use addnode=<IP> to explicitly force a connection to any node (thin block or not). That is what I am using to connect, so I do get connections to thin block compatible nodes. Does connect-thinblock do something different?

And WRT preferring thinblock nodes I am not suggesting starving normal nodes. Nodes are iterated through in an arbitrary order right now... I am just suggesting that we iterate through them in the order where thinblock nodes go first.

 

[Peter Tschipper]

connect-thinblock does something differently yes...it will always wait for an inv to show up from a thinblock node before doing a getdata so that you will always get a thinblock. But, if you use addnode then you'll get connected but not necessarily get a thinblock everytime, it depends on which node sends the inv for the block first. Nothing wrong with that either so that , as you're saying, you don't starve the network...that's a actually a good idea. I really had made the connect-thinblock functionality to help with testing but then realized that some people will always want thinblocks (raspberry pi users for instance), but nothing wrong with also using addnode.

OK, I think what I hear you saying is you want to iterate through vNodes in sorted order with thinblock nodes first, so that we would potentially get their inv first? It's possible, but I don't know that we'd get a big benefit there...thinblocks will propagate much faster, I believe, and we should get those inv's way ahead of any other blocks anyway, but hmmm...I don't know how much the sorted order will benefit us...it's late, i'll sleep on it and look at it tomorrow.

 

[Peter Tschipper]

@theZerg I'm a little sleepy so not sure if this is going to make sense, but in instead of using connect-thinblock which requires a config setting, what about using a timer such that when a block inv first arrives for a regular block we start the timer...if we have thinblock nodes connected to us and if we don't receive an inv from a thinblock node in lets say 5 seconds after the first inv for a regular block, then we just do a getdata for the regular block...that way, we don't need the config setting (although we can still leave connect-thinblock there as an option) and so there is a way for a block to get into the thinblock network of nodes while at the same time 100% preferring thinblock nodes over regular nodes. And once it's in the network it will propagate rapidly among the nodes without having to wait for the timeout?

 

[sickpig]

so after a running session of a few hours with this settings:

debug=thin
connect-thinblock=66.222.25.72
connect-thinblock=104.197.99.28
connect-thinblock=96.54.102.88

this is my debug.log: http://pastebin.com/TcBZHTKN

stat about my node as a thinblock provider:

thin size (a) block size (b) ratio (b)/(a)
61565 553841 9.0
19585 799100 40.8
22635 799100 35.3
21589 997166 46.2
167880 997166 5.9
4203 251332 59.8
4290 251332 58.6
37253 724785 19.5
10928 934178 85.5
93279 995104 10.7
5421 70027 12.9
20491 999910 48.8
9485 53286 5.6
21499 996194 46.3
14967 995197 66.5
27221 50446 1.9
29646 50446 1.7
4746 52831 11.1
15796 749086 47.4
38621 702631 18.2
18356 49210 2.7

stats about my node being a thinblock receiver:

Received (a) Reassembled (b) ratio (b)/(a)
330656 949204 2.9
457644 514393 1.1
32110 903051 28.1
132595 553841 4.2
262014 949048 3.6
22635 799100 35.3
3667 148377 40.5
835024 934575 1.1
26314 999959 38.0
170767 997166 5.8
4427 251332 56.8
17714 827044 46.7
34177 998139 29.2
30475 999848 32.8
83276 959184 11.5
24875 724785 29.1
219194 934178 4.3
298752 995104 3.3
2525 70027 27.7
75830 645588 8.5
69100 729828 10.6
266708 708944 2.7
62896 934384 14.9
754 77427 102.7
105373 934378 8.9
43241 999910 23.1
130873 934480 7.1
68570 959036 14.0
5427 53286 9.8
58158 996194 17.1
316814 995197 3.1
25462 50446 2.0
1540 52831 34.3
33859 999961 29.5
99371 933114 9.4
91356 934548 10.2
41226 933262 22.6
44363 749086 16.9
64504 932819 14.5
57215 959093 16.8
85775 949128 11.1
27398 934470 34.1
38821 934225 24.1
43171 749043 17.4
16838 513665 30.5
79101 702631 8.9
67861 998122 14.7
40604 996179 24.5
24741 949165 38.4
9818 564802 57.5
43892 749172 17.1
511011 934582 1.8
543329 934508 1.7
49780 949119 19.1
9248 243637 26.3
27142 934054 34.4
53738 694306 12.9
23118 999911 43.3
57995 999871 17.2
67380 999839 14.8
33541 583103 17.4
23450 999844 42.6
43053 999941 23.2
18356 49210 2.7
403077 998199 2.5
64626 934543 14.5
33488 999887 29.9
22737 797705 35.1
38574 99815 2.6
8719 389386 44.7
10717 572179 53.4
14240 455341 32.0
6706 352220 52.5
22179 781588 35.2
8029 368993 46.0
9831 380089 38.7
16660 690770 41.5
21986 749157 34.1
34370 999866 29.1
106435 749191 7.0
38693 987930 25.5
174050 988100 5.7
64085 997105 15.6
253032 930911 3.7
184100 949147 5.2
221279 995202 4.5
86401 592751 6.9

edit: add comment a few stats

 

[sickpig]

@sickpig our slack channel is a good place for fast discussion.

@Aquent @TrevinHofmann
can we have an implementation channel please, and invite sickpig & Peter Tschipper?

EDIT: Trevin has created:
https://bitcoinunlimited.slack.com/messages/implementation/files/

thanks @solex

I've just sent a slack subscription request to @TrevinHofmann via email.

 

[Peter Tschipper]

@sickpig the results will get better over time...there is a lot of spam out there right now so it takes much longer for the mempools to get closer in sync, 12 to 24 hours right now I would say. The more in sync the pools are the smaller the thinblocks get.

 

[theZerg]

Well I think that those ratios look great. @sickpig How did you generate this data? Parsing the logs?

@theZerg I'm a little sleepy so not sure if this is going to make sense, but in instead of using connect-thinblock which requires a config setting, what about using a timer such that when a block inv first arrives for a regular block we start the timer...if we have thinblock nodes connected to us and if we don't receive an inv from a thinblock node in lets say 5 seconds after the first inv for a regular block, then we just do a getdata for the regular block...that way, we don't need the config setting (although we can still leave connect-thinblock there as an option) and so there is a way for a block to get into the thinblock network of nodes while at the same time 100% preferring thinblock nodes over regular nodes. And once it's in the network it will propagate rapidly among the nodes without having to wait for the timeout?

Yes, that makes sense to me... right now I am testing just preferring thin block nodes in the send/receive loop but if that is not enough the timer seems like a real good solution. I mean lots of nodes (anyone but miners) will likely be happy to sacrifice latency for a bandwidth reduction.

 

[theZerg]

I'm getting a bunch of logs like this:
2016-02-16 12:18:53 Reassembled thin block for 00000000000000000180feaf10b6516ce517e14167f2e76675cac420433399f8 (749157 bytes)
2016-02-16 12:18:54 Removing Thinblock in flight 00000000000000000180feaf10b6516ce517e14167f2e76675cac420433399f8 peer=459

Does this mean that the thin block was superceded by a full block from another node or is that normal? If normal, I am thinking that I may add some code to proactively remove the thin block from the node so that the "Removing in flight" message is not issued.

 

[Peter Tschipper]

@theZerg that's normal...it was really just informational while i was coding, probably not necessary now. it's just a tracking map that tracks how many thinblocks are in flight since we can only download one per peer at a time. it's just tracking the hash of the block and the count. I can remove that message if you think it's confusing. All that happens when the thinblock is received is that the tracking hash entry is removed, it's not a thinblock that's being removed.

 

[sickpig]

Well I think that those ratios look great. @sickpig How did you generate this data? Parsing the logs?
.

yep.

received blocks:

grep -A 10  "received thin" debug.log  | grep -E '(received|Reassembled)'|  awk '{if ($3 == "Reassembled") print $3 " " $8; else print $3 " " $7} ' | sed -e 's/(//' | grep -v peer | awk '{key=$0; getline; print key ", " $0;}' | sed -e 's/,//' | awk '{ ratio = $4 / $2 ; print $2 " " $4 " " ratio }

sent blocks:

grep  "Sent xthinblock" debug.log  | awk '{ ratio = $11 / $7 ; print $7 " " $11 " " ratio}'

[doublepost=1455640137,1455639203][/doublepost]

I'm getting a bunch of logs like this:
2016-02-16 12:18:53 Reassembled thin block for 00000000000000000180feaf10b6516ce517e14167f2e76675cac420433399f8 (749157 bytes)
2016-02-16 12:18:54 Removing Thinblock in flight 00000000000000000180feaf10b6516ce517e14167f2e76675cac420433399f8 peer=459

Does this mean that the thin block was superceded by a full block from another node or is that normal?

nope.

I'm not sure how to detect when you'll receive a normal block rather than "thin" block. But I've found a few occurrences where my node send a regular block rather then a "thin" one to a thin-enabled peer (66.222.25.72)

$> grep -B1 "Sent regular block instead" debug.log 
2016-02-16 04:26:20 Removing Thinblock in flight 000000000000000005e300ba34cb2a994d167ee00a799c55059ef8c386433b6a  peer=226
2016-02-16 04:26:20 Sent regular block instead - xthinblock size: 249 vs block size: 208 => tx hashes: 1 transactions: 1  peerid=2
--
2016-02-16 04:26:54 Removing Thinblock in flight 000000000000000006afc7734aef25812e2c4e9077a439176ebe3591b8a2490e  peer=226
2016-02-16 04:26:54 Sent regular block instead - xthinblock size: 270 vs block size: 229 => tx hashes: 1 transactions: 1  peerid=2
--
2016-02-16 08:30:04 Removing Thinblock in flight 00000000000000000705cf47ce51de007cf37df37f1c224a99803728d53dc4a0  peer=226
2016-02-16 08:30:04 Sent regular block instead - xthinblock size: 249 vs block size: 208 => tx hashes: 1 transactions: 1  peerid=2
--
2016-02-16 10:23:12 Removing Thinblock in flight 0000000000000000006d275729d380a140959996dba961ba16d7f365ede302ec  peer=226
2016-02-16 10:23:12 Sent regular block instead - xthinblock size: 270 vs block size: 229 => tx hashes: 1 transactions: 1  peerid=2

 

[Peter Tschipper]

If you don't see re-assembled block but you see processedblock then you've received a regular block...you may also see "requesting regular block"

Regular blocks are sent if they are smaller than a thinblock as happens with very small blocks or just after you start up your node.

 

[Peter Tschipper]

I saw this a few times in the log...it's just an improperly formatted LogPrint statement...i'll push up the changes later.

************************
EXCEPTION: St13runtime_error
tinyformat: Not enough conversion specifiers in format string
C:\Temp\bitcoin-qt buip010.exe in ProcessMessages()

 

[Peter Tschipper]

@theZerg I submitted PR11 for a couple of bug fixes...i'll submit a separate PR for the preferential downloader when it's ready.

 

[Peter Tschipper]

@theZerg The preferential downloader is ready for testing. I opened PR12 which also includes the other two bug fixes from PR11.

https://github.com/BitcoinUnlimited/BitcoinUnlimited/pull/12


It works as follows:

The preferential downloader works for nodes that are casually connected to other thinblock nodes or by the use of "addnode=<ip>"

If a new block announcement has arrived by "inv" and it's not from a thinblock node then a 10 second timer is started. Every following inv is checked until either it gets one from a thinblock node or the timer is exceeded. If the timer is exceeded then a regular block is downloaded instead.

You can still use "connect-thinblock=<ip>" for those that always want thinblocks and it will override the timer functionality described above.

 

[sickpig]

@Peter Tschipper @theZerg

I got this new warning while compiling branch BUIP010:

unlimited.cpp: In function ‘bool ClearThinBlockTimer(uint256)’:
unlimited.cpp:583:1: warning: no return statement in function returning non-void [-Wreturn-type]
}

it seems to me that this function type has to be void.

 

[Peter Tschipper]

you r right, that's what happens with copy and paste ...i'll fix that and push it.

 

[sickpig]

another stats dump before starting a new test session on the latest modification.

let me say first that the results are very promising.

those are cumulative stats that aggregate a two running sessions of approx 20hs each. Once I finished with the last set of test I'll try to get a longer test session (72hs at least).

stats related to sent blocks:

# thinblocks sent: 112

Distribution of size(block) / size (thinblock)

Min.    :  1.009
1st Qu. :  7.036
Median  : 35.006
Mean    : 35.018
3rd Qu. : 49.258
Max.    : 154.684

stats related to received blocks:

# received thinblocks: 174

Distribution of size(block) / size (thinblock)

Min.    :  1.119
1st Qu. :  4.886
Median  : 14.585
Mean    : 20.116
3rd Qu. : 31.442
Max.    :138.694

edit: add graphics of freq distribution

 

[Peter Tschipper]

@sickpig It's great to see the distribution breakdown...thanks again.

 

[_mr_e]

Hey guys, Is this stuff basically subchains?

 

[sickpig]

@_mr_e

nope.

it uses bloom filter to reduce the amount of data nodes have to exchange to send/receive blocks to each other.

see: https://bitco.in/forum/threads/buip010-passed-xtreme-thinblocks.774/