Hello all,
A reference client for Datastream Compression is available for anybody who wants to compile and try it out. You'll need to
connect to a client that supports compression which can be found on bitnodes and using version 80002.
Reference Client:
you will need "LZO 2.09" to compile this.
https://github.com/ptschip/bitcoin/tree/BUIP017_compress
Bitnodes:
https://bitnodes.21.co/nodes/?q=80002
The client uses LZO compression with two levels of compression. 0 is off, 1 is fast, 2 is maximum compression. Be sure to run the compression.py python test script after you compile to make sure it's working correctly for you.
(./rpc-tests.py compression)
Additional Findings:
1) Most of the compression benefits would come from nodes doing IBD from our node. It's a frequents occurence and is the biggest bandwidth consumer by far, next is transactions and then inv messages. However we don't compress inv's as they are not compressible with LZO however there will be a followup BUIP that will hopefully deal with those in the near future. Also, there was a Core bug which was fixed which was constantly downloading headers from unsync'd nodes which also accounted for a significant amount of bandwidth savings.
2) Bloom filters could not be compressed to any extent using LZO compression. We'll need some kind of Adaptive Range Encoder or Arithmetic Encoding, for which there is no suitable portable open source at this time. It would be an interesting project for someone to produce such a compressor in c/c++ or asm.
Updates to getnetworkinfo:
getnetworkinfo now gives the following information. Xthin % compression over last 24hrs
as well as datastream compressionstats.
Thinblock stats was also updated to include outgoing xthins as well since there is a significant
bandwidth savings from those xthins, which we were not factoring in before.
*Potential compression refers to how much additional compression would be realized had the
other connected peers also been supporting compression.
Sample results from getnetworkinfo:
"thinblockstats": {
"enabled": true,
"summary": "326 thin blocks have saved 214.99MB of bandwidth",
"summary": "Compression (last 24hrs) is: 94.8%"
},
"compressionstats": {
"enabled": true,
"cmp level": 2,
"summary": "Compression has saved 8.95MB of bandwidth",
"summary": "Compression is: 20.1%",
"summary": "Potential Compression could save an additional 354.16MB of bandwidth"
},