BUIP017 (passed): Datastream Compression

[awemany]

I just took sometime to update myself on stuff. Great that there is all this work being done!

I have one question for @Peter Tschipper though:

Isn't LZO1x a GPL-licensed library? Isn't that creating a 'non-legal' license mess with the MIT licensed Bitcoin source?

 

[Peter Tschipper]

@awemany Good question. As far as I know, and I just looked it up again,

Berkely DB that bitcoin uses is GNU General Public LIcense which is the same as LZO so I think there is no issue there.

 

[awemany]

First of all: IANAL.
BDB is Sleepy Cat License (and now apparently dual licensed with AGPL). Apparently, SCL allows redistribution 'as long as the whole is open source licensed'. Whatever that means.

But isn't the usual reading that GPL means the whole program has to be GPL again?

Again, I am not sure, that's just the usual reading of the GPL AFAIK.

EDIT: See also here.

EDIT2: So, IOW, it seems like (as I expected) it would be allowed to mix in GPLed code, but then the whole is under the GPL. I would be fine with Bitcoin Unlimited being GPL- instead of MIT-licensed, but I think this is something that others might not like, and I think warrants a closer look and maybe a discussion regarding the BUIP.

 

[Peter Tschipper]

@awemany Ugggh...it appears you might be right (probably most certainly are right). I must admit i didn't really give it too much thought. Where to go from here...have to find a portable, suitable compression library that has an MIT license...fastLZ is one but i don't know how well it will compress in this situation...will have to do some research

I don't think we want to have GPL because we need other projects to support this and also we need other projects who may want to create a proprietary product as well...I think that's the point of the MIT license, at least now that I"m reading up on it.

 

[bitcartel]

lz4 might be a good compromise and it's BSD.

https://quixdb.github.io/squash-benchmark/#results

http://stackoverflow.com/questions/1359903/free-non-gpl-data-compression-libraries

 

[Peter Tschipper]

@bitcartel thanks for that...here's a quote from one of the posters...looks like i'll have to do a little work and some testing with lz4 and see how it does with compressing blocks and tx's.



"Due to issues with LZO licensing we decided to evaluate the suggestions in this thread that are BSD/MIT licensed, are very similar to LZO (for easier integration) and are suitable to run on low end hardware (think inexpensive 32bit MCUs, not Pentium2).

We tested LZ4, LZF, FastLZ and compared the results to LZO. We compressed graphical data. I cannot post exact results, but LZ4 was considerably faster than others (wins in 7 tests, with up to 40% margin against the second best), while matching the size (5-2-2 wins for LZF-LZ4-LZO, margins were small)."

 

[awemany]

There's also this, BSD-like license:

http://google.github.io/snappy/

I have only played with the python bindings a while ago.

Snappy is supposedly (also?) one of the fastest compression libraries.

 

[Peter Tschipper]

@awemany I've looked at snappy in the past, yes it's really fast but loses on the compression side. Also i've seen benchmarks showing lz4 being faster (depends on who's benchmark you'r using) and with lz4 you can have control over the compression settings whereas. If i remember, snappy is just a one size fits all. Anyway, have to see the real results with blocks and tx's.

I'm working on getting an lz4 version up and running and i'll do what i can for any other suggestions so that we get this right...

 

[Peter Tschipper]

@awemany @bitcartel Zlib also has an open license for any purposes as well as commercial and it gets the best compression (30%) for blocks and is faster than LZO at higher compression although slower than LZO at the lower compression ratio's. Just something else to add to the mix.

 

[awemany]

@Peter Tschipper : Sounds great, I was just throwing in a suggestion. Thanks for doing all this!

 

[bitcartel]

@Peter Tschipper Another one to consider, LZFSE, just released by Apple:

• https://github.com/lzfse/lzfse
• https://news.ycombinator.com/item?id=11944975

 

[Peter Tschipper]

@bitcartel Interesting...I like the fact that they claim Zlib like compression.

It's licensed under Apple but after reading the terms, it seems that it would be compatible with the MIT license. I"m not 100% sure on that...anybody else see any issues with the license?