Core moving from LevelDB to SQLite

[sickpig]

as per thread title. see:

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011604.html

They change db gears way too frequently for my taste considering how critical piece of the infrastructure the db is. Last time that we've changed db we had an hardfork, just saying

p.s. not properly on topic since is not protocol related...

 

[solex]

Agreed. LevelDB is well proven, and the old BDB is still used for the wallet. That never got changed. This is "devs gotta dev" syndrome. We need IBLT before another change of db.

This is a comparison;

https://leveldb.googlecode.com/svn/trunk/doc/benchmark.html

LevelDB looks pretty good.

 

[Bloomie]

p.s. not properly on topic since is not protocol related...

Maybe this section should be renamed then, because it's intended to include exactly these kinds of topics.

 

[sickpig]

@Bloomie a rename could be worthy, but I've no fancy new name, sorry. I will think about it.
[doublepost=1445608505][/doublepost]regardless it seem a fast-tracked issue:

https://github.com/bitcoin/bitcoin/pull/6873

Jeff just merged the above PR, aimed to introduce an agnostic db wrapper. mainly renaming thing, though.

 

[theZerg]

Its nice to have a generic interface and could be an automatic merge...

 

[sickpig]

@theZerg definitely.

I was just surprised by the quick way it went into the tree, taking into account that using different dbs to store block data is not as straightforward as it seems.

I don't follow bitcoin core dev process with enough details, but it could be that since Feb 2013 a lot tests infrastructure has been added to avoid hard forks due to db usage/settings corner cases.

 

[solex]

Interesting discussion during the bitcoin dev meeting:

19:20:10 <sipa> #topic leveldb replacement
19:20:27 <gmaxwell> Holy crap.
19:20:35 <jgarzik> LMDB simply doesn't work on 32-bit
19:20:40 <gmaxwell> stop.
19:20:48 * gmaxwell bangs gavel
19:20:50 <mcelrath> Why is 32-bit such a big deal?
19:21:07 <mcelrath> Want to keep running on raspbery pis?
19:21:30 <gmaxwell> I don't think that I was clear enough. We are _NOT_ replacing leveldb at this time. People should happily test other things, the code is abstracted in a way that specifically facilitates that. Have fun, report results.
http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-10-29-19.02.log.html

Looks like Pieter wants to progress LevelDB replacement, but Greg is pulling rank (the new Bitcoin CEO?) and delaying such a project.

Both Gavin and Mike not present, and no discussion of the block limit.
Time-waster BTC-Drak misses some of the meeting due to daylight saving time change (poetic justice there).

 

[theZerg]

wow gmaxwell really thinks he's in charge doesn't he? He clearly feels like he's the chairman of the meeting now with that "bangs gavel" reference.

OTOH, LMDB is working for me on 32 bit windows (different app, much lighter use than bitcoin) so "simply doesn't work" is hyperbole.

 

[solex]

that "gavel bang" must be a button in the meeting tool. Yet, he was the only one to use it.

 

[theZerg]

You use IRC much? I don't but I thought that he was just using the standard IRC way to specify an action. You just type: "/me bangs gavel" for example.

 

[solex]

nope. You are probably right.

 

[davecgh]

As a point of data, we originally were using SQLite with btcd way back in the very early stages of development and it simply couldn't handle the volume of updates necessary without tanking performance. I would be extremely surprised if Core finds differing results.

 

[theZerg]

Hard for a sql db to complete in performance with a persistent hash table unless you are able to take advantage of the superior data organization.

 

[bitcartel]

@theZerg My understanding is that LevelDB is no longer maintained, and that the LevelDB community has rallied around a fork called RocksDB. Both LevelDB and RocksDB are not ACID compliant.

@davecgh Sqlite4 is supposed to be faster than LevelDB. Release date unknown.

For BU I think it would be great if users could decide for themselves what database they wanted to run. First, by all means have sane defaults, but if a user has the resources to run an Oracle database, why not let them? Second, and perhaps more importantly, the network will be more robust in the long run when it is no longer reliant on a single piece of software and its choice of storage engine where bugs are somehow considered consensus critical.

 

[davecgh]

@bitcartel:

Yes, I've seen the discussions about sqlite4 and it certainly may end up being, but as you noted it's ETA is unknown.

Speaking of allowing a choice of which database to use, that is exactly the approach btcd uses. There is a database package that provides an interface which the rest of the code uses. That allows different backends to be written for it.

If you're interested, see my branch here (related PR that discusses the rationale and key changes) for the new database interface that will be getting merged once the rest of the infrastructure required on top of it is done.

 

[molecular]

As a point of data, we originally were using SQLite with btcd way back in the very early stages of development and it simply couldn't handle the volume of updates necessary without tanking performance. I would be extremely surprised if Core finds differing results.

Thank you for reporting this, as this was my suspicion.

I do a lot of db stuff at work and I've been playing with bitcoin-abe (tools that puts blockchain into SQL-dbs like mysql or postgresql). Performance of bitcoin-abe is god-awful, it's close to impossible nowadays to sync the bitcoin blockchain (gave up after 2 weeks last time I tried, still before block 210,000 on a fucking 8-core machine with SSDs)

I just cannot imagine a SQL db beating leveldb for this use case, let alone mysql, let alone mysql-lite.

I really don't get why they're doing this? Want to switch to oracle at some point???

 

[bitcartel]

Regarding ABE (1) the database schema is not optimal as it is designed to support multiple (alt)coins in one database (2) the application is written in Python (3) the initial parsing of raw block data to populate the databse is single threaded.

 

[theZerg]

https://en.wikipedia.org/wiki/LevelDB

Read "bugs and reliability". I wonder if this is still accurate. The project looks active given a 2 minute glance at http://leveldb.org/...

 

[bitcartel]

@theZerg FWIW, much more active over at Rocksdb (the LevleDB fork).
https://github.com/google/leveldb/graphs/contributors
vs
https://github.com/facebook/rocksdb/graphs/contributors

 

[jl777]

Since the vast majority of data in the DB (after some reasonable number of confirmations) never changes, my feeling is that using a DB is overkill and will prevent scalability. Regardless of what the benchmarks say, something is taking a lot of time and the DB that is doing HDD writes is the likely bottleneck.

In order to stream data to the HDD, every seek must be carefully budgeted. This makes doing a parallel sync quite tricky and full control of the writes to the HDD is required. I do not know of a DB that allows this.

James