iguana - parallel sync full BTC blockchain in 30 minutes, uses half the space, but starts up instant

jl777

Active Member
Feb 26, 2016
279
345
chrome app will be a bit later, I still want to get the realtime handling much more solid. Achieving instant-on has its own challenges, even with 95% of the data being permanent read only.

there are about 5 different states:

initially, there are no spendvectors or balance files, so this is the incomplete state
iguana detects if things are incomplete and then generates the utxo that is needed and ALL balances (any change could affect any balance file)

assuming things generate properly, then we get to the initial state. basically have all spendvectors and all account files as of a certain bundle. The spendvectors are permanent, but the account files are subject to change

immediately after achieving initial state (or on the next start if it is still in the initial state), all the accounts files are verified via SHA256. this does take a while. it also creates a crc32 in parallel and puts these into a file, along with SHA256 of all blockhashes

let's call this the havecrc state
on startup, if the crc file is found, then only the crc32 is calculated, this goes a LOT faster, so in 10 to 15 seconds vs minute or two. and since only the local node is creating the crc file, if it exists, we can assume that the dataset has been validated

there is even an additional state! the readonly state
if all the files are from DB/ro/..., and all files must be from there, then even the crc step is skipped and just the contents of the crc file is assumed to be valid and compared to the SHA256 inside the files.
the DB/ro/BTC dir is meant for a squashfs volume to be mounted, but the way it is done, if the files are there it is assumed to be read-only and allows uncompressed files to be put there (for faster performance).

James
 
  • Like
Reactions: rha

jl777

Active Member
Feb 26, 2016
279
345
took a bit of work, but got iguana to validate the bundles on-the-fly so that the "lossless codec" that was theoretical is now verifiable.

I am brute force going from the compact bundle formats to the raw serialized data. During this process all the txids within a block are created and the txid verified. Then all the txids for the block have their merkleroot calculated and this is verified against the block.

what this means is that given a sequence of blockhashes that are verified, all the other data is cross checked against it. I am also doing a PoW validation of the blockhashes and of course the backward link matching, so what we get is a fully verified blockchain.

All from the parallel sync'ed bundle files!

More and more of the RPC is now getting filled out and now that the data appears to be correct, once the bugs with using that data is fixed, then iguana will be working as a drop in replacement for bitcoind (other than mining)

James

P.S. I also spent a few days tweaking the chrome app version. it actually ran the first time I tried it after not testing a build for a while, but it needed some performance tuning.