iguana - parallel sync full BTC blockchain in 30 minutes, uses half the space, but starts up instant

[jl777]

iguana is bitcoin core, client, RPC server and more.

Can we extract the blockchain sync optimized code easily? -> unfortunately easily is not a word I would use to describe it. possible, certainly and most interactions internally are via queues, so with a bit of work it can be interfaced to.

The difficulty is that I had performance as only goal....

So we end up with direct dependencies internally without regard to code purity and separation into formal layers in some cases.

With iguana you can do syncs at the bundle level, where each bundle is 2000 blocks worth of data, just half the size and already fully indexed and ready to be added to parallel searches. So it is kind of like 2GB blocks, and I am working on using the main bittorrent network to sync them, so I dont see any quadratic misbehavior at all.

It should stream in at whatever bandwidth you have and since the hard work of creating all the data is already done, it wont need a lot of CPU time. Once I get the lossless codec functionality debugged, it would allow full reconstruction of any tx directly, so all nodes can validate everything locally. After the validation all the sigs can be truncated away as there is no point to keep around the 15GB of data.

For the current blocks, these types of advantages wont exist, but there are alternate methods

 

[rha]

James, how much RAM is required to run iguana for blockchain syncing?

 

[jl777]

The more RAM you configure for it the better, but even with 8 cores it doesnt go above 32GB during sync. At 4GB, it will mostly serialize the processing toward the end when it is just the large bundles. I havent optimized the minimum memory setting, as my target machine is my 3 year old laptop with 4GB of RAM. virtual memory will work too and it shouldnt be too slow, but of course slower than using actual RAM.

I would suggest to run iguana on a low resource computer as one of the few things running for the time it takes to sync, depending on bandwidth available it is either done over lunch break or about 6 hrs if you have a 20mbps connection. I am working on cutting the time in half, but that requires installed base of iguana nodes first.

one step at a time

 

[Chronos]

It seems obvious to me that any block of sufficient age is pretty much impossibly unlikely to contain an invalid transaction. For the conservative node, 6 months would be a sufficient buffer. For a typical user, 2 weeks seems to be more than enough. In order for a bad transaction to get through, it would require no node to properly validate the block for the entire span of time.

Given this, wouldn't it make sense to skip validation on blocks older than a given threshold?

 

[jl777]

I am validating everything other than the signatures at 50megabytes/sec speed using 8 cores, and I am currently tweaking the parallel sync to provide more time for a sequential signature validation.

I am pretty sure that if you skip validation, a bootstrapping node could be vulnerable to a fake chain from the attacker. Since everything other than the signatures must be properly processed, I think as long as it wont slow things down noticeably, might as well verify all the sigs. Maybe all the sigs from the most recent checkpoint would be a good compromise

 

[Bloomie]

Can you require validation of older blocks from bootstrapping nodes only?

 

[jl777]

I dont understand the question. iguana creates read only bundle files out of each 2000 blocks. Once it is made and validated, then it never changes, so there is no need to re-validate. At most a crc checksum if you worry that the filesystem read only attribute and latest update time cannot be trusted.

For the vast majority of data, even the search indexing is part of the read-only files, so it has the "instant-on" behavior. I just hate to be waiting for many minutes each time the bitcoind starts up.

The data from the most recent bundle you have, to the current will need to be validated, but that is not re-validation.

I plan to make different operational modes, but at first I just need to get the mainstream case working. Based on feedbacks, I can make whatever other modes that are in demand.

 

[Richy_T]

I am validating everything other than the signatures at 50megabytes/sec speed using 8 cores, and I am currently tweaking the parallel sync to provide more time for a sequential signature validation.

I am pretty sure that if you skip validation, a bootstrapping node could be vulnerable to a fake chain from the attacker. Since everything other than the signatures must be properly processed, I think as long as it wont slow things down noticeably, might as well verify all the sigs. Maybe all the sigs from the most recent checkpoint would be a good compromise

What I think you could do would be to have the node operate in "unsafe" mode until the blockchain is fully validated. Checkpoints would also be an option but I think the user would have to really understand what they are doing.

Vulnerability risks are probably low for a newly synchronizing node. You can't spend anything based on an invalid blockchain since other nodes will simply invalidate that transaction. You *could* think you have received a transaction based on an invalid chain but that could perhaps be mitigated by fully downloading the chain before wallet creation. But even if not, it seems the attack vector is slim. The point would be to allow the user to decide their risk level, somewhat like zero-conf.

 

[jl777]

yes. I am thinking having a precalculated checkpoint and if the dataset matches that, then just a warning until the sigs are fully validated. Of course, many users might just choose to rely on the checkpoint and skip even downloading the sigs, so I put those in separate files.

If the checkpoint isnt matched, then no money tx allowed. If it passes a warning for each tx until all the sigs are verified. I think that allows most users the quickest path to useful operation and before any large deposits are fully confirmed, all the sigs would be verified anyway

 

[rha]

Why signature validation has to be sequential?
The other threads (processors) could "prevalidate" transactions having valid unspent inputs, while main thread would go sequentially and perform either normal validation of a not validated yet transaction, or just check if the inputs of the "prevalidated" transaction are still unspent.

Maybe it was considered long ago and people calculated the gain to be too low, but I would give it a try. The trickiest thing is how to choose the transaction for the "prevalidation".

Another possibility would be to validate in parallel all transactions from examined block. If I am not mistaken, their inputs have to be in earlier blocks, so the transactions from the block don't depend on each other.

 

[Chronos]

Another possibility would be to validate in parallel all transactions from examined block. If I am not mistaken, their inputs have to be in earlier blocks, so the transactions from the block don't depend on each other.

But are you mistaken on this? I had thought that 0-conf utxo's could be respent in the same block.

 

[jl777]

during parallel sync it is possible for the txid being spent to simply not be known at all. So it needs to wait until it is there.

Since the first three passes are done in half hour or so, it doesnt seem worth the effort to partially validate the sigs as that would require having different second or third stage data files that wont be deterministically the same for all nodes

so dealing with that mess just to save a bit of time is not worth it, but it would be possible.

I am tuning things to frontload the serial scan (fourth pass) so it is happening earlier than random statistics would indicate from a parallel sync. That way the CPU load is maximized doing the sig validation, so arguably doing it in earlier passes wont gain anything for all the troubles it creates
[doublepost=1457653625][/doublepost]

But are you mistaken on this? I had thought that 0-conf utxo's could be respent in the same block.

correct. any utxo in a block can be spent in that block
that is why i process the unspents first

in any case paralellizing the current tx processing has little point as it is reasonably fast. At some point I would probably have to speed things up, but not until the 10x capacity increase would it be a limiting issue

 

[rha]

Thank you for the answers. If the CPU load is maximized, then OK. It's an interesting problem though.

 

[jl777]

yes. the problem is all parts of the system affect the other parts. So even just a few extra HDD seeks will kill the streaming of data and as it is it lost 50% of throughput due to the additional workload of saving all the sigs.

to be random accessing these to process them as soon as all the data is available would most likely slow things down much more due to the swapping in/out of data. Unless you have 64GB RAM
[doublepost=1457717781][/doublepost]to give an example of how sensitive things are, going to fixed buffer allocation and internally allocating portions of it, versus malloc, sped things up 40%.

There is also the issue of tmp files explosion. Cant have 1 file per block, as 400,000 files is bound to wreak havoc on most normal file systems. So you need to group them together, but then you cant easily delete them, which means total HDD space used cannot be kept small.

So, how to group things, when the blocks are coming in at random order?

My solution to that was to have a file per peer per bundle and it concatenates all block data to that file. When the bundle processing is done, then all the peer/bundle tmp files can be safely deleted. Still there can be quite a few number of tmp files if you have 100 peers, so controlling the request pattern of the peers to limit the number of peer/bundle tmp files is needed.

All the while creating the append only data structures. Maybe somebody more skilled than me would be able to weave in just-in-time sig validation, but I am at the limits of what I can get to work at close to full speed.

As it is, I will have to deal with the pubkeys and sigs in separate files as the 4GB limit of 32bit is very close to exceeded.

Performance engineering is an iterative process. Everything can work but there is a large range of performance within the universe of working solutions.

 

[solex]

@jl777
James, please be reassured that your presence and postings here are very welcome. And don't take the silence on some of your threads as people not being interested. Reading what you write is like being fire-hosed with ideas. If I can distill it down into a one-liner, is this right: you have written a new implementation of Bitcoin basically from scratch?

If so, do you have a prototype which can participate as a node in the Bitcoin network?
https://bitnodes.21.co/nodes/?q=iguana does not show anything yet...

Is this very far away?

A key attribute of BU is that it is a patch-set on a fork of Core. The major advantage of this is inheriting lots of the good work still being done by Core Dev, even though their primary direction is misguided. Therefore, to benefit from some of your ideas we need specific and focused self-contained changes. @Peter Tschipper's Xthin & Xpress Validation is huge for BU, but we want to build on that initial success. A major goal for BU would be to implement sub-chains to help 0-conf and block propagation (I think you have read @Peter R's paper). This would also be a self-contained change.

To make the leap to an advanced and optimized implementation like Iguana is just too great at the present time, but it is not impossible in the long-term. The first steps are collaboration on smaller changes.

 

[jl777]

yes, I have written a bitcoin core from scratch. literally. I cant stand the C++ code, it is so full of invisible things that eat up system resources, it is impossible to make high performance at the level C can if using C++, classes, big libraries, etc. Try running a full DB package from a JS interpreter in a webpage.

I assume only stdio, readonly mapped files, bsd sockets, system clock and a few other standard system functions. openssl/libsecp256k1 is the only crypto dependency, the rest is all in my repo. [add on services utilize curl, but that dependency is outside of the bitcoind equivalent] No boost, not even libevent. All from scratch, from the protocol, blockchain, scripts, signing, tx construction, RPC, etc. I started writing iguana last November, but I had written several iterations of MGW which gave me a good understanding of the blockchain and multisig.

I am testing with iguana so it functions as a node, but it is not ready yet. it also says it is a normal node, dont want to stand out.

I expect in the following months to have a fully self-contained iguana that will be able to run on very low resources. If people here would be able to help test, that would speed up the development process. It would require command line unix skills though. Even though iguana can run as a chrome app, that cross compile is just done after everything is working and the crazy fast speed is only for native version

Also, once iguana is ready and it runs a node via one click install, then i would imagine it wont take long for there to be more iguana nodes than any other. Iguana is a multicoind and can talk to many coins at the same time, so all the altcoin peoples will want to run iguana just so they can get all there coins running on the same computer, not to mention use the atomic swap and maybe play some decentralized poker

 

[solex]

Thanks. That is really good to know, and you have gone a long way in a few months. I'm sure you will keep us posted on when an alpha release is ready for general use.

 

[jl777]

the first 90% takes 90% the time, the last 10% also takes 90%

 

[sickpig]

@jl777

so it is possible to run iguana as a stand alone binary without wrapping it inside a chrome extension, right?

another question (sorry if you have already explained it, feel free to point me to prev answer if any): you said that chrome will limit storage size to 10GB, so I guess it's not possible to iguana to have access to all the block size while running has a chrome plugin, right?

or did you find a really efficient way of store the block chain?

 

[jl777]

iguana codebase is very portable as it has almost no external dependencies. So native compile is a much easier case than cross compiled into a chrome app. The build team has it built for chrome app, android, unix, osx, win64. I think win32 still has some issues to be solved. Also emscripten could also be supported for non-chrome browsers.

I cant get the validated blockchain dataset to much less than 20GB... So to have a full chain from within chrome app would require some extra effort, ie separate chrome apps that act as file servers, but really if someone is just running from webpage they probably want convenience vs. full blockchain. So my plan is to default the chrome app to be a BTC pruning node, but it can be full nodes to most altcoins

Think of iguana as a 1000 horsepower engine, that gets 1000 miles to the gallon, in a much smaller footprint.

Assuming enough time to develop, the proper question is "what is iguana not able to do"

James