Currently blocks are stored on disk sequentially in 100MB files. This is inflexible, has created problems and will create additional issues in the future. For example:
1. pruning: Currently pruning removes all old blocks. The whole 100MB file is removed or it is not. But if blocks could be individually removed, this would allow the node to prune blocks based on a pseudo-random selection criteria (say block hash has certain bits set). If every node pruned blocks based on a different selection criteria, a group of pruned nodes would still probabilistically contain all blocks. This is a way to shard the blockchain history.
2. block download (current): Currently we expect the block files to be filled in oldest to newest order. This means if we get a block out of order, we drop it. This slows down synchronization because we can't (for example) download N blocks at once and store whatever we receive -- one slow node feeding us a block slows down the whole resync.
3. block download (future): Random access block storage opens the door for downloading the blockchain in the reverse order. This feature would get a user a viable SPV-like wallet right away that would slowly transition into a full-node.
TODO:
1. I would like the current block load and save to be wrapped in a class that can handle both the old sequential file format and a new database simultaneously.
2. Blocks located in existing files are left there for compatibility with other full nodes
3. A configuration flag can be set to use the old sequential file format exclusively.
4. Use leveldb as the database since we are already using it for other purposes. Use the block hash as the key unless you propose a different and more compelling key.
5. Newly downloaded blocks are stored in the database (so if you deleted the .bitcoin and resynced the blockchain, they'd all be stored in the database ). Store it in a versioned data structure with at least 3 fields { version, block height, block }
6. CBlockIndex contains fields that specify the location of the block:
nFile and nDataPos.
These need to be supplemented with the leveldb key. Perhaps nFile==-1 and/or a block hash of 0 means that the block does not exist in that storage area.
7. "Reindex" becomes more interesting because the general order of the blocks are lost. It is possible to just iterate thru all blocks and grab their block height first, but that makes two passes. What is the performance penalty of two passes? How can this be optimized? Perhaps using the previous CBlockIndex header chain as a hint?
1. pruning: Currently pruning removes all old blocks. The whole 100MB file is removed or it is not. But if blocks could be individually removed, this would allow the node to prune blocks based on a pseudo-random selection criteria (say block hash has certain bits set). If every node pruned blocks based on a different selection criteria, a group of pruned nodes would still probabilistically contain all blocks. This is a way to shard the blockchain history.
2. block download (current): Currently we expect the block files to be filled in oldest to newest order. This means if we get a block out of order, we drop it. This slows down synchronization because we can't (for example) download N blocks at once and store whatever we receive -- one slow node feeding us a block slows down the whole resync.
3. block download (future): Random access block storage opens the door for downloading the blockchain in the reverse order. This feature would get a user a viable SPV-like wallet right away that would slowly transition into a full-node.
TODO:
1. I would like the current block load and save to be wrapped in a class that can handle both the old sequential file format and a new database simultaneously.
2. Blocks located in existing files are left there for compatibility with other full nodes
3. A configuration flag can be set to use the old sequential file format exclusively.
4. Use leveldb as the database since we are already using it for other purposes. Use the block hash as the key unless you propose a different and more compelling key.
5. Newly downloaded blocks are stored in the database (so if you deleted the .bitcoin and resynced the blockchain, they'd all be stored in the database ). Store it in a versioned data structure with at least 3 fields { version, block height, block }
6. CBlockIndex contains fields that specify the location of the block:
nFile and nDataPos.
These need to be supplemented with the leveldb key. Perhaps nFile==-1 and/or a block hash of 0 means that the block does not exist in that storage area.
7. "Reindex" becomes more interesting because the general order of the blocks are lost. It is possible to just iterate thru all blocks and grab their block height first, but that makes two passes. What is the performance penalty of two passes? How can this be optimized? Perhaps using the previous CBlockIndex header chain as a hint?
