BUIP023: (passed) Miner block creation latency optimization

[theZerg]

The mining of empty or pre-validation blocks are a mechanism that can naturally or artifically reduce the transaction capacity of the bitcoin network (see www.bitcoinunlimited.info/1txn). These blocks are mined when miners have received the headers of the next likely block but have not yet produced the next block template (a block template is a block that has not yet been mined -- its SHA256 does not meet the current difficulty).

During this interval, the miner must download and validate the full block and generate and communicate the next block template to the mining pool software. By optimizing any of these 4 steps, the average transaction committment capacity of the network is increased since the proportion of empty blocks is decreased.

This BUIP addresses the second two steps (generation and communication of the next block template) which will collectively be called "miner block creation". Two areas will be examined.

First the CreateNewBlock() function will be optimized -- in particular, it should be possible to skip validation of the transactions in the new block since they consist of transactions from the mempool. Transactions are validated when admitted to the mempool. But there may be more optimizations that become apparent as performance measurement tools are applied to the code.

Second, the getblocktemplate RPC call will be deprecated (it will still be available, but the new interface is preferred) in favor of a binary protocol that organizes data in a type,length,value (TLV) format. Endian will be that of the sending node, and specified in the binary protocol header. This will avoid endian swaps for the common case where the bitcoin daemon and pool server share the same CPU architecture. This protocol will be capable of "pushing" new blocks to the pool as soon as a block is found, rather then relying on the RPC (remote procedure call) polling techniques, and separate event channels currently used.

Extensions to this binary protocol that further optimise latency can be considered and implemented if they will be effective. For example, the pool software could provide the bitcoin daemon with the coinbase output, allowing the bitcoin daemon to create the full and exact binary block data required to pass to miners. This eliminates the step where the bitcoin pool parses the block, modifies it, and generates a new block.

The bitcoin daemon could also provide a miner interface directly, rather then passing the block to the pool and having the pool pass the block to miners. This would save a network "hop" and data translation time.

The "client side" (pool side) code will be provided as a shared library, and the open source pool software "ckpool" will be modified to show how a pool would take advantage of this new library/protocol.

=============================================================================
Addendum for Funded Development
Revision 5th November 2016 by solex

1. Project Title

Miner Block Creation Latency Optimization

2. Bitcoin Address

3G3FYxhLNRTa3hG2WD2kiKkayqy8h97SBx

3. Motivation

Miners must download and validate the full block and generate and communicate the next block template to the mining pool software. By optimizing two of these four steps (generation and communication of the next block template), the average transaction commitment capacity of the network is increased since the proportion of empty blocks is decreased

4. Objectives

The objective of this project is to enhance the Bitcoin Unlimited full node implementation for improved SHA256 mining.

5. Project Duration

All phases to be performed on a "best endeavours" for "earliest delivery" basis.

6. Project Team

Developers who wish to work on this project must reply to this thread in order to co-ordinate with the BU Developer who has oversight of this project, and accepts or executes work within its description.

7. Summary of Work Completed to Date

To be provided

8. Description of Activities

The technical detail of this BUIP is above.

PHASE 1

Design & coding completed.

PHASE 2

Testing on the BU testnet completed.

PHASE 3

Documented & Implemented in a formal BU release.

9. Anticipated Challenges and Uncertainties

Standard development issues only.

10. Budget

The budget for the project is $20,000, 1/3 for Phase 1, 1/3 for Phase 2 and 1/3 for Phase 3. The BU President and Secretary will administer these funds at the recommendation of the BU Developer when each phase is complete.

11. Impact

Improved profitability for miners using BU software, ergo an incentive for more to utilize BU.

 

[solex]

Now that miners are using BU this enhancement should get some priority.

What do people think of a $25k project fund to get miner optimization designed, coded, tested, and implemented in a formal release? perhaps 25% payable at each of the four milestones?

edit: revision 23/Oct/16
$20k project fund
designed & coded, tested on the BU test-net, documented & implemented in a formal BU release.
1/3rd payable at each of the three milestones.

 

[Peter Tschipper]

@solex The first milestone is actually a very easy one.. It's really just an extension of XVAL. Probably just a few hours , if that, to code it up and a couple days testing. Not sure it merits 5K price tag.

 

[solex]

@Peter Tschipper
Great. Just what we need to know. I have revised the proposal above.

 

[awemany]

Is there data available from preliminary tests showing improvements? I am not at all opposed to this, but I do wonder how much such a low level improvement will actually help.

In my personal experience, improvements on wire-protocol encoding details (such as avoiding htons/ntohs) are usually not that big of a deal in the context of the whole thing. But I might be totally wrong in this case.

 

[Peter R]

@awemany: Now you've got me wondering if I should reverse my vote.

I don't think we have hard data, but you could probably get something by plotting the fraction of empty blocks versus typical mempool sizes. When mempool swells, get_block_template becomes slow, and thus miners mine on empty blocks for a greater fraction of time. I'd make the plot right now if someone can point me to the raw data.

Is making an improvement here worth $20k?

 

[solex]

@awemany @Peter R
Guys, think about this in full context.
BU has to attract further mining power to succeed in its aims. Out of 36 BUIPs so far BUIP023 is the most miner-focused. Only BUIP010 (Xthin) compares in miner-focus.
The mempool is continually choked, and has been since May this year. It will only get worse if we are in a protracted arm-wrestle with the Segwiteers over the direction of scaling.

So, anything to help the miners in a congested network situation is beneficial, anything to attract miners to using BU, giving an "edge" is beneficial. $20k is good value if it helps ensure that the BU mining software is the best available.

 

[Christoph Bergmann]

If it helps the miners, even a little, and it demonstrates that BU is the perfect client for miners and care for miners I think this is worth the investment. Did you consult miners?

 

[Inca]

Unsure how much any of this costs but optimisations such as this are probably important for when network throughput really begins to ramp up. ACK.

 

[sickpig]

@solex where we are on this?