BUIP002 (passed): Multi-BIP Scaling Enabler

[solex]

BUIP002: Multi-BIP Scaling Enabler
Proposer: Andrew Clifford

Submitted on: 30/12/2015

Summary

The first version BU client for full-node owners has user configurable settings for the Excessive Block Size and Excessive Acceptance Depth (EBS & EAD) which are the low-level rules implemented by BUIP001. When used in the Bitcoin network an emergent block size limit becomes effective across all connected nodes.

Ironically, the complete flexibility of this solution may put off many potential new users who would prefer their settings to reflect the same as an existing Bitcoin Improvement Proposal (BIP) proposed for Bitcoin Core by a noted cryptocurrency expert. These are either not yet implemented, or implemented just in the XT client (BIP101). A number of BIPs have wide recognition and conceivably a significant portion of the BU network user-base may select a single BIP which determines the emergent block limit. Notably they all have programmatic nuances and are not able to be replicated with the low-level BU settings (by set and forget), except for the Bitcoin Core 1MB fixed-limit which is offered for the sake of completeness.

The final list of supported BIPs will be at the discretion of the BU lead developer.

The publicity and public relations initiative gained from allowing users to select and use their preferred BIP is potentially a significant benefit to BU, especially as the BIPs were proposed to be universal in the Core and XT clients. Mining-power may be attracted as many miners like both BIP100 and BIP248.

The following extensions are proposed for the Bitcoin client:

Section A :- Interface Changes

1. Addition of the label "Bitcoin Core BIP Implementation" and an associated drop-down box in the Settings>Unlimited window Network tab. The drop-down list will initially comprise just the deactivation setting, which leaves the existing EBS & EAD settings fully usable:

• <none>

This setting is the only one visible unless the user selects the drop-down which is populated with the short-names of the BIP Implementations which have been committed separately. The recommended or working BIP list and their short-names appears in Section B.

If one of the BIPs is selected the drop-down closes and both the EBS & EAD fields are greyed and blanked. Neither can be updated.

Internally EAD defaults to 5259600, which is 100 years, + a sequence number which may help with transparency of BIP popularity when signalling channels are available.

However, a new check-box is available, default unchecked, "Enable BU acceptance depth" which will then make the EAD modifiable again, and the user may enter a low value as an override.

2. Addition of two labels and fields on the Settings>Unlimited window Mining tab:

"Future Generated Size"
"Preferred Activation Block Height" [only multiples of 12000 allowed)​

Both are used for BUIP005 information & BIP100 Block votes

3. Command line support
utilizes short-names in lowercase e.g. -bip101

4. Remote Procedure Call support.
​

It is recommended that all the above comprise a single commit to the Bitcoin Unlimited repository. "BUIP002 Interfaces"

Section B :- BIP Implementations

Most scaling BIPs have incomplete specifications, have either no official number, not been formally proposed, or are withdrawn. Recognizing that most may never have a complete specification the working principle here is to adhere to the latest modifications announced by the original authors. Although BIP100 is withdrawn by its author it is included here as it was, for a while, the BIP of preference by >50% of the Bitcoin network hashing power.

BIP short-names are in italics. The long-name description with the author and technical detail is best collated into an informational page on the bitcoinunlimited.info website for user reference.

Some BIPs may be dropped while others are added by the BU lead developer during the development phase of BUIP002:

5. BIP Recommended or Working List.​

• BIP100
  Jeff Garzik (formally withdrawn) miner vote of voting blocks from previous 12,000 blocks. EAD=5259601
• BIP101
  Gavin Andresen (existing implementation: XT) 8MB doubling bi-annually for 20 years from 1st Aug 2015, smoothed, EAD=5259602
• BIP102
  Jeff Garzik (formal specification) 1MB, then 2MB from 5th May 2016, EAD=5259603
• BIP103
  Pieter Wuille (informal specification) 1MB increasing at 4.4% per 97 days from 1st Jan 2017 until July 2063, stepped, EAD=5259604
• BIP202
  Jeff Garzik (formal specification) 1MB, then 2MB at 5th May 2016 +20 bytes per 10 minutes, EAD=5259605
• BIP248
  Adam Back (informal specification) 2MB, then 4MB from 25th Aug 2017, 8MB from 25th Aug 2019, EAD=5259606
  
• BitPayAdaptive
  BitPay developers (pending formal specification) MGS=soft_limit, EBS=hard_limit, EAD=5259607
• Core1MB
  Satoshi Nakamoto (existing implementation: Core) 1MB fixed, EAD=5259608

For the avoidance of doubt, in every BIP, any reference to the maximum block size is implemented as the "excessive block size" in Bitcoin Unlimited.

It is recommended that each BIP Implementation comprises a single commit to the Bitcoin Unlimited repository. e.g. "BUIP002 BIP101"

BIP Specification Notes:

Flag dates trigger at 0:00 UTC
-------------------------

BIP100 details
As BIP100 is withdrawn a few implementation details are tailored for BU. Because there is no initial mining threshold needed to trigger BIP100 itself in BU, for this implementation a vote count occurs at block height multiples of 12,000 (about every 12 weeks). One block vote is cast in each block.
The original proposal for 20th percentile is replaced here with MEDIAN.

Voting:​

A miner votes by setting the Future Generate Size and this is used in the coinbase scriptSig, e.g. /BV3/ but only if their Proposed Activation Block Height < previous miner vote count block height + 12000, i.e. a desire to increase the max generate size within the next 12000 blocks.
If no FGS value or the ABH is too far in future then BV vote is set to PreviousMinerVoteResult.

Counting:​

Only a subset of the previous 12,000 blocks are expected to have votes.
Votes are ignored where the block with the BV vote is larger than the MinerVoteResult from the previous BIP100 vote count.
A BV number < 100 000 is seen as MB, a number larger is bytes.
Floor=0.5*PreviousMinerVoteResult
Ceiling=2*PreviousMinerVoteResult
Votes outside floor or ceiling are brought in and all votes sorted.
MinerVoteResult = median value of the votes (this was originally 20th percentile, but median is considered better where BIP100 is opt-in)

Adherence:​

Mining nodes which activate BIP100 are choosing to update their own mining limit (maxGenerateSize) to the new MinerVoteResult
Non-mining nodes which select BIP100 are choosing to update their Excessive Block Size to the MinerVoteResult.
-------------------------

BIP102 flag date
BIP102 was intended to start on 11th November, 2015, but if this were to be implemented by Core it would likely match the date proposed for BIP202.
-------------------------

Note i) BIP Mining thresholds are not used because there is no requirement for a universal implementation. This decision is devolved to the individual users in BU.

Note ii) For blocks >1MB the SIGOPS counting and limiting will follow BIP101's methodology for all the BIP implementations.

Note iii) If a future commit is made to Bitcoin Core for one of the scaling BIPs already applied to BU, and there are differences in the implementation, then a new BUIP may be required to synchronize BU with the Core implementation of the BIP concerned.

 

[solex]

Please review and comment so that this can be modified as necessary before voting.

 

[Wary]

@solex

Let me put it in simpler words, just to make sure I understand it correctly.

Currently I vote for BU by installing full BU node. The result - number of BU nodes in XTNodes.com. This vote is for people that want to know what node owners think of core developers

I also vote for specific blocksize, but changing the settings. This vote is not for people, but for their nodes, and result of this vote is not visible until somebody will risk their 25BTC by making >1M block and trying to send it through my node.

You are suggesting making one more vote possible: preference of specific BIP. Is it vote for people or for nodes? That is what would be result of this vote - a number on the XTNodes.com, or ability of node to accept blocks, defined by selected BIP? Or both?

Sorry for the laymen's question, but I don't know much about Bitcoin protocol.

 

[Zangelbert Bingledack]

I think acceptance depth should reset to OFF (or 100 years; I think there should be a clean OFF option for acceptance depth, but that's another issue), so that a user can "panic" over to a "safe" BIP.

It should be like they are entrusting that dev (Pieter, Adam, etc.) to be the authority they want to follow - however, they should also be allowed to build on that if they wish.

I therefore suggest that every time a BIP is selected the user is prompted something like, "BIP103 defaults to oversized block acceptance depth = OFF. Modify this BIP to set your own acceptance depth?" [If they opt to set their own, the field should load with the depth setting they currently had but allow them to adjust it before clicking OK.]

Reasoning:

That way we don't get complaints about people forgetting about acceptance depth and thinking they have their settings fully "in Pieter's hands" or whatever. People can change their mind and fully entrust Pieter, etc. on the spur of the moment without having to worry about whether they need to clean up other settings.

I interpret part of the attraction of this BUIP to be making BU into a platform that, as near as possible, let's users who want to trust a specific dev or team entrust their settings to whichever dev they trust without having to switch to the client that dev is affiliated with. Therefore I think BIPs set should default to wiping any "roll your own" aspects unless the user specifically opts for them, making it clear that it's a modification of the BIP.

 

[solex]

@Wary
You have it right except that the preference for a BIP supersedes the manual Excessive Block Size and disables it, (It is for the ability of the node to accept blocks.) in fact I realise that I never mentioned that - so I will do so.

@Zangelbert Bingledack
Interestingly, your point leads to the same conclusion. The clicking of the radio button to select the BIP list should switch off both the Excessive Block Size and Excessive Acceptance Depth fields. Basically, the names grey out and contents goes blank. The user is in the hands of the developer's decision. I don't think that there is any hint text for user settings.

The only downside is that someone couldn't later have a BIP with a sensible acceptance depth e.g. 4. Which would have been nice to get people to experiment with it
I suspect that even if BU became the reference client that it would not be possible to remove the BIP list anytime soon as many users would want to keep their setting.

 

[theZerg]

Hi solex,

Thanks for the submission!

As the first technical submission, I am going to describe the process which you can read in the Articles for everyone:

In the first period the Developer (me for the next 2 weeks at least) has a maximum of 2 weeks to work with the Proposer to work on the BUIP. As we get bigger, we may have to actually enforce the exclusivity of this period, yet the communications must remain public. But at this point I welcome all of your constructive and non-judgemental (i.e. no statements like "I support"/"I don't support") comments. Please reserve judgement until your comment period begins.

So let me move forward with my comments:

1. Let us restructure the BIP portion into 2 sections:
a) Framework to enable the addition of BIP-mimicing behavior (mostly changing the GUI & adding RPC calls, cmdline options)
b) Implementation of different BIPs.

Let's specify that code for A should be committed first, and then committed for individual BIPs (B) separately when and if patches are implemented for that specific BIP.

This allows different engineers to work on their favorite BIP.

2. Are the limits in BIPs implemented as part of consensus (i.e. as specified in the BIP), or in the "unlimited" fashion (i.e. they modify the "excessive" parameter) and so with a by-hand modification to the "accept depth", your node will follow consensus if it deviates from your BIP. It seems like the latter, but I would like this to be explicitly specified just below the list of BIPs. Something like "In every BIP, any reference to the maximum block size is implemented as the "excessive block size" in Bitcoin Unlimited". I really like the idea that accept depth continues to apply, because this holds to our core idea that BU remains flexible, but agree with you and ZB that changing to BIPXXX should pop the accept depth to infinity. Note that BU client behavior will still differ slightly from a classic client -- in particular it won't drop connections from clients that send it an excessive block.

3. Should the coinbase scriptSig be broken out into a separate BIP? I think that it may be less controversial than the other proposed changes... although maybe they are all pretty uncontroversial. And should we also post other parameters? maybe accept depth because that indicates how strongly this node will hold to the specified excessive block. Should we compress it by specifying the number in MB, i.e: /EB2.0/AD4/? (clients could either add more precision or keep 1 decimal point and round down). If we do that we can cram both numbers in the same size as the original EB specification.


4. Are all of these BIPs fully specified? I was under the impression that they were not. If they are not, we should add a note "when specified" to the BIP list.

5. Isn't BIP100 formally retracted? Maybe we should take it off (we can always add it back as a separate BUIP if it rises from the dead).

If you can't edit your top posting, post the full BUIP again (with edits) and I'll replace the top with it.

 

[solex]

@theZerg
Thanks for your positive and detailed reply! Much appreciated, and the clarity on procedures I have now done an initial edit to incorporate the thrust of your points.

I like the idea of each BIP being self-contained. While a few are simple, BIP100 is trickier, though I think it is worth the effort. Even if it draws in 5% hashing power this will be far more than what XT has yet achieved.

1. Let us restructure the BIP portion into 2 sections:
a) Framework to enable the addition of BIP-mimicing behavior (mostly changing the GUI & adding RPC calls, cmdline options)
b) Implementation of different BIPs.

Let's specify that code for A should be committed first, and then committed for individual BIPs (B) separately when and if patches are implemented for that specific BIP.

This allows different engineers to work on their favorite BIP.

Agreed. Much nicer. Done that now.

2. Are the limits in BIPs implemented as part of consensus (i.e. as specified in the BIP), or in the "unlimited" fashion (i.e. they modify the "excessive" parameter) and so with a by-hand modification to the "accept depth", your node will follow consensus if it deviates from your BIP. It seems like the latter, but I would like this to be explicitly specified just below the list of BIPs. Something like "In every BIP, any reference to the maximum block size is implemented as the "excessive block size" in Bitcoin Unlimited". I really like the idea that accept depth continues to apply, because this holds to our core idea that BU remains flexible, but agree with you and ZB that changing to BIPXXX should pop the accept depth to infinity. Note that BU client behavior will still differ slightly from a classic client -- in particular it won't drop connections from clients that send it an excessive block.

I think that all the BIPs should use the EBS and EAD parameters such that BU is performing an emulation as much as an implementation. I like the idea that users have the choice to modify the EAD down from infinity to a useful number like 4. But I also realize that there is a balance to be struck so that users have no reason to think that the BIPs aren't being implemented properly or rigorously (beyond the slightly differing connection behavior etc). Maybe there could be a concept of a EAD "override", i.e. a check-box in order to manually override it.

3. Should the coinbase scriptSig be broken out into a separate BIP? I think that it may be less controversial than the other proposed changes... although maybe they are all pretty uncontroversial. And should we also post other parameters? maybe accept depth because that indicates how strongly this node will hold to the specified excessive block. Should we compress it by specifying the number in MB, i.e: /EB2.0/AD4/? (clients could either add more precision or keep 1 decimal point and round down). If we do that we can cram both numbers in the same size as the original EB specification.

i think you mean a separate BUIP, and yes, this makes sense as the only overlap with BUIP002 is the miner voting usage. Agreed with formats like /EB2.0/AD4/ with the EB as MB to 1dp. It would be nice to have this live before too many BU blocks are mined so that as complete a history is possible for future analysis.
I have removed this from BUIP002 for now, but happy to re-insert as a Section C if you think it is OK.

 

[theZerg]

WRT the procedure, my (the Developer's) only power here, if solex and I can't come to an agreement, is to delay the process by 2 weeks and propose a competitive BUIP for a 3-way vote. But in this case we agree so I can sign off on it which starts the 2 week minimum public discussion time right away.

I REALLY think that we should allow the EAD override, because that keeps with the BU philosophy of being flexible (and still working) in the face of other scaling solutions "winning"... it sounds like you don't care so let's propose it that way and see if people complain.

I'll go either way on Section C -- but I also think its important to get done soon so if you choose to pull it out can you issue the BUIP tonight?

I have completed my technical revew. Let's move to public discussion.

 

[solex]

We are in agreement.
I will add an EAD override option, and create BUIP005 tonight for coinbase scriptsig information as this gives the flexibility of an earlier release if there is a delay in the coding and testing of BUIP002.

 

[freetrader]

I like the concept of being able to emulate other BIPs while running BU, but from a software development point of view this is problematic in several ways.

a) more complexity -> more bugs -> inherent risk for BU's core features

b) as BIPs become outdated, BU code will end up cluttered with legacy code paths that need to be removed - tedious, ungrateful and dangerous (see (a)) work

c) required amount of testing effort would quickly increase

d) "canonical" BIP implementors could criticize any perceived slight incompatibility and tell people to run their code anyway

In my experience, projects that add feature on top of feature become unmaintainable, even if there are lots of programmers involved.

I would strongly recommend sticking to simplicity - carefully extend the fundamental parameters of BU if needed, offer "profile" parameter sets to users wishing to emulate other BIPs to an extent, but focus on improving BU over emulation of other proposals.

Now KISS!

 

[solex]

@freetrader

The Multi-BIP proposal is really a one-off to drive home to the whole ecosystem exactly why BU is fundamentally different from Core and XT. There has been years of hard-fought debate about the 1MB simply because of the perceived required universality of all solutions. BU transcends this by demonstrating that there is a solution that does not require a universal top-down limit. What better way to do that than to support all the major proposals which received significant debate: simultaneously!

By supporting them all the importance of their differences is put into true context. It is safe for BU to do as all they achieve is internally alter the EBS & EAD settings. In theory someone could mimic BIP103 (for example) by changing their settings manually at the right times.

The problem of the 1MB as a scaling issue is a mirage. The real issues are deeper such as excessive SIGOPS per block and UTXO set bloat. These are the problems which should have had years of community debate, instead an incredible amount of time and effort has been wasted in the wrong direction.

...focus on improving BU over emulation of other proposals. Now KISS!

Absolutely, but after the above one-off demonstration, which should also attract more full node owners to BU, especially miners.

 

[freetrader]

@solex

Thanks for your reply. I do agree that presenting BIP emulation profiles would be useful to show BU's universality - but like you said, it's really a one-off.

If cast in code then its usefulness is quickly outlived - as soon as the blocksize limit debate resolves a clear winner (which I hope will be BU itself!) That would leave the obsolete code to remove - hopefully this would be limited to the GUI but it's not clear to me. For example, I'm still not entirely sure how to reconcile your previous two statements:

Notably they all have programmatic nuances and are not able to be replicated with the low-level BU settings, except for the Bitcoin Core 1MB fixed-limit which is offered for the sake of completeness.

and

It is safe for BU to do as all they achieve is internally alter the EBS & EAD settings.

I didn't get the impression initially that all BIPs could be easily emulated using the current BU settings.

For example, how to achieve compatibility with smoothly scaling proposals like BIP101? I don't think it's really feasible for the user to keep changing settings here, and a BU profile would probably lead through scaling by milestones. I think that's ok and presumably we agree that it's not necessary to aim for 100% compatibility.

As a BU user, I'd be perfectly satisfied if these "BIP emulations" came only in the form of parameter profile advice in documentation (e.g. website), and where necessary with disclaimers pointing out the differences between BU profiles and BIPs.

If it is decided to make them available through the GUI, then I would suggest that we think about being able to specify these settings in some kind of policy descriptor files which can be user-modified and read in at program start to dynamically present and describe the choices. I could see this being very useful e.g. to miners, who might want to periodically adjust their policies and release them to supportive node operators and end users.

 

[solex]

@freetrader

I do appreciate your feedback.
I will clarify the two quotes. The first I mean "set and forget" and have added that phrase now in the OP. If a user is prepared to return to their computer and later manually change (or set up a script to do it) their EBS setting at the right time then BIP102, BIP103 and BIP248 can be replicated on BU as it exists. The programmatic nuances are those time interval changes, the smooth scaling of BIP101 and BIP202, plus the miner setting of coinbase BV strings (block votes) in the blockchain then scanning/real-time collation of them to execute a count for BIP100.

BUIP002 will enable BU to internally first modify the EAD and then modify the EBS at appropriate intervals to handle all these BIPs.

Byte-by-byte and second-by-second compatibility with Core is the goal for supporting the BIPs. BU is about about user choice and a user which wants their BU to run a BIP fully compatible with Core's block limit (after the same BIP) then they will be able to do it.

The GUI approach I have described is probably the simplest possible for users.

 

[theZerg]

One good thing about tis idea is its a pretty self contained task... you are just setting the EBS periodically. Great little projects for various devs. If we vote yes does not mean we must dev now.

 

[bitcartel]

@solex

As a BU user, I'd be perfectly satisfied if these "BIP emulations" came only in the form of parameter profile advice in documentation (e.g. website), and where necessary with disclaimers pointing out the differences between BU profiles and BIPs.

Agreed.

 

[solex]

@bitcartel
Indeed. That is what I am saying in the OP.

BIP short-names are in italics. The long-name description with the author and technical detail is best collated into an informational page on the bitcoinunlimited.info website for user reference.

The BIPs need to be supported in the code, but all the supporting user info is documented elsewhere. They can then be activated with a command line argument e.g. -bip101

 

[Aquent]

I find the need for a delicate balance on this one. On the one hand it can make BU even more confusing and complicated, on the other hand it may conveniently allow for setting the option once and forgetting about it.

I tend to lean towards too complicated, too confusing, and doubt we will have the time to add it at this point, but the miners might like the 2-4-8 so. Difficult trade off I find it.

 

[Peter Tschipper]

@bitcartel
Indeed. That is what I am saying in the OP.

The BIPs need to be supported in the code, but all the supporting user info is documented elsewhere. They can then be activated with a command line argument e.g. -bip101

@solex I'm still a little unclear as to what this BUIP is supposed to do. Are we actually going to put in the entire code required for each of the BIP's, or are we talking about just setting the version string so that miners and others can see what block size we're supporting on our node, and these BIP selections are then just shortcuts for non-tech's who know they want to support BIP100 for example, but don't really know what block size that refers to?

 

[theZerg]

As far as I understand it, this BUIP allows patches that simulate a particular BIP be added to BU. For example, if you select BIP101, then the "excessive block size" will automatically track the block size limit increases as defined in BIP101.

Since the community seems to be coalescing to a single 2MB kick-the-can, this BUIP may be less important then it would have been a month ago. However, please remember that a BUIP does not mandate that BU resources be assigned to a job, it simply allows a patch (should someone spend the effort to make one) to be added to the BU official release without any further voting.

So I recommend that we vote YES to this BUIP, even though I think that most of the BIPs specified will not end up being supported by BU. However, a YES vote gives us the flexibility to rapidly deploy a patch if a "scaling" BIP becomes popular.

 

[solex]

@Peter Tschipper
I agree with @theZerg's view. BUIP002 has been the victim of events in a good way. It was the best we could do to try and break the mind-set that the 1MB was immutable. However, Classic is going further by actively recruiting mining-support for a hard-fork.

The "future generate size" and "activation block height" are still useful so that in the long-term miners may use them to signal preference for block limit changes, also simulating the BIP101 is likely to be wanted by some users, even if most of the other alternatives are never done.

> these BIP selections are then just shortcuts for non-tech's...

Yes that is a big part of it.