BUIP142: (closed) Create Bitcoin Cash testdata

[Jonathan Silverblood]

BUIP142: Create Bitcoin Cash testdata
Submitted by: Jonathan Silverblood
Date: 2020/01/08

Summary
The purpose of this BUIP is to develop and maintain a public set of test cases and test data to complement the public Bitcoin Cash specification from BUIP121.

Proposal
This BUIP proposes that we use bitcoin unlimited funds to:

• Compile a public set of structured testing data
• Add new test data twice a year in preparation for scheduled hard forks.
• Maintain this testing data as a long-term commitment.

Motivation
Having proper and extensive testing data is helps ensure that the various network participants remain in consensus during upgrades and lowers the cost of entry for new participants.

Budget
A maximum expenditure cap of $5,000 per month, for a period no longer than 6 months, is proposed for the technical author and programming work involved.

Background
Currently there is a lot of information on how Bitcoin and it's derivates operate, but validating that production code is consistent with the overall consensus is still cause for concern.

As a community we embrace the concept of multiple implementations and strength through diversity, but if we cannot act on it in practice it will remain an idea and the benefits will be lost. Building up a set of testing data is beneficial for the Bitcoin Cash ecosystem and helps foster the environment in which multiple implementations will be a reality.

 

[freetrader]

I like the idea of this BUIP a lot, but I think it is extremely under-budgeted.

[Note: the question below referred to the original amount of $5K total. The BUIP has since been changed to make this a monthly amount.]

Could you clarify how you arrive at a $5K budget for this?

 

[Jonathan Silverblood]

I wasn't sure of how much work it would be, or how much or how well organized data already exist, and compared to the effort to document the entire protocol otherwise it felt better to have a significantly lower sum.

If there is clear support and need for a larger budget we could either revise the number or extend it later once initial work has been done.

 

[AdrianX]

• Maintain a this testing data as a long-term commitment

should say "this" not "a this".

Re budget, I think it's okay. BU can't bleed cash forever, and if honest people are doing this, we'll figure out what it costs as we go and can add to the budget in time.

We also now are paying a couple of fulltime salaries, so maybe the BUIP should have the backing of the membership to have those who are employed to lend support where their skill can be of benefit.

Maybe we should publish all our technical works with a money button paywall (after the Abstract) or money buttons equivalent that works effectively for BCH.

We should also have a faucet for none bitcoin users to foster adoption and use so as not to inhibit using BSV or BCH to read, get access.

 

[Jonathan Silverblood]

> Re budget, I think it's okay. BU can't bleed cash forever, and if honest people are doing this, we'll figure out what it costs as we go and can add to the budget in time.

I agree. Feels weird to request BU funding over and over for things that aren't specific to BU, but I think they do benefit the goals stated by BU.

Also thanks for pointing out the spelling misstake.

I don't think it would be a good idea to require a payment to access this kind of documentation though, given that the public benefit of this work matches the goals of BU.

 

[freetrader]

Re budget, I think it's okay. BU can't bleed cash forever, and if honest people are doing this, we'll figure out what it costs as we go and can add to the budget in time.

It might help to take a more professional approach to estimating this work.
@AdrianX, I accept you are not a software developer, and you have no idea how to do such estimates.

Honest people would admit that and not throw up unsubstantiated fiscal arguments which look like they are intended to stop BU from getting the work done that is proposed here.

Roughly, we should start from a knowledge (or estimates) of the size of the spec, roughly the number of requirements not covered by tests, a good idea of how many tests on average are needed to cover a requirement (existing tests should be used to get some idea, though I believe it would be an underestimate), and how long it takes a or tester on average to design a test case and create the data for it, and a dev to create a functioning test based on those.
Then multiply that by the hourly rate of a dev (you can use some ranges too), and you get a better idea of what this BUIP would realistically cost, if the plan was to do it right.

 

[Jonathan Silverblood]

@freetrader I'd be willing to update this with a better estimate but I don't know the specific details. If someone provides me with a good estimate I'll update the BUIP.

If not, I could update the BUIP to state that we're setting aside this amount for an initial effort to get the ball rolling, but expect a follow-up BUIP later requesting an extension to the budget if the quality and value provided so far matches our expectations.

 

[solex]

@Jonathan Silverblood
If the testdata is used by the BU test software, then it will benefit BU.
I have allocated ref 142 and updated the OP with it.

 

[AdrianX]

I accept you are not a software developer, and you have no idea how to do such estimates.

I don't think you do. I employ many people and I know how salaries and quality of work can vary. eg. the Greg Maxwells of the world think they are worth a lot when in fact they are not very productive at all.

 

[freetrader]

@freetrader I'd be willing to update this with a better estimate but I don't know the specific details. If someone provides me with a good estimate I'll update the BUIP.

If not, I could update the BUIP to state that we're setting aside this amount for an initial effort to get the ball rolling, but expect a follow-up BUIP later requesting an extension to the budget if the quality and value provided so far matches our expectations.

I've had a look a the (BUIP121) spec taking shape on reference.cash, but it's simply too early to use that for any meaningful input. From what I've seen that spec is not yet in a shape from which one could proceed to do the work proposed by this BUIP.
I'll wait for the second deliverable of the spec to make further comments. That update might change things entirely.

IMO it's not a bad approach to start small.

Take a small set of requirements, with a small budget, to demonstrate that a professional result can be achieved. Then that makes further funding a sensible proposition.

 

[Jonathan Silverblood]

I have revised the budget for this BUIP to be more in-line with my expectations, based on the result of the $20,000 budget approved in BUIP121.

I have also written a complementary BUIP to extend the budget for BUIP121 with the intent of allowing the current parties to continue and hopefully complete, the work over the course of 6 months.

 

[theZerg]

I have grave concerns with this proposal.

First as freetrader said, the spec isn't near completion.

Second, we HAVE multiple public and exhaustive test suites for the spec in the form of the QA tests for our full nodes. Given the maturity of the protocol, any of these suites will be quite accurate. So its more about labeling one of them "official" and I don't think BU wants to piss off the community by doing that.

Third, the main and testnet blockchains are great test datasets although certainly one could build another blockchain with really crazy tx to hit a few more edge cases.

Fourth, where our public test suites would fail is in the interfaces: how to tell the software under test to execute a function, how to store data like test blockchains. But we already have these interfaces in the json RPC command set and hex data dumped from executing them. Rather than waste time and money creating yet ANOTHER interface we should simply label a subset of these interfaces as necessary for test. Anyone who wants to test but doesn't want these interfaces can implement their own translator.

Fifth, I frankly don't want BU to be in a janitorial role always testing and maintaining other people's stuff. This is both a bad role for BU to be in, and it is not best practices. People must build their own comprehensive suite for their proposed features.

The industry as a whole needs to step up and supply the manpower and funding for this.


Finally, as a general comment: The BU funds are in many ways constrained by the articles to be used for BU. They aren't a general purpose fund to be drawn on for development in BCH, BSV or any other blockchain.

We sponsor conferences and other outreach because that has the specific purpose of advancing the BU brand name and node counts while simultaneously being generally good for the community. So this proposal doesn't really work unless its basically framed as "BU is making its own test suite completely accurate to the spec for the purposes of ensuring our software quality, and by the way anyone else can use it and so we're calling it the "official BCH test suite"". We have enough tension with ABC on very important issues that may drive use cases and adoption forwards. We don't need to create additional issues to fight over.

 

[Jonathan Silverblood]

@theZerg I agree that there are things to be concerned with here, but I see things differently.

1) The spec being incomplete is true, but I don't see this as a problem - instead I believe we should be more proactive and try to put in this work as soon as possible to complement the spec.

2) Having the data means it's simply a matter of lower costs to complete this, that is a good thing.

3) The full blockchain is good for some testing, but not for all types of testing. Having more specific/custom data to makes it easier to test without needing the full machinery.

4) To get a diverse and resistent ecosystem I believe we need to make sure that people are encouraged to build things differently. Declaring a specific interface comes with expectations on the structure of the code that is being tested. This might make sense if what you want to validate is strictly a full-node software, but not all software that interacts with the peer to peer network needs or even wants to work in the same way.

5) I don't see this as being in a janitorial role. To me, this is about laying the foundations for the type of ecosystem we are trying to build. This BUIP is not to make BU write tests for other software - just to supply test data and test cases for the specification that others can build their test suites from.

With regards to tension with ABC, this is in no way a means of saying that BU's way of doing this is the "Official" way of doing it. Given how much ABC talks about importance of being able to maintain and do things the right way, having this information accompanied with the community specification from BUIP121 aught to be seen in positive light.

There's still a problem here though, and that is that we have a lack of talent able to produce the work required for this, and that talent might be better used elsewhere (I don't think so, but I'm not the expert here). Perhaps this would be an opportunity for some of the ABC devs to translate both theirs and BU's testing into something generic for the specification?

> The BU funds are in many ways constrained by the articles to be used for BU.

Yes, and I think this fits well with the BU vision:

> The guiding principle for Bitcoin Unlimited is that the evolution of the network is decided by the code people freely choose to run.

You cannot freely choose if your choices are restricted. By enabling and encouraging diversity you also empower the free choice.

> Instead we believe that the free market and dynamic network will eliminate bad actors, that differing implementations and behavior make the network more robust against attackers

To get a more robust network, it is important to have differing implementations. This BUIP helps provide confidence to those who build these alternative implementations.

> Values and beliefs: adoption is paramount

To get adoption, I believe that we need to enable productive builders at the very foundation. We could've had more and better point-of-sale products if it was easier and had less friction to develop stable and secure point-of-sale products. The same goes for all aspects of adoption that interacts with the network.

I personally only run BU nodes because that is what have worked out best for me, but running a BU node is not enough to get us to global peer-to-peer money adoption. I've had a great experience working with BU the few times I've done so, but I am just one of the few that actually reached out and asked for my needs to be considered. Most developers don't go straight to the source and ask for help: they read public documentation and if it's good they use it, if not they try alternatives.

This BUIP should be seen as an effort to shift from a fragmented state of incomplete and sometimes incorrect documentation to a more consistent and useful state. It should not be seen as a stand-alone task, it is complimentary to BUIP121's work and it's purpose is to help spur adoption by reducing friction in creating the usecases that are valuable to people not yet initiated.

 

[freetrader]

From what I've seen that spec is not yet in a shape from which one could proceed to do the work proposed by this BUIP.

Okay, I've had another look since PR#2 and subsequent minor fixups.
I'm going to qualify this statement a bit, because my reservation about "shape" isn't really referring to completeness.

Some work is needed on making requirements easily identifiable and traceable across the spec.
Currently it's a pretty good compilation of requirements and descriptions intermingled, with some sections using conventions like SHALL/SHOULD/etc. but others not.

Perhaps a formal identifier scheme clashes with how the writers wanted to present the spec, and to an extent can be worked around since it's a versioned file hierarchy and one can always point at a text location.
However, an absence of identifiers makes for inefficient discussing on spec matters, and increases possibility of misunderstandings.
If one didn't want to change the presentation to accomodate some kind of requirement ids, then one could still embed spanning identifiers that would be removed prior to presentation.

Some more work is obviously needed on completeness, but I think it's a good start, and I do agree with Jonathan that work on creating test cases can happen in parallel with spec completion or improvement work, as long as the people doing both are communicating...

 

[solex]

@Jonathan Silverblood, the next vote is announced, you can have this included if you consider it ready.

 

[Jonathan Silverblood]

please include it.