Bitcoin Unlimited - Development discussion

[theZerg]

@awemany

You can look at and fork from what I've done here:
https://github.com/gandrewstone/BitcoinUnlimited

There are 2 branches "block_unlimited" and "config_and_stats".

The block_unlimited branch just removes block size limitation from the validation. The idea is to make the smallest number of changes. There is one issue still pending:

1. Block buffered file.
main.cpp:3363
CBufferedFile blkdat(fileIn, 2*MAX_BLOCK_SIZE, MAX_BLOCK_SIZE+8, SER_DISK, CLIENT_VERSION);

This created a ring buffered file reader with a total buffer size of 2*MAX_BLOCK_SIZE and a max rewind of MAX_BLOCK_SIZE+8.

We need to determine whether the limits above are essential or just optimal. And if essential we need to choose to either rewrite CBufferedFile to handle any rewind amount (by reloading the file at that point), or to add a defacto max block size of something very large by having a large buffer here.

------

The config_and_stats branch adds a new dialog box called "unlimited". Even though the items in this dialog may be better placed in the Options dialog, I felt that isolating our changes would make merging with subsequent core releases much easier. So I plan to put every GUI change in this "Unlimited" dialog box. Right now there are a couple of items designed, but nothing is implemented.

I'll probably be working on it nights, EST (right now trying to add a configurable miner block size) so if you have any progress (so long as it basically runs -- it doesn't have to fully work) submit a pull so we don't repeat each other's work.

 

[awemany]

Hi @theZerg,

ok, I am looking at your patches, mainly to first figure out what is going on. Please excuse my stupid questions and remarks below

block_unlimited: AFAIR, Gavin introduced a txn size limitation with BIP101, I think 100kB, due to the non-linear validation time for large transactions? your patch do not have that limit - but I think we should at least introduce a command line option to set this similar as Gavin is doing? As far as I can seen, you are removing the limit of 1MB per transaction even and that may make the validation potentially much worse.

config_and_stats: Ok, that introduces a couple of other (useful) things besides max block size. I didn't see that it was you that posted the traffic shaping patch to BitcoinXT.
Certainly nice to have, BUT: What is the general idea here now, are we intending to advertise a 'minimum change' and a 'bitcoin unlimited ++' (with new features) variant?

If we have traffic shaping, we might be easily attacked as having 'funny stuff' in BitcoinUnlimited - thoughts?

@ the configurability: I like that more than the 'simpler' patch because it refrains from touching the core code paths and rather simply tweaks a value. I might have missed it - and I still need to actually compile your code - but: is there also a cmdline option for blocksize in addition to the GUI?

Last but not least: In 04f8556, I see a lot of formatting and whitespace changes all over the place - why is that, and is that intended?
[doublepost=1446974948,1446974145][/doublepost]Oh and here's something I'll make a BUIP if we don't clear agreement before: Please lets have an old-style Mailing List like bitcoin-dev. If there's one thing I really dislike about XT, it is the google groups fancy but laggy and slow google/javascript/ajax/'webapp' interface to their mailing list. And their interface to browse threads, I find really cumbersome, bordering on unusable...

 

[theZerg]

First of all, thanks for looking! I really appreciate some feedback and development help!

Also please understand that nobody has decided what will or won't be released. That will be decided by voting on a BUIP. So these two branches may be released, may not be released, or only portions my be released.

RE: block_unlimited

We believe that a block with large transactions will be at a disadvantage because the time required to verify will cause the block to propagate across the network more slowly (see Peter R's paper). This fact (and the economic incentive to produce blocks that become part of the main chain) will naturally limit these parameters. On the other hand, from an engineering perspective if a particular size will crash the program it may make sense to put a limit in. But it may make more sense to cause the "limit" to be an emergent property of the system itself: try { b = new char [size]; } catch OutOfMemory { rejectBlock() }; This will allow the limit to expand naturally as the underlying technology does.

> BUT: What is the general idea here now, are we intending to advertise a 'minimum change' and a 'bitcoin unlimited ++' (with new features) variant?

That is one of the proposals on the table. WRT the ++ variant, we could go down an endless road of speculation if we start debating what % of people will think a particular feature is "funny stuff". Therefore, I'd recommend that the ++ variant has everything we think is useful and follows the Articles philosophy -- but its all turned off. Then the user can go in there and turn on what is desired. If a user is using a full node client, I think he is (or wants to be) a "power user" so we should not be scared to introduce stuff.

@ the configurability: I like that more than the 'simpler' patch because it refrains from touching the core code paths and rather simply tweaks a value. I might have missed it - and I still need to actually compile your code - but: is there also a cmdline option for blocksize in addition to the GUI?

Yes, to make merges as simple as possible I think we want to refrain from touching existing stuff as much as possible. Unfortunately we DO need to go in there and add "hooks" like replacing constants with global variables, or adding function calls. Fortunately, these changes once made should not themselves change so they will be easy to repeatedly merge into Core releases.

In fact, I'm thinking of moving the XT traffic shaping stuff into the Unlimited dialog (in the original XT patch it is in the Options dialog for this reason.

Good eyes; I haven't added the cmdline option for maxGeneratedSize yet -- but the code is all set to do so; should be just 2 or 3 LOC.


Last but not least: In 04f8556, I see a lot of formatting and whitespace changes all over the place - why is that, and is that intended?

The XT devs wanted me to clean up formatting and Gavin told me that the project adhered to the def in a clang-format file so all I had to do was to run that. I did and it turned out that the project does *NOT* adhere to its own formatting rules (sigh). I think that we should run all of the exclusively BU files through the formatter periodically to solve this problem.

Oh and here's something I'll make a BUIP if we don't clear agreement before: Please lets have an old-style Mailing List like bitcoin-dev. If there's one thing I really dislike about XT, it is the google groups fancy but laggy and slow google/javascript/ajax/'webapp' interface to their mailing list. And their interface to browse threads, I find really cumbersome, bordering on unusable...

The Secretary will propose a BUIP detailing this kind of stuff and everyone will get to argue about it and vote on it then. Its weird though that you have issues with more modern message boards. What is your internet access quality? How good is your access to this message board?

 

[theZerg]

ok I added a command line option: use -blockmaxsize=2000000 for 2MB block (for example)

But I'm still scripting something that generates enough txns to actually create a large block

 

[awemany]

RE: block_unlimited

We believe that a block with large transactions will be at a disadvantage because the time required to verify will cause the block to propagate across the network more slowly (see Peter R's paper). This fact (and the economic incentive to produce blocks that become part of the main chain) will naturally limit these parameters. On the other hand, from an engineering perspective if a particular size will crash the program it may make sense to put a limit in. But it may make more sense to cause the "limit" to be an emergent property of the system itself: try { b = new char [size]; } catch OutOfMemory { rejectBlock() }; This will allow the limit to expand naturally as the underlying technology does.

Understood - but then I think we might want similar configurability to maximum blocksize for maximum transaction size?
A better way to structure transactions would make this problem go away - so I think we're going to have a kludgy interim solution no matter what?

> BUT: What is the general idea here now, are we intending to advertise a 'minimum change' and a 'bitcoin unlimited ++' (with new features) variant?

That is one of the proposals on the table. WRT the ++ variant, we could go down an endless road of speculation if we start debating what % of people will think a particular feature is "funny stuff". Therefore, I'd recommend that the ++ variant has everything we think is useful and follows the Articles philosophy -- but its all turned off. Then the user can go in there and turn on what is desired. If a user is using a full node client, I think he is (or wants to be) a "power user" so we should not be scared to introduce stuff.

I like this idea of having features available as options. I think this is easier to sell even if BU++ is going to be the default advertised client and contains a lot of such options. They could be activated by default only as soon as a majority uses them.

Last but not least: In 04f8556, I see a lot of formatting and whitespace changes all over the place - why is that, and is that intended?

The XT devs wanted me to clean up formatting and Gavin told me that the project adhered to the def in a clang-format file so all I had to do was to run that. I did and it turned out that the project does *NOT* adhere to its own formatting rules (sigh). I think that we should run all of the exclusively BU files through the formatter periodically to solve this problem.

We should IMO also separate the reformatting and the new code resp. code changes into separate patches. It is difficult to read and because they are also multi-line, 'git show -w' doesn't hide those changes.

Oh and here's something I'll make a BUIP if we don't clear agreement before: Please lets have an old-style Mailing List like bitcoin-dev. If there's one thing I really dislike about XT, it is the google groups fancy but laggy and slow google/javascript/ajax/'webapp' interface to their mailing list. And their interface to browse threads, I find really cumbersome, bordering on unusable...

The Secretary will propose a BUIP detailing this kind of stuff and everyone will get to argue about it and vote on it then. Its weird though that you have issues with more modern message boards. What is your internet access quality? How good is your access to this message board?

I have no problem using this forum, but the google groups stuff irks me for various reasons. This is probably totally a matter of taste... and I am of course going to follow the majority should be decide on google groups. Lets just say I'd like to have the selection of the ML frontend at least up as an informal vote...

That all said, here's a bug I noted:

src/univalue didn't get built on my system, I had to do a manual './configure && make' in there. I then got a complained that the lib cannot be relocated and I eventually added a line "CXXFLAGS += -fPIC" to src/univalue/Makefile.am.

 

[theZerg]

> Understood - but then I think we might want similar configurability to maximum blocksize for maximum transaction size?

Yes, definitely. Can you add it to config_and_stats?

> I like this idea of having features available as options. I think this is easier to sell even if BU++ is going to be the default advertised client and contains a lot of such options. They could be activated by default only as soon as a majority uses them.

I do to. I think that the options will be attractive to many users.

> We should IMO also separate the reformatting and the new code resp. code changes into separate patches. It is difficult to read and because they are also multi-line, 'git show -w' doesn't hide those changes.

Definitely! I just did not do it for the traffic shaping because I just took a diff of my patch to XT and applied it here. We should probably revert the spacing where it differs from Core. But the easiest time to do that will be when we pull the latest Core in.


> That all said, here's a bug I noted:

> src/univalue didn't get built on my system, I had to do a manual './configure && make' in there. I then
> got a complained that the lib cannot be relocated and I eventually added a line "CXXFLAGS += -fPIC" > to src/univalue/Makefile.am.

Weird, this hasn't happened to me. This is a problem in Core

 

[awemany]

Yes, definitely. Can you add it to config_and_stats?

I'll look into it.

EDIT: You seem to use maxGeneratedBlock for the blocksize limit. There was the time when bitcoind was used for mining or as a miner frontend. A miner might well want to keep a separate (lower) blocksize limit. Shouldn't we rename it something better / add a new global variable?

 

[theZerg]

what do you mean? I created the global variable "maxGeneratedBlock". It is meant to be the largest block that a miner will generate...

 

[awemany]

what do you mean? I created the global variable "maxGeneratedBlock". It is meant to be the largest block that a miner will generate...

Yes, nevermind, I am/was confused...

 

[awemany]

Hey @theZerg,

I PRed you a couple of minor changes on github.

I am looking through the list of commits from XT's only-bigblocks branch and I am wondering whether we should include most or at least a couple of these things into BU:

- the blocksize tests in 4c6108?
- the p2p message size limit?
- if we want to go the optional-configurable-blocksize-limit (OCBL) route, we might want to include/use Gavin's changes to have a nice and central function to determine current blocksize? This would also allow later extension for people to automatically set their maximum accepted blocksize from sources they trust etc...
- what about fork-indicating bits, should we advertise sth.?

Furthermore, how do we generally deal with fixes on BU, such as 05ec1d14? Do we rebase regularly or do we cherry pick on top, what about incidences where we might need to write our own?

 

[theZerg]

Great! I merged them. One of your patches (miner.cpp) it ended up with "unsigned uint64_t" (which I am fixing now) -- not sure whether this was a merge error or a typo on your part but please make sure you recompile before submitting a pull regardless of how trivial the change may seem.

I haven't looked at the blocksize tests but I think it makes sense to cherry-pick them from XT. Can you do it?

P2P message size limit: OK by me if it is small, but we should make it configurable.

Blocksize limit: Hmm... I thought that we've been discussing having an optional-configurable "excessive block" limit (will not be relayed). But I am ok with allowing a configurable blocksize limit. And if we do that a function is the way to go so we can make the limit automatically "track" BIP101, for example.

I think that we want patches that advertise the ability to accept large blocks, whatever fork-indicating that means (probably BIP101 at this point)

WRT: 05ec1d14 Is that on Core? I don't see it using github.

I think that we should periodically rebase off of Core and cherry-pick from XT. We are a small team and can't be re-implementing or even evaluating all the work that Core does on a piece by piece basis. I've been trying to keep my changes isolated. I am going to rebase to 11.2 tonight since it was just released and we'll see how it goes.

 

[Peter R]

Awesome work @theZerg and @awemany!

I was wondering if you guys have considered running a node on Test Net when jtoomin is doing his BIP101 tests.

A cool test would be to set the limit to 1 MB, but use the meta-cog stuff to allow BU to follow the BIP101 chain when it becomes longer. And then confirm that BU forks back to the Core chain when jtoomin stops mining the 101 chain.

If my talk for Hong Kong gets accepted, this would be something exciting to talk about: Bitcoin Unlimited works on test net and handles the BIP101 fork gracefully.

 

[theZerg]

Funny, I was just about to post about this.

I'm running 3 nodes on testnet and have written the scripts to generate 1MB+ of transactions in about a minute

However, its painful to wait for jtoomim to do his mining thing (for example no 1MB+ blocks at all this AM), and I need to get mining to work with BitcoinUnlimited to fulfill our vision of having a BU mining pool.

So, can anyone donate (or let me borrow, but honestly is it worth the shipping to send it back) an obsolete miner that uses the modern protocol (I think that that's called "stratum", right)? That way I'll be able to use BU to generate my own large block forks.

 

[Peter R]

If I had a miner I would be happy to send it to you but I don't. @cypherdoc: do you have an old miner kicking around?

 

[cypherdoc]

@Peter R

bummer. i just threw all my old ones away.
[doublepost=1447612219][/doublepost]

Funny, I was just about to post about this.

I'm running 3 nodes on testnet and have written the scripts to generate 1MB+ of transactions in about a minute

However, its painful to wait for jtoomim to do his mining thing (for example no 1MB+ blocks at all this AM), and I need to get mining to work with BitcoinUnlimited to fulfill our vision of having a BU mining pool.

So, can anyone donate (or let me borrow, but honestly is it worth the shipping to send it back) an obsolete miner that uses the modern protocol (I think that that's called "stratum", right)? That way I'll be able to use BU to generate my own large block forks.

the hashing miner doesn't use stratum. it uses, most often, cgminer which then communicates with the pool server which has both stratum and bitcoind loaded.
[doublepost=1447612401][/doublepost]actually, i saved one for posterity sakes on the offhand chance it will be considered a valuable relic one day. the original Avalon ASIC from Spring 2013. i can donate it to you if you wanna pay shipping. it could be worth alot some day

 

[theZerg]

What, after all the work I'm putting in to realize our collective vision?

Actually you're on the other side of the country. It may be cheaper to just to ebay. What about this one:
http://www.ebay.com/itm/Bitmain-Antminer-U3-USB-63GH-s-ASIC-Bitcoin-Miner-/131647669813?hash=item1ea6d00635:g:tcwAAOSwTapV5e7y

Or someone recommend something to me on ebay... I need something that works with the "latest" mining software whatever that is (I've never mined before).

 

[Peter R]

I can pitch in some coin to help subsidize the cost (that miner is only $40).

 

[theZerg]

thx! but will it do what I need? Paging @Melbustus

 

[theZerg]

Rebased the changes back to 0.11.2 (0.11cfg_stats branch). Loaded regtest on 3 computers, used 2 to mine different forks by setting the mining size and excessive block size parameter. Sent coins on only one fork, and then mined the other 2 in a leapfrog manner. Saw coins automatically added, reduced to "pending" and then added back as the forks overtook each other.

Found O(n^2) txn validation issue (EDIT: oops, its only on in -regtest mode -- its a sanity check that validates the entire txn set every time a new txn appears) and a few memory leaks (in common code).

 

[Zangelbert Bingledack]

@theZerg and @awemany

Some have asked me if BU is in active development. On the one hand I'd like people to recognize the awesome work you are doing and perhaps offer a hand, but on the other I thought you might want to keep it under wraps for now so I've avoided pointing people to the repo.

Thoughts?