BUIP037: Hardfork SegWit

[Mengerian]

I have a question about how the transaction is included in the Merkle tree in a block. Is the entire transaction, including witness, hashed when calculating the Merkle leaf node for the transaction in a block?

 

[Justus Ranvier]

I think we could reserve a tag number for block height for inputs if you feel strongly about this. It could be made mandatory in the future via soft fork if that ends up being very important. I don't think there is consensus to make it mandatory, and I don't think we can reach it in the kind of time frame we are aiming for here.

I think something should be reserved for a fraud proofs before the blocks get much bigger.

Extending outpoints to include block heights is a conceptually clean way to make "input never existed" proofs compact.

The alternative to doing this is to reserve a branch of the merkle tree for future extension. For example, you could say the right child of the merkle root will consist of a placeholder value now, and then in the future change the placeholder value to the root of a auxiliary proof tree.

 

[deadalnix]

I have a question about how the transaction is included in the Merkle tree in a block. Is the entire transaction, including witness, hashed when calculating the Merkle leaf node for the transaction in a block?

Yes. If not, then one cannot prove the witness are correct.

 

[Mengerian]

OK, good. So in this proposal, there is only one Merkle tree in a block for the transaction data, not two separate Merkle trees like in Segregated Witness?

And the Merkle leaf node for the transaction is different from txid.

Is this correct?

 

[deadalnix]

This is correct.

 

[Mengerian]

OK, that good to know. I think this is the right way to do it, better than having two separate Merkle trees per block like Segregated Witness.

Incidentally, maybe it's beyond the scope of this proposal, but this makes me think of a way to retroactively solve malleability for the old transaction format when spent by this new transaction type. Simply define a "normalized txid" for the old transactions, calculated in a non-malleable way, the the new transaction format would use to reference old transactions. The concept is similar to BIP 140.

 

[deadalnix]

I don't think it is desirable to change the old format at this stage. Considering this can spend UTXO created by the old format, we can come up with a plan to weed out the old format over time. Doing otherwise would require to do a double indexing of the UTXO set, which doesn't seems like it pays for itself for a temporary measure.

 

[Mengerian]

I'm not sure you understand what I am saying. I am not suggesting to change the old transaction format.

What I'm saying is that the node could compute a second "normalized txid", which is non-malleable, for all old transactions. This would be in addition to the regular txid which remains as is. The new transaction format would use this "normalized txid" when referencing transactions in its inputs. So old transactions would appear non-malleable when they are inputs to the new transaction format. Transactions in the old format would still use the old txid to reference other old transactions, and would still be malleable, unchanged from current operation.

Yes, it would require a double-indexing, so that would be a cost. But it might be a nice benefit for the new transaction format to see old transactions as non-malleable.

 

[deadalnix]

Yes i understood you. That would mean that you would have to be able to load the UTXO from the old od to spend it with old transaction, and from the normalized txid to spend with this transaction. So one needs to double index the whole UTXO while old tx are weeded out.

 

[Mengerian]

OK, cool, we are on the same page.

I am not necessarily advocating this idea, I just wanted to point out the possibility. It seems interesting that old transactions could appear non-malleable to a new format, but I'm not sure if any use cases would benefit from this.

As you say, it would add to computation cost until old transactions are weeded out.

 

[deadalnix]

Updated once again, improved out encoding, which is important because we need to keep them hot in the UTXO set. Next step is to use some form of variable int encoding to reduce the size of the thing as a whole. Size is not critical as we can use a different format on to transmit on the network and to hash, and only the hashed one needs to be canonical, but t the end, if we can use the same for both, at least as a first implementation, that'd be a plus to have it compact enough.

 

[Mengerian]

This new transaction format is a very good and important proposal.

The only thing missing is a cool sounding name. We have Segwit, FlexTrans... What shall we call this?

 

[deadalnix]

Really, I do think hard fork SegWit is the best here. It takes most of SegWit has to offer and adds metadata/options fields to also gets the extensibility benefit of FlexTrans. Really, there is a saying in French which goes as "le surdoué copie, le génie pille", which roughly translate as "the gifted copies, the genius pillages" and that's very much what I did here: rip the best parts of SegWit and FlexTrans and make something out of it. There is some sauce of mine on top of it, but most of the credit is due to core and tomz.

 

[deadalnix]

Added variable length encoding, which makes the transaction smaller. Also made a BUIP to have a unified address format based on rusty's work: https://bitco.in/forum/threads/buip045-unified-addresses-format-for-buip037.1725/

 

[Christoph Bergmann]

This seems controversial. I don't get the technical stuff inside, so please excuse if I don't contribute to this discussion and get some things wrong. But I don't like this proposal on a conceptional kind.

1. Bitcoin Unlimited should not be a hardfork-only project. If some of you think it should be, to sharpen the contrast to softfork-only-core, I'd like to see a BUIP (and vote against. Imo hardforks and softforks have, as all the tools, their own advantages and disadvantages. You write about it in your extension point BUIP. Since Core spent a year of work and testing with several developers on building SegWit as a SoftFork I think we need serious reasons, why it must be done as a hard- instead of a softfork, to spent time and money to duplicating Core's work (and somehow insulting some members of the team we maybe don't want to insult).

2. Imo SegWit is a way bigger change than BUIP001 or any other increase of blocksize-capacity. The latter just changes a variable to limit the pace of the growth of the database of transactions, while SegWit alters the transaction format itself.
Doing the needed capacity increase as a softfork does seriously limit its scope while making the whole endeavour more complicated, what should make the hardfork the better tool to achieve this. I'm not sure it is so with SegWit. That it doesn't effect the existing UTXO set is imo more an advantage than a disadvantage, as it leaves the choice to use the old or the new transaction format. As a first reaction, I like SegWit as a hardfork less than as a softfork, and I think the steps for actors of the ecosystem to do it as a hardfork is bigger than as a softfork; most likely you would just waste time for another upgrade that is never implemented.

3. If you want to go with this proposal, you should spent more work to elaborate why it should be done as a hardfork and not as a softfork. Also you should elaborate the goal, activation frame and politics.

4. Imo it would be a better investment of time / funds to make a in-depth review of Core SegWit-Softfork. As I said earlier, I think it should be a good way forward to make the activation of SegWit dependent on the activation of Unlimited, something like "activate SegWit if 75% of miners vote of AB1.1" or "if blocks > 1.0 MB". A deep review of SegWit could also help to formulate conditions how to make it a better part of Unlimited's roadmap. For example if we use Unlimited the alleged scaling advantage of SegWit would not be needed and it can pass as nothing more than an enhancement of the transaction format which enables fraud proofs and lightning and other things. There can even be a discssion of the discount of Signatures and its (right now very controversial) effect on growth of UTXO-set and Blockchain-size (which are inverse). This could be a wise step to reach network-wide consensus.

 

[adamstgbit]

i like this. but question..

segwit softfork changes blocksize to "block weight" how does this segwit proposal effect blocksize?
[doublepost=1484498014,1484497251][/doublepost]@Christoph Bergmann

look what happening with the segwit softfork now, they are waiting for 95% backing.
what's the point of adding code debt which allows old nodes to SORTA keep working, when you require 95% to upgrade anyway?
[doublepost=1484498709][/doublepost]one more question / comment

i think this :

keeps thing close to SegWit to minimize sunk cost for actor supporting it.

is a huge bonus

but your use of flex trans, makes it impossible for your segwit TX formate to be compatible with the core-segwit.

is it possible that we perfectly mimic core-segwit-TX format? and keep most of the HF advantages

I'd want to make it as easy as possible for implementations that already support segwit softfork, to make the switch to supporting segwit hardfork.

 

[Christoph Bergmann]

Miners as full and paid participants of the network have other duties to diligence than normal nodes which purpose is to verify payment and control money creation (you can compare non-mining-nodes with medieval coin balances of traders). It's way easier to concertize miners than miners + all nodes.

 

[deadalnix]

i like this. but question..

segwit softfork changes blocksize to "block weight" how does this segwit proposal effect blocksize?
[doublepost=1484498014,1484497251][/doublepost]@Christoph Bergmann

It doesn't do anything about block size. If it did, that'd be specified. SegWit never was about block size until luke-jr figured out it could make it as a soft fork by using a block extension - which they then decided to sell as a block size increase. And everybody at core cheered, but everybody else was left wondering who the fuck their dealer was.

look what happening with the segwit softfork now, they are waiting for 95% backing.
what's the point of adding code debt which allows old nodes to SORTA keep working, when you require 95% to upgrade anyway?

It is a hard fork, so I don't think this questions makes any sense.

but your use of flex trans, makes it impossible for your segwit TX formate to be compatible with the core-segwit.

Indeed, to be 100% compatible, I'd have to also include the parts of SegWit we don't want.

is it possible that we perfectly mimic core-segwit-TX format? and keep most of the HF advantages

I'd want to make it as easy as possible for implementations that already support segwit softfork, to make the switch to supporting segwit hardfork.

That wouldn't be the right tradeof IMO. Some parts of the SF SegWit are just soft fork kludge, such as the fact that you require an input script, but it's empty because data are in the witness fields, or the fact that output script must look like everyone can spend to old nodes - or that wouldn't be a soft fork. None of this is desirable to keep.

In addition, SegWit format is not very extensible, because that'd be a hard fork anyway and they don't want this. If we are to introduce a new format, it needs to be extensible. I don't think upgrading using soft fork and hard fork is a very happy situation, so making this extensible using the mechanism used in BUIP039 is a huge plus. We'd have a sane path forward to upgrade the protocol.
[doublepost=1484502146][/doublepost]@Christoph Bergmann
1. The BUIP process is here to decide what BU is. What you think BU should be is irrelevent to the merit of this BUIP.
2. I'm not sure there is anything of substance in there. I'm afraid what you like and don't like is not a very good argument in general.
3. SegWit is 4 BIPs. That wouldn't be reasonable to go over all this in this proposal. You can find a good sum up on SegWit and its shortcoming here: https://medium.com/the-publius-letters/segregated-witness-a-fork-too-far-87d6e57a4179 . You get the same benefit without the shortcomings doing as a hard fork - but, obviously, that's a hard fork, which some would argue is a disadvantage.
4. See 3. No need to redo the work which is already done.

 

[Christoph Bergmann]

1. right. Since there is no BUIP saying "allways hard-, never softfork" you should explain why the cost of a hardfork and the somehow redundant work is worth it. And I think the resistance against SWSF is not because the SF part but because that it is sold as a Scaling Solution.
2. explained that it is (all we know by know) not possible to scale onchain with a softfork, so hardforking is out of question for this. With SegWit it is different. Anyway ... what I like may be no argument for you, but a preview of my voting ...
3. Thank you, great read, will go through it again. It would be better to have something like this done by you and tailored for the HF/SF question.
4. This is enough? Is there some internal discussion of BU to accept this?

My key argument is untouched: you need to explain that a HF has enought advantages to legitimate the redundant work, the political trouble and the uncertainty if it ever gets accepted.

 

[deadalnix]

I don't want to be harsh, but you seem to have an opinion while you clearly lack the background to understand the problem, as your question reveals. I suggest you proceed in order, document yourself, learn about the problem, evaluate the solution and then form an opinion, rather than starting with your opinion and working back from there.

Please also consider that this BUIP is a specification fo what is proposed, not a tutorial on SegWit. It is not meant to be a tutorial on SegWit and won't be morphed into one.