Block building
And with that result, we smoothly move into the next stage in Bitcoin's operation, which is block building. The bunches a.k.a. blocks need to be build! This is where the miner selects transactions. Where the economics happens. Where stuff is rejected or accepted based on fees. A huge and without questions in itself alone quite interesting playing field where I am sure miner's like
@jtoomim can say much more about the nitty gritty details than I ever will.
But one observation I like to point out here: A miner has to "select a transaction set", build a block, that is, which will be accepted by other miners as
valid.
And valid means that it extends the current state of the chain with a new set of transactions
under the invariant of partial ordering.
Given that in the previous part, memory pool ingress, the partial ordering had to be checked anyways, it becomes
relatively easy for the miner to select transactions now that will form such a set. Most of the work has already been done. He can select from the partially ordered mempool and only needs to make sure that he doesn't create any gaps in the transaction strings.
I like to observe that he cannot simply select transactions by lexical order across some UTXO shards. They are interdependent.
He has to honor the partial order criterion at this stage.
So he arrives at a block, a "transaction set" for the lexical ordering proponents, which he now wants to communicate with his peers.
Block transception
This is the area where, it seems, the "canonical" ordering arguments originated and then got
unduly applied to the other main areas of Bitcoin as described above. But after more consideration, I think they only really apply to this area. [Minor remark aside: I have been accused of being ignorant of the state of the art in distributed systems and that I might please want to talk to a computer science professor about all this who will explain the obviousness of all what they propose to me. Apart from noting that this is an appeal to authority, I really like to have a CS professor point out to me where I am going wrong in
any of this. CS profs, please get to your keyboards!
I might see the light then. But that's the point, isn't it? I might be just an ignorant BU member, but I think I am not a complete fool. I have enough technical knowledge to understand the big picture, I believe. The saying of "If you cannot explain it to a 5 year old, you haven't understood it", applies as far as I can see. Let me be this 5-year old repeatedly asking "why is that?" and tickling those with superior knowledge enough that they come down to us and explain it in layman's terms.]
To get my block, my "transaction set" to a peer, I have to somehow encode it into a string of bytes, send it out on my network link and have the peer on the other side disassemble it and arrive at the same data structure as I did.
And with the OTI validation scheme, I can do the validation on the other end in any order that I receive the 'transactions' in, at least as far as the question of transaction validation goes.
I also have to validate the structure of a block, and here's where the trouble starts.
Besides obvious needs that are not removed or touched by transaction ordering (having a valid merkle tree, for example, and having not too many bytes for those 1MB-lovers), lexical ("canonical") ordering intends to change the ordering in the block validity from following the topological to following the lexical order.
The proponents argue here: By ordering it lexicographically, I can throw away a lot of information that I would otherwise have to transmit to my peer, which makes block transmission less efficient.
But it should be pointed out that this is,
as-is, very much a red herring. Gavin Andresen's two-step sorting scheme as
@Peter R. referenced it above would allow the same, while still leaving the block topologically ordered. As far as I can see, Gavin's scheme as above has a bit of a specification bug (it doesn't quite work the way it is described, at least how I read it), but it can be fixed. What you arrive at, then, is, as far as I as a non-CS person can see, a simple sort along a certain axis (he proposed the minimum of (input-txid, index) ), followed by a topological sort by Kahn's algorithm.
(Let me remark that I also like to avoid using the word "canonical ordering" for yet another reason. The earliest so-called canonical ordering was named like that AFAIR by Gavin Andresen for his initial IBLT propagation proposal. The current use is a redefinition of terms which we should avoid IMO.)
The complexity of this is an O(n log n) lexical sorting step, followed by an O(n) (in terms of block size) topological sorting step, giving you O(n log n) overall complexity.
Ok. Now, here it becomes murky and, given the lack of data, makes many of us argue from a sense of
gut-feeling. Let me still light up the following area as best as I can, as I believe there's still analysis and categorization that can be done that helps to form a clearer view.
So, first, it should be noted that Gavin's approach will destroy the existing partial order (with the sorting step) and then rebuild it with Kahn's algorithm (see also below on more thoughts on that).
On the receiving side, with Gavin's proposal, I can use (pretty much) any scheme to validate. I can use the old, sequential validation (as long as my H/W can keep up). Or I can use OTI. Or other schemes. I cannot use a scheme that would
depend on lexical order, but I haven't seen such a one. If you have one that is efficient, point it out!
I have demonstrated that there can be, in principle, advantages to validation by OTI when the topological order is kept:
https://github.com/awemany/valorder
I also like to remark that we have an interesting situation here: The receiving node could of course recreate the topological order as well. From any order. So I could also go and do the proposed lexical ordering and then resort to topological in blocks.
To view this in a different light, I think we basically have to define a "cut point" which is the amount of ordering or lack there of we do when we create a block. And there, a lot of scenarios that can be imagined with the trade-offs that I can see as they have been discussed:
(SEQ meaning variants of sequential validation)
sort lexical -> [BLOCK] -> OTI
(Work on the sender, efficient transmission, potentially less efficient validation, only OTI. Needs fork.)
sort lexical -> [BLOCK] -> resort topological -> OTI, SEQ
(Work on the sender, efficient transmission, work on the receiver, unlikely to be a competetive contender. Needs fork.)
This is the scenario that I meant when I said we might start building 'adapters' from lexical to topological ordering and vice versa into Bitcoin.
keep topological -> [BLOCK] -> OTI, SEQ
(No work on the sender, any validation possible on the receiver, less efficient transmission. The current situation. Needs no fork.)
sort into two sets (Tom Zander) -> [BLOCK] -> OTI, SEQ
(Work on the sender, any validation scheme possible on the receiver, relatively efficient transmission. Needs no fork but changes in the mining and network code.)
sort Gavin-canonical -> [BLOCK] -> OTI, SEQ
(Work on the sender, any validation scheme on the receiver, efficient transmission. Needs no fork but changes in the mining and network code.)
And for the unknown unknowns (as least to the best of my knowledge):
sort ??? -> [BLOCK] -> OTI, SEQ
(Maybe less work on the sender, any validation scheme on the receiver, efficient transmission, needs no fork but changes in the mining and network code.)
To explain this last part and why I am proposing, apart from status quo arguments, to keep it all as it is for now:
Yes, Gavin's algorithm resorts
everything. This makes it the same complexity as lexical ordering in big-O notation, which is O(n log n). However, it realistically adds another step (topological sorting) which likely makes it slower on the sending side compared to pure lexical.
Now it might, on the receiving side, regain these losses by yielding faster validation.
It throws away information, only to regain it.
What I am meaning with 'sort ???' is that I feel that there are unknown unknowns that might explored and which could connect the left and the right side of [BLOCK] in the best manner.
This is an area, however, where I really think input from computer science experts is needed.
One of my questions here would be: Is it possible to
post-sort, to update a partially ordered set in an efficient manner into a unique presentation?
I mean this in the sense of: The set is already partially ordered. Can this be exploited? Does a sorting algorithm exist (or can one be developed) that uses the existing information and throws away exactly that information (the potential reorderings) that would be necessary for block transmission without extra order encoding bits?
I think this is one of the key questions to be answered here.
EDIT: The above ideas on what needs to happen there are actually inaccurate and I think they need to be more detailed - there's going to be a reconstruction sorting that has to happen in any case and then there's also the question of further sorting required (or not) for the merkle tree. A lot of questions remain.