adlrocha

Last week I shared a model that can help us reason about the security of shards assuming an honest majority in the beacon chain. The model evaluates what are the trade-offs in terms of the number of shards, the number of farmers per shard, and the proportion of malicious farmers in the system. But if you recall from the overall design of the system, shards are periodically submitting segments and blocks to the upper layers of the hierarchy and to the beacon chain. We need to verify that these are valid, available, and correctly encoded before they are included in super segments and the global history of the system. Can we do so without relying on fraud proofs? This has been one of my focuses for the week, let’s jump right into it.

This week has been all about objectively assessing the security of the system. After all of the work around the membership allocation protocol and its security there are still two questions that we need to answer to understand the feasibility of the protocol: (i) what is the security bound of the protocol as a whole (from beacon chain to shards), and (ii) how can we ensure that plots are uniquely identified and that farmers cannot cheat by committing the same plot to different history sizes to try and game the shard allocation mechanism.

Last week I shared a high-level of how I was thinking farmer membership selection should work. After some discussions early in the week, we realised there were still some blind spots and attacks that we weren’t protecting against (or if we were, we didn’t have an objective measure of how robust they were). Thus, this week has been exclusively focused on modeling the membership selection protocol so we can reason objectively about its design.

This week has been another of those weeks where I am pretty happy with the progress made. The highlights of the week are the following: (i) we now have a pretty good sense of the end-to-end operation to commit shard segments into the global history of the beacon chain (as described in the discussion of PR267); (ii) and Nazar had an idea to tackle the verification of the availability and correctness of shard segments without requiring an independent data availability mechanism, by leveraging the longest-chain rule and farmers membership allocation (which was an issue that was really bugging me). Let’s jump into the details of these two topics.

I started the week thinking about the mechanics of shard segment commitment into the global history of the beacon chain. If you recall from previous updates, we already had a pretty good idea of how the information about child shard segments flow up to the beacon chain, but there were still a few questions that were really bugging me.

I am pretty happy with the progress this week. Funnily, the main culprit for all of this progress has been the the work that Nazar has been doing on the definition of the block structure for our hierarchical consensus. I’ll let Nazar dig deeper into what he’s been doing here, but let me share in this post what this block structure entails, and how this has unblock several lines of work, and solved many issues for me.

This week has been mainly focused on clearing the fog around shard archiving and trying to start fleshing the low-level details for the protocol. It feels like in the past few weeks we’ve been surfacing more questions than answers, and I honestly think this is a good sign. It means that we are getting to the point where we can start to see the details of the protocol and how it will work in practice.

We keep iterating on the best way to discuss and make progress on the design of the protocol. Using issues for discussions have shown less efficient than originally expected. The inability to make in-line threads, and having to quote every single detail of the spec that we want to discuss about was really cumbersome. I started the shard block submission issue as an attempt to start iterating the low-level details of specific protocol mechanisms in a way that is narrow enough and easy to track, but it didn’t fulfill all our needs. The solution? Creating discussion PRs that I don’t expect to get merged, but gives us all that we need to have low-level discussions about specific parts of the protocol, track our progress, open ideas, and discussions, and have them public so anyone can contribute or follow along.

We started last week with two PRs that attempted to describe in detail the operation of sharded archiving (PR192), and the data availability layer of the system (PR193). When I started writing this spec, it was meant to be for a broader audience, but we realised after a few rounds of feedback that the project is still in a really early stage and in constant change, so it would be more efficient to focus on detailing the parts of the protocol that are currently under-defined instead of trying to give a deep overview of the overall operation of the protocol from the get-go. The actual goal behind this protocol specification is to unblock the implementation of a prototype that can help us gain certainty about the design decisions that we are making, and surface potential blind spots in the design, and not to have a reference spec (just yet).

Over the past weeks, my updates have highlighted many of the ideas emerging from our open design discussions. Now that we have a clearer direction for the design, I wanted to consolidate these ideas into a draft spec. This will serve as a foundation for implementing the first few prototypes, while also providing a structured way to gather feedback and uncover potential blind spots. I expect this spec to suffer significant changes, but it felt like the perfect way to consolidate the ideas, get feedback from the community, and unblock Nazar in case he wants to start prototyping some of the ideas we’ve been discussing.