Blog

This week has been mainly focused on clearing the fog around shard archiving and trying to start fleshing the low-level details for the protocol. It feels like in the past few weeks we’ve been surfacing more questions than answers, and I honestly think this is a good sign. It means that we are getting to the point where we can start to see the details of the protocol and how it will work in practice.

The discussions with Alfonso and attempts to start building an actual blockchain led to spending a big part of last week working on block structure. That work is not done yet. However, I like to have some sort of accomplishment at the end of the week if possible, so I spent some time to finally implement the formatting of addresses, which will be the main part of this update.

This week was very similar to the last one with a bunch of refactoring in cleanups. There were important archiver improvements/fixes (depends on point of view) and more work on Merkle Trees. Two more crucial crates were moved from subspace to crates.

We keep iterating on the best way to discuss and make progress on the design of the protocol. Using issues for discussions have shown less efficient than originally expected. The inability to make in-line threads, and having to quote every single detail of the spec that we want to discuss about was really cumbersome. I started the shard block submission issue as an attempt to start iterating the low-level details of specific protocol mechanisms in a way that is narrow enough and easy to track, but it didn’t fulfill all our needs. The solution? Creating discussion PRs that I don’t expect to get merged, but gives us all that we need to have low-level discussions about specific parts of the protocol, track our progress, open ideas, and discussions, and have them public so anyone can contribute or follow along.

The last week was lighter on major changes, but there was a lot of cleanups and refactoring done to prepare Subspace components reuse for building a new blockchain from scratch. Also some improvements based on new developer feedback.

We started last week with two PRs that attempted to describe in detail the operation of sharded archiving (PR192), and the data availability layer of the system (PR193). When I started writing this spec, it was meant to be for a broader audience, but we realised after a few rounds of feedback that the project is still in a really early stage and in constant change, so it would be more efficient to focus on detailing the parts of the protocol that are currently under-defined instead of trying to give a deep overview of the overall operation of the protocol from the get-go. The actual goal behind this protocol specification is to unblock the implementation of a prototype that can help us gain certainty about the design decisions that we are making, and surface potential blind spots in the design, and not to have a reference spec (just yet).

Most blockchain implementations are pieces of software that include the logic to support many different roles/features, possibly all at once: bootstrap node, block producer, RPC node, archival node, light client, etc. That is one way to do it, but one thing I learned over the years is that you can do a lot of interesting optimizations if you can apply additional constraints during the design phase. So why is basically everyone trying to combine all into one? Let’s consider different roles separately first and see what is special or interesting about them.

Over the past weeks, my updates have highlighted many of the ideas emerging from our open design discussions. Now that we have a clearer direction for the design, I wanted to consolidate these ideas into a draft spec. This will serve as a foundation for implementing the first few prototypes, while also providing a structured way to gather feedback and uncover potential blind spots. I expect this spec to suffer significant changes, but it felt like the perfect way to consolidate the ideas, get feedback from the community, and unblock Nazar in case he wants to start prototyping some of the ideas we’ve been discussing.

Last time I mentioned that I was looking into Merkle Trees to replace KZG. This week it happened, the whole codebase is basically free from KZG. The only place that is not fully fixed and where I am looking for help is GPU plotting, it broke with all these changes and isn’t the highest priority for me to fix right now.

Last week was lighter on code changes and more heavy on research. Specifically, I’ve been looking into commitment schemes generally and Blake3 hash function in particular, which was already used in the codebase, but turns out can be applied in more interesting ways than just a hash function.