status-update

The state of the codebase is slowing approaching the state in which block production might be finally possible. Exciting!

This week I continued working with the client database and integrating it closer with the rest of the node. A key integration point that was missing completely and still not implemented was state management. The core parts of the consensus do not involve state management, but transaction processing will. So I was considering various ways to process transactions and came up with an idea I’ll be pursuing that should work nicely, but is also a bit unlike most of the blockchains out there.

The biggest update since the last blog post is that an initial prototype of the database was merged. It lays the foundation in terms of fundamental architecture and will now be extended to support more features. There were also updates in a few other areas.

After adventures with rust-gpu, which I still monitor periodically, I moved on to the client database implementation, which is required for proper blockchain operation, and which is one of the bigger undertakings. Unfortunately, the database as such isn’t quite ready yet, but I did some preparation and would like to share some details about the database architecture.

This week has been mainly focused on refining a bit the design for plot identification and sector expiration. I think that I finally have a model with which I am comfortable with, and that I think solves all of our previous problems. Nazar had this idea to drastically simplify how plot IDs were derived, and how sectors were linked to plots. The high-level idea made sense, but there were still some details that weren’t clear. This week I managed to come up with a design that I think satisfies all of our requirements.

I have to admit that I am a bit disappointed with my progress this week. If you recall from last week’s update, I started the week with a base proposal to handle the linking of sectors to plots based on history window ranges. The idea was to limit the number of parallel sectors that could be created in parallel, linked to the same plot, and hence allocated to the same shard. While the approach seemed quite elegant because it didn’t require any changes to how piece selection and expiration currently works in Subspace, it turned out to be pretty complicated (even to explain) and not the most effective solution to prevent the attack I was trying to mitigate.

GPU plotting was one of the items on the roadmap last week and that turned into a week+ long side quest, so let me share some details about that.

As mentioned on last week’s status update, one of the key pieces that I was missing to have the detailed operation of plot membership allocation was the impact of sector expiration on the protocol. By having a unique plot identifier, we are able to uniquely link sectors to plots, but these sectors need to expire in a way that does not require farmers to re-plot while archiving the most recent history. Fortunately, we can leverage the current expiration mechanism of the Subspace protocol, and build a layer on top of it to adapt it to the sharded version while maintaining the original guarantees in terms of plot expiration, sector re-plotting, and history archiving.

Last week I shared a model that can help us reason about the security of shards assuming an honest majority in the beacon chain. The model evaluates what are the trade-offs in terms of the number of shards, the number of farmers per shard, and the proportion of malicious farmers in the system. But if you recall from the overall design of the system, shards are periodically submitting segments and blocks to the upper layers of the hierarchy and to the beacon chain. We need to verify that these are valid, available, and correctly encoded before they are included in super segments and the global history of the system. Can we do so without relying on fraud proofs? This has been one of my focuses for the week, let’s jump right into it.

This was a lighter week on meaningful changes, but there are still few things to share. First of all, last week’s block import was (and still is) and incomplete prototype, but this week it was extended to become a bit more complete. I was also working on bringing up more components frm Subspace, including farmer, which lead me to attempt GPU plotting in Rust, so let’s get into it.

This week has been all about objectively assessing the security of the system. After all of the work around the membership allocation protocol and its security there are still two questions that we need to answer to understand the feasibility of the protocol: (i) what is the security bound of the protocol as a whole (from beacon chain to shards), and (ii) how can we ensure that plots are uniquely identified and that farmers cannot cheat by committing the same plot to different history sizes to try and game the shard allocation mechanism.

There was no update last week again since I didn’t feel like there was enough to share at the time, but now that more things have settled I’d like to share what I’ve been busy with.

Last week I shared a high-level of how I was thinking farmer membership selection should work. After some discussions early in the week, we realised there were still some blind spots and attacks that we weren’t protecting against (or if we were, we didn’t have an objective measure of how robust they were). Thus, this week has been exclusively focused on modeling the membership selection protocol so we can reason objectively about its design.

There have not been an update from me last week, how come? Well, it didn’t seem like there was anything particularly substantial to share, mostly some refactoring, so I decided to skip it. This week though I have a few words to say about the path towards block production (not there yet, but getting closer). Some this is a boring process, I procrastinated some too, diving into various topics with some interesting performance improvements and new learnings.

This week has been another of those weeks where I am pretty happy with the progress made. The highlights of the week are the following: (i) we now have a pretty good sense of the end-to-end operation to commit shard segments into the global history of the beacon chain (as described in the discussion of PR267); (ii) and Nazar had an idea to tackle the verification of the availability and correctness of shard segments without requiring an independent data availability mechanism, by leveraging the longest-chain rule and farmers membership allocation (which was an issue that was really bugging me). Let’s jump into the details of these two topics.

I started the week thinking about the mechanics of shard segment commitment into the global history of the beacon chain. If you recall from previous updates, we already had a pretty good idea of how the information about child shard segments flow up to the beacon chain, but there were still a few questions that were really bugging me.

The question might seem somewhat obvious: you have a header and a body with transactions, many blockchains have it, what might be so difficult about it? Well, as I mentioned in the previous update, there are some complications and part of the challenge is related to the fact that we’re dealing with sharded architecture, which most blockchains don’t need to deal with.

I am pretty happy with the progress this week. Funnily, the main culprit for all of this progress has been the the work that Nazar has been doing on the definition of the block structure for our hierarchical consensus. I’ll let Nazar dig deeper into what he’s been doing here, but let me share in this post what this block structure entails, and how this has unblock several lines of work, and solved many issues for me.

This week has been mainly focused on clearing the fog around shard archiving and trying to start fleshing the low-level details for the protocol. It feels like in the past few weeks we’ve been surfacing more questions than answers, and I honestly think this is a good sign. It means that we are getting to the point where we can start to see the details of the protocol and how it will work in practice.

The discussions with Alfonso and attempts to start building an actual blockchain led to spending a big part of last week working on block structure. That work is not done yet. However, I like to have some sort of accomplishment at the end of the week if possible, so I spent some time to finally implement the formatting of addresses, which will be the main part of this update.

This week was very similar to the last one with a bunch of refactoring in cleanups. There were important archiver improvements/fixes (depends on point of view) and more work on Merkle Trees. Two more crucial crates were moved from subspace to crates.

We keep iterating on the best way to discuss and make progress on the design of the protocol. Using issues for discussions have shown less efficient than originally expected. The inability to make in-line threads, and having to quote every single detail of the spec that we want to discuss about was really cumbersome. I started the shard block submission issue as an attempt to start iterating the low-level details of specific protocol mechanisms in a way that is narrow enough and easy to track, but it didn’t fulfill all our needs. The solution? Creating discussion PRs that I don’t expect to get merged, but gives us all that we need to have low-level discussions about specific parts of the protocol, track our progress, open ideas, and discussions, and have them public so anyone can contribute or follow along.

The last week was lighter on major changes, but there was a lot of cleanups and refactoring done to prepare Subspace components reuse for building a new blockchain from scratch. Also some improvements based on new developer feedback.

We started last week with two PRs that attempted to describe in detail the operation of sharded archiving (PR192), and the data availability layer of the system (PR193). When I started writing this spec, it was meant to be for a broader audience, but we realised after a few rounds of feedback that the project is still in a really early stage and in constant change, so it would be more efficient to focus on detailing the parts of the protocol that are currently under-defined instead of trying to give a deep overview of the overall operation of the protocol from the get-go. The actual goal behind this protocol specification is to unblock the implementation of a prototype that can help us gain certainty about the design decisions that we are making, and surface potential blind spots in the design, and not to have a reference spec (just yet).

Over the past weeks, my updates have highlighted many of the ideas emerging from our open design discussions. Now that we have a clearer direction for the design, I wanted to consolidate these ideas into a draft spec. This will serve as a foundation for implementing the first few prototypes, while also providing a structured way to gather feedback and uncover potential blind spots. I expect this spec to suffer significant changes, but it felt like the perfect way to consolidate the ideas, get feedback from the community, and unblock Nazar in case he wants to start prototyping some of the ideas we’ve been discussing.

Last time I mentioned that I was looking into Merkle Trees to replace KZG. This week it happened, the whole codebase is basically free from KZG. The only place that is not fully fixed and where I am looking for help is GPU plotting, it broke with all these changes and isn’t the highest priority for me to fix right now.

Last week was lighter on code changes and more heavy on research. Specifically, I’ve been looking into commitment schemes generally and Blake3 hash function in particular, which was already used in the codebase, but turns out can be applied in more interesting ways than just a hash function.

This week has been another good week of progress. I finally have a good idea of how shard archiving should work in the happy path, and I’ve started writing a low-level spec for it (that I am hoping to push to this repo soon). Unfortunately, there is still a slight gap in the spec that we need to fill before we can move forward: the data availability problem.

The majority of last week I spent tinkering with Subspace codebase after importing it here in preparation for building an actual blockchain.

This week I’ve gone a bit deeper into the design of the multi-shard Subspace protocol idea which I briefly introduced in my last update. The protocol is conformed by the following parts: Sharded archiving, responsible for creating a global canonical history of the whole system, and of creating the history records that will eventually become part of farmers’ plots. Sharded plotting, which takes records from the global history and seals them in plots that include segments of the history of every shard of the system, and that will be used for the farming process. And finally, merged farming, which is the protocol responsible for challenging farmer plots, and deriving the corresponding winning tickets that elect block proposers in specific shards. Let me introduce the high-level operation behind each of these sub-protocols, while digging deep in the one that I’ve focused the most on this week: sharded archiving.

After a lot of thinking, this week I came to the realisation that a sharded architecture like the one we are trying to build can be designed leveraging the current design of the Subspace protocol and all its underlying mechanisms as a base. While this was the idea from the beginning, either for lack of familiarity with the protocol or plain ignorance, I was missing the big picture of how this could be done.

The majority of the last two weeks I’ve been busy with the installation of the antivirus system update for my immune system. It was neither pleasant nor quick, but now that it is slowly approaching 100%, I’m back with another update of what I managed to do since the last update.

Overall, I am really happy with the progress I’ve made this week. I’ve been mainly focused on unravelling one of the papers that I mentioned in my last update, Proof-of-Stake Sidechains. While is true that we don’t want our system to have anything to do with PoS, and there is no 1:1 matching of the concepts from the paper with what we are trying to build, the paper presents a framework that can come pretty handy to evaluate the correctness of our designs. This paper is from 2018, and after a first pass the first thing that I did is to check if there were any follow-up papers that built upon the abstractions of this paper. I came across Proof-of-Work Sidechains from the same authors, but without a doubt, the most complete proposal is the one that I started with. Let’s dive right into it.

The plan was to get to transaction pool implementation, but it didn’t quite happen. I did a lot of investigation around performance though. For example, transaction processing was several orders of magnitude slower than direct method calls without a transaction, which concerned me, but after optimizations of last week the difference is ~10x. And it makes sense given how much more work the wallet has to do on top of the method call itself.

I want to kick-off my first weekly update in the project thanking Nazar for the warm welcome and the opportunity to work with him on this exciting project. I was really pumped to see other teams actively working on a similar problem to the one I started researching more than three years ago. For several reasons, I wasn’t actively contributing to this problem any more, but this opportunity was the perfect excuse to get back to the game.

The big change from the last update is that Alfonso de la Rocha has joined me as a part-time researcher to help with sharding designing. Code-wise, there were also a bunch of performance benchmarks and optimizations.

The most important progress from last week is initial work on transactions. I’ve spent quite some time thinking about the design and even implemented an initial wallet contract alongside with related infrastructure.

It was a challenging week working on storage access checks for slots, but it is over, and I’m quite happy with how things are looking right now. Some extra refactoring also allowed running tests under Miri and spotted some things that violate the Rust safety rules.

Last week felt a bit less productive with a lot of time spent thinking about how to approach slots conflict resolution in the native execution environment, but still managed to land a few improvements, especially on the documentation side. Also conducted four separate interviews.

After a lot of refactoring and preparation, native execution environment is finally functional and can be used for purposes like writing tests and debugging.

Last week was busy with refactoring with the primary goal of being able to run contracts in test execution environment. The environment is not quite ready yet, but a lot of progress was done, and it’ll hopefully be ready next week.

Last week was busy on various improvements for contracts infrastructure, trying to clarify existing API and ensuring everything that might be built is actually possible. First system contracts were introduced, trait support was added and more, below is a recap of key updates.

Hello, world 👋! This is the beginning of hopefully successful thing I call “Project Abundance”. After writing initial set of constraints and thinking about it for quite a while, it is finally time to dedicate all my time to it and see where it leads us.