nazar-pc

In the part 1 I shared some background information, performance improvements and future opportunities. Since then, I was pursuing various approaches. Some worked out nicely, others were not so fruitful. Overall, I have achieved a substantial performance improvement on CPU with a few more options still remaining on the table, all while becoming substantially more GPU-friendly.

In the last update I shared that I plan to work on GPU plotting some more, so that is what I did. The “easier” parts of it were done earlier. Now it was time for matching logic and that is more complex, so I decided to dedicate the whole blog post to it.

The state of the codebase is slowing approaching the state in which block production might be finally possible. Exciting!

This week I continued working with the client database and integrating it closer with the rest of the node. A key integration point that was missing completely and still not implemented was state management. The core parts of the consensus do not involve state management, but transaction processing will. So I was considering various ways to process transactions and came up with an idea I’ll be pursuing that should work nicely, but is also a bit unlike most of the blockchains out there.

The biggest update since the last blog post is that an initial prototype of the database was merged. It lays the foundation in terms of fundamental architecture and will now be extended to support more features. There were also updates in a few other areas.

After adventures with rust-gpu, which I still monitor periodically, I moved on to the client database implementation, which is required for proper blockchain operation, and which is one of the bigger undertakings. Unfortunately, the database as such isn’t quite ready yet, but I did some preparation and would like to share some details about the database architecture.

There are buzzwords in any industry that are thrown around easily, and blockchains are no exception. In this post, I want to focus on “scalability”. Turns out when you say “blockchain scalability” different people hear different things. The prevalent opinion seems to be that scalable blockchains are able to process more transactions than non-scalable ones or something along those lines. Essentially making the ability to scale equivalent to peak performance. Sure, peak performance is an important metric, although it is often a theoretical one. But I don’t think that is the most useful property, especially without clarifying the conditions under which it can be achieved.

GPU plotting was one of the items on the roadmap last week and that turned into a week+ long side quest, so let me share some details about that.

This was a lighter week on meaningful changes, but there are still few things to share. First of all, last week’s block import was (and still is) and incomplete prototype, but this week it was extended to become a bit more complete. I was also working on bringing up more components frm Subspace, including farmer, which lead me to attempt GPU plotting in Rust, so let’s get into it.

There was no update last week again since I didn’t feel like there was enough to share at the time, but now that more things have settled I’d like to share what I’ve been busy with.

There have not been an update from me last week, how come? Well, it didn’t seem like there was anything particularly substantial to share, mostly some refactoring, so I decided to skip it. This week though I have a few words to say about the path towards block production (not there yet, but getting closer). Some this is a boring process, I procrastinated some too, diving into various topics with some interesting performance improvements and new learnings.

The question might seem somewhat obvious: you have a header and a body with transactions, many blockchains have it, what might be so difficult about it? Well, as I mentioned in the previous update, there are some complications and part of the challenge is related to the fact that we’re dealing with sharded architecture, which most blockchains don’t need to deal with.

The discussions with Alfonso and attempts to start building an actual blockchain led to spending a big part of last week working on block structure. That work is not done yet. However, I like to have some sort of accomplishment at the end of the week if possible, so I spent some time to finally implement the formatting of addresses, which will be the main part of this update.

This week was very similar to the last one with a bunch of refactoring in cleanups. There were important archiver improvements/fixes (depends on point of view) and more work on Merkle Trees. Two more crucial crates were moved from subspace to crates.

The last week was lighter on major changes, but there was a lot of cleanups and refactoring done to prepare Subspace components reuse for building a new blockchain from scratch. Also some improvements based on new developer feedback.

Most blockchain implementations are pieces of software that include the logic to support many different roles/features, possibly all at once: bootstrap node, block producer, RPC node, archival node, light client, etc. That is one way to do it, but one thing I learned over the years is that you can do a lot of interesting optimizations if you can apply additional constraints during the design phase. So why is basically everyone trying to combine all into one? Let’s consider different roles separately first and see what is special or interesting about them.

Last time I mentioned that I was looking into Merkle Trees to replace KZG. This week it happened, the whole codebase is basically free from KZG. The only place that is not fully fixed and where I am looking for help is GPU plotting, it broke with all these changes and isn’t the highest priority for me to fix right now.

Last week was lighter on code changes and more heavy on research. Specifically, I’ve been looking into commitment schemes generally and Blake3 hash function in particular, which was already used in the codebase, but turns out can be applied in more interesting ways than just a hash function.

Welcome post mentioned briefly initial set of constraints that led to the creation of this project, but I figured it might be helpful to have a short writeup about it that might be helpful for sharing. In short: we’re building a blockchain. By “we” I really mean just me and Alfonso so far, but I hope more people will join over time if they find it interesting.

The majority of last week I spent tinkering with Subspace codebase after importing it here in preparation for building an actual blockchain.

The majority of the last two weeks I’ve been busy with the installation of the antivirus system update for my immune system. It was neither pleasant nor quick, but now that it is slowly approaching 100%, I’m back with another update of what I managed to do since the last update.

The plan was to get to transaction pool implementation, but it didn’t quite happen. I did a lot of investigation around performance though. For example, transaction processing was several orders of magnitude slower than direct method calls without a transaction, which concerned me, but after optimizations of last week the difference is ~10x. And it makes sense given how much more work the wallet has to do on top of the method call itself.

The big change from the last update is that Alfonso de la Rocha has joined me as a part-time researcher to help with sharding designing. Code-wise, there were also a bunch of performance benchmarks and optimizations.

The most important progress from last week is initial work on transactions. I’ve spent quite some time thinking about the design and even implemented an initial wallet contract alongside with related infrastructure.

It was a challenging week working on storage access checks for slots, but it is over, and I’m quite happy with how things are looking right now. Some extra refactoring also allowed running tests under Miri and spotted some things that violate the Rust safety rules.

Last week felt a bit less productive with a lot of time spent thinking about how to approach slots conflict resolution in the native execution environment, but still managed to land a few improvements, especially on the documentation side. Also conducted four separate interviews.

After a lot of refactoring and preparation, native execution environment is finally functional and can be used for purposes like writing tests and debugging.

Last week was busy with refactoring with the primary goal of being able to run contracts in test execution environment. The environment is not quite ready yet, but a lot of progress was done, and it’ll hopefully be ready next week.

Last week was busy on various improvements for contracts infrastructure, trying to clarify existing API and ensuring everything that might be built is actually possible. First system contracts were introduced, trait support was added and more, below is a recap of key updates.

Hello, world 👋! This is the beginning of hopefully successful thing I call “Project Abundance”. After writing initial set of constraints and thinking about it for quite a while, it is finally time to dedicate all my time to it and see where it leads us.