A Deterministic Mapping for Plot Lifecycle Management

This week has been mainly focused on refining a bit the design for plot identification and sector expiration. I think that I finally have a model with which I am comfortable with, and that I think solves all of our previous problems. Nazar had this idea to drastically simplify how plot IDs were derived, and how sectors were linked to plots. The high-level idea made sense, but there were still some details that weren’t clear. This week I managed to come up with a design that I think satisfies all of our requirements.

Apart from all of this work around plot and sector lifecycle management, I also started working on a draft specification for the protocol that fits all the pieces together. Our protocol builds upon the Subspace protocol, so I am using the existing specification as a base and extending it with all the sharding specifics. I can’t wait to share the first draft with you all for feedback and suggestions.

A Deterministic Mapping for Plot Lifecycle Management

#

The core of this proposal is based on the simplification idea for sector expiration and plot identification from this [zulip discussion]. The basic idea is to simplify plot identification by removing scheduled, network-wide epochs. Instead, we introduce a deterministic, cyclical mapping function. This approach decouples long-term shard assignment from the specific history_size by mapping any history_size to a fixed set of “history classes”. This allows a farmer to continuously update sectors with new history while having the flexibility to remain in the same shard, making a plot epoch parameter unnecessary. This also limits the number of sectors that can be created in parallel with the same sector index and different committed history sizes (artifically increasing the storage of a plot).

The operation of the protocol is based on the concept of a History Modulo (HISTORY_MODULO), a new global protocol parameter that defines the total number of unique history classes. For example, HISTORY_MODULO could be set to 2048. This parameter governs the trade-off between farmer flexibility and shard selection potential.

A farmer’s assignment to a shard is determined not by the exact history_size they are plotting, but by the class that history maps to.

• History Class Calculation: When a farmer creates a sector, they commit to a committed_history_size. From this, a history_class is derived using a simple modulo operation:

  history_class = committed_history_size % HISTORY_MODULO
  

• Plot ID Derivation: The stable plot_id is then created by hashing the farmer’s public key with this class. This is the crucial step that provides stability.

  plot_id = hash(public_key_hash, history_class)
  

• Shard Allocation: This plot_id is then used as the input to the Verifiable Random Function (VRF) that determines the shard assignment.

  assigned_shard = vrf_output_to_shard(VRF(plot_id, randomness))
  

Because history_class is the result of a modulo operation, many different history_size values will map to the same class. This ensures the plot_id and the resulting shard assignment can remain stable even as the farmer plots new history.

Sector and Plot Identification

#

Sectors must still be uniquely and verifiably tied to the specific history they contain.

• Sector ID: The sector_id is derived simply from this plot_id and the sector’s index within the plot.

  sector_id = hash(plot_id, sector_index)
  

This hierarchical ID structure ensures that a farmer’s shard assignment can be stable, while still enforcing that each sector is uniquely tied to its specific committed_history_size for verification and expiration.

Expiration and Re-Plotting Cadence

#

This model creates a flexible and predictable lifecycle for farmers, giving them direct control over their re-plotting strategy.

Farmer Lifecycle Example:

1. Joining: A farmer starts plotting when the current history size is H_1. They calculate history_class_1 = H_1 % HISTORY_MODULO. This determines a plot_id_1 and assigns them to a corresponding shard. They begin filling their drive with sectors committed to H_1.

2. Maintenance: As time passes and the chain grows to a history size of H_2, their initial sectors from H_1 begin to expire. The farmer uses the newly freed space to plot new sectors. They now have a choice:

• Re-use Existing Plot: The farmer can choose to remain in the same plot (and thus the same shard). To do this, they find a committed_history_size (H_commit) that is less than or equal to H_2 and also satisfies H_commit % HISTORY_MODULO == history_class_1. A farmer can easily calculate the most recent history size that meets this criterion.
• Utilize a New Plot: Alternatively, the farmer can commit to the latest history, H_2. This will generate a new class, history_class_2 = H_2 % HISTORY_MODULO, creating a new plot_id and likely assigning them to a new shard.

This model removes the concept of a mandatory, network-wide “Epoch Turnover.” Instead, re-shuffling becomes a strategic choice made by the farmer.

Benefits and Parameter Tuning

#

This deterministic mapping provides significant advantages:

• Simplicity: The logic is extremely straightforward, relying on a single new parameter (HISTORY_MODULO) and simple modulo arithmetic.
• Farmer Quality of Life: Farmers are never forced into a disruptive, network-wide re-plotting event. They have fine-grained control over which plots they maintain, reducing operational overhead.
• Flexibility: Farmers can choose to commit to a slightly older history size to preserve their shard assignment or switch to a new one if it is more advantageous.

The primary consideration is tuning the HISTORY_MODULO parameter, which involves a direct trade-off between farmer flexibility and preventing strategic shard selection.

A model for parameter tuning

#

The key to this design is selecting an optimal value for \(M\), the HISTORY_MODULO. This choice involves balancing two competing factors: History Lag and Shard Selection Power.

Let’s define the key parameters:

• \(M\): The HISTORY_MODULO, the number of distinct history classes. This is the value we want to determine.
• \(S\): The total number of shards in the network.
• \(H_{current}\): The current history_size of the blockchain.

Modeling Farmer Flexibility (History Lag)

#

When a farmer wants to re-use an existing plot, they may not be able to use \(H_{current}\). This creates a “lag” between their plotted history and the most current state of the network. We can quantify this as the Average History Lag ((L_H)):

$$ L_H = \frac{M}{2} $$

A larger \(M\) gives the farmer more classes to choose from, reducing the average lag for any given class.

Modeling Security (Shard Selection Power)

#

A farmer could try to gain an advantage by choosing which history class to plot. With \(M\) possible classes, they can generate \(M\) different plot_ids and calculate the resulting shard for each, choosing the one that is most favorable. We can model this as the Shard Selection Power (\(P_{select}\)), the probability of landing in one specific target shard:

$$ P_{select}(M, S) = 1 - \left(1 - \frac{1}{S}\right)^M $$

A smaller \(M\) significantly reduces this probability, making it difficult for farmers to strategically choose their shard.

The Optimization Model

#

The goal is to choose a value for \(M\) that provides a good balance. For example, if a network has \(S=1000\) shards and we decide that no farmer should have more than a 5% chance of selecting a specific shard, we would solve for \(M\):

$$ 0.05 = 1 - \left(1 - \frac{1}{1000}\right)^M $$

$$ \ln(0.95) = M \cdot \ln(0.999) \implies M \approx 51.3 $$

Based on this, a protocol designer might choose \(M=50\). This value would provide a low risk of shard selection while ensuring the average history lag is only 25 segments—a very reasonable trade-off for farmer flexibility.

An opportunity to simplify sector expiration?

#

Sector expiration has been the main thing messing with me when thinking about the design of this part of the protocol. With this new approach to bind sectors into specific plots we may be able to conceptually simplify the existing formula for sector expiration to make it more intuitive. The logic is fundamentally sound.

Essentially, the current model can be thought of as:

Expiration Point = Base Lifetime + Random Additional Lifetime

Where:

• Base Lifetime = history_size + MIN_SECTOR_LIFETIME
• Random Additional Lifetime = A random value between 0 and 3 * history_size

This conceptual model preserves all the benefits of the original design while being easier to reason about. The current implementation, while verbose, appears to be a necessary complexity to ensure long-term network health and fairness. What I want to do next is to figure out how to simplify this conceptual model into a more intuitive formula that can be easily understood and reasoned about on top of the current lifecycle management design. (Worst case, we can keep the current sector expiration, as it wouldn’t break in any way with this new design.)

Spec in progress and next steps

#

The “protocol overview” section of a specification is always the harder for me. I try to give a high-level overview of the operation of the protocl that gives readers a core intuition of how all the piece fit together. This way, one can have a clear mental model for when we jump into the low-level details (where is easy to get lost into the weeds).

As mentioned above, this week I started working on the specification for the protocol, and I already managed to have a protocol overview section that I am happy with. I hope to make more progress on it in the coming days, and to figure a way to start sharing small pieces of it so I can start getting feedback and improvement suggestions.

Throughout all of my more theoretical work from the past few weeks, I have been sharing theoretical models for different parts of the protocol, but I haven’t shared the specific protocol parameters for the various components (e.g. the optimal HISTORY_MODULO value). As part of my spec’ing efforts, I am hoping to also recommend specific values for these parameters based on the theoretical models and target system requirements. Hopefully, this will provide a more concrete foundation for the protocol and help guide implementation efforts.

On another personal note, next week I will be giving a talk at the Web3Summit in Berlin, so if you are around and you want to chat about this project, blockchain scalability, or honestly anything web3-related, feel free to reach out to me. I will be around the event and would love to meet you in person.