Shift Proof-of-Capacity

Motivation

Right now there is no verification system in place to ascertain with mathematical certainty that a storage node has the storage space available that it claims it has. Since this data is used to determine the pricing for token locking, it is of great importance that such a verification system is in place in order to prevent prices being manipulated. Research on this matter is currently ongoing and a first design is ready.

Introduction

In our token locking model the amount of storage space offered by each storage provider is summed and this figure is then used to calculate the Y variable in the model’s pricing formula. This is done by subtracting the total amount of storage space that is locked at the time of calculation, while also taking into account a buffer factor, which yields the Y variable: Y = [byte] = total amount of the platform’s storage space that is on offer (supply). Currently, the Y variable is the only variable in the pricing formula that is not being calculated based on data that was directly registered and verified at the blockchain. The raw data for the Y variable, the sum of storage space offered by each storage provider, is currently reported by the storage cluster called Phoenix. The current work flow is:

1. Phoenix peers broadcast their StorageMax value from the ipfs/config every three minutes using a push notification. Since this value can be edited by its operator without any IPFS default restrictions or validations in place, Phoenix reports the StorageMax unless the value exceeds its storage cap of 20 GB, in that case Phoenix will report a value of 20 GB.
2. Using Phoenix-core’s gossip protocol, the push notifications are relayed through the entire storage network, whereby each peer that relays the message adds its own storage value to the relay chain.
3. Each Phoenix peer collects the push notifications and keeps a cache of each individual peer’s value for StorageMax based on their public key. Each peer then sums up these values of all peers to get a single value for TotalStorage which is available at the Phoenix /stats endpoint.
4. Every new block, each Shift (blockchain) node (i.e. block producer) randomly selects a Phoenix peer and makes a request at the Phoenix /stats API call and keeps the value for TotalStorage in memory.
5. At every 10th block, each Shift node adds to a mem_table the most common figure of all the 10 values for TotalStorage that were kept in memory.
6. At every 10th block, each Shift node stores in block, the mode average of the last 10 values for TotalStorage that were selected and written to the mem_table, i.e. the mode average measured from data collected from 100 blocks.
7. Each Shift node uses the value for TotalStorage (raw data) from the last matured block to calculate the Y variable once a token locking transaction is being made.

While the process of collecting the raw data is already decentralized (Phoenix peers independently collect data and are randomly being selected), two aspects can justly be criticized and are open for improvement:

1. There’s no true consensus between the Phoenix peers on the value for TotalStorage. The way consensus on the value for TotalStorage is being enforced at the blockchain level is easily vulnerable to attacks as the process depends on messages being relayed properly between Phoenix peers.
2. The reported value for StorageMax is in no way being validated with the physical amount of disk space that is available to IPFS. This allows any malicious node operator to easily produce fake data, indirectly resulting in price manipulation for token locking. The intermediary solution of a storage cap of 20 GB is limited and does not contribute to a long term, sustainable storage ecosystem.

Problem 1 can be addressed by allowing storage node operators to have their offered storage space transacted at the blockchain. Problem 2 can be addressed by introducing a Proof-of-Capacity (PoC) algorithm, whereby the PoC challenges are being created and their results are being processed and stored at the blockchain level, killing two birds with one stone.

Roadmap

• MILESTONE 1: Introduce a new blockchain transaction type: the ability to register as a storage provider (role) (register_storage_provider).
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 2: Introduce a new blockchain transaction type: the ability to register the amount of offered storage space as a storage provider (provide_storage/withdraw_storage).
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 3: Introduce a new Phoenix feature: on-the-fly generation of plot files (plotting) within IPFS.
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 4: Prevent plotting interference from file uploading: allow Phoenix to enforce IPFS to collect garbage frequently and remove any files that were uploaded to IPFS, and/or allow Phoenix to block file uploading/pinning to peers with unconfirmed storage space.
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 5: Verify plotting at blockchain: allow Shift nodes to reproduce plot files (hash comparison of a random shard of a random plot file) using Zero Knowledge Proofs.
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 6: Allow Shift nodes to confirm (twice) the pre-registered amount of offered storage space in database (Pre-registered → Unconfirmed → Confirmed).
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 7: Allow Shift nodes to sum all Confirmed amounts of offered storage space to calculate the platform’s total amount of available storage.
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 8: Prevent Phoenix from pinning files to peers with Unconfirmed storage space. Allow Phoenix to only select peers with Confirmed storage space.
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 9: Allow a Phoenix peer to carefully handle plotting and pinning in specific scenarios, i.e. withdrawal and/or later-on addition of storage space.
  • Unit tests included.
  • Code delivery in docker container.
• MILESTONE 10: Introduce a new Phoenix feature: Distributed Hash Table (DHT).
  • Unit tests included.
  • Code delivery in docker container.