Skip to content

Latest commit

 

History

History
82 lines (79 loc) · 4.02 KB

feature_checklist.md

File metadata and controls

82 lines (79 loc) · 4.02 KB

Feature checklist

  • File hasher
    • use sha2-256 as it is more commonly used, faster than sha3-256, both no known hacks (should be easy to switch)
    • Takes a file path and read
    • File manifest to a certain size - currently using a constant of 1MB
    • Hash each chunk as leaves (nodes)
    • Produce a merkle tree
    • construct and write a file_manifest.yaml (root, nodes)
    • Unit tests: same file same hash, different file different hash, big temp file same/modified
    • last chunk lengths,
  • Analyze merkle tree vs hash list for memory usage and verification runtime
  • Manifest builder / publisher - CLI
    • Take a file, use File hasher to get the file_manifest, publish file_manifest to IPFS
      • later, take a list of files, use File hasher to hash all files and get root hashes
    • Construct a manifest with metainfo using YAML builder
      • vectorize
    • May include a status endpoint for the "canonical" publisher, but recognize the endpoint may change later on
    • Publish manifest to IPFS, receive a IPFS hash for the manifest
  • IPFS client
    • Connect to an IPFS gateway
    • Post files
    • Cat files
  • YAML parser and builder
    • Deserialize and serialize yaml files
  • Manifest server
    • require operator mnemonic
    • Use a generic path
    • Initialize service; for one Bundle, take (ipfs_hash, local_path)
      • Take a Bundle IPFS hash and get the file using IPFS client
      • Parse yaml file for all the file_manifest hashes using Yaml parser, construct the Bundle object
        • Take metainfo of file_manifest and search for access by the local_path
        • Verify local file against the chunk hashes
      • vectorize service for multiple bundles
      • Once verified, add to file to the service availability endpoint
    • Route / for "Ready to roll!"
    • Route /operator for operator info
    • Route /status for availability
      • verification for availability
    • Route /bundles/id/:id for a Bundle using IPFS hash with range requests
    • Route /health for general health
    • Route /version for Bundle server version
    • Configure and check free query auth token
    • (?) Server Certificate
    • Upon receiving a service request (ipfs_hash, range, receipt)
      • start off with request as (ipfs_hash, range)
      • Check if ipfs_hash is available
      • Check if range is valid against the Bundle and the specific file_manifest
      • TAP: Valid and store receipt
      • Read in the requested chunk
        • Add tests
      • Construct response and respond
        • determine if streaming is necessary
    • Start with free service and requiring a free query auth token
      • default pricing, allow updates for pricing per byte
    • Runs TAP agent for receipt management
      • Integration testing
  • File Download Client
    • Take private key/mneomic for wallet connections
    • Request using ipfs_hash
      • take budget for the overall bundle/file
        • construct receipts using budget and chunk sizes
        • add receipt to request
      • add free_token to request
      • File discovery and matching (Gateway?)
        • Read bundle manifest
        • Ping indexer endpoints data availability
        • Select indexers based on pricing
        • Select indexers based on performances
        • Parallel requests
        • Use random endpoints
      • Construct and send requests to indexer endpoints
        • Parallelize requests
        • Multiple connections (HTTPS over HTTP2)
    • Wait for the responses (For now, assume that the response chunks correspond with the verifiable chunks)
      • Keeps track of the downloaded and missing pieces,
      • continually requesting missing pieces until the complete file is obtained
      • Upon receiving a response, verify the chunk data in the file_manifest
        • if failed, blacklist the indexer
    • Once all file has been received and verified, terminate