Agree on a sketch of the overall wallet architecture #7

str4d · 2024-04-24T23:19:44Z

No description provided.

str4d · 2024-04-24T23:20:13Z

Rough (and incomplete, hence not put into the top post) sketches from a meeting with Daira-Emma and Kris:

Current light client wallet architecture:

flowchart
    zcashdConsensus

    zcashdConsensus -- Chain tip change --> lightwalletd
    zcashdConsensus -- New blocks --> lightwalletd

    subgraph Indexer
    zcashdDb
    lightwalletd
    end

    zcashdConsensus <-- Stuff --> zcashdDb
    zcashdDb<-- UpdateChainTip --> lightwalletd
    zcashdDb<-- GetUTXOs --> lightwalletd
    zcashdDb<-- GetTransaction --> lightwalletd

    subgraph Mobile wallet app
    Scanner
    Wallet
    end

    Wallet -- Register UFVK --> Scanner
    Wallet -- (LC) [nf], ScanRange --> Scanner
    lightwalletd <-- UpdateChainTip --> Wallet
    lightwalletd <-- GetUTXOs --> Wallet
    lightwalletd <-- GetTransaction --> Wallet
    Scanner <-- CompactBlockRange --> lightwalletd
    Scanner -- ScannedBlock --> Wallet

Sketch of possible desired architecture:

flowchart
    FullValidator
    Indexer
    Scanner
    Wallet

    FullValidator -- Chain tip change --> Indexer
    FullValidator -- New blocks --> Indexer

    Wallet -- Register UFVK --> Scanner
    Wallet -- (LC) [nf], ScanRange --> Scanner
    Scanner <-- (Compact)BlockRange --> Indexer
    Scanner -- ScannedBlock --> Wallet

Wallet
- Trusted with privacy and spend authority.
Scanner
- Trusted with privacy in perpetuity. Has access to user-specific data.
Indexer
- Can observe usage patterns for the duration it is used.
- Trusted with correctness of the chain data, but that trust can be removed with bandwidth increase (Fly Client proofs).
Full Validator
- Trusted with correctness of the chain data.

LC: Light Client

str4d · 2024-04-24T23:20:18Z

See also https://github.com/oxarbitrage/zebra-grpc-scan-spec/blob/main/client_integration.md

jackgavigan · 2024-04-26T10:04:08Z

Should the Scanner obtain the blocks from the Indexer, or direct from Zebra's database? (Assuming the scanner is running on the same machine as Zebra.)

I assume that t-address balances and transactions will be obtained by querying the Indexer?

ETA: Also, in the diagram above, is it the Indexer's job to index transparent addresses and transactions (e.g. for the block explorer use case)? I think it's worth being pedantically explicit about the functionality of each component to ensure that we don't have differing understandings/expectations.

jackgavigan · 2024-05-16T15:58:29Z

Temporary diagram for discussion

   flowchart

   subgraph Zcash_P2P_Network
   Other_nodes
   end
   Other_nodes -- chain state -->   FullValidator
   subgraph Zebra
      FullValidator
      Indexer
      TxBroadcaster
  subgraph State
      NonFinalizedState
      FinalizedState
  end
      FullValidator -- Chain tip change --> NonFinalizedState;
      FullValidator -- New blocks --> NonFinalizedState;
      FullValidator -- Finalized blocks --> FinalizedState
      State -- transparent TX info --> Indexer
   end

   State -- Read Zebra State --> Scanner
   Wallet -- T-addrs query --> Indexer
   Indexer --T-addrs state --> Wallet
   Wallet -- New transaction --> TxBroadcaster
   TxBroadcaster -- New Transaction --> Other_nodes
   Indexer --> BlockExplorer

   subgraph CLI_Wallet
      Scanner
      Wallet -- Register viewing key --> Scanner
      Scanner -- Shielded TX scan results  --> Wallet
   end

mpguerra · 2024-07-03T11:25:08Z

Are we settled on this design for the wallet architecture? What else is required in order to close out this issue?

pacu · 2024-07-03T13:12:47Z

Although risky in terms of scope creep, I think that a general purpose approach would be much more helpful in terms of requirement elicitation. Many kind of users, either they are individuals, DAOs, Protocols or other Organizations (like CEXs, Asset Custodians, Banks, etc) need to make use of a Zcash Wallet and currently their needs are not met by our tooling. It's better to think of that beforehand.

This is a layered approach where different application domains can be added horizontally as needed. The CLI can be the first one, but others come to mind.

Principles:

Applications never connect to the Zebra Wallet process directly.
Only the transformation layer connects to the wallet
Applications don't have a contract API with the wallet, they do with the Data Transformation Layer (DTL) logic module of their preference
DTL can be coded independently from wallet as long as interfaces are agreed beforehand.
DTL can stub data for modules to be developed independently from Wallet as well.
DTL modules can be worked independently
Testing with high degree of scope
- unit testing
- layer communication testing (Example: Client Facing API <-> CLI Transformation API)
- end-to-end

flowchart

   subgraph Zcash_P2P_Network
   Other_nodes
   end
   Other_nodes -- chain state -->   FullValidator
   subgraph Zebra
      FullValidator
      Indexer
      TxBroadcaster
  subgraph State
      NonFinalizedState
      FinalizedState
  end
      FullValidator -- Chain tip change --> NonFinalizedState;
      FullValidator -- New blocks --> NonFinalizedState;
      FullValidator -- Finalized blocks --> FinalizedState
      State -- transparent TX info --> Indexer
   end

   State -- Read Zebra State --> Scanner
   Wallet -- T-addrs query --> Indexer
   Indexer --T-addrs state --> Wallet
   Wallet -- New transaction --> TxBroadcaster
   TxBroadcaster -- New Transaction --> Other_nodes
   Indexer --> BlockExplorer
   
   subgraph Data Transformation Layer
      CLI_API <--> CoreTransformationLogic
      ZcashSnapAPI <--> CoreTransformationLogic
      B2B_API <--> CoreTransformationLogic

      CoreTransformationLogic
   end
   
   CoreTransformationLogic -- works with native types --> Wallet
   
   subgraph Wallet CLI
        CLIClientLogic -- mediates formats and lifecycle -->  CLI_API
   end

   subgraph Zcash Metamask Snap
      BackendLogic <--> ZcashSnapAPI
   end
   subgraph Zcash Avax Bridge
      ZavaxDataLayer <--> B2B_API
   end
   subgraph Zcash CEX Infrastructure
      CEX_Backend <--> B2B_API
   end
   subgraph Wallet Backend
      Scanner
      Wallet -- Register viewing key --> Scanner
      Scanner -- Shielded TX scan results  --> Wallet
   end

str4d · 2024-07-18T15:50:47Z

@pacu that is compatible with the sketch we have in this issue, but you should open a separate issue for discussing it.

zancas · 2024-07-26T13:24:58Z

Apologies in advance, if this question is naive or out-of-scope.

Should we explicitly consider an API that offers guidance to mobile developers in this discussion?

It's not explicit in the title of this issue that we're focused on CLI-only wallets, and it seems like explicit consideration of other interface cases, even if there's no requirement to implement (or it's already implemented) might be useful in this context.

I am specifically thinking about APIs that encourage non-CLI developers (e.g. mobile) to prefer QR code representations of addresses (or payment requests etc.) over strings.

arya2 · 2024-09-03T17:46:10Z

Updated temporary diagram based on recent discussions (see note at the end about the block explorer use case):

flowchart

subgraph Zcash_P2P_Network
    Other_nodes
end

Other_nodes -- chain state -->   FullValidator

subgraph Zebra
    FullValidator
    TxBroadcaster
    RpcServer
    subgraph State
        NonFinalizedState
        FinalizedState
    end
    FullValidator -- Chain tip change --> NonFinalizedState;
    FullValidator -- New blocks --> NonFinalizedState;
    FullValidator -- Finalized blocks --> FinalizedState

end

State -- transparent TX and nullifier info --> Indexer

subgraph Zebroid
    Indexer
    IndexerRpcServer
end

ReadStateService --> Scanner
RemoteReadStateService --> Scanner
State --> ReadStateService --> Wallet
IndexerRpcServer --T-addrs state --> RemoteReadStateService --> Wallet
Wallet -- New transaction --> TxBroadcaster
TxBroadcaster -- New Transaction --> Other_nodes
FullValidator --> RpcServer
Indexer --> IndexerRpcServer --> BlockExplorer
RpcServer -- NonFinalizedState via getblock method --> Indexer

subgraph CLI_Wallet
    Scanner["Scanner<br>(zcash_client_backend)"]
    ReadStateService["ReadStateService<br>(zebra-state)"]
    RemoteReadStateService["RemoteReadStateService<br>(zebra-state)"]
    Wallet -- Register viewing key --> Scanner
    Scanner -- Shielded TX scan results  --> Wallet
    WalletRpcServer --> Wallet
end

WalletRpcServer --> Client

The indexer RPC server is shown supporting the block explorer use case here as we've discussed moving some of the RPC methods currently supported in Zebra to the indexer RPC server so that Zebra cannot be called directly with those queries.

conradoplg · 2024-09-05T16:30:10Z

We just discussed this in the Zcashd Deprecation Collaboration meeting. I'll try to update the figure later but here are the gist of it while it's fresh in my mind.

Current zcash_client_backend-backed lightwallets work like this (where "->" is function calls inside process, and "-->" is across processes):

zcash_client_backed -> block_source --> lightwalletd --> zcashd

The proposal is to first support this:

zcash_client_backed -> block_source --> ("Zebroid": something equivalent to lightwalletd using zebra-state) --> zebrad (and zebra's DB directly)

And then eventually (but not required for zcashd deprecation) we could migrate to:

zcash_client_backed -> (another block source using zebra-state) --> zebrad (and zebra's DB directly)

Please confirm if this is accurate.

idky137 · 2024-09-16T17:26:09Z

Hey this is our (Zingolabs) current understanding and plan for Zaino (previously Zebroid). It is very similar to Arya's diagram but also includes the "light" wallet architecture. I think from our point of view we plan to carry on using the lightwallet gRPC service for our "light" wallet (Zingo Mobile) but we will also be implementing a remote endpoint for the ReadStateService in Zaino for wallet consumption. My understanding is that this is planned to be consumed by "full node" wallet implementations not "light" wallets though.

jackgavigan · 2024-09-25T09:29:22Z

Annotated diagram based on discussions during the Arborist call on 19 Sept 2024.

idky137 · 2024-09-25T16:50:41Z

I've opened an issue in the Zaino repo (here zingolabs/zaino#69) to finalise the red "gRPC" in the diagram above. I believe there are currently a couple differing ideas regarding this so have put forward an initial option there.

str4d added the S-in-progress Status: Work is currently in progress on this item. label Apr 30, 2024

daira added this to the Zcashd wallet replacement milestone May 4, 2024

mpguerra mentioned this issue May 29, 2024

Tracking: Enable zebra-scan to run as a separate independent process from zebrad ZcashFoundation/zebra#8573

Closed

9 tasks

pacu mentioned this issue Jul 3, 2024

[Misc] Wallet architecture design pacu/zcash-dev-rel-engineer#146

Closed

nuttycom self-assigned this Sep 20, 2024

idky137 mentioned this issue Sep 25, 2024

2.5 - Leverage implementation to contextualise and finalise full architecture of Zaino zingolabs/zaino#57

Open

3 tasks

pacu mentioned this issue Oct 1, 2024

GRPCs Required by Light Clients pacu/zcash-dev-rel-engineer#162

Closed

nuttycom transferred this issue from zcash/librustzcash Oct 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Agree on a sketch of the overall wallet architecture #7

Agree on a sketch of the overall wallet architecture #7

str4d commented Apr 24, 2024

str4d commented Apr 24, 2024 •

edited

Loading

str4d commented Apr 24, 2024

jackgavigan commented Apr 26, 2024 •

edited

Loading

jackgavigan commented May 16, 2024 •

edited

Loading

mpguerra commented Jul 3, 2024

pacu commented Jul 3, 2024 •

edited

Loading

str4d commented Jul 18, 2024

zancas commented Jul 26, 2024

arya2 commented Sep 3, 2024 •

edited

Loading

conradoplg commented Sep 5, 2024 •

edited

Loading

idky137 commented Sep 16, 2024 •

edited

Loading

jackgavigan commented Sep 25, 2024

idky137 commented Sep 25, 2024

Agree on a sketch of the overall wallet architecture #7

Agree on a sketch of the overall wallet architecture #7

Comments

str4d commented Apr 24, 2024

str4d commented Apr 24, 2024 • edited Loading

str4d commented Apr 24, 2024

jackgavigan commented Apr 26, 2024 • edited Loading

jackgavigan commented May 16, 2024 • edited Loading

mpguerra commented Jul 3, 2024

pacu commented Jul 3, 2024 • edited Loading

str4d commented Jul 18, 2024

zancas commented Jul 26, 2024

arya2 commented Sep 3, 2024 • edited Loading

conradoplg commented Sep 5, 2024 • edited Loading

idky137 commented Sep 16, 2024 • edited Loading

jackgavigan commented Sep 25, 2024

idky137 commented Sep 25, 2024

str4d commented Apr 24, 2024 •

edited

Loading

jackgavigan commented Apr 26, 2024 •

edited

Loading

jackgavigan commented May 16, 2024 •

edited

Loading

pacu commented Jul 3, 2024 •

edited

Loading

arya2 commented Sep 3, 2024 •

edited

Loading

conradoplg commented Sep 5, 2024 •

edited

Loading

idky137 commented Sep 16, 2024 •

edited

Loading