Integrate substrate connect with Apps

[wirednkod]

Summary:

This PR is introducing the (experimental) light client (smoldot) into the Polkadot JS apps throush substrate-connect.

Visible Changes:

A new "server" is introduced under 'Live Relays & Parachains', 'Polkadot' and 'Kusama' and under 'Test Relays & Parachains', 'Westend' as can be seen below:

Description:

Substrate connect nodes can be connected either through in-browser light client or through extension.

Development process:

In order to make this as less intrusive as possible the following steps took place:

1. A new type is introduced called 'substrate-connect' and default naming the existing to 'json-rpc' (see PR #476)
2. Following the existing implementation of using the URL parameter rpc for existing nodes (e.g. ?rpc=wss%3A%2F%2Frpc.polkadot.io#/explorer) - a new url param is introduced called sc (from substrate-connect) that has the name of the node and the characteristic -substrate-connect following (e.g. ?sc=kusama-substrate-connect#/explorer). This way it is easier a differentiation between the two, without altering existing implementation.
3. Based on api type passed in early stage of the app (Api.tsx) a choice is made on whether substrate-connect's Detector will be used or existing implementation).
4. Based on new type introduced (1. above), a new function is introduced (packages/apps-config/src/endpoints/util.ts) for endpoints that receives the url and the type ('json-rpc or substrate-connect - defaults to first if none given) and returns the url type, reducing this way duplication of code and typos).
5. All fallback methods (e.g. initSettings.ts) are updated as per bullets here.
6. Finally all endpoints are updated using function mentioned on bullet 4 and 3 new endpoints for substrate-connect are introduced as mentioned on Visible Changes above.

[jacogr]

Nice to have this. some small comments all around.

[jacogr]

As for the test failure here - https://github.com/polkadot-js/apps/pull/5644/checks?check_run_id=2875618334#step:3:518

Adapt this - https://github.com/polkadot-js/apps/blob/master/jest.config.cjs#L28

[wirednkod]

I am not sure why the CIs started to break (I don't understand the error but something seems to be wrong with .yarn).
@jacogr any insights on this?

[jacogr]

It is a yarn.lock merge issue. Just copy the yarn.lock from master and replace it here. There should be no updates needed to it at all.

[jacogr]

Played through. It seems to do what it says on the tin, which is great.

There is a small issue - when the user swaps to a light client (and he has nothing), the screen just goes white and stays that way. We need some of fallback for that.

Apart from that, seems great.

EDIT: We also need somewhere to prompt people how and what to add. (But can do that as a follow-up, not even sure how/where and what something like that would look like)

[wirednkod]

There is a small issue - when the user swaps to a light client (and he has nothing)

What do you mean by "He has nothing" ?

EDIT: We also need somewhere to prompt people how and what to add. (But can do that as a follow-up, not even sure how/where and what something like that would look like)

There are for now the substrate-connect landing-page and api docs - Im not sure if that is what we could use though

[jacogr]

So this is what happens for me...

Just dead as soon as I swap. (clone -> yarn -> yarn start)

EDIT: Yes, will figure out a place to link through to that. That works.

[wirednkod]

@jacogr This is a little tricky.
When you are testing in localhost - then the SmoldotProvider (the one that runs the light client in the opened tab and not the extension) will never run correctly. You need to make an actual build of the apps and then navigate into the build dir and run an http-server in order to see it work without the extension (simulates the actual static files on server in a way)

Steps:
a) install globally an http-server (npm install http-server -g)
b) Make a build of the apps (cd ./apps && yarn build)
c) Navigate in the build dir ($ apps > cd packages/apps/build)
d) Run the server (http-server)

Please try to test with Westend as I am not sure why Polkadot/Kusama not always respond (I'll cc @tomaka here)

You should see something like the lines below running in your console:

Chain of light client is = westend
content_script.js:159 script added
main.98207691.js:2 [smoldot_js] Loaded chain spec #0: Westend
main.98207691.js:2 [smoldot_js] Genesis hash of Westend is: 0xe143…423e
main.98207691.js:2 [smoldot_js] Using light checkpoint starting at #5509229
main.98207691.js:2 [smoldot_js] Initialization complete
main.98207691.js:2 [network] Connected to 12D3KooWR3UGwwSP5wdBMk2JXXuzXoscPSudv8hmQkzfZTBzSbeE
main.98207691.js:2 [network] Connected to 12D3KooWQeuYtdKUgC9A2wNchxQDXWHbotZWUHxnraBwiV15GH4K
main.98207691.js:2 [sync-verify] GrandPa warp sync finished to #6350101
polkadot.02.8b5b3183.js:1 web3Enable: Enabled 0 extensions: 
polkadot.02.8b5b3183.js:1 web3Accounts: Found 0 addresses: 
main.98207691.js:19 chain: Westend (Live), {"ss58Format":42,"tokenDecimals":[12],"tokenSymbol":["WND"]}

[tomaka]

Polkadot/Kusama not always respond (I'll cc @tomaka here)

To give a little overview:

Since smoldot is a light client, it doesn't store the state of the chain. Instead, it queries it from other nodes.
PolkadotJS doesn't really behave properly when it asks for the storage value through a JSON-RPC request and an error is returned. The UI more or less stops working.
Unfortunately, while smoldot is far away from the head of the chain, storage requests over the network typically fail because nodes prune old blocks.

In order to circumvent that, we only report substrate-connect as "connected" when it has synced to the head of the chain. Once at the head of the chain, all storage queries normally succeed.

This works well for Westend. For Kusama and Polkadot, unfortunately, there is only one "WebSocket secure" node to connect to at the moment for each chain (cc https://github.com/paritytech/devops/issues/948).
This node is unfortunately often full.

When the node is full, smoldot can't sync to the head of the chain at all, meaning that no "connected" event is reported to PolkadotJS.

It should work when using the substrate-connect browser extension, as this unlocks more nodes to connect to.

[jacogr]

The point is that I'm still in white-screen mode on a local server -

So, the issue is that people are going to click this, get exactly the same and have no way of recovering.

The only reason I could recover here to swap to westend is because there is a bug where a "remove params" and refreshing doesn't actually load the light client again. (So it connected to the default chain and then allowed recovered after the params were removed)

[wirednkod]

The white screen has to do with the await detect.connect that exists in packages/react-api/src/Api.tsx.

Due to the fact that this exists in a Component (<Api />) prior to <Apps />, the await (expecting Promise) is blocking the React flow to "draw" the components and expect Api to become ready ( the code freezes until Promise of detector is resolved - so flow never reaches thus never draws the "waiting" state - instead we see a white page)

I will be working on this to resolve it asap - if there is something that can be used (e.g. useApi) and I am not aware of please for your feedback @jacogr

cc @raoulmillais @tomaka

[jacogr]

catch-22 - the useApi uses the context, so actually uses the <Api /> wrapper. There should be a way with minimal changes to get it to behave inside the context provider.

[wirednkod]

@jacogr After further investigation and re-tries it was fairly easy to fix the "await" part but due to the fact that api was undefined a huge amount of fixes should take place across all Apps codebase in order to make sure that there will be no breaking.

Instead we proceed with a workaround on the substrate-connect side that would resolve the issue and make the Apps code clearer. Substrate-connect now, through the Detector besides the connect() it now provides one, called provider() that, depending on whether the user has the substrate connect extension installed or not, returns a provider of ProviderInterface type, that is used by passed into the ApiPromise.

cc @raoulmillais @tomaka

[jacogr]

Seems sane after resolving the conflicts. Will just do a final play-through.

[jacogr]

Nice! Seems to do exactly what it says on the tin, this is an excellent addition.

Thank you.

[polkadot-js-bot]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.