Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mobile: Add support for plugins #9652

Conversation

personalizedrefrigerator
Copy link
Collaborator

@personalizedrefrigerator personalizedrefrigerator commented Jan 4, 2024

Summary

This pull request:

  1. Adds a mobile-only default plugin that displays a message box
  2. Loads and runs the plugin from a jpl file on mobile.
    • Plugins are run within a background WebView

Demo APK

Plugin settings GUI

screenshot: Cards shown for different plugins (custom vimrc, etc.)

What's working

  • Loading plugins from JPL files
  • Running plugins in a background WebView
  • joplin.views.dialogs.showMessageBox API
  • joplin.plugins.register API
  • Data API
    • Tested with getting a list of folders and adding a note to the first one.
  • CodeMirror 6 content scripts
    • Currently, the example "CodeMirror line numbers" plugin loads and adds line numbers to the editor
    • Some remaining issues (see below)
  • Dialogs
  • Markdown-it content scripts

To-do

  • Connect to the plugin repository
    • For now, this is only enabled on Android (and iOS in dev mode)
    • To enable this on iOS in production, extra work may be required to comply with iOS AppStore guidelines. For example, we might use a plugin repository in which all plugins are pre-reviewed by Joplin team members. Additionally, such plugins may need additional restrictions to comply with the requirement that code not included in the bundle "only use capabilities available in a standard WebKit view"1 which may limit what APIs such plugins can use.
    • Verify that updates work correctly
      • Checked & some bugs were fixed — it will need to be tested again.
  • Allow loading plugins from .jpl files
    • Currently disabled on iOS release builds (enabled in development builds) for reasons mentioned above.
  • Fix plugins don't reload correctly with React Native's fast refresh.
    • Follow-up: Implement PluginRunner.stop for desktop?
  • Add GUI for enabling/disabling plugins
    • Currently, this uses PluginRunner.stop when a plugin is disabled (rather than requiring an app restart). This reuses the logic that adds support for React Native's fast refresh.
  • Test on iOS
  • Get plugin API working
    • CodeMirror 6 content scripts
      • Load content scripts into editor
      • postMessage
      • Fix content scripts not added if the plugin finishes loading while the editor is open. (Editor needs to be closed & re-opened).
    • Toolbar buttons
    • WebView dialogs
    • Panels (e.g. as used by the favorites plugin).
      • We also need some way to move/resize/switch between panels. As this could get complicated, it may be best left for a follow-up pull request.
    • Show plugin error messages in the log and link to them in settings.
    • Imaging API?
    • Clipboard API
    • Renderer content scripts
      • Run renderer within the note viewer iframe (DOM API access & allows us to avoid evaling plugin code in the main process).
        • Check performance: How much slower (if at all?) is it?
      • Support webviewApi.postMessage
  • Fix error messages when there are multiple plugins registered
  • Document plugin development workflow
    • Will plugin authors need to run a development version of Joplin to load & develop plugins?
  • Move refactoring changes to separate pull requests. These changes are currently part of this pull request, but should be moved to separate pull requests.
    • Mobile: Markdown toolbar refactor
    • Mobile: CodeMirror: Moved padding from the outside of the editor to the inside.
    • @joplin/default-plugins: Added support for building for just one application and refactored default plugins configuration file from JSON to TypeScript.

Notes

  • The expo-asset library is used to load local assets (used to load JPL files)
  • Plugins are run within iframes within a single WebView
  • RemoteMessenger and tarExtract were originally created for other pull requests
  • We work around this buffer bug to support tarExtract
    • This might instead be a React Native Hermes bug? (Upgrading RN might help.)
  • Whether or not a plugin is running on mobile can be found through the versionInfo().platform plugin API.

Code structure

The below explanation(s) may be converted into files under readme/dev/spec.

Explanation of RemoteMessenger

On mobile, not only is the background page running in a separate process, but so are the renderer and dialogs.

To simplify communication between these processes, a RemoteMessenger class is introduced.

The RemoteMessenger<LocalInterface, RemoteInterface> class

The RemoteMessenger class simplifies communication over postMessage. Its job is to convert asynchronous method calls to messages, then send these messages to another RemoteMessenger that handles them.

flowchart
	RemoteMessenger1<--postMessage-->RemoteMessenger2
Loading

For example, if we have

// Dialogs
export interface MainProcApi {
	onSubmit: ()=> void;
	onDismiss: ()=> void;
	onError: (message: string)=> Promise<void>;
}

export interface WebViewApi {
	setCss: (css: string)=> void;
	closeDialog: ()=> Promise<void>;
	setButtons: (buttons: ButtonSpec[])=> void;
}

We might then create messengers like this:

In the WebView:

const webViewApiImpl: WebViewApi = {
	// ... some implementation here ...
	setCss: css => {} // ...
};

// Different messageChannelIds allow us to have multiple messengers communicate over the same channel.
// Different IDs prevent the wrong messenger from acting on a message.
const messageChannelId = 'test-channel';

const messenger = new WebViewToRNMessenger<WebViewApi, MainProcApi>(messageChannelId, webViewApiImpl);

In the main process:

const mainProcApiImpl: WebViewApi = {
	// ... some implementation here ...
	closeDialog: () => {} // ...
};

const messageChannelId = 'test-channel';
const messenger = new WebViewToRNMessenger<MainProcApi, WebViewApi>(messageChannelId, mainProcApiImpl);

// We can now use the messenger.
// Messages are all asynchronous.
await messenger.remoteApi.setCss('* { color: red; }');

To call messenger.remoteApi.setCss(...), we use a process similar to the following:

First: Queue the method call and wait for both messengers to be ready.

When a messenger is ready, it sends a message with kind: RemoteReady.

flowchart
	postMessage1(["postMessage({ kind: RemoteReady, ... })"])
	rm1--1-->postMessage1--2-->rm2
	subgraph MainProcess
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
	end
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
	end
Loading

When a messenger receives a message with kind: RemoteReady, it replies with the same message type.

flowchart
	postMessage1(["postMessage({ kind: RemoteReady, ... })"])
	rm2--3-->postMessage1--4-->rm1
	subgraph MainProcess
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
	end
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
	end
Loading

Second: Send all queued messages

After both messengers are ready, we wend all queued messages. In this case, that's the setCss message:

{
	kind: MessageType.InvokeMethod,
	methodPath: ['setCss'],
	arguments: {
		serializable: ['* { color: red; }'],

		// If there were callbacks, we would assign them
		// IDs and send the IDs here.
		callbacks: [ null ],
	},
}
flowchart
	postMessage(["postMessage({ kind: InvokeMethod, ... })"])
	rm1--2-->postMessage--3-->rm2
	subgraph MainProcess
		call(["await m1.remoteApi.setCss('...')"])
		call--1-->rm1
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
	end
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
		webViewApiImpl["webViewApiImpl.setCss"]
		rm2--4-->webViewApiImpl
	end
Loading

After handling the message, a result is returned also by postMessage, this time with the kind ReturnValueResponse:

flowchart
	postMessage(["postMessage({ kind: ReturnValueResponse, ... })"])
	rm2--6-->postMessage--7-->rm1
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
		webViewApiImpl["webViewApiImpl.setCss"]
		webViewApiImpl--5-->rm2
	end
	subgraph MainProcess
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
		calll(["await m1.remoteApi.setCss('...')"])
		rm1--8-->calll
	end
Loading

After receiving the response, the setCss call resolves.

On mobile, we address the same problem in similar, but more generalized way. We define a RemoteMessenger class that handles postMessage communication.

RemoteMessenger and callbacks

Suppose we call a method in a way similar to the following:

messenger.remoteApi.joplin.plugins.register({
	onStart: async () => {
		console.log('testing');
	},
	test: 'test',
});

We can't send callbacks over postMessage. As such, we assign the onStart callback an ID and send the ID instead. The message might look like this:

{
	kind: MessageType.InvokeMethod,
	methodPath: ['joplin', 'plugins', 'register'],
	arguments: {
		serializable: [
			{
				onStart: null,
				test: 'test',
			}
		],
		callbacks: [
			{
				onStart: 'some-generated-id-for-onStart',
				test: null,
			}
		],
	},
	respondWithId: 'another-autogenerated-id',
}

Note: As before, the respondWithId connects a method call to its return value (the return value has the same ID).

The arguments.callbacks object contains only callback IDs and the arguments.serializable object contains only the serializable arguments. The two objects otherwise should have the same structure. These two objects are merged by the RemoteMessenger that receives the message:

flowchart
	callbacks[arguments.callbacks]
	serializable[arguments.serializable]

	callbacks--"only callbacks"-->original
	serializable--"only properties not in callbacks"-->original
Loading

Callbacks are called by sending an InvokeMethod message similar to the following:

{
	kind: MessageType.InvokeMethod,
	methodPath: ['__callbacks', 'callback-id-here'],
	arguments: { ... },
	respondWithId: 'some-autogenerated-id-here',
}

Footnotes

  1. https://developer.apple.com/app-store/review/guidelines/

Comment on lines 12 to 18
// TODO: Might not work (and `relative` is broken in path-browserify)
const isSubdirectoryOrSame = (parent: string, possibleChild: string) => {
possibleChild = normalize(possibleChild);
parent = normalize(parent);

return possibleChild.startsWith(parent);
};
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Replace with resolvePathWithinDir once related PR is merged.

@laurent22
Copy link
Owner

@personalizedrefrigerator, please let me know when this is ready to merge (there are few conflicts and the PR is still a draft)

@personalizedrefrigerator
Copy link
Collaborator Author

Closing -- all functionality, except default plugins, has been moved into separate pull requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mobile All mobile platforms plugins Anything related to Joplin's plugin system
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants