Search result from API is stale after sync

[gineer01]

Operating system

Linux

Joplin version

CLI 3.0.1 from NPM + Joplin Desktop 3.1.24

Desktop version info

Joplin for Desktop

Copyright © 2016-2025 Laurent Cozic
Joplin 3.1.24 (prod, darwin)

Client ID: 41fd86eba6dd407a94d2655b86ad878a
Sync Version: 3
Profile Version: 47
Keychain Supported: Yes

Revision: d581264

Backup: 1.4.2
Joplin DDDot: 0.4.2
macOS theme: 1.5.8
Markdown Table: Colorize: 1.2.0
Quick Links: 1.3.2

Current behaviour

I use CLI to start API server to automate creating some notes. However, after syncing, the search result doesn't return new notes.

In my setup, I have 2 clients: my Joplin Desktop syncs to WebDAV and a CLI client syncs to the same WebDAV.

Steps to reproduce:

• Start API server on CLI client: joplin --profile /joplin/profile server start
• On Joplin Desktop app, create a new note (e.g., "Foobar") in notebook "Welcome!"
• Sync from Joplin Desktop app to WebDAV
• Calling API from CLI should not return this "Foobar" note yet (use note ID from the desktop client)

01:52:28: Request: GET /notes/87f0ba5654be445fb6f17ca10f412b04?token=<>
01:52:28: Error: Not Found

• Call sync on CLI client: joplin --profile /joplin/profile sync
• Call API from CLI returns "GET /notes/87f0ba5654be445fb6f17ca10f412b04" OK. Thus, the note data is there after syncing. The log below also shows 87f0ba5654be445fb6f17ca10f412b04 was fetched.
• Call API for search for query notebook:"Welcome!" doesn't return the note. It returns other notes in the same notebook

01:54:19: Request: GET /search?query=notebook%3A%22Welcome%21%22&fields=created_time%2Ctitle%2Cparent_id%2Cuser_updated_time%2Cid&page=1&token=<>

Thus, the search result is stale for some reason. The note data is already there after sync but the search result doesn't reflect the data. Perhaps, the SQLite full-text search needs update to reflect the new data. Is there a command to force SQLite to update?

Expected behaviour

After sync, the search result should show the new notes to reflect the data Joplin has.

Logs

2025-01-13 01:53:14: Synchronizer: Sync: starting: Starting synchronisation to target 6... supportsAccurateTimestamp = false; supportsMultiPut = false} [1736761994055]
2025-01-13 01:53:14: Synchronizer: Indexing resources...
2025-01-13 01:53:14: ResourceService::indexNoteResources: Start
2025-01-13 01:53:14: ResourceService::indexNoteResources: Completed
2025-01-13 01:53:14: Synchronizer: Sync target remote info: {"version":3,"e2ee":{"value":false,"updatedTime":0},"activeMasterKeyId":{"value":"","updatedTime":0},"masterKeys":[],"ppk":{"value":null,"updatedTime":0},"appMinVersion":"3.0.0"}
2025-01-13 01:53:14: Synchronizer: Sync target is already setup - checking it...
2025-01-13 01:53:14: Synchronizer: Sync target local info: {"version":3,"e2ee":{"value":false,"updatedTime":0},"activeMasterKeyId":{"value":"","updatedTime":0},"masterKeys":[],"ppk":{"value":null,"updatedTime":0},"appMinVersion":"3.0.0"}
2025-01-13 01:53:18: Synchronizer: BasicDelta: Report: {"timestamp":1736760874000,"older":68598,"newer":3,"equal":1}
2025-01-13 01:53:18: Synchronizer: supportsDeltaWithItems =  <false>
2025-01-13 01:53:18: Synchronizer: Sync: fetchingTotal: Fetching delta items from sync target
2025-01-13 01:53:18: Synchronizer: Sync: fetchingProcessed: Processing fetched item
2025-01-13 01:53:18: Synchronizer: Sync: createLocal: remote exists but local does not: (Remote 573cb160e8ae464f8e0959e622b57e1b.md)
2025-01-13 01:53:18: Synchronizer: Sync: fetchingProcessed: Processing fetched item
2025-01-13 01:53:18: Synchronizer: Sync: createLocal: remote exists but local does not: (Remote 87f0ba5654be445fb6f17ca10f412b04.md)
2025-01-13 01:53:18: Synchronizer: Sync: fetchingProcessed: Processing fetched item
2025-01-13 01:53:18: Synchronizer: Sync: createLocal: remote exists but local does not: (Remote 19a952e3e5094df381285960a3109086.md)
2025-01-13 01:53:18: Synchronizer: Sync: finished: Synchronisation finished [1736761994055]
2025-01-13 01:53:18: Synchronizer: Operations completed: 
2025-01-13 01:53:18: Synchronizer: fetchingTotal: 3
2025-01-13 01:53:18: Synchronizer: fetchingProcessed: 3
2025-01-13 01:53:18: Synchronizer: createLocal: 3
2025-01-13 01:53:18: Synchronizer: Total folders: 72
2025-01-13 01:53:18: Synchronizer: Total notes: 20928
2025-01-13 01:53:18: Synchronizer: Total resources: 19024
2025-01-13 01:53:18: checkDisabledSyncItemsNotification: No errors: Hiding notification
2025-01-13 01:53:18: ResourceFetcher: Auto-add resources: Mode: always
2025-01-13 01:53:18: ResourceFetcher: Auto-added resources: 0
2025-01-13 01:53:18: checkDisabledSyncItemsNotification: No errors: Hiding notification

[laurent22]

The search engine would need to index the note first, and I don't think that can be triggered via CLI. You'd need to start the TUI and let it run for some time, a few seconds probably, until indexing is done

[bsavant]

I had this same issue when trying to run the CLI and use the API. Ended up having to run a full desktop environment to keep Joplin Desktop running and use the API that way. It would be a welcome functionality change if at all possible for automation!

[laurent22]

Adding something like a joplin search --index command would be relatively easy if someone's interested in creating a pull request

[gineer01]

joplin search --index command would be very useful.

In addition, should sync and API calls trigger indexing automatically? For example, API calls that update data (e.g., POST, PUT) should trigger indexing so that the next API call to /search API will reflect the new data instead of waiting for indexing by a separate joplin search --index or TUI. Similarly, after a joplin sync, Joplin should know any note has changed by the sync and can add indexing at the end?

In these cases, API and sync code have more information what has changed and can be more efficient. If we expect users to call joplin search --index, I think many will just call that command "just to be sure" to avoid stale results. This is less efficient because if no data has changed, calling joplin search --index is just a waste?

[laurent22]

Or maybe we don't need joplin search --index - instead just change the existing joplin search and call indexing every time it starts

[gineer01]

What does joplin search do? I don't see it in https://joplinapp.org/help/apps/terminal/#commands .

I use joplin server to start the API and use search API to get result. Calling indexing every time joplin server starts is not enough. API calls (POST/PUT/DELETE calls) can update notes after joplin server starts. I think we need the proposed joplin search --index to index new/updated/deleted notes again. Ideally, if API calls indexing after each update (POST/PUT/DELETE calls), it's more efficient. Don't need to index for read-only operations like GET /notes or GET /search.