Huge data loss because of malfunctioning/badly designed synchronization feature

[aardbol]

Describe the bug
The app doesn't store records in the db that is defined in the Database path setting. The Synchronization button in the left menu has no effect to update that db with the latest records, hence keep increasing the gap of the live data and the data that is stored in that db. As a result, any backups made of that db will be useless and there's a huge risk of the Synchonization feature in the settings to cause an enormous data loss as it doesn't warn about a bigger db being overwritten by a smaller one or any similar security check.

To Reproduce

I often pressed the Synchronization button in the left menu, expecting that the original file (as shown in Database path setting) gets the latest updates written into it, that for a reason I don't understand aren't directly written to that db. I experienced this data gap issue with an older version before, so I decided from that point on to do manual syncs as explained. This worked in the beginning, hence why I didn't doubt its functionality later on.

Wanting to copy the latest version of the db to another device, I did the same thing again and copied the db, seeing that its history was more than 7 months old. So I Synchronized again, and saw that the size nor the last Modification Date changed for the db on the original device. So I went to Settings -> Synchronization, which is unfortunately very poorly documented in the app on what it actually does, and enabled Sync and without much thinking pressed the Download button, hoping that it would download the changes to the original db. Even though it should "Download... from the cloud storage", I never set up anything like that so I assumed it would cause no data loss. And without any warning about the huge data discrepancy, the app just overwrote the original db and removing more than 7 months of data...

Expected behavior
There's a few issues here imo.

It's unclear why the original db doesn't get updated with the latest records. This should be basic behavior. It's also unclear why the Synchronize feature in the left menu doesn't perform that data sync

The Synchonization setting should clearly explain what's happening, and where it is downloading form and uploading to. There should be a warning about the risk of data loss, when downloading a db which doesn't have the latest live records.
When enabling sync - in its current way of working -, it should upload the latest records immediately, and would've prevented Downloading an older db to overwrite 7 months of extra records as well. But generally, it shouldn't upload anything anywhere externally without a manual setup first, which also involves considering the security and privacy risks.

The app needs a proper backup feature. This flaw also shows again why syncing is not a backup. One issue or design flaw can cause a huge data loss. I have a backup process myself of my desktop computer, where the db gets synced to very often, but this is useless for the reason explained above: the db that always got synced, never got any updates from the app.

[guanlisheng]

hi @aardbol
sorry to hear your data was lost and great you finally saved it with your own data backup.

in short, the app will temporarily store records in the db defined in the Database path setting as a cache, and keep it synced to Remote path periodically based on your synchronization setting (e.g. the interval to 5 minutes and ensure “Sync on start” from https://moneymanagerex.org/docs/features/usecloud/).

No idea why your sync did not work while can you share some information as the issue template suggested?
Device Information:

• Device: [e.g. LG G7 ThinQ]
• OS: [e.g. Android 8.1]
• Storage: [e.g. Phone storage + Syncthing]
• Synchronization: [e.g. Enabled with 5min interval and Sync on Start]
• Encrypted: [e.g. Yes]
• App Version: [e.g. 2023.12.06]

[aardbol]

I'm afraid you misunderstood the backup part, due to the app not syncing the cache to the db by default or after me pressing the Synchronize button, the db in the backup also had +7 months of difference resulting in me having that huge data loss.

Fairphone 4
Android 13
Sync was disabled
Latest app

[guanlisheng]

The synchronization should be enabled from the very beginning. along with upload and download, all these should have better hints.

may I have your cloud provider and specific version?
we know DropBox application has some issue on its SAF support which caused MMEX can't fetch correct meta to perform the sync.

[aardbol]

Yes, I agree that the hints should be much clearer. But aside from that, it should have sane default settings to avoid such diasters, as well as have additional features for a perfect situation:

• enable cache to persistent syncing by default
• offer automatic periodic sync but also a sync on app start setting
• warn the user that an older version will overwrite a newer one on a manual download, so he's aware of the risk
• add a proper backup feature that keeps multiple versions that's also configurable

I can't stress enough how much this has inconvenienced me so I wanna help as much as I can to get this to work reliabily for everyone. I think I've used the app for at least 10 years and always have been happy, so I can also not thank you enough for continuing to maintain this app. If I had the skill and time, I'd even implement these features myself.

As for your question, I don't use a cloud storage for this due to the sensitivity of the data and never have set it up as mentioned earlier. I store the persistent db in a folder that's synced with many devices and some of them with an active automatic backup.

[guanlisheng]

the first two are in place and you can check them out there `Main Menu -> Settings -> Synchronization`. no matter cloud storage or phone storage, this should be always set up early. the last two would be improved a lot. sorry and thank you again. Leonardo ***@***.***> 于2024年9月20日周五 03:21写道：

…

Yes, I agree that the hints should be much clearer. But aside from that, it should have sane default settings to avoid such diasters, as well as have additional features for a perfect situation: - enable cache to persistent syncing by default - offer automatic periodic sync but also a sync on app start setting - warn the user that an older version will overwrite a newer one on a manual download, so he's aware of the risk - add a proper backup feature that keeps multiple versions that's also configurable I can't stress enough how much this has inconvenienced me so I wanna help as much as I can to get this to work reliabily for everyone. I think I've used the app for at least 10 years and always have been happy, so I can also not thank you enough for continuing to maintain this app. If I had the skill and time, I'd even implement these features myself. As for your question, I don't use a cloud storage for this due to the sensitivity of the data and never have set it up as mentioned earlier. I store the persistent db in a folder that's synced with many devices and some of them with an active automatic backup. — Reply to this email directly, view it on GitHub <#1827 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFQINM7TZOYWK7Q7DLTZCTZXMP3DAVCNFSM6AAAAABOH3IXLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRSGAYDMMBRGE> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- Lisheng

[guanlisheng]

as a quick response to avoid misconfiguration, the sync is always enabled, and sync service is always on after #1838

[aardbol]

That's a great change!

Are you also interested in implementing the last two items from my list in #1827 (comment)?

[guanlisheng]

yup while need a little bit of time
No.3 will consider populating an alerting dialog to double-confirm
No.4 will incline to re-use attachment folder Attachment Preview

[guanlisheng]

hi @aardbol , refer to #1853 for No.3

[aardbol]

Looking good. I'll make a contribution to the docs too in the next days. Besides that, I think it's fair for me to make a donation after having used it for so long. What's the best way for that?

[guanlisheng]

thanks, @aardbol for both. my personal PayPal via email would be great.

[aardbol]

I kept monitoring the sync functionality and it still doesn't seem to work flawlessly. The last modification date of my DB is more than 2 weeks ago. When I try to sync manually, it gives me the error: Not online. My DB is not stored somewhere online, it's stored locally and then automatically synced to remote locations via an independent process.

[guanlisheng]

To conform to Android rules and SAF, only the app's sandbox is internal while the Phone's storage and external service are both external or remote.

you can give it a try after make it online.

may i know some details about your independent process, Syncthing?

[aardbol]

I did grant file system permission though.

I'm using resilio sync

[guanlisheng]

the "Not online" issue for the phone's storage will be refined in the coming release.

by the way, possible to write up something about MMEX + Resilio setup?

[aardbol]

Thank you.

Regarding the docs, sure but it's a simple approach. I tried to put it on the wiki, but it seems the real docs are on an external website. Steps:

• Create a sync folder using the Create folder option (alternatively, you can explore the Add backup option but then changes made on other devices won't be synced back to this device)
• Create a new MMX db and add it to the folder that was created in the previous step
• Go to the folder settings by pressing the i button
  • Deactivate the Selective Sync option (which then will make sure all the files in the folder are always synced)
  • Set up the Allowed Network settings if needed (note that connections are encrypted and a new peer will always need to be approved before syncing works)
  • Go to Preferences and set options as necessary. It's recommended to keep Use archive enabled, so deleted files will be kept in the archive folder until manually removed from there.
• Now start peering that folder with other devices. It's recommended to have at least two peers on different locations, and a backup process running on them keep different versions of the db.

[wolfsolver]

Hi, keep this to simple rule:
/Docs refer to website, this is automatically update after merge
/Doc refer to MD page on GitHub

[Ronkn]

Thank you.

Regarding the docs, sure but it's a simple approach. I tried to put it on the wiki, but it seems the real docs are on an external website. Steps:

• Create a sync folder using the Create folder option (alternatively, you can explore the Add backup option but then changes made on other devices won't be synced back to this device)

• Create a new MMX db and add it to the folder that was created in the previous step

• Go to the folder settings by pressing the i button

  • Deactivate the Selective Sync option (which then will make sure all the files in the folder are always synced)
  • Set up the Allowed Network settings if needed (note that connections are encrypted and a new peer will always need to be approved before syncing works)
  • Go to Preferences and set options as necessary. It's recommended to keep Use archive enabled, so deleted files will be kept in the archive folder until manually removed from there.

• Now start peering that folder with other devices. It's recommended to have at least two peers on different locations, and a backup process running on them keep different versions of the db.

Does this work for more than 1 user of the same database? Id like it if my wife and I could use the app on our phones and use syncthing or resilio to sync to a local server in our home). Not sure how well this method handles conflicts of sync data if there are two inputs to the database.

[aardbol]

Thank you.
Regarding the docs, sure but it's a simple approach. I tried to put it on the wiki, but it seems the real docs are on an external website. Steps:

• Create a sync folder using the Create folder option (alternatively, you can explore the Add backup option but then changes made on other devices won't be synced back to this device)

• Create a new MMX db and add it to the folder that was created in the previous step

• Go to the folder settings by pressing the i button

  • Deactivate the Selective Sync option (which then will make sure all the files in the folder are always synced)
  • Set up the Allowed Network settings if needed (note that connections are encrypted and a new peer will always need to be approved before syncing works)
  • Go to Preferences and set options as necessary. It's recommended to keep Use archive enabled, so deleted files will be kept in the archive folder until manually removed from there.

• Now start peering that folder with other devices. It's recommended to have at least two peers on different locations, and a backup process running on them keep different versions of the db.

Does this work for more than 1 user of the same database? Id like it if my wife and I could use the app on our phones and use syncthing or resilio to sync to a local server in our home). Not sure how well this method handles conflicts of sync data if there are two inputs to the database.

Sorry for the late reply.

I wouldn't recommend it for your use case because a syncing app isn't smart enough to understand WHAT updates are made in your file, it's more about that AN update happened and the modification time of your file has changed. So, in other words, you make a change today at time 13:00 and your wife makes a change at 14:00 without syncing your changes first, that would mean your change would get lost as soon as she syncs because her modification time is newer than yours. That's usually how these apps work AFAIK. I would recommend you to use two separate databases for that or use an online alternative.

[guanlisheng]

the behavior comment is correct, this is exactly how/what the cloud provider does today for file level sync.
refer to https://moneymanagerex.org/docs/features/usecloud/