Sync is broken

[mourningsun75]

Please fix the sync with taskwarrior or at least caldav.

The taskwarrior sync was the reason to use mirakel, but with version 3 the sync ist broken.

It is more important to have a stable sync then a nice ui...

[matthiasbeyer]

I totally agree. Taskwarrior sync is more important than anything else. I'd help if I could, but I'm not a Android dev at all...

[jspricke]

For me it works if I do the initial syn in 2.8.1 and upgrade afterwards to 3.0. Both versions are available in F-Droid.

[mourningsun75]

nothing new? is mirakel dead?

[itadventurer]

Mirakel is not dead, we just have actually absolutely no time to contribute to Mirakel. But we would be very happy to help new devs and merge pull requests ;)

[vonpupp]

I also really would like to use sync since I only use Mirakel due to it's taskwarrior integration, which is no longer working =(

I read somewhere else that the taskwarrior server is buggy, probably on the new dialog warning on Mirakel, Is this feature something doable or should it be dismissed at all and it will never happen? What would be needed to implement it? Any guidance please?

Regards.

[Bugsbane]

DavDroid 0.8+ is supposed to sync tasks, so if I have DavDroid (0.9.4) set up to sync tasks, from owncloud (8.1.3), will these show up in Mirakel?

[matthiasbeyer]

@Bugsbane I don't think so. DavDroid syncs ical tasks, taskwarrior has its own format and needs taskserver therefor. I may be wrong, but I'M pretty sure...

[jspricke]

Mirakel has support for ical tasks, but I haven't tried it. Afaik you need to install Mirakel first and DavDroid afterwards.

[einalex]

Has someone tried that (davdroid sync) successfully? It didn't work for me.

[vonpupp]

I don't think alternative sync methods will properly work, this is why a taskwarrior exists. I remember to have asked a while ago why it wasn't possible to sync against dropbox for instance, and a developer explained that the syncing process is not that simple. I think the right way to go is with the taskwarrior server sync for android, the problem is that I have no clue how to start.

If anyone comes with an alternate working solution, I would be very glad to hear it. I currently moved to todoist =(. I would prefer to use taskwarrior + mirakel with sync working properly, however fixing this seems to be beyond my knowledge. It would be nice if someone could point to the documentation on the sync and what seems to fail so we can try to fix it by ourselves.

Regards.

[Bugsbane]

@jspricke I managed to get Mirakel to see the sync account set up in DavDroid and switch on "Use as default account", but the "Information" panel remains greyed out with an empty server field and the name of the davdroid account for the username, instead of the username actually setup in DavDroid. It also says "Sync manually only" under "Misc" and as far as I can tell, no syncing actually seems to happen, even after DavDroid syncs again.

[tylermenezes]

I just spent a while looking into this, and there are a few things going wrong:

1. When you hit the sync button, the app crashes in the MirakelInternalContentProvider.withTransaction() function because db is null. I changed if (!db.inTransaction()) { to if (db != null && !db.inTransaction()) { and that seems to get sync working enough to download tasks and mark them complete.
2. With that fix, however, it's totally impossible to create new tasks in Mirakel because of a crash in Semantic.createStubTask: currentList.get().isSpecial() returns true, but currentList.get().toSpecial() is absent.
3. (It's also impossible to mark the default "First task" as complete for some reason, and it actually gets synced to taskwarrior, which is annoying, but not a huge issue right now.)

I really have no idea how the app is structured, and there aren't a lot of comments, so I'm sort of just guessing. @azapps, is it possible you could briefly explain what happens when I try to create a task in Mirakel so I have somewhere to start? Or any other advice on what I should look into?

I also can't get the debugger to attach to the sync process (only to the main activity), which is making it more difficult to debug -- I'm not really an Android developer, so if anyone can give me any help on making that happen, it would be appreciated.

[weiznich]

@tylermenezes

1. The problem here is that db should not be null at this point. If db is null noting can be executed, so we need to solve the problem why db is null. Can you post a stacktrace for this?
2. see the following patch. The 3.0 Release does not contain this bug. It was caused by some heavy refactoring of the list classes.
3. Its working for me, maybe the patch for 2 has fixed this.

To attach the debugger to the sync process insert somewhere in the sync process waitForDebugger and then attach the debugger as soon as the process has started.

About the task creation:

1. If you touch the + button a new temporary task is created. To create a task you will need a list and a initial name. Done here
2. The user can change the initial setting.
3. On the first save the Semantics are applied again and may change some properties according to the rules.See this function and this one

[jspricke]

Here is a stacktrace of the current HEAD using assembleFdroidDebug (excluding play-services)

E/ACRA    (18419): de.azapps.mirakelandroid fatal error : null
E/ACRA    (18419): java.lang.NullPointerException
E/ACRA    (18419):      at de.azapps.mirakel.model.provider.MirakelInternalContentProvider.withTransaction(MirakelInternalContentProvider.java:451)
E/ACRA    (18419):      at de.azapps.mirakel.model.task.Task.unsafeSave(Task.java:638)
E/ACRA    (18419):      at de.azapps.mirakel.model.task.Task.save(Task.java:576)
E/ACRA    (18419):      at de.azapps.mirakel.model.task.Task.save(Task.java:571)
E/ACRA    (18419):      at de.azapps.mirakel.sync.taskwarrior.model.TaskWarriorTaskSerializer.serialize(TaskWarriorTaskSerializer.java:114)
E/ACRA    (18419):      at de.azapps.mirakel.sync.taskwarrior.model.TaskWarriorTaskSerializer.serialize(TaskWarriorTaskSerializer.java:51)
E/ACRA    (18419):      at com.google.gson.TreeTypeAdapter.write(TreeTypeAdapter.java:70)
E/ACRA    (18419):      at com.google.gson.Gson.toJson(Gson.java:593)
E/ACRA    (18419):      at com.google.gson.Gson.toJson(Gson.java:572)
E/ACRA    (18419):      at com.google.gson.Gson.toJson(Gson.java:527)
E/ACRA    (18419):      at com.google.gson.Gson.toJson(Gson.java:507)
E/ACRA    (18419):      at de.azapps.mirakel.sync.taskwarrior.TaskWarriorSync.taskToJson(TaskWarriorSync.java:599)
E/ACRA    (18419):      at de.azapps.mirakel.sync.taskwarrior.TaskWarriorSync.getMsg(TaskWarriorSync.java:573)
E/ACRA    (18419):      at de.azapps.mirakel.sync.taskwarrior.TaskWarriorSync.sync(TaskWarriorSync.java:543)
E/ACRA    (18419):      at de.azapps.mirakel.sync.taskwarrior.services.SyncAdapter.onPerformSync(SyncAdapter.java:116)
E/ACRA    (18419):      at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:259)

[jspricke]

Same crash (modulo line numbers) happens when using the current releases in Google and F-Droid. Steps to reproduce:

1. Generate a fresh account on https://www.freecinc.com/.
2. Use a freshly installed Mirakel
3. Setup the Freecinc account
4. Press sync

same steps work fine using Mirakel 2.8.2 from F-Droid.

[emembrives]

I looked into it a bit.

• db in withTransaction (inside model/src/main/java/de/azapps/mirakel/model/provider/MirakelInternalContentProvider.java) is null because getWritableDatabase() returns null.
• getWritableDatabase() returns null because dbHelper is null (database is also null, but would get filled-in if dbHelper was fine). This is suggesting that onCreate is not called.
• DatabaseHelper.getDatabaseHelper() needs a Context, which is unavailable in the getWritableDatabase() static method. However, as DatabaseHelper is a singleton, the Context is only needed to create the object the first time. I added a method to get an existing DatabaseHelper to the DatabaseHelper class. It turns out that there is already existing DatabaseHelper, so we can get one and create the database object.
• However, the code now fails to acquire a lock on the database, so syncing still fails.

Now entering the realm of speculation:

• In the Mirakel app, it seems that DatabaseHelper is created twice: one in de.azapps.mirakelandroid in MirakelContentProvider.java and one in de.azapps.mirakelandroid:sync in Mirakel.java.
• It would seem that these are two different processes used by the app.
• However, my understanding is that if one process accesses the database and locks it, the other won't be able to access it. Hence the issue.

[weiznich]

@emembrives You are near the real problem.
In android a sync adapter and a process are running in two different processes. This would be no problem if database transactions are build with the help of sqlite. But they are implemented on top of sqlite, so there is no way to synchronize this across multiple processes.
As I understood this at the beginning of the year a started to rewrite how transactions are handled in mirakel. This means rewriting some of the central model code of mirakel. To verify that there is no new bug introduced by the rewrite, a decide to fix the tests first.
Because of this not all test are executed, so I could not verify everything is working again.
For anyone is interested in this: I've uploaded the current version of the patch, but be warned that there may some new bugs in mirakel itself.

[emembrives]

@weiznich Thanks for looking into this!

I tried to look at your patch, but I get a 504 error. I'll try to test it another day.

[weiznich]

@emembrives Gerrit is up again

[rosshadden]

👍 This has always been the biggest draw to Mirakel. Now that it no longer works, most of the prose written about this app (including the official website) is completely wrong.

[lenucksi]

I've tried DAVDroid however, with no good results. I do have very satisfactory results with @dmfs CalDAV Syncadapters.
Collaboration should absolutely be striven for!

[weiznich]

@lenucksi Please open a new issue for this. This issue is for sync with taskd.
If possible add the following informations to the new issue:

• What did you do exactly
• What did happen
• What did you expect to happen
• If there was an crash, please attach a stacktrace(from adb)

[mourningsun75]

Any news here?

[weiznich]

There is this patch which should fix this problem. It's not merged yet because it touches most of Mirakels internal database handling. Before merging we will need to get our testsuite running again. This is blocked by an issue in robolectric.

[radfish]

Looks like I just re-traced @emembrives's steps (wasn't aware of this ticket).
While the big fix is in the works, the patch that fixed the crash on sync for me is the following three lines. Perhaps this introduces the possibility of data loss if the update sequence that was supposed to be in the transaction gets interrupted, but it seems to at least work.

diff --git a/model/src/main/java/de/azapps/mirakel/model/provider/MirakelInternalContentProvider.java b/model/src/main/java/de/azapps/mirakel/model/provider/MirakelInternalContentProvider.java
index 3b99272..5de7de9 100644
--- a/model/src/main/java/de/azapps/mirakel/model/provider/MirakelInternalContentProvider.java
+++ b/model/src/main/java/de/azapps/mirakel/model/provider/MirakelInternalContentProvider.java
@@ -449,6 +452,11 @@ public class MirakelInternalContentProvider extends ContentProvider implements
     public static void withTransaction(final Context ctx, final DBTransaction what) {
         if (what != null) {
             final SQLiteDatabase db = getWritableDatabase();
+            if (db == null) {
+                what.exec();
+                return;
+            }
+
             if (!db.inTransaction()) {
                 db.beginTransaction();
                 try {

Note that calling DatabaseHelper.getDatabaseHelper from withTransaction (even having passed a context object to it from all call sites), does not work due to a deadlock (probably the same one that was mentioned earlier in the ticket).

[p-vitt]

Any news on this topic? Would be really nice to get rid of this problem.

[vonpupp]

It would be awesome to have Mirakel back but since it has been a while, there is a possibility of not having Mirakel back, so to all of you interested in an alternative, check out taskwarriorandroid. I installed it recently and sync seems to work fine.

[itadventurer]

Hi guys,

Sorry for the bad news :(

It was a great journey to build Mirakel and experience such great support from our community (YOU!), but we are not able to continue the development any further :(

We do not see any way to support Mirakel any further and am searching for a new maintainer. We would be more than happy if Mirakel could be reborn and of course we would also provide help and guidance for new maintainers if needed. If you are interested, send us an e-mail to [email protected]

Best,
Georg and Anatoly

[Bugsbane]

Well, thanks for all of the effort over the years. Ironically, other than sync, 3.0 was the first Foss task management app that ever managed to actually do all that I needed it to. Thanks for opting out in the right way, too (ie announcing it definitively so we users don't hold our breathes waiting, offering it for a new maintainer, and offering to help with the transition).

Just out of interest, what was the main problem? Trouble turning it into income to support your work? Loss of interest? Going into a different area of software development?

[weiznich]

Just out of interest, what was the main problem? Trouble turning it into income to support your work? Loss of interest? Going into a different area of software development?

Its a combination out of different factors. The main factor is in my opinion that we are both studying in a subject not related to any mobile app development. So our interests shifted slightly. In combination with this, there was no one that really had tried to support us while developing new features. Also we noticed that some parts of the source code need a complete rewrite to be more maintainable, but there was no time beside our studies to achieve such a big rewrite.

Well, thanks for all of the effort over the years. Ironically, other than sync, 3.0 was the first Foss task management app that ever managed to actually do all that I needed it to.

There is this patch which may solve you problems. I never had the time to do the required in depth testing, so it is quite likely that something is broken, because this changes some internal widely used internal functionality.

[bepolymathe]

What's the alternative now ?

[Bugsbane]

The best alternative I've managed to find so far is Tasks. It also doesn't have CalDav sync in the main release yet, however there's work on it and it seems to be getting close, so I encourage everyone to go and help test!