Certain messages are not available after re-login

[daniel-abramov]

Steps to reproduce

1. I was using Element 1.11.8 on Mac.
2. Logged out.
3. Reset the password.
4. Logged in again.
5. In all chats, I could only see the last 2-3 messages (and no loading animation or something), the content of all of which was not shown due to the error: ** Unable to decrypt: The sender's device has not sent us the keys for this message. **.
6. After about 20 seconds, messages in the chat got loaded, and I could see their content, except that some messages (in my case, 2 last messages from the 1:1 chat) were not shown due to ** Unable to decrypt: The sender's device has not sent us the keys for this message. ** error. Clicking on "Re-request encryption keys" only says "Key request sent.", but the content of the message is still not shown.
7. Meanwhile, all other messages (the content of which has been successfully decrypted) have a grey shield icon. Each incoming message has a tooltip The authenticity of this encrypted message can't be guaranteed on this device, while all outgoing messages have Encrypted by a deleted session.
8. The other person in the 1:1 chat gets online, and we start talking, but the aforementioned 2 messages are still not available. Once that other person got online, one of those 2 messages turned into an ** Unable to decrypt: Error: OLM.UNKNOWN_MESSAGE_INDEX ** error.

The issue is related to several other issues, so I was not sure if it exists, but with a new reproduction scenario or a new one, I think it might be related to these:

• https://github.com/vector-im/element-web/issues?q=is%3Aissue+is%3Aopen+OLM.UNKNOWN_MESSAGE_INDEX
• https://github.com/vector-im/element-web/issues?q=is%3Aopen+is%3Aissue+sort%3Areactions-%2B1-desc+*+Unable+to+decrypt%3A+The+sender%27s+device+has+not+sent+us+the+keys+for+this+message.+**

There are quite a few related issues, so I decided to create a new one to describe the reproduction steps (so that in the worst case, I can just provide more info for the dev folks to be added to the tracking issue).

Outcome

What did you expect?

I just wanted to see my messages after logging in.

What happened instead?

Certain messages cannot be decrypted and have quite an "unfriendly" (from the user perspective) error. I have to ask the person to re-send the messages, telling them that what they sent to me is not readable on my device for some unknown reason. That's a bit embarrassing.

Operating system

Mac OS Big Sur

Application version

Element 1.11.8

How did you install the app?

From the official website

Homeserver

matrix.org

Will you send logs?

Yes

[uhoreg]

It sounds like the undecryptable events were sent while you were logged out. In this case, the sender doesn't have any device to send the decryption keys to, so you are unable to decrypt the message. We are working on a solution to this, but we're working through some issues.

[uhoreg]

FWIW, this will be fixed when #22711 gets implemented

[daniel-abramov]

It sounds like the undecryptable events were sent while you were logged out. In this case, the sender doesn't have any device to send the decryption keys to, so you are unable to decrypt the message.

Yeah, it could be, just in case: the sender was online at the time I logged in again. I.e. I could exchange new messages with the sender, but those 2 other messages remained encrypted and I could not see the content of them in any of the active sessions (albeit those 2 messages had different errors shown as mentioned in the issue description), so I had to ask the sender to re-send them.

I've encountered the same issue again, this time in element-web (not in Element Desktop) on a different device with even more odd behavior: I had 2 sessions opened: one session in Chrome and another in Firefox. Both sessions had a 1:1 chat open. The thing is: one of the sessions (in Chrome) could not display two last messages with the same error whereas the other session (Firefox) displayed them just fine. I tried to click on "Request session keys", tried restarting both browsers, refreshing the page etc, but nothing helps: the session in Firefox shows the messages whereas the session in Chrome does not. Quite odd. All other messages are shown just fine. I've sent the debug logs.

[uhoreg]

Hmm. If you had multiple sessions open, and one of them can decrypt the messages, your other devices should be able to get the keys from that device. So this does sound (at least partially) like a different issue.

[feroom]

I encounter a similar problem if the interlocutor has a bad connection or at the time of sending a message, he loses connection.

{ "type": "m.room.message", "content": { "msgtype": "m.bad.encrypted", "body": "** Unable to decrypt: decryption key withheld **" } }

Perhaps there is a need for a mechanism for re-requesting the keys from the interlocutor or verifying the state of the keys for decryption.

[pizdjuk]

I encounter a similar problem if the interlocutor has a bad connection or at the time of sending a message, he loses connection.

{ "type": "m.room.message", "content": { "msgtype": "m.bad.encrypted", "body": "** Unable to decrypt: decryption key withheld **" } }

Perhaps there is a need for a mechanism for re-requesting the keys from the interlocutor or verifying the state of the keys for decryption.

same here. Was about to connect to VPN, and in this momen my interlocutor sent me a message. This was unable to decrypt. But the worse thing, that ALL of messages after it in this session were unable to decrypt.

[menturion]

@uhoreg

FWIW, this will be fixed when #22711 gets implemented

When will device hydration finally be implemented?
There are so many bug reports here that stem from the lack of this long-awaited and absolutely essential feature.

[vertigo220]

Not sure if my issue is related to this or different and should be its own, separate issue, but I have what seems to be a similar problem. The other night, I was in a DM with a friend in Element Desktop and I signed out to try to switch servers. I ended up not switching (so nothing changed, except that I changed the server and tried unsuccessfully to log into the new one, then changed it back to matrix.org, not sure if that would affect this), and when I logged back in, it said I needed to verify the device and I chose to do so via passphrase, which I supplied and it accepted. I then got a message saying the following:

Your new device is now verified. It has access to your encrypted messages, and other users will see it as trusted.

Once it loaded, I reentered the DM and it showed the following message:

Decrypting messages...
Please wait as we try to decrypt your messages. This may take a few moments.

which then changed to

Open another device to load encrypted messages
This device is requesting decryption keys from your other devices. Opening one of your other devices may speed this up.

I tried the "Resend key requests" button to no avail, and after ~38 hours nothing has changed and a few of the messages sent by my friend show as "Unable to decrypt message." It just periodically keeps cycling between the "Decrypting messags..." and "Open another device to load encrypted messages" banner at the top.

I originally didn't realize it was just these messages and thought they were all encrypted (or maybe they were and now they're not, but I think I just didn't notice), so I thought it was a different issue. So unfortunately, I'm not sure, but looking at the timestamps, I think maybe the messages from them that I can't decrypt were sent when I was signed out. But since they were online, once I signed in it should have been able to do a key exchange if necessary to allow me to see those messages. Also, I just checked Element Web to see if that would get them to decrypt (it didn't), and that only shows one "message" (a call from them) with a timestamp in the range of messages that won't decrypt in Element Desktop (four total).

Edit/Update: I was getting really annoyed at having the banner constantly at the top, made even worse by it constantly changing back and forth, so I reopened Element Web yet again, and this time it finally decrypted the messages, which is good, but also sort of worse, since it means it could and should have done so the first time I opened Element Web, and really when I entered my passphrase originally. Also, the four messages that said they couldn't be decrypted in Element Desktop are now just the one "message" that is the voice call from my friend, as shown in Element Web, so not sure what that's about, showing four when it's really one.

Update 2: This has now happened multiple times and is pretty much a regular occurrence every time I logout and log back in.

[mrx23dot]

How come this is not an issue in Signal and Wire? they just work out of the box.

Their servers cache the encrypted message until I login again.
And no need to fiddle with the keys after I joined a room.

[uhoreg]

I'm pretty sure this is also an issue with Signal and Wire, because they use a similar encryption system to us. Have you actually tried it? IIRC with Signal, you need to wait for the sender to re-send messages that were sent while you were offline. I don't know how Wire behaves.

[mrx23dot]

I double checked, Wire does warn that it might loose history if I was offline for long time,
but I tried, no issue for short times. I logged out, sent messages from another user to common group, I logged in, I saw the encrypted messages.

Which is what I would expect, since I have the old keys stored in permanent storage,
server stores encrypted messages till I'm offline,
I login, everything is given to continue where I left off.

Given the server+client storage has enough capacity to look back.
Or what am I missing?
I admit it can get complicated when I'm logged into to multiple devices with same user.

[uhoreg]

Was your sending device online when you logged back in? The message may have been re-sent to your new device when you logged back in. Or perhaps they've implemented something equivalent to our plan for dehydrated devices.

Wire uses an encryption system similar to what we're using. The problem described in this issue is a limitation of their encryption system too. They may have worked around the limitation in some way, and may have already implemented things that we're planning on implementing, but I can guarantee you that it's something that they had to figure out how to solve at some point.

It's not just a matter of server/client storage capacity. It's a matter of making a tradeoff between convenience and the security properties that the underlying encryption system is supposed to give you. If you are able to log in to a new device and read old encrypted messages, whether in Element, Wire, Signal, WhatsApp (they all use similar encryption systems), then the program that you're using is working around limitations of the encryption system in some way, in favour of convenience.

I should note that this issue is a limitation of the encryption system by design. It's a feature called "post-compromise security" or "future secrecy". Basically, if somebody compromises one of your devices, they should not be able to read messages that are sent to you after the compromise has been resolved.

[richvdh]

resolving in favour of element-hq/element-meta#245