Different FPS of Replica office0

[jinmaodaomaye2021]

Hi,

Thanks for your great work. Not sure if the algorithm was tested with large view movements.

Since the Replica office0 video has high FPS, I tested the algorithm at different FPS settings.

The algorithm works fine when the FPS was reduced to 1/2, 1/3, 1/5, 1/10 and 1/20 with modifications of parameters. But the algorithm failed to estimate poses at 1/30.

There is still a large view overlapping between frames by visually inspecting the 1/30 FPS data. However, the results are very bad (wrong poses).

In my experiment, I modified the following parameters at 1/30:

• mapping_iters = 500
• mapping_keyframe_every = 1
• mapping-map_every = 1
• mapping_n_pixels = 0.1
• mapping_first_iters = 500
• tracking_iters = 500

Any idea why the algorithm doesn't work for low FPS videos ?
What I observed is that if the pose is incorrect from tracking, it is hard to correct in mapping.

[HengyiWang]

Hi @jinmaodaomaye2021, thanks for your interest in our work. Interesting question.

For each scene, there are 2000 frames in total. What you mean is that Co-SLAM can run with only 100 frames and fail at 65 frames. I assume in Replica, the video is simulated with 10-30Hz. Then, in your setting, it would be a 0.3-1Hz video sequence. There are several reasons that might result in losing track in this case:

1. In the tracking process, we only sample 1024 pixels for optimizing camera poses. In your case, part of the samples will be in unobserved regions which would lead to losing track. To address this, I would recommend filtering out outliers from the samples or ensuring that the sampling is performed only in overlapping regions to maintain tracking robustness.

2. The truncation distance is 10cm. You may need to check whether the depth search range is large enough to hit the truncation region.

3. It appears that you set n_pixels=0.1, and you have 65 keyframes in total. This means you are trying to reconstruct the scene with a pixel count of 6.5 frames. I think this amount of pixels can not have enough overlapping for BA. (BTW, if you do not increase #samples for tracking or #keyframe pixels for BA, simply increasing #iters for tracking and mapping might not work very well. 500 tracking and mapping iterations are too much for 1024 tracking samples and 6.5 frame pixels)

I hope these suggestions help. If you have further questions or need more assistance, feel free to ask:)

[jinmaodaomaye2021]

@HengyiWang Thanks for your reply.

Please check the logs below which explains why I chose 100-500 tracking/mapping iterations for low FPS videos (1/20 FPS, tracking_iters=400, mapping_iters=100, tracking_samples=1024, mapping_samples=2048).

I printed out the losses every 10 tracking/mapping iterations. It seems the tracking stage takes many steps to converge for large camera movements (constant_speed=True, camera pose learning rate = 1e-3). If the tracking/mapping stage takes > 100 iterations, the processing time would be very high (3-8 seconds per frame on my machine if mapping_map_every=1, mapping_keyframe_every=1).

tracking iter: 0 loss: 1.481
tracking iter: 10 loss: 1.470
tracking iter: 20 loss: 1.455
tracking iter: 30 loss: 1.438
tracking iter: 40 loss: 1.431
tracking iter: 50 loss: 1.413
tracking iter: 60 loss: 1.401
tracking iter: 70 loss: 1.372
tracking iter: 80 loss: 1.331
tracking iter: 90 loss: 1.265
tracking iter: 100 loss: 1.178
tracking iter: 110 loss: 1.133
tracking iter: 120 loss: 1.118
tracking iter: 130 loss: 1.110
tracking iter: 140 loss: 1.098
tracking iter: 150 loss: 1.087
tracking iter: 160 loss: 1.070
tracking iter: 170 loss: 1.041
tracking iter: 180 loss: 1.011
tracking iter: 190 loss: 0.952
tracking iter: 200 loss: 0.893
tracking iter: 210 loss: 0.792
tracking iter: 220 loss: 0.671
tracking iter: 230 loss: 0.552
tracking iter: 240 loss: 0.446
tracking iter: 250 loss: 0.373
tracking iter: 260 loss: 0.320
tracking iter: 270 loss: 0.278
tracking iter: 280 loss: 0.246
tracking iter: 290 loss: 0.227
tracking iter: 300 loss: 0.222
tracking iter: 310 loss: 0.221
tracking iter: 320 loss: 0.222
tracking iter: 330 loss: 0.215
tracking iter: 340 loss: 0.204
tracking iter: 350 loss: 0.199
tracking iter: 360 loss: 0.187
tracking iter: 370 loss: 0.183
tracking iter: 380 loss: 0.181
tracking iter: 390 loss: 0.182

Best loss: 0.020, Last loss: 0.020

mapping iter 0 loss: 0.148
mapping iter 1 loss: 0.147
mapping iter 2 loss: 0.147
mapping iter 3 loss: 0.141
mapping iter 4 loss: 0.136
mapping iter 5 loss: 0.141
mapping iter 6 loss: 0.146
mapping iter 7 loss: 0.142
mapping iter 8 loss: 0.138
mapping iter 9 loss: 0.135
mapping iter 10 loss: 0.136
mapping iter 11 loss: 0.137
mapping iter 12 loss: 0.135
mapping iter 13 loss: 0.135
mapping iter 14 loss: 0.132
mapping iter 15 loss: 0.134
mapping iter 16 loss: 0.133
mapping iter 17 loss: 0.134
mapping iter 18 loss: 0.132
mapping iter 19 loss: 0.134
mapping iter 20 loss: 0.131
mapping iter 21 loss: 0.127
mapping iter 22 loss: 0.132
mapping iter 23 loss: 0.128
mapping iter 24 loss: 0.136
mapping iter 25 loss: 0.130
mapping iter 26 loss: 0.133
mapping iter 27 loss: 0.126
mapping iter 28 loss: 0.128
mapping iter 29 loss: 0.131
mapping iter 30 loss: 0.127
mapping iter 31 loss: 0.125
mapping iter 32 loss: 0.129
mapping iter 33 loss: 0.128
mapping iter 34 loss: 0.132
mapping iter 35 loss: 0.129
mapping iter 36 loss: 0.125
mapping iter 37 loss: 0.127
mapping iter 38 loss: 0.127
mapping iter 39 loss: 0.130
mapping iter 40 loss: 0.126
mapping iter 41 loss: 0.128
mapping iter 42 loss: 0.124
mapping iter 43 loss: 0.126
mapping iter 44 loss: 0.122
mapping iter 45 loss: 0.128
mapping iter 46 loss: 0.128
mapping iter 47 loss: 0.118
mapping iter 48 loss: 0.123
mapping iter 49 loss: 0.125
mapping iter 50 loss: 0.129
mapping iter 51 loss: 0.125
mapping iter 52 loss: 0.121
mapping iter 53 loss: 0.126
mapping iter 54 loss: 0.128
mapping iter 55 loss: 0.122
mapping iter 56 loss: 0.125
mapping iter 57 loss: 0.125
mapping iter 58 loss: 0.121
mapping iter 59 loss: 0.122
mapping iter 60 loss: 0.121
mapping iter 61 loss: 0.120
mapping iter 62 loss: 0.125
mapping iter 63 loss: 0.118
mapping iter 64 loss: 0.118
mapping iter 65 loss: 0.117
mapping iter 66 loss: 0.120
mapping iter 67 loss: 0.123
mapping iter 68 loss: 0.120
mapping iter 69 loss: 0.124
mapping iter 70 loss: 0.122
mapping iter 71 loss: 0.117
mapping iter 72 loss: 0.119
mapping iter 73 loss: 0.120
mapping iter 74 loss: 0.120
mapping iter 75 loss: 0.114
mapping iter 76 loss: 0.118
mapping iter 77 loss: 0.120
mapping iter 78 loss: 0.115
mapping iter 79 loss: 0.121
mapping iter 80 loss: 0.117
mapping iter 81 loss: 0.119
mapping iter 82 loss: 0.116
mapping iter 83 loss: 0.122
mapping iter 84 loss: 0.113
mapping iter 85 loss: 0.113
mapping iter 86 loss: 0.120
mapping iter 87 loss: 0.118
mapping iter 88 loss: 0.118
mapping iter 89 loss: 0.117
mapping iter 90 loss: 0.111
mapping iter 91 loss: 0.119
mapping iter 92 loss: 0.112
mapping iter 93 loss: 0.118
mapping iter 94 loss: 0.116
mapping iter 95 loss: 0.116
mapping iter 96 loss: 0.115
mapping iter 97 loss: 0.111
mapping iter 98 loss: 0.112
mapping iter 99 loss: 0.112

Update current pose

add keyframe: 2

[HengyiWang]

@jinmaodaomaye2021, thanks for providing the log. I assumed you have tried different lr for this. Then, can you try 1 to see if that help? A simple way is to select several frames, and manually set a sample region that ensures the overlapping.