Visualizing saved reconstruction.

[pytholic]

Hi. Thank you for the amazing work.

I am running the code on remote server in --disable_vis mode. I have the saved reconstruction results in .npy format. I am not sure how to reconstruct the scene in Open3d using these .npy files.

Before that I was using visualization mode and directly saving .ply files and I was able to visualize them in open3d. But I don't want to do that anymore and want to use the default .npy files.

Thank you~

[pytholic]

Can we please get some guide on this?

[bkhanal-11]

I do not know how famailer you are with Open3D, but you can use RGBD interation to construct .ply files from data produced in .npy files.

You can check this for more info.

[pytholic]

@bkhanal-11 Hi. I wrote the following code.

import glob
import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt


class CameraPose:
    def __init__(self, meta, mat):
        self.metadata = meta
        self.pose = mat

    def __str__(self):
        return (
            "Metadata : "
            + " ".join(map(str, self.metadata))
            + "\n"
            + "Pose : "
            + "\n"
            + np.array_str(self.pose)
        )


def read_trajectory(filename):
    traj = []
    with open(filename, "r", encoding="utf-8") as f:
        metastr = f.readline()
        while metastr:
            metadata = list(map(int, metastr.split()))
            mat = np.zeros(shape=(4, 4))
            for i in range(4):
                matstr = f.readline()
                mat[i, :] = np.fromstring(matstr, dtype=float, sep=" \t")
            traj.append(CameraPose(metadata, mat))
            metastr = f.readline()
    return traj


volume = o3d.pipelines.integration.ScalableTSDFVolume(
    voxel_length=4.0 / 512.0,
    sdf_trunc=0.04,
    color_type=o3d.pipelines.integration.TSDFVolumeColorType.RGB8,
)

intrinsics = np.load("./results/intrinsics.npy")

fx = intrinsics[0][0]
fy = intrinsics[0][1]
cx = intrinsics[0][2]
cy = intrinsics[0][3]

color_images = np.load("./results/images.npy")
depth_images = np.load("./results/disps.npy")
camera_poses = read_trajectory("./results/log.txt")
camera_intrinsic = o3d.camera.PinholeCameraIntrinsic(
    width=576, height=336, fx=fx, fy=fy, cx=cx, cy=cy
)

for i in range(len(color_images)):
    print(f"Integrate {i}-th image into the volume.")
    color = color_images[i]
    color = np.ascontiguousarray(color.transpose(1, 2, 0))
    color = o3d.geometry.Image((color).astype(np.uint8))
    depth = depth_images[i]
    depth = np.ascontiguousarray(depth.transpose(0, 1))
    depth = o3d.geometry.Image((depth * 255))
    rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(
        color, depth, depth_trunc=4.0, convert_rgb_to_intensity=False
    )

    volume.integrate(rgbd, camera_intrinsic, np.linalg.inv(camera_poses[i].pose))

print("Extract a triangle mesh from the volume and visualize it.")
mesh = volume.extract_triangle_mesh()
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])

However, the resulting map is not correct. I am missing something but not sure what it is. Here is the result.

[pytholic]

@bkhanal-11 This is how my log.txt looks like.

0 0 1 
-1.0 0.0 0.0 0.0 
0.0 -1.0 0.0 0.0 
0.0 0.0 1.0 0.0 
0.0 0.0 0.0 1.0 
1 1 2 
-0.9988476022589705 -0.02794569745350285 0.039019295936009724 -0.017698440700769424 
0.02813569752382185 -0.9995947404348644 0.004328674139191881 -0.010660745203495026 
0.03888251517523815 0.005421520892940287 0.9992290814046871 -0.010660745203495026 
0.0 0.0 0.0 1.0 
2 2 3 
-0.9921915766540266 -0.07212852491360724 0.10175141822394249 -0.15529966354370117 
0.0741547246722205 -0.9971140443786379 0.016268353071603008 -0.02430793270468712 
0.10028435583670908 0.02368667128708258 0.9946768267018962 -0.02430793270468712 
0.0 0.0 0.0 1.0 
3 3 4 
-0.9858344461233062 -0.09958461872798406 0.1349568395804975 -0.229265496134758 
0.10741816380243378 -0.9928507594990048 0.05204524423578825 -0.11546353995800018 
0.1288090948732565 0.06580481042485715 0.9894836754605307 -0.11546353995800018 
0.0 0.0 0.0 1.0 
4 4 5 
-0.9800208043395229 -0.11316477936030064 0.16356330815329656 -0.30861613154411316 
0.12554535795960764 -0.9897944798567042 0.06741847476761405 -0.17945930361747742 
0.1542646626953489 0.08660612194026414 0.9842264950131948 -0.17945930361747742 
0.0 0.0 0.0 1.0 

[bkhanal-11]

@pytholic Hi, I also tried to write script for visualization and end up with similar result as yours. It may be due to either intrinsic or disparity (depth). I am also stuck on same problem now.

[kelvinyankey6]

@bkhanal-11 @pytholic Hi. Sorry just want to ask if you ever figured out why this was happening? Also running this on a remote server and have to use --disable_vis but im also getting a mess after trying reconstruction?

[pytholic]

@kelvinyankey6 My workaround was to save points and poses in .ply file on the fly.
https://github.com/pytholic/ml_cv_scripts_guides/blob/main/droid_slam/vis_headless.py

[kelvinyankey6]

Ah @pytholic I did the same to save the points. My environment doesn't have access to an x server. But I didn't know I could do the same for the camera poses. This is really helpful. Thank you very much for getting back to me

[NOTGOOOOD]

@kelvinyankey6 My workaround was to save points and poses in .ply file on the fly. https://github.com/pytholic/ml_cv_scripts_guides/blob/main/droid_slam/vis_headless.py

Hello, thanks for your code! I can't save the '.ply' in that way. Because I running this on a remote server, this means it won't go into 'droid_visualization' function.

I had a similar problem when I used open3d to visualize the saved reconstruction in sfm_bench dataset.

My approach is to use pose, intrinsics and depth to directly recover the 3D coordinates of the point.

def visual(disp, K, T, image):

    depth = 1.0 / disp.squeeze()    # h, w
    height, width = depth.shape
    img = cv2.resize(cv2.cvtColor(image, cv2.COLOR_BGR2RGB), dsize=(width, height), interpolation=cv2.INTER_LINEAR)
    fx, fy, cx, cy = K
    intrinsics_matrix = np.array([[fx, 0, cx],
                                  [0, fy, cy],
                                  [0, 0, 1]])
    intrinsics_matrix_homo = np.zeros((3, 4), dtype=np.float32)
    intrinsics_matrix_homo[:3, :3] = intrinsics_matrix
    intrinsics_matrix_homo[2,3] = 1

    pose = pose_quaternion_to_matrix(T)

    points = []
    rgbs = []
    for v in range(height):
        for u in range(width):
            d = depth[v, u]

            if d==0: continue

            x_cam = (u - cx) / fx * d
            y_cam = (v - cy) / fy * d
            z_cam = d
            points.append([x_cam, y_cam, z_cam])
            rgbs.append(img[v, u, :])
    
    rgbs = np.array(rgbs)
    points = np.array(points)
    ones = np.ones((points.shape[0], 1))
    points = np.hstack((points, ones))  # homo

    pc_world = (pose @ points.T).T

    return points, pc_world, intrinsics_matrix_homo, pose, rgbs

  points, pc_world, intrinsics_matrix_homo, pose, rgbs = visual(disp, K, T, image)
  pcd = o3d.geometry.PointCloud()
  pcd.points = o3d.utility.Vector3dVector(pc_world[:, 3])
  pcd.colors = o3d.utility.Vector3dVector(rgbs / 255.0)
  o3d.visualization.draw_geometries([pcd])

[pytholic]

@kelvinyankey6 My workaround was to save points and poses in .ply file on the fly. https://github.com/pytholic/ml_cv_scripts_guides/blob/main/droid_slam/vis_headless.py

Hello, thanks for your code! I can't save the '.ply' in that way. Because I running this on a remote server, this means it won't go into 'droid_visualization' function.

I had a similar problem when I used open3d to visualize the saved reconstruction in sfm_bench dataset.

My approach is to use pose, intrinsics and depth to directly recover the 3D coordinates of the point.

def visual(disp, K, T, image):



    depth = 1.0 / disp.squeeze()    # h, w

    height, width = depth.shape

    img = cv2.resize(cv2.cvtColor(image, cv2.COLOR_BGR2RGB), dsize=(width, height), interpolation=cv2.INTER_LINEAR)

    fx, fy, cx, cy = K

    intrinsics_matrix = np.array([[fx, 0, cx],

                                  [0, fy, cy],

                                  [0, 0, 1]])

    intrinsics_matrix_homo = np.zeros((3, 4), dtype=np.float32)

    intrinsics_matrix_homo[:3, :3] = intrinsics_matrix

    intrinsics_matrix_homo[2,3] = 1



    pose = pose_quaternion_to_matrix(T)



    points = []

    rgbs = []

    for v in range(height):

        for u in range(width):

            d = depth[v, u]



            if d==0: continue



            x_cam = (u - cx) / fx * d

            y_cam = (v - cy) / fy * d

            z_cam = d

            points.append([x_cam, y_cam, z_cam])

            rgbs.append(img[v, u, :])

    

    rgbs = np.array(rgbs)

    points = np.array(points)

    ones = np.ones((points.shape[0], 1))

    points = np.hstack((points, ones))  # homo



    pc_world = (pose @ points.T).T



    return points, pc_world, intrinsics_matrix_homo, pose, rgbs

  points, pc_world, intrinsics_matrix_homo, pose, rgbs = visual(disp, K, T, image)

  pcd = o3d.geometry.PointCloud()

  pcd.points = o3d.utility.Vector3dVector(pc_world[:, 3])

  pcd.colors = o3d.utility.Vector3dVector(rgbs / 255.0)

  o3d.visualization.draw_geometries([pcd])

Use open3d headless rendering and it works.

[kelvinyankey6]

@pytholic forgot to respond to this. But I used your method. It worked. Thanks

[pytholic]

@kelvinyankey6 cool!
@NOTGOOOOD please try it with headless rendering, will work hopefully!

[ERGOWHO]

@kelvinyankey6 cool! @NOTGOOOOD please try it with headless rendering, will work hopefully!

Hi thanks for sharing your vis_headless.py code, I can do visualization on my device so I would like to ask how to use your code? like shall I download ml_cv_scripts_guides/droid_slam/ folder and place the files under the original DROIDSLAM project under and then just run python demo.py --imagedir= data/exampledata --calib=calib/exampledata.txt --reconstruction_path=exampledata?

Or is there any other special step to use your vis_headless.py to get ply?

[ERGOWHO]

@kelvinyankey6 cool! @NOTGOOOOD please try it with headless rendering, will work hopefully!

Thank you so much for sharing your code, I was able to get the point cloud ply file I want.
For those may have same question with me: I used demo.py to replace the original Droidslam demo.py and vis_headless.py to replace the original code visuallization.py. and simply change line 38 in droid.py to set the save path:
self.visualizer = Process(target=droid_visualization, args=(self.video,"saveplypath"))

[Zunhammer]

HI, thanks for your work and explanations. I'm running the code in a docker container and the headless script is not working there, I get the error:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/DROID-SLAM/vis_headless.py", line 175, in droid_visualization
vis.get_render_option().load_from_json("misc/renderoption.json")
AttributeError: 'NoneType' object has no attribute 'load_from_json'

So it seems it cannot create the open3d visualizer. Is th code supposed to able to run in docker or am I missing something?

Thanks

[pytholic]

@Zunhammer Not sure. I think I have used it with docker in the past but that was my custom docker.

Here is my docker for Droid Slam:
#98

I created PR but no one is maintaining. You can check this:
http://www.open3d.org/docs/0.16.0/docker.html

[HtutLynn]

@pytholic @Zunhammer , Hi, i am also using the docker provided by @pytholic and trying to generate ply files from the docker container. However, I am having the same problem as @Zunhammer. Is there anyway to solve this problem?

[Zunhammer]

First, thanks to @pytholic for the response and the hints. I tried with my own docker but had no luck, didn't have time to test the one from pytholic yet. I'll probably do this after christmas. So in short, I don't have a solution yet.

[HtutLynn]

@Zunhammer , I solved this issue already so may be it would be helpful to you too. @pytholic's Dockerfile works great. But to use it in a headless environment and still to be able to save ply files, you also need to use open3d headless. After building the Docker image with @pytholic's Dockerfile, remove the open3d from the container and recompile the open3d headless and install it as described - http://www.open3d.org/docs/latest/tutorial/Advanced/headless_rendering.html. And use #75 (comment) this method and it worked out well.

[pytholic]

As mentioned by @HtutLynn , if you are using my headless script, you need to install open3D-headless in your container. I used headless because I was working with a remote machine. Also, I did not add open3d-headless in the Dockerfile to keep it more generalized.

If you want to use it without a headless, then you might have to pick some parts from my script (parts that save .ply files) and add them to the original script. Also, in that case, you should use normal open3d, and you might also have to use X11 forwarding for the container.

[jmunshi1990]

@pytholic I am trying to use your implementation of visualization code, but the demo.py does not generate any log.txt camera trajectory for pose. Do you know how to get the log.txt ?

[pytholic]

@jmunshi1990 Hi. It has been a long time since I used this. I don't remember how I was saving the final point cloud but maybe you can take a look at this part from the official demo for camera pose.

https://github.com/pytholic/DROID-SLAM/blob/f68a25fa230730e3eaaaa6ab03a818ca5da6954f/demo.py#L59

[jmunshi1990]

@pytholic I am using the highlighted part of the demo script to generate the reconstructions, but the only problem is that I am not able to generate the log txt file which has the trajectory needed to reproduce your below code which you have posted earlier. I am trying to find the read_trajectory input file out of the saved files which I am not getting. Any suggestion how to do that?

import glob
import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt


class CameraPose:
    def __init__(self, meta, mat):
        self.metadata = meta
        self.pose = mat

    def __str__(self):
        return (
            "Metadata : "
            + " ".join(map(str, self.metadata))
            + "\n"
            + "Pose : "
            + "\n"
            + np.array_str(self.pose)
        )


def read_trajectory(filename):
    traj = []
    with open(filename, "r", encoding="utf-8") as f:
        metastr = f.readline()
        while metastr:
            metadata = list(map(int, metastr.split()))
            mat = np.zeros(shape=(4, 4))
            for i in range(4):
                matstr = f.readline()
                mat[i, :] = np.fromstring(matstr, dtype=float, sep=" \t")
            traj.append(CameraPose(metadata, mat))
            metastr = f.readline()
    return traj


volume = o3d.pipelines.integration.ScalableTSDFVolume(
    voxel_length=4.0 / 512.0,
    sdf_trunc=0.04,
    color_type=o3d.pipelines.integration.TSDFVolumeColorType.RGB8,
)

intrinsics = np.load("./results/intrinsics.npy")

fx = intrinsics[0][0]
fy = intrinsics[0][1]
cx = intrinsics[0][2]
cy = intrinsics[0][3]

color_images = np.load("./results/images.npy")
depth_images = np.load("./results/disps.npy")
camera_poses = read_trajectory("./results/log.txt")
camera_intrinsic = o3d.camera.PinholeCameraIntrinsic(
    width=576, height=336, fx=fx, fy=fy, cx=cx, cy=cy
)

for i in range(len(color_images)):
    print(f"Integrate {i}-th image into the volume.")
    color = color_images[i]
    color = np.ascontiguousarray(color.transpose(1, 2, 0))
    color = o3d.geometry.Image((color).astype(np.uint8))
    depth = depth_images[i]
    depth = np.ascontiguousarray(depth.transpose(0, 1))
    depth = o3d.geometry.Image((depth * 255))
    rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(
        color, depth, depth_trunc=4.0, convert_rgb_to_intensity=False
    )

    volume.integrate(rgbd, camera_intrinsic, np.linalg.inv(camera_poses[i].pose))

print("Extract a triangle mesh from the volume and visualize it.")
mesh = volume.extract_triangle_mesh()
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])