[Question] Is there any way to exclude an object during the Replicator (SDG) Recording Process?

[StewardMelvinTang]

Hi everyone,

I’ve been testing out Isaac Sim 5.0, especially the new 3DGS USDZ rendering support, which is amazing!

I’m trying to use 3DGS models to create synthetic datasets using the built-in Replicator tools in Isaac Sim. But I haven’t been able to get it working directly. As a workaround, I convert the 3DGS into a mesh using CloudCompare or MeshLab, then import the .fbx file back into Isaac Sim. ( so like the 3dgs for visual, collision mesh for physics & interaction)

This method works. The object shows up correctly in depth, RGB, point clouds, and segmentation captures. However, I’m running into a problem with the RGB images. I want to hide the collision mesh from the RGB camera, but still want them show up in depth and segmentation.

I tried using a transparent material (by changing the opacity), but once I do that, the object disappears from everything including depth and segmentation,which is not what I want.

I heard there’s a feature called multi render source? that might let me control what object shows up in each parameters (like RGB or depth), but I’m not sure how to use it.

Has anyone found a way to hide an object in RGB but still keep it visible in depth and segmentation? I’d really appreciate any help. Thanks!

[RandomOakForest]

Thank you for posting this. The visibility of an object can be set, as in the following code:

import omni.replicator.core as rep

def hide_object():
    objects = rep.get.prims(path_pattern="/World/MyObject")  # Set path as needed
    with objects:
        rep.modify.visibility([False])  # Exclude by making invisible
    return objects.node

rep.randomizer.register(hide_object)

with rep.trigger.on_frame(num_frames=100):
    rep.randomizer.hide_object()

Here's an example in the docs.

With the standard workflow in NVIDIA Omniverse Replicator (including the Isaac Sim integration), setting a prim’s visibility to False (via rep.modify.visibility([False])) will make that object invisible in all rendered outputs—RGB, depth, and segmentation. There is no built-in, per-channel visibility control: visibility is a global stage property per prim per frame.

In theory, creating a completely custom rendering pipeline or annotator might allow for modality-dependent masking, but this is outside the scope of standard Replicator SDG workflows and not currently documented for user-side implementation in Isaac Sim or Replicator Python APIs. See comments here.

[StewardMelvinTang]

Alright, I got it. Thank you so much for your answer. Really appreciate that

[StewardMelvinTang]

This is a bit off-topic, but I’m currently trying to implement 3D Gaussian Splatting (3DGS) in a robotics example (like Franka's pick and place task or cube stacking task). Do you think it's possible to replace the default "Cube" with something else (the cube is an object from DynamicCuboid from isaacsim.core.api.objects), and I wnt to change it to like USDZ primitives (as references) that contains my 3dgs objects or something else? I’m curious on how are the 3D gaussian splatting objects loaded from USDZ files? Are they using dynamic rigid primitives? Do you happen to know exactly where the source code for this is?

Thanks again in advance, if you'd like me to post this on a new issue, please feel free to let me know.

[superjuices]

Alright, I got it. Thank you so much for your answer. Really appreciate that

Hi, I recently encountered a similar problem. I overlapped the mesh (hide) and 3dgs, but could not render the depth map. Can you share in detail how you did it?

[RandomOakForest]

Thank you for following up, I'll move this post to our Discussions section for the team and others to follow up.