[Question] Procedurally Generate Obstacles in Curriculum

[ologandavid]

Question

I was wondering if there were a way to procedurally generate obstacles in a curriculum during training with Isaac Lab. The conventional workflow seems to require an upfront declaration of all assets, terrains and robots in the initial configuration. Is there a wrapper or class I could inherit from that would allow for me to add obstacles of varying size during training? Like if I wanted to spawn a stack of blocks prims in the world at randomized locations.

Thanks!

[ooctipus]

Our TerrainGenerator should be generating terrain procedurally, and the procedural script is written source/isaaclab/isaaclab/terrains/trimesh/mesh_terrains.py or source/isaaclab/isaaclab/terrains/height_field/hf_terrains.py You can implement your custom procedures as well.

I see that you are asking for add obstacles DURING TRAINING Unfortunately that is a lot more tricker is do, and we also currently don't have a good practice suggestion for that either. I suppose you can define more rigidbodies shared across all environments, and handles the rigidbodies obstacle arrangement during the training by calling reset. You should be mindful that too many rigidbodies of complex shapes can impact simulation performance.

It is definitly possible and worth a try and if you find a way to do it, I'd love to see it!

[ologandavid]

Hi @ooctipus,
Thanks for pointing me to mesh_terrains.py its a good example of both structure and developing custom procedures.

With regards to the second point, is the process of adding and removing prims expensive even if the operation occurs during an env reset instead of mid episode? Would it be best to prespawn all of the objects at initialization, and then toggle their collisions/visibility during curriculum transitions/ resets?

Also I noticed that the terrain curriculum, simply changes the initial spawn pose of the robot to a more difficult terrain origin. Was this done to avoid dynamically importing meshes during the training loop?

Thanks!

[ooctipus]

Hi @ologandavid

whether or not having more rigidbodies will add more burden to simulation still needs testing. But it is known that if you have one shared triangle mesh fixed terrain, the simulation is very very fast, and it is very easy to do Raycasting(height scanner) with just one mesh. Having more meshes will requires a more well-rounded raycaster support if using height scanner is intended.

So say if you want create a more dynamic scenario where obstacle reset every reset, here are some questions to think about

1. Do all robot share the same obstacles, or are the obstacle independent to each robot per environment
2. How many obstacles are you going to simulation? does it makes more sense to reset them during reset, or does it make more sense to create more terrain tiles, and spawn the robot to the tiles

If you after answering these two questions you still find operation during reset make sense then definitely give it a try and I am very curious how far you get : )) , maybe that will make it to isaaclab as the first dynamic obstacle example!

[ologandavid]

Hi @ooctipus,
Since I'm trying to do a curriculum that increases the number of the objects and their properties in the scene, I decided to go about this using method 2. Similar to the way terrains are loaded into the scene and curriculum changes the env id of the robot to a harder terrain as training progresses, I figured I could spawn all the objects upfront in their respective envs, and change the env id's as the policy improves. I had a couple of questions (some might be obvious but humor me) on the structure of my code (if there's an easier way to do this) and implementation details.

In my current setup, I've written a custom env (ObstacleEnv) that inherits from ManagerBasedRLEnv and a custom class ObstacleSpawner that directly creates obstacles and joints between them using isaacsim.core.utils.prims once given an id.

Question 1: Where in the ObstacleEnv should an instance of my custom class ObstacleSpawner be created and called? In the init, or the reset? Is the simulator picky about which one I choose? Should the spawning be done in the init, and the curriculum update in the reset?

Question 2: If I wanted to keep track of collisions with the obstacles, say for example to assign rewards based on contact would I also have to attach a sensor to each object? (If that's the case, things might get to expensive computationally.)

Question 3: Do I have to use sim_utils to spawn simple prims into the scene? Does sim utils allow for jointing prims to each other and to the world? I've directly been spawning objects using isaacsim libraries but have been getting these warnings:

2025-08-11 02:35:54 [10,505ms] [Error] [omni.physx.plugin] PhysX error: PxRigidDynamic::setGlobalPose(): it is illegal to call this method if PxSceneFlag::eENABLE_DIRECT_GPU_API is enabled!, FILE /builds/omniverse/physics/physx/source/physx/src/NpRigidDynamic.cpp, LINE 102
2025-08-11 02:35:54 [10,505ms] [Error] [omni.physx.plugin] PhysX error: PxRigidDynamic::setLinearVelocity(): it is illegal to call this method if PxSceneFlag::eENABLE_DIRECT_GPU_API is enabled!, FILE /builds/omniverse/physics/physx/source/physx/src/NpRigidDynamic.cpp, LINE 239
2025-08-11 02:35:54 [10,505ms] [Error] [omni.physx.plugin] PhysX error: PxRigidDynamic::setAngularVelocity(): it is illegal to call this method if PxSceneFlag::eENABLE_DIRECT_GPU_API is enabled!, FILE /builds/omniverse/physics/physx/source/physx/src/NpRigidDynamic.cpp, LINE 287

Thanks,
David

[ooctipus]

Hi @ologandavid

I see you are doing a lot of customization, and ran into issue with some very advanced operations there. The error you are seeing is because once simulation is running, you can't spawn or delete any objects. I do not recommend creating your own ObstacleEnv nor your ObstacleSpawner unless you really know what you are doing. If you just stick with our example of rough terrain, you will get question2 and question3 resolve for free, and you have freedom to define your obstacles through mesh_terrain or height field.

[ologandavid]

Hi @ooctipus,
Little confused about your suggestion here.
I'm not trying to spawn or delete objects once simulation is running but rather spawn them all up front.
Are you recommending I build the obstacles into the terrain and spawn it like a terrain, or build a custom class with the same structure as the terrain generator/importer and define my custom functions in a mesh terrain like file. My obstacles can't be approximated as trimeshes so building it into the terrain isn't exactly an option.

Thanks!

[ooctipus]

I don't recommend any custom classes, custom classes over asset are advanced isaaclab topics, if running into issue, we have limited capacity to review it. what I do recommend is to write your own custom mesh_terrain, and use existing terrain generator to spawn them as sub-terrain. Or if terrain Importer or terrain generator doesn't give exactly what you need because they are static, then RigidObject is probably something you are looking for : ))

[RandomOakForest]

Thank you for following up. I'll move this post to our Discussions.