[hal metal] ray tracing acceleration structures #8071

Lichtso · 2025-08-09T13:19:29Z

Connections
Fixes: #7402
Depends on: #5641
Supersedes: #7660

Description
Implements the missing ray tracing acceleration structures in the HAL metal backend.

Testing
The examples ray_scene, ray_shadows, ray_cube_compute, ray_cube_fragment and ray_traced_triangle all work.
That is if invoked via cargo run --bin wgpu-examples ray_traced_triangle, but not via cargo xtask test ray_traced_triangle, still current CI runner is too old to catch that as it does not support hardware ray tracing.

Squash or Rebase?
Squash

Checklist

Run cargo fmt.
Run taplo format.
Run cargo clippy --tests.
Run cargo xtask test to run tests.
If this contains user-facing changes, add a CHANGELOG.md entry.

This removes the manual implementation introduced in 7b00140.

This is no longer necessary, since Winit can now properly issue redraw events on resize.

To keep the diff smaller and easier to review, this uses a temporary fork of `objc2-metal` and `objc2-quartz-core` whose methods use the naming scheme of the `metal` crate. One particular difficult part with this is that the `metal` crate has several methods where the order of the arguments are swapped relative to the corresponding Objective-C methods. This includes most perilously (since these have both an offset and an index argument, both of which are integers): - `set_bytes` - `set_vertex_bytes` - `set_fragment_bytes` - `set_buffer` - `set_vertex_buffer` - `set_fragment_buffer` - `set_threadgroup_memory_length` But also: - `set_vertex_texture` - `set_fragment_texture` - `set_sampler_state` - `set_vertex_sampler_state` - `set_fragment_sampler_state` Another noteworthy thing to mention is that `objc2-metal` does not (yet) provide a fallback for MTLCopyAllDevices, so we have to do that ourselves: madsmtm/objc2@3543940

…ation_structure_build_sizes().

…n_structure().

Vecvec

All of the migrated things look good, but I'm not sure the residency sets would work in complex scenarios. Please correct me if I've gotten anything wrong.

Vecvec · 2025-08-09T20:34:41Z

wgpu-hal/src/metal/command.rs

+        self.state.residency_sets.insert(dst.residency_set.clone());
+        dst.residency_set.requestResidency();
+        dst.residency_set.removeAllAllocations();


The clone of the residency set (and the other clone in build_acceleration_structures) feel like they are trying to duplicate the residency set, but it seems from reading the docs that they are instead duplicating the Retained, which will still reference the same residency set. If not, won't this edit the residency set that we've previously added to command encoders?

This is supposed to clone the Arc, not be a deep copy. I can use .retained() to make it more clear. And yes, it does edit what we will add to command encoders state (self.state) which is then later committed in when the queue is submitted.

The line self.state.residency_sets.insert(dst.residency_set.clone()); registers the residency_set for a delayed commitment.

You might want to use Arc::clone() to make it explicit.

Well, it is not a Rust Arc but a objc2 Retained, so the Clone impl for that would be: https://docs.rs/objc2/latest/objc2/rc/struct.Retained.html#method.retain

My concern is that this may still allow state leaks. I think a user could create an encoder which built a empty TLAS without ever submitting it and thereby clear the residency set in another command encoder, which could leave BLASes un-resident.

Vecvec · 2025-08-09T20:52:29Z

wgpu-hal/src/metal/command.rs

+                        }
+                    }
+                    super::BufferLikeResource::AccelerationStructure(ptr) => {
+                        encoder.setVertexAccelerationStructure_atBufferIndex(


Shouldn't dependencies be inserted somewhere in set bind group? (not necessarily here)

This here is set_bind_group().

By inserted I assume you mean residency_set.makeResident(), and yes we should probably move that part here.

Vecvec · 2025-08-10T03:27:43Z

wgpu-hal/src/metal/mod.rs

-pub struct AccelerationStructure;
+pub struct AccelerationStructure {
+    raw: Retained<ProtocolObject<dyn MTLAccelerationStructure>>,
+    residency_set: Retained<ProtocolObject<dyn MTLResidencySet>>,


I'm not sure about putting the residency sets into the acceleration structure, I thought the main benefit of residency sets was that one could make them after encoding for before execution and so use the data for validating submissions (validate_acceleration_structure_actions, AsAction::UseTlas), then send the dependencies down to either the command buffer (e.g device.set_command_buffer_blas_dependencies(cmd_buf, blases)), or as an extra argument in queue.submit.

Residency sets behave like command buffers: You can pre record a bunch of actions without an immediate effect and then commit them later. And that is how it is used here: The residency set is recorded once when its corresponding acceleration structure is build (because that is when the dependencies can actually change) and then that recording is saved to be committed later, when the build action is submitted in a queue.

When we turn our dependency list into a residency set is kinda irrelevant, but we have to allocate residency sets from the device thus it is easiest to have them allocated together with the acceleration structure.

Lichtso · 2025-08-10T11:23:57Z

wgpu-hal/src/metal/mod.rs

@@ -929,6 +932,7 @@ struct CommandState {
    index: Option<IndexState>,
    raw_wg_size: MTLSize,
    stage_infos: MultiStageData<PipelineStageInfo>,
+    residency_sets: HashSet<Retained<ProtocolObject<dyn MTLResidencySet>>>,


@Vecvec we don't need extra interfaces like set_command_buffer_blas_dependencies because we can just store that in the encoder state here, from which command buffers are then emitted alongside residency sets (see CommandBuffer::residency_sets).

Lichtso · 2025-08-10T15:48:29Z

Open problems remaining:

How to solve the issue with the tests not passing because the metal debug layer breaks acceleration structures if no window is present.
When to call residency_set.endResidency()? Possibly when the next set_bind_group() call happens.
What to do about a bindless usage of TLAS and its residency_set.

Vecvec · 2025-08-10T19:29:48Z

@Lichtso I think that we think of MTLResidencySets differently. I think you think of them like a encoder, where you stage commands and then commit them to run those commands. I think of them like an array of resources which get made resident when commit gets called, and requestResidency tells the driver that you are likely to try to make these resource resident, and to maybe try make them resident if it can so you don't have to block on residency commit.

Vecvec · 2025-08-10T19:36:24Z

I've drawn an image of what I think could happen based on my understanding of residency sets. I've used green to be user space and red to be device space.

Edit: it's not very clear, but the resources in the residency set are only made resident at queue submit (when it gets committed), not when they are added to the residency set

MarijnS95 · 2025-08-10T21:32:11Z

Yes, supposedly MTLResidencySet doesn't make a "shadow copy" whenever it is attached to anything. Instead commit() simply applies all the add/remove changes, that any command buffer to which it is attached can subsequently read to know which resources need to be made or kept resident. (Or the rest of the system can read to evict live resources when starved for memory, based on running command buffers?).

In other words it should be fine to remove live resources from a residency set, as long as those are not committed until all pending command buffers that are still referencing these resources have finished.

@Vecvec afaik the device space you have drawn isn't affected until a command buffer referencing a certain MTLResidencySet is submitted and starts running (and it'll reference the most recently committed state), or explicit residency is requested ahead of time using requestResidency() (which doesn't preclude attaching it to a queue or command buffer).

Vecvec · 2025-08-10T23:29:01Z

@Vecvec afaik the device space you have drawn isn't affected until a command buffer referencing a certain MTLResidencySet is submitted and starts running (and it'll reference the most recently committed state), or explicit residency is requested ahead of time using requestResidency() (which doesn't preclude attaching it to a queue or command buffer).

I don't think it was very clear that what is put in the residency set is not the same as what is made resident, I was meaning calling addAllocation which does happen in the encoder calls.

madsmtm and others added 9 commits August 9, 2025 14:56

[metal] Use raw-window-metal to do layer creation

e983e9c

This removes the manual implementation introduced in 7b00140.

[examples] Do not request redraw on resize

a276b59

This is no longer necessary, since Winit can now properly issue redraw events on resize.

Use normal objc2 naming scheme

41b0f1b

BRANCH BEGIN MARKER

1c0b31d

Fixes index_buffer label in ray_traced_triangle example.

fa258bf

Fixes min_push_constant_size in ray_shadows example.

fa3435b

Ensures there always is Some() buffer reference, even for get_acceler…

36005d0

…ation_structure_build_sizes().

Adds BuildAccelerationStructureDescriptor::dependencies.

8b5f9d3

Lichtso requested a review from a team as a code owner August 9, 2025 13:19

Lichtso mentioned this pull request Aug 9, 2025

[hal metal] ray tracing acceleration structures #7660

Closed

5 tasks

Lichtso force-pushed the objc2_metal/ray_tracing_acceleration_structures branch from e1218ce to edeaaba Compare August 9, 2025 13:21

Lichtso added 18 commits August 9, 2025 15:22

Updates CHANGELOG.

2228d60

Adds feature detection.

8b5659b

Adds Capabilities Limits.

2e6410b

Adds necessary objc2-metal features in Cargo.toml

2616505

Sets raw_tlas_instance_size.

01def90

Sets ray_tracing_scratch_buffer_alignment.

d7240d3

Adds conv::map_index_format().

c9e0d51

Adds conv::map_acceleration_structure_descriptor().

f25ea54

Implements AccelerationStructure.

4219ab1

Implements resource binding.

f605812

Adds CommandState::acceleration_structure_builder.

42fae2d

Implements CommandEncoder::copy_acceleration_structure_to_acceleratio…

bd1dbb0

…n_structure().

Implements CommandEncoder::build_acceleration_structures().

33cc31a

Implements CommandEncoder::place_acceleration_structure_barrier().

99dd885

Implements CommandEncoder::read_acceleration_structure_compact_size().

836b870

Implements Device::get_acceleration_structure_build_sizes().

61d27e2

Implements Device::get_acceleration_structure_device_address().

d294535

Implements Device::create_acceleration_structure().

c95eb8f

Lichtso added 2 commits August 9, 2025 15:22

Implements Device::destroy_acceleration_structure().

393aa1c

Implements Device::tlas_instance_to_bytes().

a201c51

Vecvec requested changes Aug 10, 2025

View reviewed changes

Lichtso commented Aug 10, 2025

View reviewed changes

Lichtso force-pushed the objc2_metal/ray_tracing_acceleration_structures branch from edeaaba to 1244389 Compare August 10, 2025 15:56

Uses MTLResidencySet for TLAS dependencies.

aa6be88

Lichtso force-pushed the objc2_metal/ray_tracing_acceleration_structures branch from 1244389 to aa6be88 Compare August 10, 2025 16:04

cwfitzgerald assigned Vecvec Aug 13, 2025

[hal metal] ray tracing acceleration structures #8071

Are you sure you want to change the base?

[hal metal] ray tracing acceleration structures #8071

Uh oh!

Conversation

Lichtso commented Aug 9, 2025

Uh oh!

Vecvec left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lichtso Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lichtso Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lichtso commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Vecvec commented Aug 10, 2025

Uh oh!

Vecvec commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarijnS95 commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Vecvec commented Aug 10, 2025

Uh oh!

Uh oh!

Lichtso Aug 10, 2025 •

edited

Loading

Lichtso Aug 10, 2025 •

edited

Loading

Lichtso commented Aug 10, 2025 •

edited

Loading

Vecvec commented Aug 10, 2025 •

edited

Loading

MarijnS95 commented Aug 10, 2025 •

edited

Loading