Skip to content

Commit bc2a87a

Browse files
weegeekpsweegeekps
authored
Merge pull request #88 from CesiumGS/draft-splat-spz
Update cesium-native branch with latest splatting extension updates
2 parents e33b398 + e5f8ce1 commit bc2a87a

File tree

5 files changed

+276
-236
lines changed

5 files changed

+276
-236
lines changed

‎extensions/2.0/Khronos/KHR_gaussian_splatting/README.md

Lines changed: 76 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ Written against the glTF 2.0 spec.
3434

3535
## Overview
3636

37-
This extension defines support for storing 3D Gaussian splats in glTF, bringing structure and conformity to the 3D Gaussian splatting space. 3D Gaussian splats are effectively fields of 3D Gaussian splats that can be treated as a point cloud for the purposes of storage. 3D Gaussian splats are defined by their position, rotation, scale, and spherical harmonics which provide both diffuse and specular color. These values are stored as values on a point primitive. Since we treat the 3D Gaussian splats as points primitives, a graceful fallback to treating the data as a sparse point cloud is possible.
37+
This extension defines support for storing 3D Gaussian splats in glTF, bringing structure and conformity to the 3D Gaussian splatting space. 3D Gaussian splatting uses fields of Gaussians that can be treated as a point cloud for the purposes of storage. 3D Gaussian splats are defined by their position, rotation, scale, and spherical harmonics which provide both diffuse and specular color. These values are stored as values on a point primitive. Since we treat the 3D Gaussian splats as points primitives, a graceful fallback to treating the data as a sparse point cloud is possible.
3838

3939
## Adding 3D Gaussian Splats to Primitives
4040

@@ -50,108 +50,102 @@ The extension must be listed in `extensionsUsed`:
5050

5151
Other extensions that depend on this extension such as 3D Gaussian splatting compression extensions may require that this extension be included in `extensionsRequired`.
5252

53-
### Geometry Type
53+
## Geometry Type
5454

5555
The `mode` of the `primitive` must be `POINTS`.
5656

57-
### Schema Example
57+
## Schema Example
5858

59-
Example shown below. This extension only affects any `primitive` nodes containting 3D Gaussian splat data.
59+
Example shown below including optional properties. This extension only affects any `primitive` nodes containting 3D Gaussian splat data.
6060

6161
```json
62-
"meshes": [{
63-
"primitives": [{
64-
"attributes": {
62+
"meshes": [{
63+
"primitives": [{
64+
"attributes": {
6565
"POSITION": 0,
6666
"COLOR_0": 1,
67-
"_SCALE": 2,
68-
"_ROTATION": 3,
69-
"_SH_DEGREE_1_COEF_0": 4,
70-
"_SH_DEGREE_1_COEF_1": 5,
71-
"_SH_DEGREE_1_COEF_2": 6
72-
},
73-
"mode": 0,
74-
"indices": 7,
75-
"extensions": {
76-
"KHR_gaussian_splatting": {}
77-
}
78-
}]
79-
}],
67+
"KHR_gaussian_splatting:SCALE": 2,
68+
"KHR_gaussian_splatting:ROTATION": 3,
69+
"KHR_gaussian_splatting:SH_DEGREE_1_COEF_0": 4,
70+
"KHR_gaussian_splatting:SH_DEGREE_1_COEF_1": 5,
71+
"KHR_gaussian_splatting:SH_DEGREE_1_COEF_2": 6
72+
},
73+
"mode": 0,
74+
"indices": 7,
75+
"extensions": {
76+
"KHR_gaussian_splatting": {
77+
"shape": "ellipsoid",
78+
"hints": {
79+
"sortingMethod": "cameraDistance",
80+
"projection": "perspective"
81+
}
82+
}
83+
}
84+
}]
85+
}],
8086
```
8187

82-
### Extension Properties
88+
## Extension Properties
8389

84-
The extension specifies no additional properties but must be included on a point primitive with an empty body to indicate that the primitive should be treated as a 3D Gaussian splatting field.
90+
### Shape
8591

86-
#### attributes
92+
Gaussian splats can have a variety of shapes and this has the potential to change over time. The `shape` property is an optional property that provides an indication to the renderer what these shapes may be. Typically `ellipsoid` refers to the shape generally considered to be a "splat" and this is considered the default value.
8793

88-
Each 3D Gaussian splat has the following attributes. At minimum the attributes must contain `POSITION`, `COLOR_0`, `_ROTATION`, and `_SCALE`. `_SH_DEGREE_ℓ_COEF_n` attributes hold the spherical harmonics data and are not required. If higher degrees of Spherical Harmonics are used then lower degrees are required implicitly.
94+
Renderers are free to ignore any values they do not recognize, but are encouraged to follow the non-normative list below.
8995

90-
| Splat Data | glTF Attribute | Accessor Type | Component Type | Required | Notes |
91-
| --- | --- | --- | --- | --- | --- |
92-
| Position | POSITION | VEC3 | float | yes | |
93-
| Color (Spherical Harmonic degree 0 (Diffuse) and alpha) | COLOR_0 | VEC4 | unsigned byte normalized or float | yes | |
94-
| Rotation | _ROTATION | VEC4 | float | yes | Rotation is a quaternion. |
95-
| Scale | _SCALE | VEC3 | float | yes | |
96-
| Spherical Harmonics degree 1 | _SH_DEGREE_1_COEF_n (n = 0 to 2) | VEC3 | float | no (yes if degree 2 or 3 are used) | |
97-
| Spherical Harmonics degree 2 | _SH_DEGREE_2_COEF_n (n = 0 to 4) | VEC3 | float | no (yes if degree 3 is used) | |
98-
| Spherical Harmonics degree 3 | _SH_DEGREE_3_COEF_n (n = 0 to 6) | VEC3 | float | no | |
96+
#### Known Shapes
9997

100-
The value of `COLOR_0` is derived by multiplying the 3 diffuse color components of the 3D Gaussian splat with the constant zeroth-order Spherical Harmonic (ℓ = 0).
98+
*This section is non-normative and not comprehensive. It may change over time.*
10199

102-
Each increasing degree of spherical harmonics requires more coeffecients. At the 1st degree, 3 sets of coeffcients are required, increasing to 5 sets for the 2nd degree, and increasing to 7 sets at the 3rd degree. With all 3 degrees, this results in 45 spherical harmonic coefficients stored in the `_SH_DEGREE_ℓ_COEF_n` attributes.
100+
- `ellipsoid` _(Default Value)_
101+
- `triangle`
102+
- `quad`
103103

104-
### Accessors
104+
### Rendering Hints
105105

106-
Required `accessors` for `POSITION`, `COLOR_0`, `_SCALE`, and `_ROTATION`:
106+
This extension provides a `hints` property that contains sub-properties that may help renderers understand how best to render the Gaussians to the screen. This property and all of it's sub-properties are optional, and renderers can choose to ignore them.
107107

108-
```json
109-
"accessors": [{
110-
"bufferView": 0,
111-
"componentType": 5126,
112-
"count": 590392,
113-
"type": "VEC3",
114-
"max": [
115-
1,
116-
1,
117-
1,
118-
],
119-
"min": [
120-
-1,
121-
-1,
122-
-1,
123-
]
124-
}, {
125-
"bufferView": 1,
126-
"componentType": 5121,
127-
"count": 590392,
128-
"type": "VEC4",
129-
"normalized": true
130-
}, {
131-
"bufferView": 2,
132-
"componentType": 5126,
133-
"count": 590392,
134-
"type": "VEC3"
135-
}, {
136-
"bufferView": 3,
137-
"componentType": 5126,
138-
"count": 590392,
139-
"type": "VEC4"
140-
}],
141-
```
108+
#### Projection
142109

143-
Spherical harmonics `accessors` all follow the pattern:
110+
The `projection` property is an optional hint that specifies how the Gaussians should be projected into the image. This is typically provided by the training process for the splats. This is a freeform string field to allow new projections to be specified as they become available. The default value is `perspective`.
144111

145-
```json
146-
"accessors": [{
147-
"bufferView": 4,
148-
"componentType": 5126,
149-
"count": 590392,
150-
"type": "VEC3"
151-
}]
152-
```
112+
Renderers are free to ignore any values they do not recognize, but are encouraged to follow the non-normative list below.
113+
114+
##### Known Projection Methods
115+
116+
*This section is non-normative and not comprehensive. It may change over time.*
117+
118+
- `perspective` _(Default Value)_: The typical 3D perspective projection based on scene depth.
119+
- `orthographic`: A orthogonal projection of splats into a scene to preserve shape and scale and reduce distortion.
120+
121+
#### Sorting Method
122+
123+
The `sortingMethod` property is an optional hint that specifies how the Gaussians should be sorted during the rendering process. This typically is provided by the training process for the splats. This is a freeform string field to allow new sorting methods to be specified as they become available. The default value is `cameraDistance`.
153124

154-
Accessors must be defined for all in-use `attributes`.
125+
Renderers are free to ignore any values they do not recognize, but are encouraged to follow the non-normative list below.
126+
127+
##### Known Sorting Methods
128+
129+
*This section is non-normative and not comprehensive. It may change over time.*
130+
131+
- `cameraDistance` _(Default Value)_: The distance between the center of the splat and the position of the camera.
132+
- `zDepth`: The projected z-depth in the camera projection.
133+
134+
## Attributes
135+
136+
Each 3D Gaussian splat has the following attributes. At minimum the attributes must contain `POSITION`, `COLOR_0`, `KHR_gaussian_splatting:ROTATION`, and `KHR_gaussian_splatting:SCALE`. `KHR_gaussian_splatting:SH_DEGREE_ℓ_COEF_n` attributes hold the spherical harmonics data and are not required. `POSITION` and `COLOR_0` are defined by the base glTF specification. If higher degrees of Spherical Harmonics are used then lower degrees are required implicitly.
137+
138+
| Splat Data | glTF Attribute | Accessor Type | Component Type | Required | Notes |
139+
| --- | --- | --- | --- | --- | --- |
140+
| Rotation | KHR_gaussian_splatting:ROTATION | VEC4 | float | yes | Rotation is a quaternion. (xyzw) |
141+
| Scale | KHR_gaussian_splatting:SCALE | VEC3 | float | yes | |
142+
| Spherical Harmonics degree 1 | KHR_gaussian_splatting:SH_DEGREE_1_COEF_n (n = 0 to 2) | VEC3 | float | no (yes if degree 2 or 3 are used) | |
143+
| Spherical Harmonics degree 2 | KHR_gaussian_splatting:SH_DEGREE_2_COEF_n (n = 0 to 4) | VEC3 | float | no (yes if degree 3 is used) | |
144+
| Spherical Harmonics degree 3 | KHR_gaussian_splatting:SH_DEGREE_3_COEF_n (n = 0 to 6) | VEC3 | float | no | |
145+
146+
The value of `COLOR_0` is derived by multiplying the 3 diffuse color components of the 3D Gaussian splat with the constant zeroth-order Spherical Harmonic (ℓ = 0) for the RGB channels. The alpha channel should contain the opacity of the splat.
147+
148+
Each increasing degree of spherical harmonics requires more coeffecients. At the 1st degree, 3 sets of coeffcients are required, increasing to 5 sets for the 2nd degree, and increasing to 7 sets at the 3rd degree. With all 3 degrees, this results in 45 spherical harmonic coefficients stored in the `_SH_DEGREE_ℓ_COEF_n` attributes.
155149

156150
## Implementation
157151

@@ -339,7 +333,7 @@ Accessed via `usampler2D`:
339333

340334
#### Sorting and Indexes
341335

342-
With the Gaussian splat attributes packed into a texture the sorting only has to act upon a separate `_INDEX` attribute created at runtime. Gaussian splats are sorted as above, but instead of sorting each vertex buffer only sort the index values. When the glTF is loaded, Gaussian splats can be indexed in the order read.
336+
With the Gaussian splat attributes packed into a texture the sorting only has to act upon the index of the splat at runtime. Gaussian splats are sorted as above, but instead of sorting each vertex buffer you can only sort the index values. When the glTF is loaded, Gaussian splats can be indexed in the order read.
343337

344338
#### Extracting Data in the Vertex Shader
345339

‎extensions/2.0/Khronos/KHR_gaussian_splatting/schema/mesh.primitive.KHR_gaussian_splatting.schema.json

Lines changed: 33 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,36 @@
11
{
2-
"$schema": "http://json-schema.org/draft-04/schema",
3-
"title": "KHR_gaussian_splatting glTF Mesh Primitive Extension",
4-
"type": "object",
5-
"description": "Data for a 3D Gaussian Splat primitive.",
6-
"allOf": [ { "$ref": "glTFProperty.schema.json" } ],
7-
"properties": {
8-
"extensions": { },
9-
"extras": { }
2+
"$schema": "http://json-schema.org/draft-04/schema",
3+
"title": "KHR_gaussian_splatting glTF Mesh Primitive Extension",
4+
"type": "object",
5+
"description": "Data for a 3D Gaussian Splat primitive.",
6+
"allOf": [
7+
{
8+
"$ref": "glTFProperty.schema.json"
109
}
10+
],
11+
"properties": {
12+
"shape": {
13+
"type": "string",
14+
"description": "The shape of the Gaussian to render. Gaussians may be available in different shapes, so this is a freeform field that defaults to ellipsoid. See the extension specification for futher known types.",
15+
"default": "ellipsoid"
16+
},
17+
"hints": {
18+
"type": "object",
19+
"description": "Optional rendering hints for rendering the 3D Gaussian splats. Renderers are free to ignore any of these.",
20+
"properties": {
21+
"projection": {
22+
"type": "string",
23+
"description": "Provides a hint specifying how to project the Gaussians to achieve a perspective correct value. This is a freeform field that defaults to perspective. See the extension specification for further known types.",
24+
"default": "perspective"
25+
},
26+
"sortingMethod": {
27+
"type": "string",
28+
"description": "Provides a hint specifying how to sort the Gaussians during rendering. This is a freeform field defaulting to cameraDistance. See the extension specification for further known types.",
29+
"default": "cameraDistance"
30+
}
31+
}
32+
},
33+
"extensions": {},
34+
"extras": {}
35+
}
1136
}

0 commit comments

Comments
 (0)