Update cesium-native branch with latest splatting extension updates

extensions/2.0/Khronos/KHR_gaussian_splatting/README.md

@@ -34,7 +34,7 @@ Written against the glTF 2.0 spec.
 
 ## Overview
 
-This extension defines support for storing 3D Gaussian splats in glTF, bringing structure and conformity to the 3D Gaussian splatting space. 3D Gaussian splats are effectively fields of 3D Gaussian splats that can be treated as a point cloud for the purposes of storage. 3D Gaussian splats are defined by their position, rotation, scale, and spherical harmonics which provide both diffuse and specular color. These values are stored as values on a point primitive. Since we treat the 3D Gaussian splats as points primitives, a graceful fallback to treating the data as a sparse point cloud is possible.
+This extension defines support for storing 3D Gaussian splats in glTF, bringing structure and conformity to the 3D Gaussian splatting space. 3D Gaussian splatting uses fields of Gaussians that can be treated as a point cloud for the purposes of storage. 3D Gaussian splats are defined by their position, rotation, scale, and spherical harmonics which provide both diffuse and specular color. These values are stored as values on a point primitive. Since we treat the 3D Gaussian splats as points primitives, a graceful fallback to treating the data as a sparse point cloud is possible.
 
 ## Adding 3D Gaussian Splats to Primitives
 
@@ -50,108 +50,102 @@ The extension must be listed in `extensionsUsed`:
 
 Other extensions that depend on this extension such as 3D Gaussian splatting compression extensions may require that this extension be included in `extensionsRequired`.
 
-### Geometry Type
+## Geometry Type
 
 The `mode` of the `primitive` must be `POINTS`.
 
-### Schema Example
+## Schema Example
 
-Example shown below. This extension only affects any `primitive` nodes containting 3D Gaussian splat data.
+Example shown below including optional properties. This extension only affects any `primitive` nodes containting 3D Gaussian splat data.
 
 ```json
-  "meshes": [{
-      "primitives": [{
-          "attributes": {
+"meshes": [{
+    "primitives": [{
+        "attributes": {
             "POSITION": 0,
             "COLOR_0": 1,
-            "_SCALE": 2,
-            "_ROTATION": 3,
-            "_SH_DEGREE_1_COEF_0": 4,
-            "_SH_DEGREE_1_COEF_1": 5,
-            "_SH_DEGREE_1_COEF_2": 6
-          },
-          "mode": 0,
-          "indices": 7,
-          "extensions": {
-            "KHR_gaussian_splatting": {}
-          }
-        }]
-    }],
+            "KHR_gaussian_splatting:SCALE": 2,
+            "KHR_gaussian_splatting:ROTATION": 3,
+            "KHR_gaussian_splatting:SH_DEGREE_1_COEF_0": 4,
+            "KHR_gaussian_splatting:SH_DEGREE_1_COEF_1": 5,
+            "KHR_gaussian_splatting:SH_DEGREE_1_COEF_2": 6
+        },
+        "mode": 0,
+        "indices": 7,
+        "extensions": {
+            "KHR_gaussian_splatting": {
+                "shape": "ellipsoid",
+                "hints": {
+                    "sortingMethod": "cameraDistance",
+                    "projection": "perspective"
+                }
+            }
+        }
+    }]
+}],
 ```
 
-### Extension Properties
+## Extension Properties
 
-The extension specifies no additional properties but must be included on a point primitive with an empty body to indicate that the primitive should be treated as a 3D Gaussian splatting field.
+### Shape
 
-#### attributes
+Gaussian splats can have a variety of shapes and this has the potential to change over time. The `shape` property is an optional property that provides an indication to the renderer what these shapes may be. Typically `ellipsoid` refers to the shape generally considered to be a "splat" and this is considered the default value.
 
-Each 3D Gaussian splat has the following attributes. At minimum the attributes must contain `POSITION`, `COLOR_0`, `_ROTATION`, and `_SCALE`. `_SH_DEGREE_ℓ_COEF_n` attributes hold the spherical harmonics data and are not required. If higher degrees of Spherical Harmonics are used then lower degrees are required implicitly.
+Renderers are free to ignore any values they do not recognize, but are encouraged to follow the non-normative list below.
 
-| Splat Data | glTF Attribute | Accessor Type | Component Type | Required | Notes |
-| --- | --- | --- | --- | --- | --- |
-| Position | POSITION | VEC3 | float | yes | |
-| Color (Spherical Harmonic degree 0 (Diffuse) and alpha) | COLOR_0 | VEC4 | unsigned byte normalized or float | yes | |
-| Rotation | _ROTATION | VEC4 | float | yes | Rotation is a quaternion. |
-| Scale | _SCALE | VEC3 | float | yes | |
-| Spherical Harmonics degree 1 | _SH_DEGREE_1_COEF_n (n = 0 to 2) | VEC3 | float | no (yes if degree 2 or 3 are used) | |
-| Spherical Harmonics degree 2 | _SH_DEGREE_2_COEF_n (n = 0 to 4) | VEC3 | float | no (yes if degree 3 is used) | |
-| Spherical Harmonics degree 3 | _SH_DEGREE_3_COEF_n (n = 0 to 6) | VEC3 | float | no | |
+#### Known Shapes
 
-The value of `COLOR_0` is derived by multiplying the 3 diffuse color components of the 3D Gaussian splat with the constant zeroth-order Spherical Harmonic (ℓ = 0).
+*This section is non-normative and not comprehensive. It may change over time.*
 
-Each increasing degree of spherical harmonics requires more coeffecients. At the 1st degree, 3 sets of coeffcients are required, increasing to 5 sets for the 2nd degree, and increasing to 7 sets at the 3rd degree. With all 3 degrees, this results in 45 spherical harmonic coefficients stored in the `_SH_DEGREE_ℓ_COEF_n` attributes.
+ - `ellipsoid` _(Default Value)_
+ - `triangle`
+ - `quad`
 
-### Accessors
+### Rendering Hints
 
-Required `accessors` for `POSITION`, `COLOR_0`, `_SCALE`, and `_ROTATION`:
+This extension provides a `hints` property that contains sub-properties that may help renderers understand how best to render the Gaussians to the screen. This property and all of it's sub-properties are optional, and renderers can choose to ignore them.
 
-```json
-  "accessors": [{
-      "bufferView": 0,
-      "componentType": 5126,
-      "count": 590392,
-      "type": "VEC3",
-      "max": [
-        1,
-        1,
-        1,
-      ],
-      "min": [
-        -1,
-        -1,
-        -1,
-      ]
-    }, {
-      "bufferView": 1,
-      "componentType": 5121,
-      "count": 590392,
-      "type": "VEC4",
-      "normalized": true
-    }, {
-      "bufferView": 2,
-      "componentType": 5126,
-      "count": 590392,
-      "type": "VEC3"
-    }, {
-      "bufferView": 3,
-      "componentType": 5126,
-      "count": 590392,
-      "type": "VEC4"
-    }],
-```
+#### Projection
 
-Spherical harmonics `accessors` all follow the pattern:
+The `projection` property is an optional hint that specifies how the Gaussians should be projected into the image. This is typically provided by the training process for the splats. This is a freeform string field to allow new projections to be specified as they become available. The default value is `perspective`.
 
-```json
-  "accessors": [{
-    "bufferView": 4,
-    "componentType": 5126,
-    "count": 590392,
-    "type": "VEC3"
-  }]
-```
+Renderers are free to ignore any values they do not recognize, but are encouraged to follow the non-normative list below.
+
+##### Known Projection Methods
+
+*This section is non-normative and not comprehensive. It may change over time.*
+
+ - `perspective` _(Default Value)_: The typical 3D perspective projection based on scene depth.
+ - `orthographic`: A orthogonal projection of splats into a scene to preserve shape and scale and reduce distortion.
+
+#### Sorting Method
+
+The `sortingMethod` property is an optional hint that specifies how the Gaussians should be sorted during the rendering process. This typically is provided by the training process for the splats. This is a freeform string field to allow new sorting methods to be specified as they become available. The default value is `cameraDistance`.
 
-Accessors must be defined for all in-use `attributes`.
+Renderers are free to ignore any values they do not recognize, but are encouraged to follow the non-normative list below.
+
+##### Known Sorting Methods
+
+*This section is non-normative and not comprehensive. It may change over time.*
+
+ - `cameraDistance` _(Default Value)_: The distance between the center of the splat and the position of the camera.
+ - `zDepth`: The projected z-depth in the camera projection.
+
+## Attributes
+
+Each 3D Gaussian splat has the following attributes. At minimum the attributes must contain `POSITION`, `COLOR_0`, `KHR_gaussian_splatting:ROTATION`, and `KHR_gaussian_splatting:SCALE`. `KHR_gaussian_splatting:SH_DEGREE_ℓ_COEF_n` attributes hold the spherical harmonics data and are not required. `POSITION` and `COLOR_0` are defined by the base glTF specification. If higher degrees of Spherical Harmonics are used then lower degrees are required implicitly.
+
+| Splat Data | glTF Attribute | Accessor Type | Component Type | Required | Notes |
+| --- | --- | --- | --- | --- | --- |
+| Rotation | KHR_gaussian_splatting:ROTATION | VEC4 | float | yes | Rotation is a quaternion. (xyzw) |
+| Scale | KHR_gaussian_splatting:SCALE | VEC3 | float | yes | |
+| Spherical Harmonics degree 1 | KHR_gaussian_splatting:SH_DEGREE_1_COEF_n (n = 0 to 2) | VEC3 | float | no (yes if degree 2 or 3 are used) | |
+| Spherical Harmonics degree 2 | KHR_gaussian_splatting:SH_DEGREE_2_COEF_n (n = 0 to 4) | VEC3 | float | no (yes if degree 3 is used) | |
+| Spherical Harmonics degree 3 | KHR_gaussian_splatting:SH_DEGREE_3_COEF_n (n = 0 to 6) | VEC3 | float | no | |
+
+The value of `COLOR_0` is derived by multiplying the 3 diffuse color components of the 3D Gaussian splat with the constant zeroth-order Spherical Harmonic (ℓ = 0) for the RGB channels. The alpha channel should contain the opacity of the splat.
+
+Each increasing degree of spherical harmonics requires more coeffecients. At the 1st degree, 3 sets of coeffcients are required, increasing to 5 sets for the 2nd degree, and increasing to 7 sets at the 3rd degree. With all 3 degrees, this results in 45 spherical harmonic coefficients stored in the `_SH_DEGREE_ℓ_COEF_n` attributes.
 
 ## Implementation
 
@@ -339,7 +333,7 @@ Accessed via `usampler2D`:
 
 #### Sorting and Indexes
 
-With the Gaussian splat attributes packed into a texture the sorting only has to act upon a separate `_INDEX` attribute created at runtime. Gaussian splats are sorted as above, but instead of sorting each vertex buffer only sort the index values. When the glTF is loaded, Gaussian splats can be indexed in the order read.
+With the Gaussian splat attributes packed into a texture the sorting only has to act upon the index of the splat at runtime. Gaussian splats are sorted as above, but instead of sorting each vertex buffer you can only sort the index values. When the glTF is loaded, Gaussian splats can be indexed in the order read.
 
 #### Extracting Data in the Vertex Shader
 

extensions/2.0/Khronos/KHR_gaussian_splatting/schema/mesh.primitive.KHR_gaussian_splatting.schema.json

@@ -1,11 +1,36 @@
 {
-    "$schema": "http://json-schema.org/draft-04/schema",
-    "title": "KHR_gaussian_splatting glTF Mesh Primitive Extension",
-    "type": "object",
-    "description": "Data for a 3D Gaussian Splat primitive.",
-    "allOf": [ { "$ref": "glTFProperty.schema.json" } ],
-    "properties": {
-        "extensions": { },
-        "extras": { }
+  "$schema": "http://json-schema.org/draft-04/schema",
+  "title": "KHR_gaussian_splatting glTF Mesh Primitive Extension",
+  "type": "object",
+  "description": "Data for a 3D Gaussian Splat primitive.",
+  "allOf": [
+    {
+      "$ref": "glTFProperty.schema.json"
     }
+  ],
+  "properties": {
+    "shape": {
+      "type": "string",
+      "description": "The shape of the Gaussian to render. Gaussians may be available in different shapes, so this is a freeform field that defaults to ellipsoid. See the extension specification for futher known types.",
+      "default": "ellipsoid"
+    },
+    "hints": {
+      "type": "object",
+      "description": "Optional rendering hints for rendering the 3D Gaussian splats. Renderers are free to ignore any of these.",
+      "properties": {
+        "projection": {
+          "type": "string",
+          "description": "Provides a hint specifying how to project the Gaussians to achieve a perspective correct value. This is a freeform field that defaults to perspective. See the extension specification for further known types.",
+          "default": "perspective"
+        },
+        "sortingMethod": {
+          "type": "string",
+          "description": "Provides a hint specifying how to sort the Gaussians during rendering. This is a freeform field defaulting to cameraDistance. See the extension specification for further known types.",
+          "default": "cameraDistance"
+        }
+      }
+    },
+    "extensions": {},
+    "extras": {}
+  }
 }