Merge pull request #1057 from koordinates/feature/point-cloud-import-override-crs

Add --override-crs option to point-cloud-import command

Author: craigds

.gitignore

@@ -78,6 +78,7 @@ docs/_build/
 *.code-workspace
 .vscode/
 .idea/
+.claude/
 
 CMakeUserPresets.json
 build*/

CLAUDE.md

@@ -0,0 +1,123 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development Commands
+
+### Building Kart
+
+Kart uses CMake for building. **Python 3.11 is required** for compatibility with CI builds.
+
+**Recommended approach (using VCPKG):**
+```bash
+# Configure with VCPKG dependency management
+cmake -B build -S . -DPython3_EXECUTABLE=/path/to/python3.11 -DUSE_VCPKG=ON
+
+# Build development version
+cmake --build build
+
+# Test the build
+./build/kart --version
+```
+
+**Alternative (using CI vendor archives for faster builds):**
+```bash
+# Download vendor-{os}-{arch}-py3.11.zip from recent CI builds
+cmake -B build -S . -DPython3_EXECUTABLE=/path/to/python3.11 -DVENDOR_ARCHIVE=/path/to/vendor-archive.zip -DUSE_VCPKG=OFF
+cmake --build build
+```
+
+**Production build:**
+```bash
+# Build bundled binary and install
+cmake --build build --target bundle
+cmake --install build
+```
+
+### Testing
+
+```bash
+# Run full test suite
+./build/venv/bin/pytest -v
+
+# Run tests excluding slow ones
+./build/venv/bin/pytest -m "not slow"
+
+# Run specific test categories
+./build/venv/bin/pytest -m "mssql"    # SQL Server tests
+./build/venv/bin/pytest -m "mysql"   # MySQL tests
+./build/venv/bin/pytest -m "e2e"     # End-to-end tests
+```
+
+### Code Quality
+
+```bash
+# Linting (via Ruff)
+ruff check kart/
+
+# Type checking
+mypy kart/
+
+# Format code
+ruff format kart/
+```
+
+**Note**: Pre-commit hooks automatically run Ruff, MyPy, and CMake formatting tools.
+
+## Architecture Overview
+
+Kart is a **distributed version control system for geospatial data** that extends Git's object model to handle tabular, point cloud, and raster datasets.
+
+### Core Components
+
+**Repository Management** (`kart/repo.py`):
+- `KartRepo`: Extends pygit2.Repository with geospatial-aware operations
+- Supports both "bare-style" (legacy) and "tidy-style" repositories
+- Built-in Git LFS integration and spatial filtering support
+
+**Dataset Abstraction** (`kart/base_dataset.py`):
+- `BaseDataset`: Abstract base for all dataset types (table, point-cloud, raster)
+- Immutable view pattern - datasets are views of git trees
+- Common diff/merge interface across all data types
+
+**Multi-Format Support**:
+- **Tabular** (`kart/tabular/`): Database-backed (GPKG, PostGIS, MySQL, SQL Server)
+- **Point Cloud** (`kart/point_cloud/`): Tile-based LAZ/LAS with PDAL integration
+- **Raster** (`kart/raster/`): Tile-based GeoTIFF with GDAL integration
+
+**Working Copy System** (`kart/working_copy.py`):
+- Multi-part architecture supporting both database and file-based backends
+- Conflict detection and resolution for concurrent edits
+- State tracking against repository HEAD
+
+**CLI Framework** (`kart/cli.py`):
+- Click-based modular command system
+- Direct Git command pass-through for compatible operations
+- Helper process architecture for environment isolation
+
+### Key Design Patterns
+
+- **Plugin Architecture**: Different strategies for tabular vs. tile-based data
+- **Repository Pattern**: Clean separation between git operations and geospatial handling
+- **Factory Pattern**: Dataset instantiation based on directory structure
+- **Streaming Processing**: Incremental handling of large datasets
+
+### Data Storage Strategy
+
+- All geospatial data stored as git objects (blobs/trees/commits)
+- Features/tiles organized in balanced hierarchical trees
+- Metadata stored separately from data for efficient operations
+- Automatic Git LFS integration for large files
+- Schema versioning with V2/V3 formats for tables
+
+### Important File Locations
+
+- **Main entry**: `kart/cli.py`
+- **Core repository logic**: `kart/repo.py`, `kart/core.py`
+- **Dataset base classes**: `kart/base_dataset.py`
+- **Format-specific implementations**: `kart/tabular/`, `kart/point_cloud/`, `kart/raster/`
+- **Working copy backends**: `kart/tabular/working_copy/`
+- **Build configuration**: `CMakeLists.txt`, `pyproject.toml`
+- **Test configuration**: `pytest.ini`
+
+When working with this codebase, understand that Kart bridges the gap between Git's file-based version control and the specialized needs of geospatial data management, requiring careful handling of both git operations and geospatial data formats.

kart/point_cloud/import_.py

@@ -7,6 +7,7 @@
     MutexOption,
     KartCommand,
 )
+from kart.crs_util import CoordinateReferenceString
 from kart.completion_shared import file_path_completer
 from kart.exceptions import InvalidOperation, INVALID_FILE_FORMAT
 from kart.lfs_util import prefix_sha256
@@ -112,6 +113,14 @@
         "the authoritative source for the given data and data is fetched from there if needed."
     ),
 )
+@click.option(
+    "--override-crs",
+    type=CoordinateReferenceString(keep_as_string=True),
+    help=(
+        "Override the CRS of all source tiles and set the dataset CRS. "
+        "Can be specified as EPSG code (e.g., EPSG:4326) or as a WKT file (e.g., @myfile.wkt)."
+    ),
+)
 @click.argument(
     "args",
     nargs=-1,
@@ -131,6 +140,7 @@ def point_cloud_import(
     num_workers,
     dataset_path,
     do_link,
+    override_crs,
     args,
 ):
     """
@@ -162,6 +172,7 @@ def point_cloud_import(
         num_workers=num_workers,
         do_link=do_link,
         sources=sources,
+        override_crs=override_crs,
     ).import_tiles()
 
 
@@ -171,6 +182,12 @@ class PointCloudImporter(TileImporter):
     CLOUD_OPTIMIZED_VARIANT = "Cloud-Optimized Point Cloud"
     CLOUD_OPTIMIZED_VARIANT_ACRONYM = "COPC"
 
+    def extract_tile_metadata(self, tile_path, **kwargs):
+        """Override to pass override_crs parameter to point cloud metadata extraction."""
+        if self.override_crs:
+            kwargs["override_crs"] = self.override_crs
+        return self.DATASET_CLASS.extract_tile_metadata(tile_path, **kwargs)
+
     def get_default_message(self):
         return f"Importing {len(self.sources)} LAZ tiles as {self.dataset_path}"
 
@@ -199,7 +216,9 @@ def get_merged_source_metadata(self, all_metadata):
         else:
             rewrite_metadata = RewriteMetadata.DROP_OPTIMIZATION
 
-        return rewrite_and_merge_metadata(all_metadata, rewrite_metadata)
+        return rewrite_and_merge_metadata(
+            all_metadata, rewrite_metadata, override_crs=self.override_crs
+        )
 
     def get_predicted_merged_metadata(self, all_metadata):
         if self.convert_to_cloud_optimized:
@@ -215,7 +234,9 @@ def get_predicted_merged_metadata(self, all_metadata):
             # For --preserve-format we allow both COPC and non-COPC tiles, so we don't need to check or store this information.
             rewrite_metadata = RewriteMetadata.DROP_OPTIMIZATION
 
-        return rewrite_and_merge_metadata(all_metadata, rewrite_metadata)
+        return rewrite_and_merge_metadata(
+            all_metadata, rewrite_metadata, override_crs=self.override_crs
+        )
 
     def get_actual_merged_metadata(self, all_metadata):
         rewrite_metadata = (
@@ -224,7 +245,9 @@ def get_actual_merged_metadata(self, all_metadata):
             else RewriteMetadata.DROP_OPTIMIZATION
         )
 
-        return rewrite_and_merge_metadata(all_metadata, rewrite_metadata)
+        return rewrite_and_merge_metadata(
+            all_metadata, rewrite_metadata, override_crs=self.override_crs
+        )
 
     def get_conversion_func(self, tile_source):
         if self.convert_to_cloud_optimized and not is_copc(tile_source.metadata):

kart/point_cloud/metadata_util.py

@@ -7,7 +7,7 @@
 
 from osgeo import osr
 
-from kart.crs_util import normalise_wkt, wkt_equal
+from kart.crs_util import normalise_wkt, wkt_equal, make_crs
 from kart.exceptions import (
     InvalidOperation,
     INVALID_FILE_FORMAT,
@@ -49,7 +49,7 @@ class RewriteMetadata(IntFlag):
 
 
 def rewrite_and_merge_metadata(
-    tile_metadata_list, rewrite_metadata=RewriteMetadata.NO_REWRITE
+    tile_metadata_list, rewrite_metadata=RewriteMetadata.NO_REWRITE, override_crs=None
 ):
     """
     Given a list of tile metadata, merges the parts we expect to be homogenous into a single piece of tile metadata in
@@ -59,18 +59,32 @@ def rewrite_and_merge_metadata(
     drop those parts of the metadata in accordance with the rewrite_metadata option. This means a) the merge will happen
     cleanly in spite of possible differences and b) we won't store any metadata that can't describe every tile in the
     dataset (ie, we won't store anything about whether tiles are COPC if we're going to allow a mix of both COPC and not).
+
+    If override_crs is provided, it will be used to override the CRS of all tiles, setting the dataset CRS.
     """
     result = {}
+    # Normalize override CRS if provided
+    normalized_override_crs = None
+    if override_crs:
+        crs_obj = make_crs(override_crs)
+        normalized_override_crs = normalise_wkt(crs_obj.ExportToWkt())
+
     for tile_metadata in tile_metadata_list:
         _merge_metadata_field(
             result, "format.json", rewrite_format(tile_metadata, rewrite_metadata)
         )
         _merge_metadata_field(
             result, "schema.json", rewrite_schema(tile_metadata, rewrite_metadata)
         )
-        _merge_metadata_field(
-            result, "crs.wkt", tile_metadata["crs.wkt"], eq_func=wkt_equal
-        )
+
+        # Handle CRS with potential override
+        if override_crs:
+            # Override CRS for all tiles if specified
+            tile_crs = normalized_override_crs
+        else:
+            tile_crs = tile_metadata["crs.wkt"]
+
+        _merge_metadata_field(result, "crs.wkt", tile_crs, eq_func=wkt_equal)
         # Don't copy anything from "tile" to the result - these fields are tile specific and needn't be merged.
     return result
 
@@ -157,7 +171,7 @@ def get_native_extent(info):
     )
 
 
-def extract_pc_tile_metadata(pc_tile_path, oid_and_size=None):
+def extract_pc_tile_metadata(pc_tile_path, oid_and_size=None, override_crs=None):
     """
     Use pdal to get any and all point-cloud metadata we can make use of in Kart.
     This includes metadata that must be dataset-homogenous and would be stored in the dataset's /meta/ folder,
@@ -178,6 +192,7 @@ def extract_pc_tile_metadata(pc_tile_path, oid_and_size=None):
 
     pc_tile_path - a pathlib.Path or a string containing the path to a file or an S3 url.
     oid_and_size - a tuple (sha256_oid, filesize) if already known, to avoid repeated work.
+    override_crs - if provided, override the CRS of the tile with this CRS.
     """
     pc_tile_path = str(pc_tile_path)
 
@@ -241,10 +256,16 @@ def extract_pc_tile_metadata(pc_tile_path, oid_and_size=None):
         "size": size,
     }
 
+    # Use override CRS if provided, otherwise use the CRS from the file
+    final_crs = compound_crs or horizontal_crs
+    if override_crs:
+        crs_obj = make_crs(override_crs)
+        final_crs = crs_obj.ExportToWkt()
+
     result = {
         "format.json": format_json,
         "schema.json": schema_json,
-        "crs.wkt": normalise_wkt(compound_crs or horizontal_crs),
+        "crs.wkt": normalise_wkt(final_crs),
         "tile": _remove_nones(tile_info),
     }
 

kart/tile/importer.py

@@ -76,6 +76,7 @@ def __init__(
         num_workers,
         do_link,
         sources,
+        override_crs=None,
     ):
         """
         repo - the Kart repo from the context.
@@ -95,6 +96,7 @@ def __init__(
         allow_empty - if True, the import commit will be created even if the dataset is not changed.
         num_workers - specify the number of workers to use, or set to None to use the number of detected cores.
         sources - paths to tiles to import.
+        override_crs - if specified, override the CRS of all source tiles and set the dataset CRS.
         """
         self.repo = repo
         self.ctx = ctx
@@ -111,6 +113,7 @@ def __init__(
         self.num_workers = num_workers
         self.do_link = do_link
         self.sources = sources
+        self.override_crs = override_crs
 
         need_to_store_tiles = not self.do_link
         need_tiles_for_wc = self.do_checkout and not self.repo.is_bare

tests/point_cloud/test_imports.py

@@ -1177,3 +1177,67 @@ def test_point_cloud_import_from_s3__convert(
                 check_tile_is_reflinked(
                     repo_path / "auckland" / f"auckland_{x}_{y}.copc.laz", repo
                 )
+
+
+def test_import_with_crs_override(
+    tmp_path,
+    chdir,
+    cli_runner,
+    data_archive_readonly,
+    check_lfs_hashes,
+    requires_pdal,
+    requires_git_lfs,
+):
+    """Test that --override-crs option overrides the CRS of all imported tiles."""
+    with data_archive_readonly("point-cloud/laz-auckland.tgz") as auckland:
+        repo_path = tmp_path / "point-cloud-repo"
+        r = cli_runner.invoke(["init", repo_path])
+        assert r.exit_code == 0, r.stderr
+
+        repo = KartRepo(repo_path)
+        with chdir(repo_path):
+            # First, import a single tile to establish the dataset CRS
+            # This tile is actually in EPSG:2193 but we override it to EPSG:2994
+            r = cli_runner.invoke(
+                [
+                    "point-cloud-import",
+                    f"{auckland}/auckland_0_0.laz",
+                    "--dataset-path=auckland",
+                    "--override-crs=EPSG:2994",
+                    "--convert-to-copc",
+                ]
+            )
+            assert r.exit_code == 0, r.stderr
+
+            # Get the original CRS from the dataset
+            r = cli_runner.invoke(["meta", "get", "auckland", "crs.wkt"])
+            assert r.exit_code == 0, r.stderr
+            original_crs = r.stdout.strip()
+            assert "2994" in original_crs
+
+            # Import another tile with --override-crs=EPSG:2994 to override its CRS
+            r = cli_runner.invoke(
+                [
+                    "point-cloud-import",
+                    f"{auckland}/auckland_0_1.laz",
+                    "--dataset-path=auckland",
+                    "--update-existing",
+                    "--override-crs=EPSG:2994",
+                    "--convert-to-copc",
+                ]
+            )
+            assert r.exit_code == 0, r.stderr
+
+            # Verify that both tiles were imported successfully
+            r = cli_runner.invoke(["data", "ls"])
+            assert r.exit_code == 0, r.stderr
+            assert "auckland" in r.stdout
+
+            # Check that the CRS is still consistent in the dataset
+            r = cli_runner.invoke(["meta", "get", "auckland", "crs.wkt"])
+            assert r.exit_code == 0, r.stderr
+            final_crs = r.stdout.strip()
+
+            assert "2994" in final_crs
+
+            check_lfs_hashes(repo, 2)