Merge pull request #109 from lsst-sqre/tickets/schema-registry

Readd schema registry support

Author: Fireye04

.pre-commit-config.yaml

@@ -1,13 +1,13 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v5.0.0
+    rev: v6.0.0
     hooks:
       - id: check-toml
       - id: check-yaml
       - id: trailing-whitespace
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.12.3
+    rev: v0.12.9
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

CHANGELOG.md

@@ -6,6 +6,13 @@ Find changes for the upcoming release in the project's [changelog.d directory](h
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-0.4.1'></a>
+## 0.4.1 (2025-08-15)
+
+### New features
+
+- Readded support for avro schemas via direct connection
+
 <a id='changelog-0.4.0'></a>
 ## 0.4.0 (2025-07-16)
 

docs/dev/add_datasources.rst

@@ -79,10 +79,6 @@ Use this as an opportunity to debug your commands so they work as intended befor
 Add Schemas
 ===========
 
-.. warning ::
-
-    Sasquatch-backpack version 0.4.0 does not support non-json schemas in its default publishing method. Support is planned, but in the meantime either opt to use the old publish method (PublishMethod.REST_API), or serialize any schema data to json before publishing.
-
 `Sasquatch <https://sasquatch.lsst.io>`__ (the wearer of the proverbial backpack), uses `Avro schemas <https://sasquatch.lsst.io/user-guide/avro.html>`__ for data serialization.
 Navigate to your Datasource's folder and create a ``schemas.py`` file for your Datasource.
 Inside, use `pydantic's AvroBaseModel <https://marcosschroh.github.io/dataclasses-avroschema/pydantic/>`__ to create an avro schema.
@@ -104,7 +100,7 @@ Below is a quick sample of usage.
         class Meta:
             """Schema metadata."""
 
-            namespace = "$namespace"
+            namespace = "Default"
             schema_name = "topic_name_goes_here"
 
 Make one such schema for each command or API call you wish to make.
@@ -128,7 +124,7 @@ While not required, giving each entry a unique ID is strongly reccommended to id
 Note 3: Meta
 ------------
 The Meta subclass is required, and must contain both namespace and schema_name values.
-The namespace will be replaced with its actual value later on when the file is parsed, so simply keep its value as shown above, in "$thing" format.
+The namespace will be replaced with its actual value later on when the file is parsed, so simply keep its value "Default" as shown above.
 The schema_name, on the other hand, should be hardcoded in.
 
 Add Configs
@@ -161,7 +157,7 @@ Add Datasources
 ===============
 Now you're finally ready to add Datasources.
 From within your ``scripts.py`` file, for each command you have, make a new Datasource class inheriting from ``sasquatchbackpack.sasquatch.Datasource``.
-These new classes will require two methods: ``get_records()`` and ``get_redis_key()``.
+These new classes will require three methods: ``get_records()``, ``assemble_schema()`` and ``get_redis_key()``.
 
 ``get_records()`` should call the Datasource's respective ``scripts.py`` function, then return the encoded results in an array.
 This should be surrounded with a "try" like so:
@@ -179,6 +175,31 @@ This should be surrounded with a "try" like so:
                 f"A connection error occurred while fetching records: {ce}"
             ) from ce
 
+``assemble_schema()`` should take one of the list items obtained by get records (or None), and the namespace. This is where that default value above is substituted. This function has two purposes. One is to create a full schema object from a provided record and the other is to provide a compliant boilerplate schema if not provided a record. This is achieved like so:
+
+.. code-block:: python
+
+    def assemble_schema(self, namespace: str, record:dict | None = None) -> AvroBaseModel:
+        """Docstring: the third"""
+        if record is None:
+            schema = {
+                "timestamp": 1,
+                "id": "default",
+                # Every object in the schema needs to be here, and provided with a boiler plate values
+                #eg: "percentage":1.0,
+                "namespace"=namespace,
+            }
+        else:
+            schema = {
+                "timestamp": record["timestamp"],
+                "id": record["id"],
+                # Again, Every object in the schema needs to be here, and provided with its record value.
+                #eg: "percentage":record["percentage"]
+                "namespace": namespace,
+            }
+
+        return CommandnameSchema.parse_obj(data=schema)
+
 ``get_redis_key()`` can safely return an empty string if your config has set uses_redis to false, and you don't intend to integrate this souce with backpack's redis instance.
 Otherwise, this method should return a unique string structured as such: ``f"{self.topic_name}:uniqueItemIdentifierHere"``.
 This identifier is best suited as an integer id number as stated above in Note #2, however can be anything that uniquely identifies this specific object.

docs/dev/publish_methods.rst

@@ -13,10 +13,6 @@ Direct Connection
 
 PublishMethod.DIRECT_CONNECTION is the current default option. With sasquatch and backpack running on the same clusters, it just makes sense to directly connect in. This runs up a connection to sasquatch when publish is called, and sends published data through that tunnel.
 
-.. warning ::
-
-    Sasquatch-backpack version 0.4.0 does not support non-json schemas in its default publishing method. Support is planned, but in the meantime either opt to use the old publish method (PublishMethod.REST_API), or serialize any schema data to json before publishing.
-
 REST_API
 ========
 

docs/documenteer.toml

@@ -9,7 +9,17 @@ package = "sasquatchbackpack"
 rst_epilog_file = "_rst_epilog.rst"
 disable_primary_sidebars = ["index", "changelog"]
 extensions = ['sphinx_click']
-nitpick_ignore = [["py:class", "faststream.kafka.KafkaBroker"]]
+nitpick_ignore = [
+	[
+		"py:class",
+		"faststream.kafka.KafkaBroker",
+	],
+	[
+		"py:class",
+		"dataclasses_avroschema.pydantic.main.AvroBaseModel",
+	],
+]
 
 [sphinx.intersphinx.projects]
+safir = "https://safir.lsst.io"
 python = "https://docs.python.org/3/"