Skip to content

Extending Delta Producer

Jump to bottom
DavidJ edited this page Mar 22, 2023 · 2 revisions

Description

This page will explain how to extend the delta producer. There is cases where we need to retrieve data on the other end.

E.g. : A worship-submission is made and sent in Loket but we need the sent date in app-worship-decisions-database so a config updated and a sync is needed to get the submission in the app-worship-decisions-database DB.

How the flow works.

When creating and sending new submissions in Loket, a new submission delta is produced and ready to be consumed. Connecting two apps for a sync will allow the consumer app to ingest and dispatch the submission delta. E.g. : We can use Loket to send worship-submissions to app-worship-decisions-database.

Relevant services

To make sure our submission is flowing correctly we'll be looking at these services.

In app-digitaal-loket

delta-producer-background-jobs-initiator-worship-submissions : delta-producer-background-jobs-initiator

delta-producer-publication-graph-maintainer-worship-submissions : delta-producer-publication-graph-maintainer

Note that services can be extended to other submissions as well. E.g : delta-producer-background-jobs-initiator-leidinggevenden & delta-producer-publication-graph-maintainer-leidinggevenden

In app-worship-decisions-database

For deltas

Note : Every cases has his own service and configuration.

Development setup

How to ingest a submission from Loket locally

You will need to log in as the same administrative unit in both apps, some rules may change so it's easier to watch changes this way.

What you will need as worship-decision-database setting in your docker-compose file

  ################################################################################
  # DELTAS
  ################################################################################
  submissions-consumer:
    image: lblod/delta-consumer:0.0.15
    environment:
      DCR_SYNC_BASE_URL: "http://0.0.0.0:90" # 0.0.0.0 is just for the sake of the example. To know your address use hostname -I in your linux terminal, it should be the first one to pickup. 
      DCR_SYNC_LOGIN_ENDPOINT: 'http://0.0.0.0:90/sync/worship-submissions/login' # don't forget DCR_SECRET_KEY in docker-compose.override.yml
      DCR_SERVICE_NAME: "submissions-consumer"
      DCR_SYNC_FILES_PATH: "/sync/worship-submissions/files"
      DCR_SYNC_DATASET_SUBJECT: "http://data.lblod.info/datasets/delta-producer/dumps/WorshipSubmissionsCacheGraphDump"
      DCR_INITIAL_SYNC_JOB_OPERATION: "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/submissions"
      DCR_JOB_CREATOR_URI: "http://data.lblod.info/services/id/submissions-consumer"
      DCR_DISABLE_INITIAL_SYNC: "true"
      DCR_KEEP_DELTA_FILES: "true"
      DCR_SECRET_KEY: "your_secret_key" # This is important as you need it to sync the app same as loket KEY.
      DCR_START_FROM_DELTA_TIMESTAMP: "2023-03-12T16:26:00.596Z"
      DCR_WAIT_FOR_INITIAL_SYNC: 'false'
      DCR_DELTA_FILE_FOLDER: "/consumer-files"
      DCR_DELTA_SYNC_JOB_OPERATION: "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/sumbissionFileSyncing"
      INGEST_GRAPH: "http://mu.semte.ch/graphs/temp/for-dispatch"
      FILE_SYNC_GRAPH: "http://mu.semte.ch/graphs/temp/original-physical-files-data"
    volumes:
      - ./config/submissions-consumer/submissions-dispatching:/config/triples-dispatching/custom-dispatching
      - ./data/files/consumer-files/submissions:/consumer-files/
      - ./data/files:/share
    restart: always
    logging: *default-logging
  files-consumer:
    image: lblod/delta-consumer-file-sync-submissions:0.5.1
    environment:
      DISABLE_AUTOMATIC_SYNC: "true"
      DISABLE_INITIAL_SYNC: "true"
      FILES_ENDPOINT_BASE_URL: "http://0.0.0.0:90"
      SYNC_LOGIN_ENDPOINT: "https://0.0.0.0:90/sync/worship-submissions/login" # don't forget SECRET_KEY in docker-compose.override.yml
      SOURCE_FILES_DATA_GRAPH: "http://mu.semte.ch/graphs/temp/original-physical-files-data"
      SECRET_KEY: "your_secret_key"
    volumes:
      - ./data/files:/share

In loket

  delta-producer-publication-graph-maintainer-worship-submissions:
    environment:
      ...
      WAIT_FOR_INITIAL_SYNC: "false"
      KEY: "your_secret_key" # This is important as you need it to sync the app.

delta-producer-publication-graph-maintainer, see delta-producer-publication-graph-maintainer will not work without providing the matching keys so make sure you have the same KEY and DCR_SECRET_KEY, also don't forget to add SYNC_LOGIN_ENDPOINT in your docker-compose files for the app-worship-decisions-database project.

For better results, you can set a timestamp with the ENV DCR_START_FROM_DELTA_TIMESTAMP in the submission-consumer service and then set the ISO date before the produced delta. This way it reduces the syncing time.

Extending config in Loket

To add new predicates, you'll need to modify in the root config folder the submission resource located in master-submissions-domain.lisp with the new predicate property and within the /delta-producer folder, you'll need to modify the export.json file corresponding to the right submission type.

config/
└── delta-producer/
    ├──
    └──submissions/
    	└── export.json

To add the new predicate check that you are in the submission Object and then you can add the new predicate to the properties Array :

{
  "export": [
    {
      "type": "http://rdf.myexperiment.org/ontologies/base/Submission", // This is the line you should look for
      "graphsFilter": [ "http://mu.semte.ch/graphs/organizations/.*/LoketLB-toezichtGebruiker" ],
      "properties": [
        // New predicates are added here
        "http://www.w3.org/1999/02/22-rdf-syntax-ns#type",
        "http://mu.semte.ch/vocabularies/core/uuid",
        "http://www.w3.org/ns/prov#atLocation",
        "http://purl.org/dc/terms/created",
        "http://purl.org/dc/terms/modified",
        "http://purl.org/pav/createdBy",
        "http://purl.org/pav/providedBy",
        "http://purl.org/dc/terms/subject",
        "http://purl.org/dc/terms/source",
        "http://www.w3.org/ns/adms#status",
        "http://www.w3.org/ns/prov#generated",
        "http://purl.org/dc/terms/hasPart"
      ],
      "additionalFilter": "?subject <http://schema.org/publication> <http://lblod.data.gift/concepts/403b71bd-5ab9-4c92-8990-4bb19d5469d1>."
    },
    ...
   ]
}

Note that if you modify a predicate for a worship-submission you'll have to change the export.json in submissions AND export.json in worship-submissions.

To check if a delta is created you will find new deltas in within Loket :

data/
└── files/
    └── deltas/

Note that each subfolders are linked to the submission type so for example if you created a worship submission, you'll find the related delta in worship-submissions subfolder.

Rules

Each submissions are dispatched in the consumer app by what we call dispatch rules. It makes sure we have the right submission linked to the right organisations, the rules are defined by the business so this is subject to change.

E.g. : Let's say CKB Aalst is sending a submission from Loket, organisations linked to CKB Aalst will also see the submission in app-worship-decisions-database including CKB Aalst.

Relevant service

Dispatcher for worship-submissions : See documentation worship-submissions-graph-dispatcher-service

Clone this wiki locally