2025.4.0

Devs

• Added: Added mock PID service and PID tests (#839)
• Added: minimal and full publication test cases for each workflow
• Added: Use CSC PID service for DOI in FEGA and generic workflows (#827)
• Added: Implemented support for Bigpicture annotation and observer metadata schemas
• Added: Included new Bigpicture related schemas
• Added: Add /ingest endpoint for starting data ingestion
• Added: Mocked Admin API service #843
• Added: Instructions on how to use the SD Submit API for BigPicture programmatic dataset submissions #815
• Added: Defined a discovery service for each workflow 512
• Added: Total size of files is included in Metax metadata #507
• Added: Tests for taxonomy search #826
• Added: Created an endpoint for getting taxonomy names and id based on NCBI database dump files #281
• Added: Endpoint for flagging file as deleted #824
• Added: BP metadata gets accession attribute added to the XML content as well before it's stored #818
• Added: Endpoint to add and remove a linked folder path to a submission #825
• Added: new format for error responses of /validate endpoint #641
• Added: Validate files against file schema #823
• Added: List of Funding references from crossref API and enum values for Funder Names in Submission schema #817
• Added: Added a POST endpoint for receiving file information #814
• Added: Added additional 'announce' endpoint for BP workflow and to openapi document #813
• Added: Added new endpoint /users/{userId}/key for generating a signing key #777
• Removed: Some XSD files which are perceivably not being utilized at the moment
• Removed: Previously implemented logic that automatically converted ISO-8601 duration string in incoming metadata to a numeric value
• Removed: Some schemas that are unnecessary for Generic use case workflow #841
• Removed: RabbitMQ message broker functionality
• Fixed: Fixed faulty Datacite unit tests (#859)
• Fixed: Addressed utcnow deprecation warnings
• Fixed: Fix pyspelling not ignoring urls (#857)
• Fixed: Update broken and permanent redirect links in documentation (#856)
• Fixed: Fixed taxonomy script file permissions and ´docker casing warnings
• Fixed: Automated changelog generation with CI jobs and github actions recognizes git LFS files
• Fixed: Add missing schema check that disallowed multiple metadata objects
• Fixed: Fixed newly posted versions of deleted files still showing up as deleted #828
• Fixed: Fixed bugs in file version creation #823
• Changed: Altered and removed some of the Github actions workflows that were either not doing the correct thing or not needed anymore
• Changed: Updated instructions in the README and the CONTRIBUTING files
• Changed: Updated OpenAPI specifications
• Changed: all required schemas are needed before publishing a submission in unit and integration tests
• Changed: Updated some EGA related XML schemas to be up-to-date
• Changed: Updated XML to JSON conversion logic to accommodate new metadata schema versions
• Changed: REMS object is refined and validated when it is added to a submission #532 #649
• Changed: GET files request returns only latest file versions #840
• Changed: Removed large taxonomy related names.json file from the repository commit history and brought back as git LFS pointer file
• Changed: Modify SDSX workflow to be the same as FEGA workflow
• Changed: Refactored POST files endpoint to address several issues #828
• Changed: BigPicture workflow no longer requires a study
• Changed: XML objects are parsed from multipart XML structure into single objects before storing so the content matches with its JSON counterpart #818
• Changed: Modified workflow for Generic use case #821
• Changed: Updated middleware to accept user signed tokens #777
• Changed: Switch to idpyoidc library instead of oidc-rp
• Changed: Updated dependencies

Users

• Added: Process information from XML file for bprems schema and add it to the rems key in a BP submission
• Changed: Submission can no longer have the same name inside the project
• Changed: Previous functionality behind POST and PUT request for adding/modifying submission files is unified and the endpoint is retitled to PATCH /v1/submissions/{submissionId}/files
• Changed: Bigpicture related metadata validation process now follows the Bigpicture metadata version 2.0.0

2024.1.0

Added

• lastModified to folder to keep track when a folder or the objects in a folder change and be able to filter via the lastModified
• connection checking and retry-mechanism for calls to Metax-service in case of server and connection errors
• Endpoint for submitting DOI information. #389
• Endpoint with swagger documentation /swagger
• added aiohttp_session as dependency and removed old way of handling cookies
• Create a new ServiceHandler class to share error handling, retry mechanism, custom request logic between service integrations.
• Integration with REMS #498
  • New rems service handler
  • New rems mock api service for integration tests
  • New API endpoint /v1/rems for the frontend to retrieve DAC and Policies
  • Submission now can have a new field, dac (changed to rems in #648), with workflowId, organizationId, and licenses (array of int)
  • Published datasets have a new field dac (changed to rems in #648) with workflowId, organizationId, resourceId, and catalogueItemId
• Pylint static checks
• Added mapping for languages between the submitter and Metax-service #514
• Added mapping for subjects from submission doi info to Metax field_of_science #556
• Run integration tests with pytest
  • run integration tests with --nocleanup option
• Support for LifeScience groups as a substitute to CSC projects #548
• Support for Bearer tokens, opening use of the API without frontend. Tokens are validated from the configured OIDC_URL
• Made PKCE settings explicit in oidcrp client auth
• Added DPOP placeholder settings for when AAI support has been implemented
• Add advanced health checks for Datacite, REMS and Metax and performance checks for API response #585
• pre-commit check to sort and remove duplicates in the dictionary
• vulture as a tox env and pre-commit hook.
• Add endpoint for fetching workflows #362
• Add checks so that submission conforms with the workflows #591
• Add new schema file.json to represent files which are linked to a submission #148
  • Add a corresponding field to submission that lists files attached to a submission #148
• Add pytest-xdist to run unit tests faster, in parallel #626
• add MessageBroker class with MQPublisher and MQConsumer separate classes functionality #148 #622
  • add cli tool for MQConsumer so that we can deploy multiple consumers independently of the web server
  • add message broker publishing to workflow
  • add rabbitmq + default config to integration tests
• File operator that does database operations for files #148
  • introduced /v1/submissions/{submissionId}/files to update and remove files in a submission #633
  • file flagged for deletion also removed from submission and check files have the status ready when being read from the submission #633
  • prevent publish if files have in submission have status added (added but no metadata object) or failed (failed in ingestion, completion, or for any other reason) #633
• Mongo indexes for file schema #148
• /files endpoint to retrieve files attached to a project #148 #627
• option to add additional members to application/problem+json #642
• Bigpicture sample, image and dataset XML schemas were added and JSON schemas for those objects were produced #445 #481 #491
• Bigpicture observation XML schema was added and JSON schema for it was produced #665
• Bigpicture staining XML schema was added and JSON schema for it was produced #666

Changed

• schema loader now matches schema files by exact match with schema #481. This means that schema file naming in metadata_backend/helpers/schemas now have rules:
  • file name starts with schema provider separated with dot or underscore (e.g. EGA.policy.xsd, ena_policy.json) or
  • if schema is local then no schema provider needs to be added (e.g users.json)
  • schema name and mongo database collection name must be the same
• migrated to variables used by motor 3 for ssl https://motor.readthedocs.io/en/stable/migrate-to-motor-3.html?highlight=ssl_certfile#renamed-uri-options
  • env vars MONGO_SSL_CLIENT_KEY and MONGO_SSL_CLIENT_CERT are replaced with MONGO_SSL_CLIENT_CERT_KEY
• Refactor folder to submission #411
• HTTP PATCH for submissions has changed to only accept 'name' and 'description' in a flat JSON object.
• Better (metax) error handling #382
  • HTTPError exceptions return a response with JSON Problem instead of an HTML page #433
  • Make Metax errors visible to user #453
  • Catch unexpected errors and return a JSON Problem instead of server crashing #453
• Fix session and authorization issues
  • Prefix API endpoint with /v1
  • Refactor authentication checking, fixing issues from #421 and remove HTTPSeeOther from the API
• Recreate DB before integration tests run and cleanup after integration tests have run #448
• Make using of discovery service (METAX) optional #467
• Refactor api handlers to share a single instance of service handlers
• Refactor metadata creation methods to parse and add separate metadata objects to db from a single XML file #525
• Validate schema definitions, updating them to JSON schema 2020-12 #581
• changed HTTP header Server to metadata
• enabled new pylint checks #597 logging-fstring-interpolation, fixme, useless-param-doc, suppress-message
• reformatted logs to style { and refactored some of the messages to be clearer #597
• exceptions now default to log.exception with stacktrace #597
• POST/PATCH/PUT/DELETE requests on published submission responds with 405 instead of 401 HTTP response #618
• Submissions must have a workflow #591
• Refactor publish endpoint into its own python class and module #591
• Refactored Mongo queries to return the value for a single field #591
• There's more clear distinction between publishing to each integration: datacite, metax, and rems #591
• Separated operator classes into their own files for better readability #625
• updated ENA XML and JSON schemas to 1.16 #628
  • XML to JSON parser had to be adjusted for assemblyGraph
• Refactor Operator -> ObjectOperator #627
• Refactor xml_object.py -> object_xml.py #627
• Refactor operators to use a common base class called BaseOperator #627
• XML validation errors now compile all the failing elements into same error message and include the line number in the error reason #630
• Changed the submission object's dac field to rems and its subsequent endpoints similarly #648
• XML to JSON parser/converter was enhanced to accommodate for the new Bigpicture XML objects #445 #481 #491
• Updated BP sample XML and JSON schema, its parsing method and its related example file and tests #650
• Updated BP image and dataset JSON schemas for better functionality #664
• Updated XML to JSON converter to parse ISO-8601 duration strings into numbers in BP sample attributes #696

Removed

• remove datacite.json to render the form from folder["doiInfo"]
  • we removed namedtype for contributors and creators we therefore allow additionalProperties
  • subjectsSchema is a given by frontend thus we allow via additionalProperties
• Removed OIDC_ENABLED testing variable which can cause misconfiguration incidents
• Removed METAX_ENABLED as superseded by the introduction of workflows #591
• Removed REMS_ENABLED as superseded by the introduction of workflows #591
• remove unused code related to change from user to project ownership caused by faulty rebase or rollback of certain features #579
• Remove dictionary de-duplication from pre-commit's sort hook #626

Fixed

• Schemas endpoint returned 400 for /v1/schemas/datacite #554
• XML delete when an object or submission is deleted #579
• small pylint issues e.g. web.HTTPSuccessful was never being raised #579
• fix Any type wherever that is possible. #579
• Published submissions and its objects cannot be altered #584
• Incorrectly marking a workflow schema as required because its step is required #591
• Incorrectly marking a workflow schema as required when it appears in the requires field of a schema that is not required #591
• deprecated syntax in github publish action for ::set-output and better default param. Dockerfile fix for github actions
• mypy complaining about default values None in service handler

v0.13.1

[0.13.1] - 2022-05-30

Changed

• migrated to variables used by motor 3 for ssl https://motor.readthedocs.io/en/stable/migrate-to-motor-3.html?highlight=ssl_certfile#renamed-uri-options #420
  • env vars MONGO_SSL_CLIENT_KEY and MONGO_SSL_CLIENT_CERT are replaced with MONGO_SSL_CLIENT_CERT_KEY
• adds required field affiliation for creator and contributor in datacite schema #399

v0.13.0

Added

• Submission endpoint update #371
  • Adds mandatory query parameter folder for submit endpoint POST
  • On actions add and modify object is added or updated to folder(submission) where it belongs with it's accession ID, schema, submission type, title and filename
  • Adds metax integration to submit endpoint
• Integration with Metax service #356 #387
  • Adds new local container for testing against mocked Metax API
  • Introduces new env vars: METAX_USER, METAX_PASS, METAX_URL for connection to Metax service
  • Introduces new env var DISCOVERY_URL for creating link to dataset inside Fairdata SD catalog
  • Adds new key metaxIdentifier to Study and Dataset collections containing metax id returned from Metax API
  • Adds new handler MetaxServiceHandler to take care of mapping Submitter metadata to Metax metadata and to connect to Metax API
  • Adds new mapper class to adjust incoming metadata to Metax schema
• Add patching of folders after object save and update operations #354
  • Adds mandatory query parameter folder for objects endpoint POST
  • Object is added or updated to folder(submission) where it belongs with it's accession ID, schema, submission type, title and filename in the case of CSV and XML upload
  • Adds configuration for mypy linting to VScode devcontainer setup
• Templates API #256
  • use ujson as default json library
• Creating draft Datacite DOI for folders #257 #332
  • created a mock web app, which would act similarly to DataCite REST API
  • altered publish_folder endpoint so that extraInfo containing the DOI data is added upon publishing
  • added datePublished key to folders which takes in the date/time, when folder is published
• DOI Publishing and deletion to Datacite #332 #369
  • create draft DOIs for both Study and Datasets and add them to the folder extraInfo when published
  • delete draft DOIs on object delete
  • update DOI info at Datacite when folder is published
• VScode Dev environment #287
  • Add VS Code development container
  • Update docker for development
• Docker-compose and docker-compose-tls files changed to use variables from .env file. #301
• Add folder querying by name #305
  • Add indexing on database initialization
  • Add new field text_name to folder collection
  • Python scripts for database operations. mongo_indexes.py for collections and indexes creation to be run if the database is destroyed and clean_db.py script with new functionality to only delete documents from collections
  • update github actions
• Add folder querying by date #308
• Add description to JSON schemas #323
  • add JSON schema spelling checker to pyspelling github action
  • optimise wordlist by adding regex ignore patterns
  • added pyspelling to pre-commit hooks (fixed syntax for scripts according to https://github.com/koalaman/shellcheck )
  • enum are sorted alphabetically, with the exception of other and unspecified values which are left at the end of the list
  • allow for accession key in referenceAlignment & process sequence as array, previously all accession keys were converted to accessionId which is not correct
  • add default gender as unknown
• Project ownership #346
  • added new collection project
  • added new key projects to user
  • added new key projectId to folder and template-* collections
  • new mandatory /userinfo value from AAI at login time sdSubmitProjects
    • user is redirected to an info page by AAI if key is missing
  • new mandatory query parameter projectId in GET /folders
  • new mandatory JSON key projectId in POST /folders and POST /templates
  • new endpoint GET /templates to replace GET /users/current {"templates":[...]}
  • new JSON keys index and tags to PATCH /templates/schema/templateId, same values as were previously used in PATCH /user which is now removed
  • WARNING: breaking change that requires fresh database, because "project" is new information that did not exist before, and it can't be migrated to existing user-owned hierarchy
• Multilevel add patch objects to support /extraInfo/datasetIdentifiers/- which needs dot notation for mongodb to work e.g. extraInfo.datasetIdentifiers #332

Changed

• Refactor auth.py package by removing custom OIDC code and replacing it with https://github.com/IdentityPython/JWTConnect-Python-OidcRP. #315
  • New mandatory ENV OIDC_URL
  • New optional ENVs OIDC_SCOPE, AUTH_METHOD
  • Added oidcrp dependency
• Use node 16+ #345
• VScode Dev environment #287
  • Adds requirements-dev.in/txt files. Now pip dependencies can be managed with pip-tools
  • README updated with tox command, development build instructions, and prettify Dockerfile.
• Update ENA XML and JSON schemas #299
• Github actions changed the use of https://git.io/misspell to rojopolis/spellcheck-github-actions #316
• Separated most of the handlers to own files inside the handlers folder #319
• allow inserting only one study in folder #332
• JSON schemas #332
  • introduce keywords required for Metax in doiInfo
  • dataset description and study studyAbstract are now mandatory
• keywords will be comma separated values, that will require splitting when adding to Metax API

Fixed

• Coveralls report #267
• Typos for functions and tests #279
• Fix spelling mistakes for JSON schemas #323
• Oidcrp does not allow empty values, prefill them in mockauth so front-end can start #333
• Fix development environment #336
  • Add env vars OIDC_URL and OIDC_URL_TEST to mock auth container
  • Adds logging configs for mock auth
  • Updates mock auth api's token endpoint with expiration configs
  • Adds config .pre-commit-config.yaml file required by pre-commit library
  • Redirect url in docker-compose is now default
  • Adds logging for doi mock api

Removed

• Removed Authlib dependency #315
• Project ownership #346
  • deprecated folders and templates keys from GET /users/current
    • as a side effect, deprecated items query parameter from the same endpoint
  • deprecated PATCH /user

Deprecated

• Deprecated ENVs ISS_URL, AUTH_URL, AUTH_REFERER, JWK_URL #315

metadata-submitter v0.11.0

• package updates
• Feature/sort folders #249
• Include DOI information in the folder schema #246

metadata-submitter v0.10.0

• PRs: #217 #223 #224 #243 #235 #227 #240 #233 #231
• package updates
• Fix misses for DAC, experiment and policy processing of XML
• fix misses in JSON Schema
• add integration tests for misses in dataset, experiment, policy
• refactor analysis and experiment schemas to adhere to XML schema
• EGA XML schemas version:1.8.0

metadata-submitter v0.9.0

PR #203 #206 #207 #208

• Refactor JSON schema Links
• refactor handlers to be more streamlined
• validate patch requests for JSON content
• use dependabot
• switch to python 3.8
• support simultaneous sessions

metadata-submitter v0.8.1

bugfix for error pages. See #202

metadata-submitter v0.8.0

see #180 #181 #183 #185 #186 #187 #189 #190 #193 #194 #195 #197 #198 #199

• TLS support
• use sub as alternative to eppn to identify users
• 'PATCHfor objects andPUT` for XML objects enabled
• delete folders and objects associated to user on user delete
• fix replace on json patch
• redirect to error pages
• general bug and fixes
• extended integration tests

metadata-submitter v0.7.1

hotfix release, see #176

Changes made

• check objects and folders exist before any operation
• integration check to see if deleted object or folder are still registered in db