v3.5.0.05 (beta)

With this beta release, we are adopting a new numbering convention for better alignment with the ServiceNow scoped application since ServiceNow does not allow alpha characters (e.g. "beta") in an application release number. Version "3.5.0.05" of the app will correspond to version "3.5.0.05" of the Java code. The first non-beta release of "3.5" will be "3.5.1".

If this is a new installation then you must install the following Update Sets in order

1. 20210423.DataPump.v1.1.xml
2. DataPump-v3.5.0.05-Upgrade.xml

If this is an upgrade then you only need to install the second Update Set.

"3.5" is a significant upgrade to SNDML, and contains a number of enhancements to support a new release of the DataPump scoped app. The Java code has been significantly refactored. If you are exclusively using the YAML command line interface, then you may want to stick with 3.4.6 for now, since it has been stable for since June 2022.

For additional details refer to

• https://github.com/gflewis/sndml3/wiki/Release-3.5

3.5.0-beta-3

3.5 pre-release with Update Set.

v3.4.6

Fix issue #34

v3.4.5

Upgrade libraries:

• mssql-jdbc to 8.4.1
• postgresql to 42.3.4
• jackson-databind to 2.12.6.1

v3.4.4

This release includes

• pom.xml specifies compiler and source versions as 1.8
• Upgrade log4j-core to 2.17.1
• Upgrade sqlite-jdbc to 3.36.0.3
• Add sqltemplate for sqlite

JUnit tests have been verified with SQLite3

v3.4.3

Upgrade log4j to 2.16.0

v3.4.2

This release includes

• Upgrade Java from 8 to 11
• Create JAR for SQLite
• New property daemon.pidfile

This release is compatible with version 1.1 of DataPump which can be downloaded from https://developer.servicenow.com/connect.do#!/share/user/content.

v3.4.1

Version 3.4.1 contains improvements for thread management, error handling and logging when running in --daemon or --scan mode.
If the property daemon.threads is less than 2, then the application will not spawn new threads and it will use a single ServiceNow session and Database connection for all jobs. If daemon.threads is greater than 1, then each job will have a unique ServiceNow session and Database connection.

The following new properties are defined.

• daemon.scope
• daemon.continue

This version also contains significant changes to the log4j2 configuration.

v3.4.0

3.4.0 is the initial release containing the new SNDML daemon utilized by the DataPump app on the ServiceNow Developer Share site. If you are using the YAML command line interface, then you may want to consider 3.2.0 (which is the current stable release).

The changes in 3.4 are very significant. The code has been completely refactored. The packages have been renamed. The JSON (json.org) and YAML (SnakeYAML) parsers have been replaced with Jackson. This affects both the YAML config files, and parsing of the JSON payloads returned by the ServiceNow REST API. A lot of new code has been added to support the new app, and some old code has been replaced or removed.

Changes in the YAML syntax include the following:

• The Tables parameter is now required to be a list of objects. The simpler syntax where this could be a list of table names is no longer supported.
• The meaning of the Name table parameter has been changed. This parameter is now used to designate the prefix that appears for properties in the metrics file. It is recommended that Source and/or Target be used instead of Name to designate the table name. Instead of {name: sys_user_group, truncate: true} you should write {source: sys_user_group, truncate: true} or {target: sys_user_group, truncate: true}. For backwards compatibility, Source and Target will default to Name if both are omitted.
• The Threads YAML global setting is no longer available. If you want to run multiple loads concurrently, it is recommended that you simply initiate multiple Java processes.
• Threads is now available as an option on partitioned loads. For example, if you are running a large historical load, you could specify {source: incident, target: incident, created: [2018-01-01, 2021-01-01], partition: month, threads: 3}. Threads is not a supported as an option for non-partitioned loads.
• Filter is now permitted for use with Action: SYNC. Use at your own risk. Records which do not match the filter will be deleted from the target table.
• Create is a new Action which can be used to create but not load a table. For example, to CREATE the Incident table, but not load any data, you can specify {target: incident, action: create}.
• Drop is available as an option for the Create Action. For example, {target: incident, action: create, drop: true}.
• Execute is a new Action. This action will execute any arbitrary SQL statement, but the primary intent is to call a stored procedure. For example, {name: "My stored procedure", action: execute, sql: "call my_stored_procedure"}.
• The SQLBefore and SQLAfter options for use in loads are deprecated. Please use the Execute action instead.
• Action: PRUNE is now deprecated. This action has been problematic because it requires access to sys_audit_delete, and because some tables are not audited. Action: SYNC appears to be a more reliable method of handling deletes.

Changes in processing behavior include the following:

• The Columns option now affects both table creation and table loading. SNDML will not alter an existing table. If columns are missing from the target table, then they will simply not be loaded.
• The loader no longer uses the getkeys API to retrieve a list of sys_ids when the Since option is specified for Action: UPDATE.

New properties in the connection profile include the following:

• servicenow.pagesize - Default page size if not overridden at table level
• datamart.autocreate - Defaults to "true" for backwards compatibility
• daemon.agent - Name of agent as configured in ServiceNow app
• daemon.interval - Polling interval in seconds
• daemon.threads - Maximum number of jobs that can run concurrently
• daemon.continue - Affects whether or not daemon will abort on an IO error

You can download the Update Set for the DataPump application from the ServiceNow Share Site at

• https://developer.servicenow.com/connect.do#!/share/contents/7145124_datapump_export_to_oracle_sql_server_mysql_or_postresql?v=1&t=PRODUCT_DETAILS

You will find the documentation here:

• https://docs.google.com/document/d/1VWbJuFpitrGSJ3WcoSL5V-8amXGOtO4b5ZRh2Fvu_g8/

3.2.0

Upgraded to latest release of Log4J2 and mysql-connecter 8.0. Because of the MySQL Connector upgrade it is recommended, beginning with this version, that the database.schema property in the connection profile property be used to specify the name of the MySQL database.

Added support for Domain Separation.

Significant redesign of Bash scripts in the bin directory. Bash scripts are now named with a sndml- prefix.