Skip to content

v3.4.0
Compare
Choose a tag to compare
@gflewis gflewis released this 28 Mar 20:19
· 346 commits to master since this release
v3.4.0
5ceb08e

3.4.0 is the initial release containing the new SNDML daemon utilized by the DataPump app on the ServiceNow Developer Share site. If you are using the YAML command line interface, then you may want to consider 3.2.0 (which is the current stable release).

The changes in 3.4 are very significant. The code has been completely refactored. The packages have been renamed. The JSON (json.org) and YAML (SnakeYAML) parsers have been replaced with Jackson. This affects both the YAML config files, and parsing of the JSON payloads returned by the ServiceNow REST API. A lot of new code has been added to support the new app, and some old code has been replaced or removed.

Changes in the YAML syntax include the following:

  • The Tables parameter is now required to be a list of objects. The simpler syntax where this could be a list of table names is no longer supported.
  • The meaning of the Name table parameter has been changed. This parameter is now used to designate the prefix that appears for properties in the metrics file. It is recommended that Source and/or Target be used instead of Name to designate the table name. Instead of {name: sys_user_group, truncate: true} you should write {source: sys_user_group, truncate: true} or {target: sys_user_group, truncate: true}. For backwards compatibility, Source and Target will default to Name if both are omitted.
  • The Threads YAML global setting is no longer available. If you want to run multiple loads concurrently, it is recommended that you simply initiate multiple Java processes.
  • Threads is now available as an option on partitioned loads. For example, if you are running a large historical load, you could specify {source: incident, target: incident, created: [2018-01-01, 2021-01-01], partition: month, threads: 3}. Threads is not a supported as an option for non-partitioned loads.
  • Filter is now permitted for use with Action: SYNC. Use at your own risk. Records which do not match the filter will be deleted from the target table.
  • Create is a new Action which can be used to create but not load a table. For example, to CREATE the Incident table, but not load any data, you can specify {target: incident, action: create}.
  • Drop is available as an option for the Create Action. For example, {target: incident, action: create, drop: true}.
  • Execute is a new Action. This action will execute any arbitrary SQL statement, but the primary intent is to call a stored procedure. For example, {name: "My stored procedure", action: execute, sql: "call my_stored_procedure"}.
  • The SQLBefore and SQLAfter options for use in loads are deprecated. Please use the Execute action instead.
  • Action: PRUNE is now deprecated. This action has been problematic because it requires access to sys_audit_delete, and because some tables are not audited. Action: SYNC appears to be a more reliable method of handling deletes.

Changes in processing behavior include the following:

  • The Columns option now affects both table creation and table loading. SNDML will not alter an existing table. If columns are missing from the target table, then they will simply not be loaded.
  • The loader no longer uses the getkeys API to retrieve a list of sys_ids when the Since option is specified for Action: UPDATE.

New properties in the connection profile include the following:

  • servicenow.pagesize - Default page size if not overridden at table level
  • datamart.autocreate - Defaults to "true" for backwards compatibility
  • daemon.agent - Name of agent as configured in ServiceNow app
  • daemon.interval - Polling interval in seconds
  • daemon.threads - Maximum number of jobs that can run concurrently
  • daemon.continue - Affects whether or not daemon will abort on an IO error

You can download the Update Set for the DataPump application from the ServiceNow Share Site at

You will find the documentation here:

Assets 5
Loading