IBM Message Hub Toolkit v3.0.1

What's new in this release and what has changed

This toolkit is a bugfix release, which is based on the Kafka toolkit 3.0.1.
The underlying Kafka toolkit fixes following issue:

• Kafka toolkit 196 - KafkaProducer: Consistent region reset can trigger addtional reset. This issue affects the MessageHubProducer operator within this toolkit.

You find the SPL documentation at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v3.0.0

What's new in this release and what has changed

• This toolkit is based on the Kafka toolkit 3.0.0
• The included Kafka client has been upgraded from version 2.2.1 to 2.3.1.
• The schema of the output port of the MessageHubProducer operator supports optional types for the error description.
• When in autonomous region, the optional input port of the MessageHubConsumer operator can be used to change the topic subscription, not only the partition assignment. To create JSON for changing topic subscriptions, the toolkit contains new SPL functions. For consistent region, this new function is not available.
• The guaranteeOrdering parameter now enables the idempotent producer when set to true, which allows a higher throughput by allowing more in-flight requests per connection.
• The MessageHubConsumer operator now enables and benefits from group management when the user does not specify a group identifier, i.e. when the operator generates a group identifier.
• Checkpoint reset of the MessageHubConsumer is optimized in consistent region when the consumer operator is the only group member.
• The MessageHubConsumer operator now uses read_committed as the default isolation.level configuration unless the user has specified a different value. In read_committed mode, the consumer will read only those transactional messages which have been successfully committed. Messages of aborted transactions are now skipped. The consumer will continue to read non-transactional messages as before.

Deprecated features

• The use of the input control port has been deprecated when the MessageHubConsumer is used in a consistent region.

Removed features

The function, which has been deprecated in toolkit version 2.x has been removed in this toolkit version. The removed function contains following items:

• The messageHubCredentialsFile operator parameter has been removed from all operators. Please use the credentialsFile parameter instead.
• The default filename etc/messagehub.json for specifying Event Streams service credentials is not read any more. Please use etc/eventstreams.json when you want to use a default filename for service credentials.
• The default name messagehub for an application configuration is not read any more. Please use eventstreams as name for the application configuration when you want to use a default.
• The property name messagehub.creds within an application configuration is not read any more. Please name the property for the Event Streams credentials eventstreams.creds.

Changes in behaviour

MessageHubConsumer operator

When the user does not specify a group identifier, and does not specify partitions to consume (via the partition parameter), the MessageHubConsumer now subscribes to the given topics and benefits from group management, for example it automatically gets assigned new partitions when the number of partitions of the subscribed topics change. In previous versions, the MessageHubConsumer operator used to self-assign all available topic partitions of the topics at startup, and the partition assignment of the operator never changed during runtime.

With default configuration, the MessageHubConsumer operator reads only committed messages when transactional messages have been produced. Messages of aborted transactions are now skipped. To revert to the old behavior, add the consumer configuration

isolation.level=read_uncommitted

Incompatible changes

• The toolkit requires at minimum Streams version 4.3. It is incompatible with older versions of IBM Streams.
• When the MessageHubConsumer operator is configured with input port, the topic, pattern, partition, and startPosition parameters used to be ignored in previous versions. Now an SPL compiler error is raised when one of these parameters is used together with the input port.
• The deprecated function has been removed.

Bug fixes

• #98 (operator model)
• #99 (documentation)

The SPL documentation can be found at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v1.9.4

What's new in this toolkit release

This toolkit release is a critical bug fix release for those users who cannot update to a recent 2.x version (but at least 2.0.2). It is based on streamsx.kafka v1.9.5, which fixes following issue:

• IBMStreams/streamsx.kafka#171 Resetting from checkpoint will fail when sequence id is >1000

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/doc/v1.9.4/spldoc/html/.

IBM Message Hub Toolkit v2.2.1

What's new in this toolkit release

This toolkit release is a bugfix release. It is based on the streamsx.kafka bugfix release 2.2.1.

It fixes issue 179 - KafkaProducer: Lost output tuples on FinalMarker reception found in the streamsx.kafka toolkit.

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v2.2.0

What's new in this toolkit release

Changes and enhancements

• The MessageHubProducer operator can be configured with an optional output port. The function of this port can be configured with the outputErrorsOnly parameter.

• The MessageHubProducer operator has following new custom metrics: nFailedTuples, nPendingTuples, and nQueueFullPause.

• Changed recovery strategy for the MessageHubProducer operator when used outside a consistent region: In previous version the operator used to abort the PE on Kafka error with loss of all buffered producer records.

  Now the input tuples are kept until all their producer records are acknowledged. On retriable Kafka errors, the internal producer instance is re-created and the failed records of the tuples are retried. Records that fail two send attempts are treated as finally failed.

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v2.1.0

What's new in this toolkit release

This toolkit release is based on streamsx.kafka v2.1.0, which fixes following issues:

• IBMStreams/streamsx.kafka#169 new optional operator parameter sslDebug. For debugging SSL issues see also the Kafka toolkit documentation

• IBMStreams/streamsx.kafka#167 changed default values for following consumer and producer configurations:

  • client.dns.lookup = use_all_dns_ips
  • reconnect.backoff.max.ms = 10000 (Kafka's default is 1000)
  • reconnect.backoff.ms = 250 (Kafka's default is 50)
  • retry.backoff.ms = 500 (Kafka's default is 100)

• Changed exception handling for the MessageHubProducer when not used in a consistent region: IBMStreams/streamsx.kafka#163 (comment). In short: Exceptions that occur asynchronous to tuple processing are not propagated to the tuple processing anymore.

For the bugs fixed in the underlying streamsx.kafka toolkit, please refer to the release note of this toolkit: https://github.com/IBMStreams/streamsx.kafka/releases/tag/v2.1.0

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v2.0.2

What's new in this toolkit release

This toolkit release is based on streamsx.kafka v2.0.1, which fixes following issue:

• IBMStreams/streamsx.kafka#171 Resetting from checkpoint will fail when sequence id is >1000

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v2.0.1

What's new in this toolkit release

This toolkit release fixes the critical bug #91 in the v2.0.0 release, which should not be used.

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v2.0.0

Do not use this release. It contains the critical bug #91.

What's new in this toolkit release

• The included Kafka client has been upgraded from version 2.1.1 to 2.2.1, #89
• This toolkit release is based on Kafka toolkit release 2.0.0
• The toolkit has enhancements for the MessageHubConsumer when it is used in an autonomous region (i.e. not part of a consistent region):
  • The MessageHubConsumer operator can now participate in a consumer group with startPosition parameter values Beginning, 'End, and Time`, issue 94 in Kafka toolkit
  • After re-launch of the PE, the MessageHubConsumer operator does not overwrite the initial fetch offset to what the startPosition parameter is, i.e. after PE re-launch the consumer starts consuming at last committed offset, issue 107 in Kafka toolkit

The new startPosition handling requires that the application always includes a JobControlPlane operator when startPosition is different from Default.

Deprecations

The toolkit version 2.x deprecates following items (#87, #88):

• The messageHubCredentialsFile parameter. Please use the credentialsFile parameter instead
• The default filename etc/messagehub.json for service credentials. Please use etc/eventstreams.json when you want to use a default filename.
• The default name messagehub for an application configuration. Please use eventstreams as name for the application configuration when you want to use a default.
• The property name messagehub.creds within an application configuration. Please name the property for the Event Streams credentials eventstreams.creds.

The deprecated items are still supported by this toolkit version, but may be removed in next major toolkit version 3.0.

Incompatible changes, which may break existing applications

The behavior of the MessageHubConsumer operator changes when

1. the operator is not used in a consistent region, and
2. the startPosition parameter is used with Beginning, End, Time, or Offset.

In all other cases the behavior of the MessageHubConsumer is unchanged. Details of the changed behavior, including sample code that breaks, can be found in the Toolkit documentation on Github.

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/

IBM Message Hub Toolkit v1.9.3

What's new in this toolkit release

This release is based on the Kafka toolkit version 1.9.4 and inherits following fixes from this toolkit:

• IBMStreams/streamsx.kafka#158 (Kafka consumer: Failed offset commit makes the PE restart when in consistent region)
• support StateHandler function resetToInitialState when the operator is reset before checkpoint creation in an autonomous region

You find the SPL documentation online at https://ibmstreams.github.io/streamsx.messagehub/docs/user/SPLDoc/