IBM Message Hub Toolkit v1.5.1

This release is a bugfix release, which is based on the Kafka toolkit release v1.5.1.

It resolves following issues

• #61 - resolve security vulnerabilities in third-party libs

The underlying Kafka toolkit v1.5.1 resolves following issues:

• Kafka 124 - KafkaConsumer does not assign to all Partitions after restart when in CR
• Kafka 123 - KafkaConsumer should keep the generated group.id accross PE restarts
• Kafka 122 - resolve security vulnerabilities in third-party libs
• Kafka 121 - KafkaConsumer throws NullPointerException when topic is missing

IBM Message Hub Toolkit v1.5.0

This toolkit release is based on the Kafka toolkit v1.5.0.

New features and enhancements:

• enable Kafka consumer groups when in consistent region, #55 , Please have a look at the sample
  MessageHubConsumerGroupWithConsistentRegion
• Kafka transactions for the MessageHubProducer when in consistent region, #53 . Please verify that your plan of the IBM Event Streams cloud service is on Kafka 1.1
• new custom metrics nAssignedPartitions, isGroupManagementActive, nPartitionRebalances, drainTimeMillis, and drainTimeMillisMax
• Operators generate a client ID that allows to identify the Job and Streams operator, when no client ID is specified. The pattern for the client Id is {C|P}-J<job-ID>-<operator name>, where C denotes a consumer operator, P a producer.

Solved issues in the underlaying Kafka toolkit:

• issue 102 - KafkaConsumer crashes when fused with other Kafka consumers
• issue 105 - Message resources for non en_US locale should return en_US message when message is not available in specific language
• issue 115 - KafkaProducer: adapt transaction timeout to consistent region period

The online version of the SPL documentation for this toolkit is available here.

IBM Message Hub Toolkit v1.5.0 beta1 pre-release

This is a release for testing consumer groups together with consistent region. Do not use this release for productive use.

This release is considered stable regarding the interface of the operators.

Notable changes compared with the alpha release:

• the consistentRegionAssignmentMode parameter has been removed from the MessageHubConsumer operator
• To enable group management for the Consumer, it is sufficient to specify a group ID, unless the partition parameter or the input port is used
• additional metrics in the MessageHubConsumer operator
• improved SPL Doc

The SPL documentation for this release can be found here:
(https://ibmstreams.github.io/streamsx.messagehub/doc/v1.5.0.beta1/spldoc/html/)

The Kafka base is tagged with the v1.5.0_beta1 tag in the Kafka toolkit repository.

IBM Message Hub Toolkit v1.5.0 alpha pre-release

This toolkit release is for testing Kafka consumer group within a Consistent Region. Do not use this release for production. It is an early pre-release, which still has instabilities.

The SPL documentation of this release can be found here: https://ibmstreams.github.io/streamsx.messagehub/doc/v1.5.0.alpha/spldoc/html/

The Kafka base can be found in GitHub here: https://github.com/IBMStreams/streamsx.kafka/tree/v1.5.0_alpha

IBM Message Hub Toolkit v1.4.2

This release is a bugfix release.

It is based on the Kafka toolkit release v1.4.2, which fixes following issues:

• Kafka issue 100 - KafkaProducer registers for governance as input/source instead of output/sink. This issue also affected the MessageHubProducer operator.

IBM Message Hub Toolkit v1.4.1

This bugfix release is based on the bugfix release 1.4.1 of the Kafka toolkit. It inherits the fixes for following issues in the Kafka toolkit:

• Kafka issue 98 - KafkaConsumer can silently stop consuming messages

IBM Message Hub Toolkit v1.4.0

Release 1.4.0 of the MessageHub toolkit is based on the Kafka Toolkit v1.4.0 and inherits the corrections and changes made in the new Kafka toolkit release:

• Monitoring of memory consumption in addition to monitoring the internal queue fill (Kafka-TK issue 91)
• New custom metrics:
  • nLowMemoryPause - Number times message polling was paused due to low memory.
  • nQueueFullPause - Number times message polling was paused due to full queue.
  • nPendingMessages - Number of pending messages to be submitted as tuples.
• Committing offsets turned from synchronous to asynchronously. This can help improve throughput.
• Offsets are committed now after tuple submission. In previous versions, offsets have been committed immediately after messages have been buffered internally - before tuple submission. (Kafka-TK issue 76)
• New operator parameter commitCount to specify the commit period in terms of number of submitted tuples when the operator does not participate in a consistent region.
• Offset commit on drain when the operator is part of a consistent region. (Kafka-TK issue 95)
• corrected Kafka-TK issue 96 - KafkaConsumer: final de-assignment via control port does not work

IBM Message Hub Toolkit v1.3.3

This toolkit release is based on the Kafka toolkit v1.3.3 and inherits following features and fixes from the new Kafka toolkit version:

• internationalization (translated messages) for de_DE, es_ES, fr_FR, it_IT, ja_JP, ko_KR, pt_BR, ru_RU, zh_CN, and zh_TW
• fix of Kafka toolkit issue 88 - Consumer op is tracing per tuple at debug level

Known issues

• When Kafka's group management is enabled (MessageHubConsumer not in consistent region and startPosition parameter unset or Default), the MessageHubConsumer can silently stop consuming messages when committing Kafka offsets fails, see Kafka issue 98. As a workaround, the consumer property enable.auto.commit=true can be used in a consumer property file or app Option.

IBM Message Hub Toolkit v1.3.2

This release is based on the Kafka toolkit v1.3.2 and contains following fixes and enhancements:

• #46 toolkit build fails in Streams Quickstart Edition
• #45 Add support for the control port for the MessageHubConsumer operator
• SPL doc improvements and corrections

Note: The corrections inherited from the v1.3.2 Kafka release are not relevant for this toolkit. They all affect the transactional Kafka producer, which is disabled in this toolkit.

Known issues

• When Kafka's group management is enabled (MessageHubConsumer not in consistent region and startPosition parameter unset or Default), the MessageHubConsumer can silently stop consuming messages when committing Kafka offsets fails, see Kafka issue 98. As a workaround, the consumer property enable.auto.commit=true can be used in a consumer property file or app Option.

IBM Message Hub Toolkit v1.3.1

This bugfix release is based on Kafka toolkit v1.3.1 that fixes following Kafka issues:

• Kafka issue 75 - KafkaConsumer: InterruptedException not handled properly
• Kafka issue 79 - Consumer commits same offset multiple times

Included are SPL doc improvements regarding for the parameters topic, partition, startPosition, and startOffset.

Known issues

• When Kafka's group management is enabled (MessageHubConsumer not in consistent region and startPosition parameter unset or Default), the MessageHubConsumer can silently stop consuming messages when committing Kafka offsets fails, see Kafka issue 98. As a workaround, the consumer property enable.auto.commit=true can be used in a consumer property file or app Option.