Skip to content

Conversation

matt-wright86
Copy link
Contributor

Add Exclusive Node with Parallel Processing Feature

Summary

This PR introduces a new feature that fills the gap between competing consumers (all nodes, full parallelism) and strict ordering (single node, single thread). The new ExclusiveNodeWithParallelism configuration allows for single-node exclusive processing while maintaining configurable parallelism for improved throughput.

Motivation

Currently, Wolverine offers two main execution patterns:

  1. Competing Consumers: All nodes process messages in parallel (default)
  2. Strict Ordering (ListenWithStrictOrdering): Single node, single thread processing

There's a missing middle ground for scenarios where you need:

  • Exclusive processing on a single node (for consistency/ordering requirements)
  • But with parallel processing within that node (for performance)

This is particularly useful for:

  • Processing ordered event streams with parallelism within partitions
  • Singleton background job processors
  • Resource-constrained operations that should run on one node but can parallelize internally

Changes

Core Implementation

  • Added ExclusiveNodeWithParallelism(int maxParallelism = 10, string? endpointName = null) to ListenerConfiguration
  • Added ExclusiveNodeWithSessionOrdering(int maxParallelSessions = 10, string? endpointName = null) for session-based ordering scenarios
  • Both methods properly validate input parameters and throw appropriate exceptions for invalid values
  • Added warnings (via Debug.WriteLine) when parallelism values exceed 100

Azure Service Bus Integration

  • Added AzureServiceBusListenerConfigurationExtensions with specialized methods:
    • ExclusiveNodeWithSessions() - Combines session requirements with exclusive node configuration
    • ExclusiveNodeWithParallelism() - For topic subscriptions without sessions

Testing

  • Comprehensive unit tests in ExclusiveNodeWithParallelismTests.cs covering:
    • Configuration state verification
    • Parameter validation (negative/zero values)
    • Local queue prevention
    • Method chaining compatibility
    • Default value testing
    • Endpoint name configuration

Documentation

  • Added detailed documentation in docs/guide/messaging/exclusive-node-processing.md including:
    • Clear use cases and scenarios
    • Code examples with snippets
    • Comparison table with other modes
    • Performance considerations and resource implications
    • Troubleshooting section
    • Implementation notes about leader election and failover

Breaking Changes

None. This change is fully backward compatible as it only adds new methods without modifying existing ones.

Testing

All tests pass:

  • 11/11 unit tests passing
  • Proper validation of configuration settings
  • Parameter validation working correctly
  • Local queue restrictions enforced

Usage Example

// Basic usage with default parallelism (10)
opts.ListenToRabbitQueue("important-jobs")
    .ExclusiveNodeWithParallelism();

// Custom parallelism
opts.ListenToRabbitQueue("background-tasks")
    .ExclusiveNodeWithParallelism(maxParallelism: 5);

// Session-based ordering (e.g., Azure Service Bus)
opts.ListenToAzureServiceBusQueue("ordered-events")
    .ExclusiveNodeWithSessionOrdering(maxParallelSessions: 5);

// Azure Service Bus with sessions
opts.ListenToAzureServiceBusQueue("user-events")
    .ExclusiveNodeWithSessions(maxParallelSessions: 8);

Checklist

  • Code follows the project's naming conventions (Pascal case for public members, camel case for private, _ prefix for private fields)
  • All tests pass
  • Documentation added
  • No breaking changes
  • Validated with .NET 9.0 SDK
  • Feature branch created (feature/exclusive-node-with-parallelism)

Notes for Reviewers

  • The implementation uses the existing ListenerScope.Exclusive with modified execution options to achieve the desired behavior
  • The warning threshold of 100 for parallelism is somewhat arbitrary and can be adjusted based on your preferences
  • The Azure Service Bus extension method was renamed from ExclusiveNodeProcessing to ExclusiveNodeWithParallelism for consistency
  • Resource implications have been documented to help users make informed decisions about parallelism levels

- Added ExclusiveNodeWithParallelism() method for single-node exclusive processing with configurable parallelism
- Added ExclusiveNodeWithSessionOrdering() for maintaining order within sessions while allowing parallel session processing
- Added Azure Service Bus specific extensions for session-based processing
- Added comprehensive unit tests covering all scenarios
- Added detailed documentation with examples and troubleshooting
- Includes validation for parameters and warnings for high parallelism values

This feature fills the gap between competing consumers and strict ordering, providing a valuable middle-ground option for scenarios requiring single-node consistency with parallel processing for improved throughput.
@jeremydmiller
Copy link
Member

Oh, sweet! Are you w/ Improving?

@matt-wright86
Copy link
Contributor Author

That's an excellent guess and the answer is yes lol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants