Building dynamic filters: Getting an Observable to use on Filter() when any subproperty of an collection change

[Dunge]

This lib is extremely powerful and flexible, but every time I try to use it I also find it very hard to find documentation and/or basic example of use cases.

I have a SourceCache on which I get a real time transformed list via Connect().AutoRefresh().Filter().Transform().SortAndBind().DisposeMany().Subscribe(). This part works great.

For the "Filter()" method, you can easily create an Observable for simple cases, like using a .WhenAny() on a string property so when the searched text changes the filter refresh.

What I'm trying to achieve is having the user build dynamic filters. Some sort of "query builder". This will be based on a collection of filter conditions that can be added and removed, and on which each one can apply different operations. And I will have a global "ApplyFilter" method that will compute everything to see if the conditions passes.

My issue is how to get the Observable to pass to Filter() so that it refreshes when the user add/remove or modify a new filter condition. How to get an observable that triggers when a collection or any of its sub objects are modified?

Here's what I came up with:

    public class DeviceListViewModel : ReactiveObject
    {
        public DeviceListViewModel()
        {
            var filter = this.WhenAnyValue(t => t.DeviceListFilters, t => t.DeviceListFilters.Filters, (t, u) => FilterDeviceList());
            _deviceListDisposable = MySoureCache
                .Connect()
                .AutoRefresh()
                .Filter(filter)
                .Transform(t => (Device)t)
                .SortAndBind(out _deviceList, SortExpressionComparer<Device>.Ascending(t => t.Name))
                .DisposeMany()
                .Subscribe();
        }

        private Func<ContainerBase, bool> FilterDeviceList()
        {
                return cnt => DeviceListFilters.ApplyFilter(cnt);
        }

        private FilterViewModel _deviceListFilters;
        public FilterViewModel DeviceListFilters
        {
            get { return _deviceListFilters; }
            set { this.RaiseAndSetIfChanged(ref _deviceListFilters, value); }
        }
    }


    public class FilterViewModel : ReactiveObject, IPropagateCollectionChanges
    {
        private ObservableCollection<FilterCondition> _filters = new ObservableCollection<FilterCondition>();
        public ObservableCollection<FilterCondition> Filters
        {
            get { return _filters; }
            set { this.RaiseAndSetIfChanged(ref _filters, value); }
        }

        public FilterViewModel()
        {
            this.PropagateCollectionChanged(Filters, nameof(Filters));
            Filters.ToObservableChangeSet()
              .WhenAnyPropertyChanged()
              .SubscribeEx(t =>
              {
                  this.RaisePropertyChanged("Filters");
              });
        }

        public bool ApplyFilter(Device device)
        {
            if (device == null)
                return false;

            foreach (var filter in Filters)
            {
                switch (filter.Filter)
                {
                     [...]
                }
            }
            return true;
        }
    }

    public class FilterCondition : ReactiveObject
    {
        private DeviceListFilter _filter;
        public DeviceListFilter Filter
        {
            get { return _filter; }
            set { this.RaiseAndSetIfChanged(ref _filter, value); }
        }

        private string _text;
        public string Text
        {
            get { return _text; }
            set { this.RaiseAndSetIfChanged(ref _text, value); }
        }
    }

As you can see, this is already getting exhaustive code and I'm sure it can be shortened to something simpler. But, it doesn't work.

I used ToObservableChangeSet/WhenAnyPropertyChanged to trigger an RaisePropertyChanged() when any of the sub properties of the FilterCondition model change, and I see it triggering, but somehow the WhenAny() observable doesn't, and the filter doesn't refresh?!? Why?

I also used IPropagateCollectionChanges to listen to the ObservableCollection add/remove events and trigger the same event.

    public static class IPropagateCollectionChangesExtensions
    {
        public static void PropagateCollectionChanged<TObj>(this TObj reactiveObject,
            INotifyCollectionChanged observableCollection,
            string collectionPropertyName)
            where TObj : IPropagateCollectionChanges
        {
            observableCollection.CollectionChanged += (obj, changeArgs) =>
            {
                reactiveObject.RaisePropertyChanged(collectionPropertyName);
            };
        }
    }

Is that a good method, is there a better method? It was taken from another thread here. I'm using Blazor, but I don't think it matters here.

[JakenVeina]

There's a bit too much missing context here for me to diagnose exactly what you're missing, but it's likely something in you filter stream logic. I.E. it's not emitting notifications for certain filter changes that you're interested in. I can definitely give you a working example of how I would do this...

public static class EntryPoint
{
    public static void Main()
    {
        using var devicesList = new DevicesListViewModel();

        using var subscription = devicesList.FilteredDevices
            .ObserveCollectionChanges()
            .Subscribe(_ =>
            {
                Console.WriteLine("\tFiltered Devices:");
                if (devicesList.FilteredDevices.Count is 0)
                    Console.WriteLine("\t\t<Empty>");
                else
                    foreach (var device in devicesList.FilteredDevices)
                        Console.WriteLine($"\t\t\"{device.Name}\"");
                Console.WriteLine();
            });

        Console.WriteLine("Adding Device #1...");
        devicesList.Devices.AddOrUpdate(new DeviceViewModel()
        {
            Id          = 1,
            Name        = "Device #1: Audio",
            Created     = DateTimeOffset.Parse("2025-01-01"),
            CreatedBy   = "Jake"
        });

        Console.WriteLine("Adding Device #2...");
        devicesList.Devices.AddOrUpdate(new DeviceViewModel()
        {
            Id          = 2,
            Name        = "Device #2: Video",
            Created     = DateTimeOffset.Parse("2025-01-02"),
            CreatedBy   = "Darrin"
        });

        Console.WriteLine("Adding Device #3...");
        devicesList.Devices.AddOrUpdate(new DeviceViewModel()
        {
            Id          = 3,
            Name        = "Device #3: Audio",
            Created     = DateTimeOffset.Parse("2025-01-03"),
            CreatedBy   = "Roland"
        });

        Console.WriteLine("Adding Device #4...");
        devicesList.Devices.AddOrUpdate(new DeviceViewModel()
        {
            Id          = 4,
            Name        = "Device #4: Video",
            Created     = DateTimeOffset.Parse("2025-01-04"),
            CreatedBy   = "Dunge"
        });

        Console.WriteLine("Adding Name Filter...");
        var nameFilter = new DeviceFilterViewModel()
        {
            Field           = DeviceField.Name,
            MatchingPattern = "Audio"
        };
        devicesList.Filters.Add(nameFilter);

        Console.WriteLine("Adding Created By Filter...");
        var createdByFilter = new DeviceFilterViewModel()
        {
            Field           = DeviceField.CreatedBy,
            MatchingPattern = "Jake"
        };
        devicesList.Filters.Add(createdByFilter);

        Console.WriteLine("Adding Device #5...");
        devicesList.Devices.AddOrUpdate(new DeviceViewModel()
        {
            Id          = 5,
            Name        = "Device #5: Audio",
            Created     = DateTimeOffset.Parse("2025-01-05"),
            CreatedBy   = "Jake"
        });

        Console.WriteLine("Removing Created By Filter...");
        devicesList.Filters.Remove(createdByFilter);

        Console.WriteLine("Changing Device #4 Name...");
        devicesList.Devices.Lookup(4).Value.Name = "Device #4: Audio";

        Console.WriteLine("Changing Name Filter Pattern...");
        nameFilter.MatchingPattern = "Video";
    }
}

public sealed class DeviceViewModel
    : INotifyPropertyChanged
{
    public DeviceViewModel()
    {
        _createdBy  = string.Empty;
        _name       = string.Empty;
    }

    public required long Id
    {
        get => _id;
        init => _id = value;
    }

    public required string Name
    {
        get => _name;
        set
        {
            if (_name == value)
                return;

            _name = value;

            PropertyChanged?.Invoke(this, new(nameof(Name)));
        }
    }

    public required DateTimeOffset Created
    {
        get => _created;
        set
        {
            if (_created == value)
                return;

            _created = value;

            PropertyChanged?.Invoke(this, new(nameof(Created)));
        }
    }

    public required string CreatedBy
    {
        get => _createdBy;
        set
        {
            if (_createdBy == value)
                return;

            _createdBy = value;

            PropertyChanged?.Invoke(this, new(nameof(CreatedBy)));
        }
    }

    public event PropertyChangedEventHandler? PropertyChanged;

    private readonly long _id;

    private DateTimeOffset  _created;
    private string          _createdBy;
    private string          _name;
}

public enum DeviceField
{
    Id,
    Name,
    Created,
    CreatedBy
}

public sealed class DeviceFilterViewModel
    : INotifyPropertyChanged
{
    public DeviceFilterViewModel()
        => _matchingPattern = string.Empty;

    public required DeviceField Field
    {
        get => _field;
        init => _field = value;
    }

    public string MatchingPattern
    {
        get => _matchingPattern;
        set
        {
            if (_matchingPattern == value)
                return;

            _matchingPattern = value;

            PropertyChanged?.Invoke(this, new(nameof(MatchingPattern)));
        }
    }

    public event PropertyChangedEventHandler? PropertyChanged;

    private readonly DeviceField _field;
    
    private string _matchingPattern;
}

public sealed class DevicesListViewModel
    : IDisposable
{
    public DevicesListViewModel()
    {
        _devices = new(static device => device.Id);

        _filters = new();

        // DynamicData equivalent of .Publish().RefCount() or .Share(), cause we're going to need to reference this stream twice.
        var filtersChanged = _filters
            .ToObservableChangeSet()
            .AsObservableList();

        var predicate = new Func<DeviceViewModel, bool>(EvaluateFilters);

        _subscription = _devices
            .Connect()
            .AutoRefresh()
            .Filter(Observable.Merge(
                    // Evaluate the filter on startup
                    Observable.Return(Unit.Default),
                    // Re-evaluate filtering upon changes to the collection of filters (E.G. filters added or removed)
                    filtersChanged
                        .Connect()
                        .Select(static _ => Unit.Default),
                    // Re-evaluate filtering upon changes to individual filters themselves
                    filtersChanged
                        .Connect()
                        .MergeMany(filter => filter.WhenAnyPropertyChanged())
                        .Select(static _ => Unit.Default))
                .Select(_ => predicate))
            .SortAndBind(out _filteredDevices, SortExpressionComparer<DeviceViewModel>.Ascending(device => device.Name))
            .Subscribe();
    }

    public ISourceCache<DeviceViewModel, long> Devices
        => _devices;

    public ReadOnlyObservableCollection<DeviceViewModel> FilteredDevices
        => _filteredDevices;

    public ObservableCollection<DeviceFilterViewModel> Filters
        => _filters;

    public void Dispose()
    {
        _devices.Dispose();
        _subscription.Dispose();
    }

    private bool EvaluateFilters(DeviceViewModel device)
        => (_filters.Count is 0)
            || _filters.All(filter =>
            {
                var subject = filter.Field switch
                {
                    DeviceField.Id          => device.Id.ToString(),
                    DeviceField.Name        => device.Name,
                    DeviceField.Created     => device.Created.ToString(),
                    DeviceField.CreatedBy   => device.CreatedBy.ToString(),
                    _                       => throw new InvalidOperationException()
                };

                return Regex.IsMatch(subject, filter.MatchingPattern);
            });

    private readonly SourceCache<DeviceViewModel, long>             _devices;
    private readonly ReadOnlyObservableCollection<DeviceViewModel>  _filteredDevices;
    private readonly ObservableCollection<DeviceFilterViewModel>    _filters;
    private readonly IDisposable                                    _subscription;
}

Output ends up as...

Adding Device #1...
        Filtered Devices:
                "Device #1: Audio"

Adding Device #2...
        Filtered Devices:
                "Device #1: Audio"
                "Device #2: Video"

Adding Device #3...
        Filtered Devices:
                "Device #1: Audio"
                "Device #2: Video"
                "Device #3: Audio"

Adding Device #4...
        Filtered Devices:
                "Device #1: Audio"
                "Device #2: Video"
                "Device #3: Audio"
                "Device #4: Video"

Adding Name Filter...
        Filtered Devices:
                "Device #1: Audio"
                "Device #3: Audio"
                "Device #4: Video"

        Filtered Devices:
                "Device #1: Audio"
                "Device #3: Audio"

Adding Created By Filter...
        Filtered Devices:
                "Device #1: Audio"

Adding Device #5...
        Filtered Devices:
                "Device #1: Audio"
                "Device #5: Audio"

Removing Created By Filter...
        Filtered Devices:
                "Device #1: Audio"
                "Device #3: Audio"
                "Device #5: Audio"

Changing Device #4 Name...
        Filtered Devices:
                "Device #1: Audio"
                "Device #3: Audio"
                "Device #4: Audio"
                "Device #5: Audio"

Changing Name Filter Pattern...
        Filtered Devices:
                "Device #3: Audio"
                "Device #4: Audio"
                "Device #5: Audio"

        Filtered Devices:
                "Device #2: Video"
                "Device #3: Audio"
                "Device #4: Audio"
                "Device #5: Audio"

        Filtered Devices:
                "Device #2: Video"
                "Device #4: Audio"
                "Device #5: Audio"

        Filtered Devices:
                "Device #2: Video"
                "Device #5: Audio"

        Filtered Devices:
                "Device #2: Video"

So yeah, no significant issue with triggering re-filtering whenever your set of filters changes, you just need to cover all the scenarios. I've commented that portion of a stream to indicate the thought process.

[Dunge]

Oh wow. This is a very complete answer. I could not have asked for any better! Thank you so much.

So my main issue was trying to fake a propertychanged event from the list and subscribe back to it via whenany, instead of actually using the methods to create a combined observable stream on the collections directly.

Nice to know I can call Connect() on a ObservableChangeSet to subscribe to the collection changed event! I used the above mechanism (subscribe to CollectionChanged/RaisePropertyChanged/subscribe to PropertyChanged) to call Blazor StateHasChanged() in many of my components as suggested in #303 and wonder if I would gain or lose performance by using those ObservableChangeSet connections everywhere instead?

The .AsObservableList() part in order to use it twice is an important concept I was missing out on.

I'll admit I'm a but loss when it comes the point of using Connect() then MergeMany() to get the propertychanged stream, since my previous version simply using WhenAnyPropertyChanged() directly on the ObservableChangeSet seemed to be working?

And finally the magic of using .Merge() to combine them all and Select() so that they return the same type (Unit) to wrap it up. Great.

[JakenVeina]

Nice to know I can call Connect() on a ObservableChangeSet to subscribe to the collection changed event!

To be specific about what's going on, you're not calling .Connect() on an "ObservableChangeSet". An "ObservableChangeSet" isn't really a thing.

var filtersChanged = _filters
    .ToObservableChangeSet()
    .AsObservableList();

.ToObservableChangeSet() is an extension method in DynamicData that bridges the INotifyCollectionChanged API and the DynamicDatta API. I.E. it converts an ObservableCollection<> into an IObservable<ChangeSet<>>, which is exactly what .Connect() on a SourceList<> returns. It's the data type that you can call DynamicData operators upon.

The thing you're calling .Connect() on is the IObservableList<> that .AsObservableList() materializes the changes into. It's equivalent to a read-only SourceList<>.

Personally, I think .ToObservableChangeSet() is a misnomer, because of confusion like this. Not quite sure what a better name would be. But it's quite old, from before I joined the project, and no living API is perfect.

wonder if I would gain or lose performance by using those ObservableChangeSet connections everywhere instead

If I'm reading and researching everything right, the IPropagateCollectionChanges implementation is just a workaround to add view-level support for INotifyCollectionChanged since it already supports INotifyPropertyChanged, yes? Except, support for INotifyPropertyChanged is something YOU added, it's not baked into Blazor. Somewhere in your View layer you have code that amounts to viewModel.PropertyChanged += (_, _) => StateHasChanged(). There shouldn't be anything stopping you from adding collection.CollectionChanged += (_, _) => StateHasChanged() in the same manner.

I don't expect you'd see any notable performance difference using your workaround to funnel .CollectionChanged events through .PropertyChanged versus just adding .CollectionChanged support directly, but the latter is definitely communicating your intent better, IMO.

The .AsObservableList() part in order to use it twice is an important concept I was missing out on.

Not really, it's just a small optimization. You could just as easily call .ToObservableChangeSet() twice, for each of the two queries you want to run from _filters. It's just going to do a bunch of the change-processing work twice.

I'll admit I'm a but loss when it comes the point of using Connect() then MergeMany() to get the propertychanged stream, since my previous version simply using WhenAnyPropertyChanged() directly on the ObservableChangeSet seemed to be working?

Turns out there is no point.

public static IObservable<TObject?> WhenAnyPropertyChanged<TObject>(
        this    IObservable<IChangeSet<TObject>>    source,
        params  string[]                            propertiesToMonitor)
    where TObject : INotifyPropertyChanged
{
    source.ThrowArgumentNullExceptionIfNull(nameof(source));

    return source.MergeMany(t => t.WhenAnyPropertyChanged(propertiesToMonitor));
}

That extension is literally just an alias for what I wrote. I was not aware that it exists. Your version is functionally identical, in that regard.

And finally the magic of using .Merge() to combine them all and Select() so that they return the same type (Unit) to wrap it up. Great.

I always put an extension method .SelectUnit() in my projects, for stuff like this.