Feature/SK-1575 | Test local updates on Dev Studio#898

Open

Dackeval wants to merge 70 commits intomasterfrom

feature/SK-1575

Commits on Apr 24, 2025

changes to log handling and num workers to allow for 1000 clients connected

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Apr 29, 2025

Updates for handling upto 1000 clients

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Updates to prolong keep alive timeout in conjunction with benjamins updates on branch: feature/SK-1353

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on May 14, 2025

updates to combiner, in order to remove the issue with combinerunavailableerror that occurs with 600+ clients connected.

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on May 19, 2025

Update to gunicorn_app.py for worker timeout increased and for a limit on status_routes.py to limit the amount of logs retrieved

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
removed fork from gunicorn_app.py

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Added cache for nr_active_clients to reduce the list_active_clients calls to the combiner. Added new methods _get_nr_active_clients_throttled,that uses COMBINER_CACHE_COOLDOWN to check against last…

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Setting _active_clients_cache and COMBINER_CACHE_COOLDOWN as instance attributes

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on May 21, 2025

Update to remove possible infinite loop for 2+ combiners

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
fix

Wrede
authored and
Sigvard Dackevall
committed
Bug/SK-1577 | Path bug fix in mnist-pytorch example (#899)

Hovstadius
authored and
Sigvard Dackevall
committed
Bug/SK-1467 | Mismatch in click.option input and method argument for repository (#875)

KatHellg
authored and
Sigvard Dackevall
committed
bump fedn

Wrede
authored and
Sigvard Dackevall
committed
Feature/SK-1449 | List projects with info in tabular format (#876)

KatHellg
authored and
Sigvard Dackevall
committed
Feature/SK-1566 | Added Telemetry Store/DTO/Route/GRPC (#897)

carl-andersson
authored and
Sigvard Dackevall
committed
Bug/SK-1582 | Running client.get_model_trail() returns only one model when should return multiple (#902)

niklastheman
authored and
Sigvard Dackevall
committed
Feature/SK-1361 | Write test for all API GET routes (#904)

niklastheman
authored and
Sigvard Dackevall
committed
fix header

Wrede
authored and
Sigvard Dackevall
committed
Bug/SK-1584 | Start session not working (api client) (#905)

niklastheman
authored and
Sigvard Dackevall
committed
Feature/SK-1585 | Start comparing response in api client tests (count, length of lists) (#903)

niklastheman
authored and
Sigvard Dackevall
committed
Bug/SK-1581 | Telemetry_bp is missing from routes (#901)

carl-andersson
authored and
Sigvard Dackevall
committed
Bug/SK-1589 | Race condition in project create (#906)

KatHellg
authored and
Sigvard Dackevall
committed
Feature/SK-1590 | Add ability to filter for null and grater than (etc) values in stores (#907)

niklastheman
authored and
Sigvard Dackevall
committed
Bugfix/SK-1591 | MongoClient was initialized before fork (#908)

carl-andersson
authored and
Sigvard Dackevall
committed
Feature/SK-1579 | Add a field 'updated_at' similar to 'committed_at' (#910)

niklastheman
authored and
Sigvard Dackevall
committed
Feature/SK-1596 | Added routes and ClientAPI methods (#911)

carl-andersson
authored and
Sigvard Dackevall
committed
Feature/SK-1598 | Integrate OpenTelemetry with controller, combiner (#915)

stefanhellander
authored and
Sigvard Dackevall
committed
Bugfix/SK-1602 | Telemetries are not getting cleared

carl-andersson
authored and
Sigvard Dackevall
committed
Feature/SK-1578 | Add tests for server functions (#900)

viktorvaladi
authored and
Sigvard Dackevall
committed
Feature/SK-1595 | Api client for server functions + example (#912)

authored and
Sigvard Dackevall
committed
fix (#914)

niklastheman
authored and
Sigvard Dackevall
committed
Feature/SK-1364 | Start existing session via API Client (#909)

niklastheman
authored and
Sigvard Dackevall
committed
Feature/SK-1605 | Improved error handling and grpc retries in python client (#919)

carl-andersson
authored and
Sigvard Dackevall
committed
Feature/SK-1353 | Extended tests in async-clients (#844)

benjaminastrand
authored and
Sigvard Dackevall
committed
Feature/SK-1261 | Split Learning in FEDn (#776)

FrankJonasmoelle
authored and
Sigvard Dackevall
committed
Feature/SK-1609 | Only init OTEL if configured, init tracer (#920)

stefanhellander
authored and
Sigvard Dackevall
committed
deps: update protobuf requirement (#845)

dependabot[bot]
authored and
Sigvard Dackevall
committed
bump fedn

Wrede
authored and
Sigvard Dackevall
committed
Docs/SK-1613 | Updating Split Learning Example Docs (#921)

FrankJonasmoelle
authored and
Sigvard Dackevall
committed
Fix/SK-1615 | Update load-test to take float factor for array size (#924)

Wrede
authored and
Sigvard Dackevall
committed
deps: bump flask from 3.1.0 to 3.1.1 in the pip group (#923)

dependabot[bot]
authored and
Sigvard Dackevall
committed
Bug/SK-1628 | sender_id index should be sender.client_id #927

niklastheman
authored and
Sigvard Dackevall
committed
bump fedn

niklastheman
authored and
Sigvard Dackevall
committed
Feature/SK-1586 | Switch from MinIO to Boto3 for storage management (#913)

Wrede
authored and
Sigvard Dackevall
committed
Updates for handling upto 1000 clients

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Updates to prolong keep alive timeout in conjunction with benjamins updates on branch: feature/SK-1353

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Update to gunicorn_app.py for worker timeout increased and for a limit on status_routes.py to limit the amount of logs retrieved

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
removed old flags

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on May 30, 2025

Update to how combiners write to round database, in order to avoid combiners writing over each others entries in rounds db

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
MongoDBRoundStore had no attribute update_one, so testing with self.server.db.round_store.collection.update_one instead

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 9, 2025

Added method to roundstore, to be able to update the entries on Round store, instead of copying and replacing.

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 10, 2025

Fixing typo in update_one method for MongoDBstore

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 11, 2025

Handling combinerunavailable error, in order not to break the thread, then also increasing the number of times and the time delay between retrying to submit in interfaces.

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Updated round logic to use session timeout, instead of waiting indefinately on combiners to finish updating.

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
added a if condition if aggregator had not been configured in roundhandler. Defaulting to fedavg

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 13, 2025

disabled otel logging, to remove transient error relating to hoenycomb logs

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 16, 2025

Every combiner needs to report their results before moving on to next round.

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 17, 2025

Added logging for identifying slow parts of the script, and added robustness to waitforit

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
formatting logs

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
try except on list_active_clients, if client id would be None, the roundhandler will still start the training round

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Update to _list_active_clients, skipping if None

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 19, 2025

Added a slight buffer to the timeout to allow combiners to respond slightly after timeout, if there would be any delay. Removed skipping clients in list_active_clients in combiner, since that intro…

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Skipping None clients in combiner and using Victors update on the session routes.

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Restored to use check_combiners_done_reporting

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Optional argument client_ids in _get_number_of_available_clients

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jun 22, 2025

Removing skip clients in _list_active_clients

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jul 3, 2025

Added max_concurrent_streams to the grpc server 10000, and updated the the grpc timeout in grpc_handler.py

Sigvard Dackevall
authored and
Sigvard Dackevall
committed

Commits on Jul 8, 2025

Adding robustness to model loading and log time taken

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Shorten the retry timeout and retry staging only once, 3x2 seconds with many clients can result in significant delay

Sigvard Dackevall
authored and
Sigvard Dackevall
committed
Shorten the retry delay in load_model_update_bytesIO and retry only once, 3x3xN clients too much when running a lot of clients

Sigvard Dackevall
authored and
Sigvard Dackevall
committed