fix(surrealdb): improve surrealdb healthchecks

[Odonno]

Fixes #794

Improve SurrealDB resources healthchecks to avoid consumption before initialized.

PR Checklist

• Created a feature/dev branch in your fork (vs. submitting directly from a commit on main)
• Based off latest main branch of toolkit
• PR doesn't include merge commits (always rebase on top of our main, if needed)
• Tests for the changes have been added (for bug fixes / features) (if applicable)
• Contains NO breaking changes
• Every new API (including internal ones) has full XML docs
• Code follows all style conventions

Other information

Note: there is also an issue with some password generation (not correctly handled by the client). I am also trying to figure out a solution.

[Odonno]

Not sure how to properly escape password but I suppose now it should work for a majority of scenario.

[aaronpowell]

Looks like there's still a failing test

[Odonno]

I can't reproduce this, and I don't see the reason why it could fail. Isn't it a random error, what if we rerun the failed jobs?

[aaronpowell]

Randomly failing tests are a concern because they either a) represent tests that aren't robust or b) represent a feature that isn't stable and could cause runtime errors to users.

Looking through the logs, it appears that the health check for databases is not passing, so I've pushed a commit with some additional log messages in the health check to see if we can get more insights.

[aaronpowell]

The new logs are reporting this:

fail: CommunityToolkit.Aspire.SurrealDb.SurrealDbHealthCheck[0]
      SurrealDB health check raised an exception. Health check had previously reported: Healthy.
      System.TimeoutException: The operation has timed out.
         at SurrealDb.Net.Internals.SurrealDbWsEngine.SendRequestAsync(String method, Object[] parameters, SurrealDbWsRequestPriority priority, CancellationToken cancellationToken) in /_/SurrealDb.Net/Internals/SurrealDbEngine.Ws.cs:line 1535
         at SurrealDb.Net.Internals.SurrealDbWsEngine.SendRequestAsync(String method, Object[] parameters, SurrealDbWsRequestPriority priority, CancellationToken cancellationToken) in /_/SurrealDb.Net/Internals/SurrealDbEngine.Ws.cs:line 1546
         at SurrealDb.Net.Internals.SurrealDbWsEngine.SendRequestAsync(String method, Object[] parameters, SurrealDbWsRequestPriority priority, CancellationToken cancellationToken) in /_/SurrealDb.Net/Internals/SurrealDbEngine.Ws.cs:line 1546
         at SurrealDb.Net.Internals.SurrealDbWsEngine.SendRequestAsync(String method, Object[] parameters, SurrealDbWsRequestPriority priority, CancellationToken cancellationToken) in /_/SurrealDb.Net/Internals/SurrealDbEngine.Ws.cs:line 1546
         at SurrealDb.Net.Internals.SurrealDbWsEngine.RawQuery(String query, IReadOnlyDictionary`2 parameters, CancellationToken cancellationToken) in /_/SurrealDb.Net/Internals/SurrealDbEngine.Ws.cs:line 798
         at CommunityToolkit.Aspire.SurrealDb.SurrealDbHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /home/runner/work/Aspire/Aspire/src/CommunityToolkit.Aspire.SurrealDb/SurrealDbHealthCheck.cs:line 29

This would suggest that _surrealdbClient.Health returns true, but then the query line, _surrealdbClient.RawQuery is timing out, which is highly confusing 🤣

[aaronpowell]

The stack trace says that the error is coming from here: https://github.com/surrealdb/surrealdb.net/blob/6b9057dd004b085276fe0c9019cd7d6950a30741/SurrealDb.Net/Internals/SurrealDbEngine.Ws.cs#L1535

Unfortunately, we don't have a stack trace on the OperationCancelledException that was the root cause of landing there, but ultimately, something in the stack doesn't like the raw query being run despite it "being healthy".