-
Notifications
You must be signed in to change notification settings - Fork 110
Ollama #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Ollama #26
Changes from 19 commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
4000be1
Initial commit
QuantumNightmare 45791ad
initial project creation
QuantumNightmare 27ec070
Ollama resource with model downloader
QuantumNightmare d689033
Packagerize
QuantumNightmare 65eb487
A couple of fixes
QuantumNightmare 20df3bb
documentation and minor tweaks
QuantumNightmare 0ab743f
release as version 1 and added change log
QuantumNightmare 28bde7a
Well that's embarrassing - I hope nobody saw that
QuantumNightmare 80c22dc
Merge pull request #1 from MindscapeHQ/initial-development
QuantumNightmare 94bf83c
Making the `ModelName` property public for read-only access.
aaronpowell 1922dd4
Merge pull request #4 from aaronpowell/aaronpowell/issue-3
QuantumNightmare 4fb1e48
Version bump
QuantumNightmare 21b32b6
Changelog entry
QuantumNightmare 7891337
Merge pull request #5 from MindscapeHQ/v1.1.0
QuantumNightmare ad3be06
Merge remote-tracking branch 'raygun/main' into ollama
aaronpowell 01b6aba
Merge branch 'main' into ollama
aaronpowell fa36e44
Adding readme to csproj
aaronpowell 860910e
docs based off the raygun package readme
aaronpowell 61c7d64
Adding ollama across all docs
aaronpowell 3a580b3
Starting tests for Ollama integration
aaronpowell 3f3d452
Adding ollama demo app
aaronpowell e09a9d7
Fixing by using the annotation rather than allocated endpoint
aaronpowell 791b40b
Using a set of constants for the container image details
aaronpowell 4209602
Code style tweak
aaronpowell cf5d37e
Removing the PublishAsContainer since we exclude from manifest anyway
aaronpowell 1903ea4
Adding initial integration test
aaronpowell 5ec43cf
Adding a WaitFor and timeout in tests
aaronpowell 23ed05a
Making volume optional
aaronpowell a5028b3
Wait for resource before resolving it?
aaronpowell 19f442e
Adding some retry logic to the lifecycle hook
aaronpowell 20307d5
Have to set the name of the endpoint, otherwise it will use 'http', w…
aaronpowell a57d253
Merge branch 'main' into ollama
aaronpowell 7380e01
Update docs/integrations/hosting-ollama.md
aaronpowell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # CommunityToolkit.Aspire.Hosting.Ollama | ||
|
|
||
| [](https://nuget.org/packages/CommunityToolkit.Aspire.Ollama/) | [>)](https://nuget.org/packages/CommunityToolkit.Aspire.Ollama/absoluteLatest) | ||
|
|
||
| ## Overview | ||
|
|
||
| An Aspire component leveraging the [Ollama](https://ollama.com) container with support for downloading a model on startup. | ||
|
|
||
| ## Usage | ||
|
|
||
| Use the static `AddOllama` method to add this container component to the application builder. | ||
|
|
||
| ```csharp | ||
| // The distributed application builder is created here | ||
|
|
||
| var ollama = builder.AddOllama(); | ||
|
|
||
| // The builder is used to build and run the app somewhere down here | ||
| ``` | ||
|
|
||
| ### Configuration | ||
|
|
||
| The AddOllama method has optional arguments to set the `name`, `port` and `modelName`. | ||
| The `name` is what gets displayed in the Aspire orchestration app against this component. | ||
| The `port` is provided randomly by Aspire. If for whatever reason you need a fixed port, you can set that here. | ||
| The `modelName` specifies what LLM to pull when it starts up. The default is `llama3`. You can also set this to null to prevent any models being pulled on startup - leaving you with a plain Ollama container to work with. | ||
|
|
||
| ## Downloading the LLM | ||
|
|
||
| When the Ollama container for this component first spins up, this component will download the LLM (llama3 unless otherwise specified). | ||
| The progress of this download will be displayed in the State column for this component on the Aspire orchestration app. | ||
| Important: Keep the Aspire orchestration app open until the download is complete, otherwise the download will be cancelled. | ||
| In the spirit of productivity, we recommend kicking off this process before heading for lunch. | ||
| This component binds a volume called "ollama" so that once the model is fully downloaded, it'll be available for subsequent runs. | ||
|
|
||
| ## Accessing the Ollama server from other Aspire components | ||
|
|
||
| You can pass the ollama component to other Aspire components in the usual way: | ||
|
|
||
| ```csharp | ||
| builder.AddMyComponent().WithReference(ollama); | ||
| ``` | ||
|
|
||
| Within that component (e.g. a web app), you can fetch the Ollama connection string from the application builder as follows. | ||
| Note that if you changed the name of the Ollama component via the `name` argument, then you'll need to use that here when specifying which connection string to get. | ||
|
|
||
| ```csharp | ||
| var connectionString = builder.Configuration.GetConnectionString("Ollama"); | ||
| ``` | ||
|
|
||
| You can then call any of the Ollama endpoints through this connection string. We recommend using the [OllamaSharp](https://www.nuget.org/packages/OllamaSharp) client to do this. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
src/CommunityToolkit.Aspire.Hosting.Ollama/CommunityToolkit.Aspire.Hosting.Ollama.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <PackageReadmeFile>hosting-ollama.md</PackageReadmeFile> | ||
| <Description>An Aspire component leveraging the Ollama container with support for downloading a model on startup.</Description> | ||
| <AdditionalPackageTags>hosting ollama ai</AdditionalPackageTags> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="OllamaSharp"/> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
31 changes: 31 additions & 0 deletions
31
src/CommunityToolkit.Aspire.Hosting.Ollama/OllamaResource.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| namespace Aspire.Hosting.ApplicationModel; | ||
|
|
||
| /// <summary> | ||
| /// A resource that represents an Ollama container. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// Constructs an <see cref="OllamaResource"/>. | ||
| /// </remarks> | ||
| /// <param name="name">The name for the resource.</param> | ||
| /// <param name="modelName">The LLM to download on initial startup.</param> | ||
| public class OllamaResource(string name, string modelName) : ContainerResource(name), IResourceWithConnectionString | ||
| { | ||
| internal const string OllamaEndpointName = "ollama"; | ||
|
|
||
| private EndpointReference? _endpointReference; | ||
|
|
||
| public string ModelName { get; internal set; } = modelName; | ||
|
|
||
| /// <summary> | ||
| /// Gets the endpoint for the Ollama server. | ||
| /// </summary> | ||
| public EndpointReference Endpoint => _endpointReference ??= new(this, OllamaEndpointName); | ||
|
|
||
| /// <summary> | ||
| /// Gets the connection string expression for the Ollama server. | ||
| /// </summary> | ||
| public ReferenceExpression ConnectionStringExpression => | ||
| ReferenceExpression.Create( | ||
| $"http://{Endpoint.Property(EndpointProperty.Host)}:{Endpoint.Property(EndpointProperty.Port)}" | ||
| ); | ||
| } |
32 changes: 32 additions & 0 deletions
32
src/CommunityToolkit.Aspire.Hosting.Ollama/OllamaResourceBuilderExtensions.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| using Aspire.Hosting.ApplicationModel; | ||
| using Aspire.Hosting.Lifecycle; | ||
| using CommunityToolkit.Aspire.Hosting.Ollama; | ||
|
|
||
| namespace Aspire.Hosting; | ||
|
|
||
| /// <summary> | ||
| /// Provides extension methods for adding an Ollama container to the application model. | ||
| /// </summary> | ||
| public static class OllamaResourceBuilderExtensions | ||
| { | ||
| /// <summary> | ||
| /// Adds the Ollama container to the application model. | ||
| /// </summary> | ||
| /// <param name="builder">The <see cref="IDistributedApplicationBuilder"/>.</param> | ||
| /// <param name="name">The name of the resource. This name will be used as the connection string name when referenced in a dependency.</param> | ||
| /// <param name="port">An optional fixed port to bind to the Ollama container. This will be provided randomly by Aspire if not set.</param> | ||
| /// <param name="modelName">The name of the LLM to download on initial startup. llama3 by default. This can be set to null to not download any models.</param> | ||
| /// <returns>A reference to the <see cref="IResourceBuilder{T}"/>.</returns> | ||
| public static IResourceBuilder<OllamaResource> AddOllama(this IDistributedApplicationBuilder builder, | ||
| string name = "Ollama", int? port = null, string modelName = "llama3") | ||
| { | ||
| builder.Services.TryAddLifecycleHook<OllamaResourceLifecycleHook>(); | ||
| var raygun = new OllamaResource(name, modelName); | ||
| return builder.AddResource(raygun) | ||
| .WithAnnotation(new ContainerImageAnnotation { Image = "ollama/ollama", Tag = "latest" }) | ||
aaronpowell marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| .WithHttpEndpoint(port, 11434, OllamaResource.OllamaEndpointName) | ||
aaronpowell marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| .WithVolume("ollama", "/root/.ollama") | ||
aaronpowell marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| .ExcludeFromManifest() | ||
| .PublishAsContainer(); | ||
aaronpowell marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| } | ||
99 changes: 99 additions & 0 deletions
99
src/CommunityToolkit.Aspire.Hosting.Ollama/OllamaResourceLifecycleHook.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| using Aspire.Hosting.ApplicationModel; | ||
| using Aspire.Hosting.Lifecycle; | ||
| using OllamaSharp; | ||
|
|
||
| namespace CommunityToolkit.Aspire.Hosting.Ollama; | ||
| internal class OllamaResourceLifecycleHook(ResourceNotificationService notificationService) : IDistributedApplicationLifecycleHook, IAsyncDisposable | ||
| { | ||
| private readonly ResourceNotificationService _notificationService = notificationService; | ||
|
|
||
| private readonly CancellationTokenSource _tokenSource = new(); | ||
|
|
||
| public Task AfterResourcesCreatedAsync(DistributedApplicationModel appModel, CancellationToken cancellationToken = default) | ||
| { | ||
| foreach (var resource in appModel.Resources.OfType<OllamaResource>()) | ||
| { | ||
| DownloadModel(resource, _tokenSource.Token); | ||
| } | ||
|
|
||
| return Task.CompletedTask; | ||
| } | ||
|
|
||
| private void DownloadModel(OllamaResource resource, CancellationToken cancellationToken) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(resource.ModelName)) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| _ = Task.Run(async () => | ||
| { | ||
| try | ||
| { | ||
| var connectionString = await resource.ConnectionStringExpression.GetValueAsync(cancellationToken); | ||
|
|
||
| if (string.IsNullOrWhiteSpace(connectionString)) | ||
| { | ||
| await _notificationService.PublishUpdateAsync(resource, state => state with { State = new ResourceStateSnapshot("No connection string", KnownResourceStateStyles.Error) }); | ||
| return; | ||
| } | ||
|
|
||
| var ollamaClient = new OllamaApiClient(new Uri(connectionString)); | ||
| var model = resource.ModelName; | ||
|
|
||
| await _notificationService.PublishUpdateAsync(resource, state => state with { State = new ResourceStateSnapshot("Checking model", KnownResourceStateStyles.Info) }); | ||
| var hasModel = await HasModelAsync(ollamaClient, model, cancellationToken); | ||
|
|
||
| if (!hasModel) | ||
| { | ||
| await PullModel(resource, ollamaClient, model, cancellationToken); | ||
| } | ||
|
|
||
| await _notificationService.PublishUpdateAsync(resource, state => state with { State = new ResourceStateSnapshot("Running", KnownResourceStateStyles.Success) }); | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| await _notificationService.PublishUpdateAsync(resource, state => state with { State = new ResourceStateSnapshot(ex.Message, KnownResourceStateStyles.Error) }); | ||
| } | ||
|
|
||
| }, cancellationToken); | ||
| } | ||
|
|
||
| private static async Task<bool> HasModelAsync(OllamaApiClient ollamaClient, string model, CancellationToken cancellationToken) | ||
| { | ||
| var localModels = await ollamaClient.ListLocalModels(cancellationToken); | ||
| return localModels.Any(m => m.Name.StartsWith(model)); | ||
| } | ||
|
|
||
| private async Task PullModel(OllamaResource resource, OllamaApiClient ollamaClient, string model, CancellationToken cancellationToken) | ||
| { | ||
| await _notificationService.PublishUpdateAsync(resource, state => state with { State = new ResourceStateSnapshot("Downloading model", KnownResourceStateStyles.Info) }); | ||
|
|
||
| long percentage = 0; | ||
|
|
||
| await ollamaClient.PullModel(model, async status => | ||
| { | ||
| if (status.Total != 0) | ||
| { | ||
| var newPercentage = (long)(status.Completed / (double)status.Total * 100); | ||
| if (newPercentage != percentage) | ||
| { | ||
| percentage = newPercentage; | ||
|
|
||
| var percentageState = percentage == 0 ? "Downloading model" : $"Downloading model {percentage} percent"; | ||
| await _notificationService.PublishUpdateAsync(resource, | ||
| state => state with | ||
| { | ||
| State = new ResourceStateSnapshot(percentageState, KnownResourceStateStyles.Info) | ||
| }); | ||
| } | ||
| } | ||
| }, cancellationToken); | ||
| } | ||
|
|
||
| public ValueTask DisposeAsync() | ||
| { | ||
| _tokenSource.Cancel(); | ||
| return default; | ||
| } | ||
| } |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.