Skip to content

feat: add ollama and llama.cpp engines #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 19, 2025
Merged

feat: add ollama and llama.cpp engines #5

merged 1 commit into from
Feb 19, 2025

Conversation

11zhouxuan
Copy link
Collaborator

@11zhouxuan 11zhouxuan commented Feb 18, 2025 •
edited
Loading

  1. Add ollama and llama.cpp engines.
  2. Support DeepSeek R1 / V3 series models.
  3. Support TGI engine with Inf2 instance.
  4. Support customize model files path.
  5. Add supported models doc.

@11zhouxuan 11zhouxuan changed the title Dev feat: add ollama and llama.cpp engines Feb 18, 2025
@AoyuQC
Copy link
Contributor

AoyuQC commented Feb 18, 2025

@11zhouxuan please reset commits to one commit after @outstandingcandy finishes reviewing, thx

@11zhouxuan
add ollama engine
03bfce2 
support multiple model files prepare methods

modify local deploy

support llama.cpp engine

add generate_supported_models_doc_cli.py; add docs/supported_models.md

fix concurrency bug

modify tgi backend
@AoyuQC AoyuQC self-requested a review February 19, 2025 00:36
@AoyuQC AoyuQC removed the request for review from outstandingcandy February 19, 2025 00:41
@11zhouxuan 11zhouxuan merged commit 80e62ee into main Feb 19, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AoyuQC AoyuQC AoyuQC approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@11zhouxuan @AoyuQC @outstandingcandy