Usage of GTE-ModernColBERT or ModernBERT embedding model for storing file chunks and vector embeddings in a vector store

[ampulsora]

Summary:
We are looking for a code example that show how to use ModernColBERT model to vectorize the chunks incrementally read from a file and how to to store them in a vector database, e.g. FAISS.

Details:
We have 2 requirements and we are looking for a solution to meet them. I

1. We are want to use lightonai/GTE-ModernColBERT-v1 embedding model with vector stores, e.g. FAISS. We are using LangChain to integrate with FAISS
2. We want to incrementally read chunks of a file (or files), create embeddings(vectorize) for each chunk using lightonai/GTE-ModernColBERT-v1 embedding model and write each chunk along with its vector into FAISS.
   We have not been able to figure out whether this can be done.

The typical process for vectorizing chunks and storing them in FAISS, using the LangChain wrapper classes, looks like this:

from uuid import uuid4
import faiss
from langchain_community.vectorstores import FAISS
from langchain_community.docstore.in_memory import InMemoryDocstore
from langchain_openai import OpenAIEmbeddings

index = faiss.IndexFlatL2(len(OpenAIEmbeddings().embed_query("hello world")))

vector_store = FAISS(
    embedding_function = OpenAIEmbeddings(),
    index = index,
    docstore = InMemoryDocstore(),
    index_to_docstore_id = {}
)

loop to read a file in chunks:
    chunk = read the next chunk from the file
    chunk_id = str(uuid4())
    vector_store.add_documents(documents = [chunk], ids = [chunk_id])

We have tried many approaches including the following:

tokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL_GTE_MODERN_COLBERT)
embedding_model = AutoModel.from_pretrained(EMBEDDING_MODEL_GTE_MODERN_COLBERT) # This returns ModenBertModel class

# Need a method to vectorize a sample string and get the length of the vector. The following code does not work bedacuse encode method does not exist We don't want to hardcode the vector dimensions to 128.
vector_dimension =  len(embedding_model.encode("hello world))

index = faiss.IndexFlatL2(vector_dimension)

# Need an embedding function to be passed to the FAISS constructor for ModernBert model. The following code does not work because  embedding_mode (ModernBertModel class) cannot be assigned to embedding_function paramter
vector_store = FAISS(
    embedding_function = embedding_model,
    index = index,
    docstore = InMemoryDocstore(),
    index_to_docstore_id = {}
)

Questions:
In general, we are looking for a code example that show how to use ModernColBERT model to vectorize the chunks incrementally read from a file and how to to store them in a vector database, e.g. FAISS.

1. We are looking for a way to create a vector/embeddings for a sample string to get the vector dimension without hardcoding it to 128.
2. We are looking for an embedding_function that can be passed as a paramter to the FAISS constructor (the same requirement for Chroma vector DB, too). The embedding function will be used by the vector DB to convert strings/documents into vectors (create embeddings) when the strings/documents are added to the vector DB.

Any help is much appreciated.

[ampulsora]

We also saw a warning message from the FAISS(VectorStore) class constructor that indicates an Embedding object needs to be passed in for the embedding_function parameter. Please let us know how to get this Embeddings instance for ModernColBERT model or whether it not possible to use ModernColBERT/ModernBERT model with FAISS and Chroma vector DBs that have these requirements.

https://api.python.langchain.com/en/latest/_modules/langchain_community/vectorstores/faiss.html

embedding_function: Union[
Callable[[str], List[float]],
Embeddings]

    if not isinstance(embedding_function, Embeddings):
        logger.warning(
            "`embedding_function` is expected to be an Embeddings object, support "
            "for passing in a function will soon be removed."
        )

[onestardao]

you are very close, but i think you're hitting a silent pitfall here.

the reason your faiss vectorstore is not behaving as expected is likely due to a mismatch between the embedding model output shape and faiss's dimensional assumption.

most moderncolbert or gte-style models return sentence-wise embeddings with hidden dims around 768 or 1024. if your faiss index was built with a fixed d, and your embedding function returns a mismatched shape — faiss will accept the insert, but silently fail at similarity search. no warning. just nonsense retrieval.

also, your usage of embedding_function as a lambda might be backfiring. faiss requires that this function returns consistent float32 numpy vectors. any async wrapping or mixed tensor type will cause subtle misbehavior.

if you want, i have mapped these issues before in a problem list i maintain. let me know if you want the link. good luck, this one is tricky to debug.

[ampulsora]

Thank you very much @onestardao for the explanation.

We would love to look at the problem list that you are maintaining.

[onestardao]

yeah this one lands in a weird limbo ~ looks like a classic case of silent mismatch. most of these colbert/gte variants spit out sentence-wise embeddings, often around dim=768 or 1024, but faiss defaults expect flat dense shapes with strict float32. if your lambda wrapper adds async/tensorflow/numpy weirdness, the faiss vectorstore might accept it, but what actually gets indexed is off ~ you’ll get valid similarity scores for garbage intent.

we’ve logged this before under problem no.3 or no.10 depending on setup. full diagnostics + patch stack here if useful:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

just tossing the rope ~ take it if you’re stuck.