FirestoreVectorStore cannot handle batches (?) #2039
Unanswered
Gustavo-KnowMe
asked this question in
Q&A
Replies: 1 comment
-
One more detail for what is worth: if I manually batch the documents and process them 25 at a time, it works just fine. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I'm exploring using llamaindex with Google Firestore.
I've run this example successfully: https://github.com/run-llama/LlamaIndexTS/tree/main/examples/storage/firestore
I decided to try and load
/sf_budget_2023_2024.pdf
from the example data instead (closer to my real application).This file is larger, so you need to configure batching, otherwise you get
Error: 3 INVALID_ARGUMENT: datastore transaction or write too big.
(Firestore will not store more than 25 txn per batch).If batching is configured (i.e. 25) you get
Error: Cannot modify a WriteBatch that has been committed. at WriteBatch.verifyNotCommitted
Now I've found that every time you configure batching, you get the error mentioned above. Looks like the implementation does not close the WriteBatch once is sent? or I'm doing something wrong...?
Happy to provide the code but is basically the standard example applied to different sample data, both available in the repos.
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions