Retrieve all user datasets #65
Conversation
| @@ -152,6 +152,57 @@ def pull(self, alias: str): | |||
| description=f"{progress.tasks[task_id].description} [rgb(25,227,160)]Done!)", | |||
| ) | |||
|
|
|||
| def pull_all(judgment_api_key: str) -> dict: | |||
There was a problem hiding this comment.
Make this a method of the EvalDataset class by adding the self parameter and removing the judgment_api_key parameter, this is already an EvalDataset attribute
There was a problem hiding this comment.
Oops I forgot to include part of my change, I actually had a @staticmethod on this because it didn't make a lot of sense to me for pull_all to be a instance method of EvalDataset because it is returning multiple 'EvalDataset'. Lmk ur thoughts on this though
There was a problem hiding this comment.
Quick clarification -- if we are retrieving multiple datasets, shouldn't this be a method under the JudgmentClient object?
There was a problem hiding this comment.
see new changes
src/judgeval/judgment_client.py
Outdated
| Returns: | ||
| EvalDataset: The retrieved dataset | ||
| """ | ||
| return EvalDataset.pull_all(self.judgment_api_key) |
There was a problem hiding this comment.
Do something similar here:
dataset = EvalDataset(judgment_api_key=self.judgment_api_key)
dataset.pull_all()
There was a problem hiding this comment.
Isn't the point that we're actually getting all user EvalDatasets? Why is this function returning a single EvalDataset?
There was a problem hiding this comment.
see new changes
alanzhang25
force-pushed
the
az-all-user-db-endpoint
branch
from
031caa9 to
160df00
Compare
src/e2etests/judgment_client_test.py
Outdated
| def test_pull_all_datasets(self, client: JudgmentClient): | ||
| dataset: EvalDataset = client.create_dataset() | ||
| # dataset.add_example(Example(input="input 1", actual_output="output 1")) | ||
| # client.push_dataset(alias="test_dataset_6", dataset=dataset, overwrite=False) |
There was a problem hiding this comment.
You need to create a unique dataset (using a random name for the alias), and then add a few examples and ground truths to it, then check the count (the count should match based on how many examples and ground truths you added).
The issue with the current test is that it relies on everyone who is running the e2etest to have the test_dataset_6 and test_dataset_7 in their account.
There was a problem hiding this comment.
I think that this file probably belongs in src/judgeval/data/datasets/ right?
There was a problem hiding this comment.
We can just reroute the imports
alanzhang25
force-pushed
the
az-all-user-db-endpoint
branch
from
1d3eb9b to
168180c
Compare
e2e test:
https://www.notion.so/18fe6c8347d680f9bb71c6361febab5f?v=18fe6c8347d6816e8a01000c1dbebebf&p=18fe6c8347d6804181a6e72d200869e6&pm=s