Merge pull request #35 from JudgmentLabs/add_dev_docs

Add developer docs

Author: SecroLoL

.gitignore

@@ -17,6 +17,8 @@ wheels/
 .installed.cfg
 *.egg-info/
 
+# APIs
+google-cloud-sdk/
 # PyInstaller
 #  Usually these files are written by a python script from a template
 #  before PyInstaller builds the exe, so as to inject date/other infos into it.

docs

@@ -0,0 +1 @@
+Subproject commit 00ab9e46926eff3a327b4646a0bd71a1f2ed0650

docs/create_dataset.ipynb renamed to judgeval_docs/create_dataset.ipynb

@@ -0,0 +1,127 @@
+# Datasets 
+
+## Quick Summary
+In most scenarios, you will have multiple `Example`s that you want to evaluate together.  
+In `judgeval`, an evaluation dataset (`EvalDataset`) is a collection of `Example`s and/or `GroundTruthExample`s that you can scale evaluations across.
+
+`Tip`:
+
+A `GroundTruthExample` is a specific type of `Example` that do not require the `actual_output` field. This is useful for creating datasets that can be dynamically updated at evaluation time by running your workflow on the `GroundTruthExample`s to create `Example`s.
+
+## Creating a Dataset
+
+Creating an `EvalDataset` is as simple as supplying a list of `Example`s and/or `GroundTruthExample`s.
+
+```
+from judgeval import (
+    EvalDataset, 
+    Example, 
+    GroundTruthExample
+)
+
+examples = [Example(input="...", actual_output="..."), Example(input="...", actual_output="..."), ...]
+ground_truth_examples = [GroundTruthExample(input="..."), GroundTruthExample(input="..."), ...]
+
+dataset = EvalDataset(examples=examples, ground_truth_examples=ground_truth_examples)
+```
+
+You can also add `Example`s and `GroundTruthExample`s to an existing `EvalDataset` using the `add_example` and `add_ground_truth_example` methods.
+
+```
+...
+
+dataset.add_example(Example(...))
+dataset.add_ground_truth(GroundTruthExample(...))
+```
+
+## Saving/Loading Datasets
+
+`judgeval` supports saving and loading datasets in the following formats:
+- JSON
+- CSV
+
+### From Judgment
+You easily can save/load an `EvalDataset` from Judgment's cloud. 
+
+```
+# Saving
+...
+from judgeval import JudgmentClient
+
+client = JudgmentClient()
+client.push_dataset(alias="my_dataset", dataset=dataset)
+```
+
+```
+# Loading
+from judgeval import JudgmentClient
+
+client = JudgmentClient()
+dataset = client.pull_dataset(alias="my_dataset")
+```
+
+### From JSON
+
+You can save/load an `EvalDataset` with a JSON file. Your JSON file should have the following structure:
+```
+{
+    "examples": [{"input": "...", "actual_output": "..."}, ...],
+    "ground_truths": [{"input": "..."}, ...]
+}
+```
+
+Here's an example of how use `judgeval` to save/load from JSON.
+
+```
+from judgeval import EvalDataset
+
+# saving
+dataset = EvalDataset(...)  # filled with examples
+dataset.save_as("json", "/path/to/save/dir", "save_name")
+
+# loading
+new_dataset = EvalDataset()
+new_dataset.add_from_json("/path/to/your/json/file.json")
+
+```
+
+### From CSV
+
+You can save/load an `EvalDataset` with a `.csv` file. Your CSV should contain rows that can be mapped to `Example`s via column names.
+TODO: this section needs to be updated because the CSV format is not yet finalized.
+
+
+Here's an example of how use `judgeval` to save/load from CSV.
+
+```
+from judgeval import EvalDataset
+
+# saving
+dataset = EvalDataset(...)  # filled with examples
+dataset.save_as("csv", "/path/to/save/dir", "save_name")
+
+# loading
+new_dataset = EvalDataset()
+new_dataset.add_from_csv("/path/to/your/csv/file.csv")
+```
+
+## Evaluate On Your Dataset
+
+You can use the `JudgmentClient` to evaluate the `Example`s and `GroundTruthExample`s in your dataset using scorers.
+
+```
+...
+
+dataset = client.pull_dataset(alias="my_dataset")
+res = client.evaluate_dataset(
+    dataset=dataset,
+    scorers=[JudgmentScorer(threshold=0.5, score_type=APIScorer.FAITHFULNESS)],
+    model="gpt-4o",
+)
+```
+
+## Conclusion 
+
+Congratulations! You've now learned how to create, save, and evaluate an `EvalDataset` in `judgeval`.
+
+You can also view and manage your datasets via Judgment's platform. Check out TODO: add link here