Skip to content

Add json correctness scorer #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 49 commits into from
Jan 21, 2025
Merged

Add json correctness scorer #37

merged 49 commits into from
Jan 21, 2025

Conversation

SecroLoL
Copy link
Contributor

Adds JSON correctness scorer

  • validates whether the LLM output adheres to a given JSON schema

Adds a custom model_dump() to EvaluationRun such that kwargs can now be passed to the init of different scorers on the backend. In the case of JSON correctness, we need to pass the schema.

  • this is extendable to new scorers in the future that require something more than just threshold.

Also, removed some telemetry stuff that is unused

JCamyre and others added 30 commits January 7, 2025 19:32
@JCamyre
Small changes.
f51cbf3
@JCamyre
Add a span_type field to traces, to specify between LLM calls, evalua…
1bd3197 
…tions, tools, etc. Tweak condense() logic to properly support structure.
@JCamyre
Pass span_type's into @judgment.observe()'s.
cf2adbb
@JCamyre
Fix span_types not being passed in for all observe() cases.
1d3f9ff
@JCamyre
Fix depth count issues with spans.
dcff0f9
@JCamyre
Add span_type to the TraceEntry 'to dictionary' function so that span…
ccc9171 
…_type shows up in the final trace.
@JCamyre
Remove debugging print statements.
fa9bec7
@JCamyre
Update prompt_scorer notebook docs to proper python version.
c57a0c7
@JCamyre
Add e2e test for editing, updating, and pushing a classifier scorer.
e3d272a
@JCamyre
Private functions for e2etests/test_prompt_scoring.py.
7fe7337
@JCamyre
Fix unit tests which was accessing old private method names.
0bb2f91
@JCamyre
Privatize methods and use new method name.
5ac335d
@JCamyre
Update unit test and unit test mock object to use private method names.
7d19cd1
@JCamyre
More privatization.
c48d072
@JCamyre
Add update functions for ClassifierScorer.
1c33164
@JCamyre
Add Judgment Client method to push classifier scorers from SDK side.
941ecbb
@JCamyre
Add sleep to make llm_call function more realistic. Pass in project n…
9a9f28e 
…ame to trace.
@JCamyre
Add project name field to traces.
ea895ef
@SecroLoL
Add new tests for JSONCorrectnessScorer
11bd9d8
@SecroLoL
remove telemetry
e26645a
@SecroLoL
Add custom model_dump() to EvaluationRun so that kwargs can be associ…
ae38e51 
…ated with scorers
@SecroLoL
Minor syntax change
97f50a3
@SecroLoL
add JSONCorrectnessScorer tests
a2d7d21
@JCamyre
Remove judgment client test changes.
78a5857
@JCamyre
Add automatic eval run name generation. Don't allow empty Trace name.…
22f8f17 
… Add type hinting for Tracer fields.
@SecroLoL
Merge branch 'main' into add_json_correctness_scorer
431067b
@SecroLoL
Remove telemetry script
bbd67b7
@SecroLoL
remove refs to telemetry
ef4934e
@andrewli2403
Merge pull request #39 from JudgmentLabs/remove_telemetry
a2c541c 
Remove telemetry
@JCamyre
Change trace and project name. Specify overwrite kwaarg.
fe07188
JCamyre and others added 19 commits January 16, 2025 13:09
@JCamyre
Add and pass arguments for logic relating to saving and overwriting t…
d756a4f 
…races.
@JCamyre
Add error handling from save trace API call.
8d214d0
@JCamyre
Remove logic related to actual_eval_run_name. Add logic for receiving…
5be3b0d 
… the results after logged (important as it contains example_id).
@JCamyre
Add comments for pull_eval. Properly handle receiving updated fetch e…
f526528 
…val API endpoint.
@JCamyre
Add new fields to ScoringResult, needed for linking between trace and…
cc66f54 
… example page.
@JCamyre
Merge branch 'joseph/improve-trace-pages' into joseph/span-level-evals
4aa109c
@JCamyre
Merge branch 'joseph/simplify-classifier-scorers' into joseph/span-le…
c9043ff 
…vel-evals
@JCamyre
Merge branch 'joseph/eval-run-name-uniqueness' into joseph/span-level…
c3f3cca 
…-evals
@JCamyre
Add demo folder. Add Patronus tracing workflow for comparison in demos.
5e00fa0
@JCamyre
Add Patronus library, needed for Patronus demo.
684b8ce
@JCamyre
Make tracer test evals make more contextual sense.
8829ab9
@JCamyre
Remove print statement.
74b62a0
@JCamyre
Merge branch 'main' into joseph/span-level-evals
951858a
@JCamyre
Merge branch 'main' of https://github.com/JudgmentLabs/judgeval into …
1d20648 
…joseph/span-level-evals
@JCamyre
Fix failing UT's.
b01c104
@JCamyre
Fix test_condense_trace UT.
22ebf64
@SecroLoL
Merge pull request #34 from JudgmentLabs/joseph/span-level-evals
4587969 
Additional features for span-level (multi-step) evaluations
@SecroLoL
Fix merge conflicts
5091c40
@SecroLoL
Fix merge conflicts
a7f4368
@SecroLoL SecroLoL merged commit 05b3ddb into main Jan 21, 2025
@SecroLoL SecroLoL deleted the add_json_correctness_scorer branch February 10, 2025 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SecroLoL @JCamyre @andrewli2403