Small scale evaluation.

[TomMonks]

Provide a simple table of time, tokens used, and success (Pass@10) rate for different models (and hardware for time).

[TomMonks]

For a series of user prompts - does the system run the correct scenario of the simulation model

Perform 10 runs.

Metrics (for model and hardware):

• Correct parameters (Pass @10)
• Correct simulation output (Pass @10)
• Time taken (mean)
• Self reflection loops (mean)
• Tokens user (mean) - to research how to access via Ollama.

Parameter generation data

The following prompt was given to Claude 3.7 Sonnet to generate 10 test cases

You are an assistant in an evaluation study of natural language interface to a computer simulation study.  You produce query and answer pairs that can be used in testing.

THE SCHEMA:

I have attached the simulation scheme that lists all of the parameters.

EXAMPLE USER INPUTS

"Simulate 14 operators, 12 nurses and 5% extra demand"
"Run scenario with double staffing and normal call volume"
"Test configuration with minimal staff"
"Double the usual demand"

OUTPUT FORMAT

{
  "example_1": {
      "user_input": "Simulate 14 operators, 12 nurses and 5% extra demand"
      "parameters":{
          "n_operators": 15,
          "n_nurses": 8,
          "mean_iat": 0.57,
          # insert other parameters
          "random_seed": 42,
          "run_length": 1000
      }
  }
}

TEST DATA GENERATION:

1. Analyse the simulation schema.  
2. Create 10 example user inputs.
3. Do not use ambigous language e.g. "high staffing" or "low demand" as these will produce unpredictable results and cannot be tested.
4. Create 10 expected parameters for the pairs
5. Use random_seed = 0 in all test cases
6. Output the input and parameters pairs in the example JSON format

Data:

Claude generated the following test cases:

{
  "example_1": {
    "user_input": "Simulate 14 operators, 12 nurses and 5% extra demand",
    "parameters": {
      "n_operators": 14,
      "n_nurses": 12,
      "mean_iat": 0.57,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_2": {
    "user_input": "Run scenario with double staffing and normal call volume",
    "parameters": {
      "n_operators": 26,
      "n_nurses": 20,
      "mean_iat": 0.6,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_3": {
    "user_input": "Test configuration with 8 operators, 6 nurses, and 20% increased demand",
    "parameters": {
      "n_operators": 8,
      "n_nurses": 6,
      "mean_iat": 0.48,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_4": {
    "user_input": "Simulate with 20 operators, 15 nurses, and half the usual call volume",
    "parameters": {
      "n_operators": 20,
      "n_nurses": 15,
      "mean_iat": 1.2,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_5": {
    "user_input": "Run with 10 operators, 8 nurses, 30% callback probability, and 10% reduced demand",
    "parameters": {
      "n_operators": 10,
      "n_nurses": 8,
      "mean_iat": 0.66,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.3,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_6": {
    "user_input": "Test scenario: 25 operators, 25 nurses, triple demand",
    "parameters": {
      "n_operators": 25,
      "n_nurses": 25,
      "mean_iat": 0.2,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_7": {
    "user_input": "Simulate 15 operators, 12 nurses, call durations 3-6-9 minutes, 2000 minute runtime",
    "parameters": {
      "n_operators": 15,
      "n_nurses": 12,
      "mean_iat": 0.6,
      "call_low": 3.0,
      "call_mode": 6.0,
      "call_high": 9.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 2000
    }
  },
  
  "example_8": {
    "user_input": "Run with 6 operators, 4 nurses, 60% callback rate, and 25% higher demand",
    "parameters": {
      "n_operators": 6,
      "n_nurses": 4,
      "mean_iat": 0.45,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.6,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_9": {
    "user_input": "Test configuration: 18 operators, 14 nurses, nurse consultations 15-25 minutes, normal demand",
    "parameters": {
      "n_operators": 18,
      "n_nurses": 14,
      "mean_iat": 0.6,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 15.0,
      "nurse_consult_high": 25.0,
      "random_seed": 0,
      "run_length": 1000
    }
  },
  
  "example_10": {
    "user_input": "Simulate 12 operators, 10 nurses, 50% callback probability, and 40% demand reduction",
    "parameters": {
      "n_operators": 12,
      "n_nurses": 10,
      "mean_iat": 0.84,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.5,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    }
  }
}

[TomMonks]

next steps:

• Use simulation model to create simulation results and add to the JSON as "expected_metrics"
• Setup as formal analysis notebook with pipeline to ensure that results are reproducible.
• Control ollama random number seed per generation. - to investigate.

[TomMonks]

Added an observability layer via Arize Phoenix. Only traces at the moment.

[TomMonks]

Added script to create evals with single replication output from parameters as expected value. create_evals.py

Sample

{
  "example_1": {
    "user_input": "Simulate 14 operators, 12 nurses and 5% extra demand",
    "parameters": {
      "n_operators": 14,
      "n_nurses": 12,
      "mean_iat": 0.57,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    },
    "expected_results": {
      "01_mean_waiting_time": 2.3504694345055652,
      "02_operator_util": 90.86219986414451,
      "03_mean_nurse_waiting_time": 4.1106430489118715,
      "04_nurse_util": 90.36357093206236,
      "05_callback_rate": 42.5531914893617
    }
  },
  "example_2": {
    "user_input": "Run scenario with double staffing and normal call volume",
    "parameters": {
      "n_operators": 26,
      "n_nurses": 20,
      "mean_iat": 0.6,
      "call_low": 5.0,
      "call_mode": 7.0,
      "call_high": 10.0,
      "callback_prob": 0.4,
      "nurse_consult_low": 10.0,
      "nurse_consult_high": 20.0,
      "random_seed": 0,
      "run_length": 1000
    },
    "expected_results": {
      "01_mean_waiting_time": 0.0,
      "02_operator_util": 46.452122409679156,
      "03_mean_nurse_waiting_time": 0.0041045663539504606,
      "04_nurse_util": 51.20346053052576,
      "05_callback_rate": 42.12121212121212
    }
  },
}