Software Engineer Intern · San Jose, CA
Built an AI-driven system that benchmarks how LLM agents perform, tailored to the mainframe environment.
- Built a distributed evaluation harness in Python running 30 concurrent multi-turn agent trajectories against IBM's ZDBA for DB2 at up to 6,000 questions/hour, cutting manual review from days to two hours.
- Designed a model-based grader on watsonx combining rubric-based scoring and pairwise comparison, with a reason-before-score contract and temperature-0 decoding for deterministic, reproducible scores.
- Extended the grader to assess tool-call efficiency against SME-defined expected calls, revealing only 82% of trajectories used tools efficiently versus a 95% estimate, and fed flagged runs into early-stopping logic.
- Validated against a 12,000-question, 15-agent dataset, establishing ground-truth benchmarks at 90% agent accuracy.









