Researcher and Engineer – LLM Evaluation

Job Description

Fall Analyst Intern | Caltech Student Investment Fund

The Tone:
This is a full-time position located in San Francisco, California, with a start date targeted for January 2026. Vals AI is a pioneering organization committed to establishing the definitive standard for evaluating Large Language Models (LLMs) on real-world, practical tasks. The company’s core offering involves developing sophisticated benchmarks designed to help diverse organizations accurately measure the true capabilities of cutting-edge foundation models, specifically focusing on their performance in economically valuable applications. This role for a researcher or engineer is integral to Vals AI’s mission, directly contributing to the advancement of LLM evaluation methodologies and ensuring their practical utility and impact across global financial institutions and major enterprise sectors. Vals AI’s significant contributions have already garnered recognition in prominent publications such as the Wall Street Journal, Washington Post, and Bloomberg.

The TL;DR
• Role: Researcher / Engineer
• Type: Full-time
• Location: In-person, San Francisco, CA

• Mission: To build and provide the authoritative standard for evaluating Large Language Models (LLMs) by developing benchmarks that precisely measure their capabilities in economically valuable, real-world tasks for various organizations.

What You’ll Actually Do
• Evaluate LLMs: Actively engage in the process of defining, developing, and refining the industry standard for evaluating the performance and utility of Large Language Models across a spectrum of real-world applications and scenarios.
• Design Benchmarks: Architect and implement comprehensive, high-fidelity benchmarks that serve to objectively quantify what advanced foundation models are truly capable of accomplishing within economically valuable tasks.
• Collaborate with Model Labs: Work closely and strategically with all major foundation model laboratories, ensuring that evaluation methodologies are cutting-edge, relevant, and effectively measure model capabilities.
• Engage with Industry Leaders: Interact and build relationships with some of the largest financial institutions and leading enterprises globally to understand their specific needs and challenges in measuring LLM performance, tailoring benchmarks accordingly.
• Drive Measurement Standards: Contribute significantly to the establishment and widespread adoption of new standards for LLM performance measurement, thereby enabling organizations to make informed decisions about foundation model integration and deployment.

The Must-Haves
• Background: Not specified in the job description.
• Experience: Not specified in the job description.
• Skills: Not specified in the job description.