Your primary responsibilities will include:

• Driving research on the effectiveness and limitations of existing LLM evaluation techniques.
• Designing and developing novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness.
• Communicating, collaborating, and building relationships with clients and peer teams to facilitate cross-functional projects.
• Collaborating with internal teams and external partners to refine metrics and create standardized evaluation protocols.
• Implementing scalable and reproducible evaluation pipelines using modern ML frameworks.
• Publishing research findings in top-tier AI conferences and contributing to open-source benchmarking initiatives.
• Mentoring and guiding research scientists and engineers, providing technical leadership across cross-functional projects.
• Staying deeply engaged with the ML research community, tracking emerging work and contributing to the advancement of LLM evaluation science.

The ideal candidate will have 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development.

You will thrive in a high-energy, fast-paced startup environment and be ready to dedicate the time and effort needed to drive impactful results.

Compensation packages at Scale for eligible roles include base salary, equity, and benefits. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position, determined by work location and additional factors, including job-related skills, experience, interview performance, and relevant education or training.

XML job scraping automation by YubHub

We're seeking a Tech Lead Manager to lead a talented team of research scientists and research engineers focused on developing and implementing novel evaluation methodologies, metrics, and benchmarks to assess the capabilities and limitations of our cutting-edge LLMs.

Key responsibilities:

• Lead a team of highly effective research scientists and research engineers on LLM evals.
• Conduct research on the effectiveness and limitations of existing LLM evaluation techniques.
• Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness.
• Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects.
• Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols.
• Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
• Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives.

Ideal candidate has 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development. Experience supporting and leading a team of research scientists and research engineers is also required.

XML job scraping automation by YubHub