{"version":"0.1","company":{"name":"YubHub","url":"https://yubhub.co","jobsUrl":"https://yubhub.co/jobs/skill/technical-qa"},"x-facet":{"type":"skill","slug":"technical-qa","display":"Technical Qa","count":1},"x-feed-size-limit":100,"x-feed-sort":"enriched_at desc","x-feed-notice":"This feed contains at most 100 jobs (the most recently enriched). For the full corpus, use the paginated /stats/by-facet endpoint or /search.","x-generator":"yubhub-xml-generator","x-rights":"Free to redistribute with attribution: \"Data by YubHub (https://yubhub.co)\"","x-schema":"Each entry in `jobs` follows https://schema.org/JobPosting. YubHub-native raw fields carry `x-` prefix.","jobs":[{"@context":"https://schema.org","@type":"JobPosting","identifier":{"@type":"PropertyValue","name":"YubHub","value":"job_87f455b9-4bf"},"title":"QA Lead, AI Agent","description":"<p>Join us on this thrilling journey to revolutionize the workforce with AI. The future of work is here, and it&#39;s at Cresta.</p>\n<p>At Cresta, shipping AI is only half the story. Ensuring that AI interacts with humans reliably, accurately, and empathetically at scale is where the real challenge lies. As the QA Lead, AI Agent, you will be the ultimate guardian of the customer experience for our AI Agent product line. This role is perfect for a strategic quality expert who loves the intersection of human psychology and machine logic.</p>\n<p>You will own the end-to-end quality strategy, from designing complex test plans for non-deterministic LLMs to building automated and scalable testing environments using Cresta&#39;s proprietary no-code test and evaluation tools. You aren&#39;t just looking for bugs; you are building the framework that allows Cresta to deploy world-class AI agents for the world&#39;s largest enterprises with total confidence.</p>\n<p><strong>Responsibilities</strong></p>\n<ul>\n<li>Architect &amp; Scale AI Evaluation Systems: Design and oversee the end-to-end framework for testing AI agent systems at scale. You will leverage LLM-driven methodologies,including automated simulations, &quot;LLM-on-LLM&quot; rubrics, and adversarial red-teaming,to ensure reliability, policy adherence, and logic across complex, multi-turn conversational flows.</li>\n</ul>\n<ul>\n<li>Drive Deployment Excellence: Partner with Forward Deployed Engineers and PMs to triage issues, identify bottlenecks, and create new test cases on the fly to address real-world deployment challenges.</li>\n</ul>\n<ul>\n<li>Be the Customer’s Voice: Conduct manual UAT and voice-call testing to represent the end-customer experience. You take it personally when an agent lacks empathy or clarity, and you excel at articulating these nuances to the engineering team and clients.</li>\n</ul>\n<ul>\n<li>Lead and Scale the Team: lead a pod of QA analysts and partners. You will define the best practices, communication loops, and shared knowledge base that allow the QA function to scale alongside our rapidly growing product line.</li>\n</ul>\n<p><strong>Requirements</strong></p>\n<ul>\n<li>5+ years of experience in Quality Engineering, Deployments, or Technical QA, ideally within an AI or high-growth SaaS environment.</li>\n</ul>\n<ul>\n<li>Systems Thinking: A strong technical intuition and curiosity about how LLMs work. While you don&#39;t need to code, you must be comfortable navigating technical concepts like LLM, RAG, prompt logic, and multi-turn conversational flows.</li>\n</ul>\n<ul>\n<li>Operational Leadership: Proven ability to large E2E technical projects through partners, and a passion for building processes that improve efficiency between QA, Engineering, and Product.</li>\n</ul>\n<ul>\n<li>The &quot;QA Nose&quot;: An uncanny ability to find the edge case and a bias toward action. You anticipate bottlenecks before they happen and deliver solutions with urgency.</li>\n</ul>\n<ul>\n<li>High Empathy: A consultative mindset with the ability to represent the &quot;human element&quot; of a customer support interaction.</li>\n</ul>\n<ul>\n<li>Startup Agility: You thrive in fast-paced environments, excel at turning ambiguity into execution, and are comfortable &quot;rolling up your sleeves&quot; to build.</li>\n</ul>\n<p><strong>Bonus Points</strong></p>\n<ul>\n<li>Experience with CCaaS (Contact Center as a Service), telephony, or STT/TTS (Speech-to-Text) technologies.</li>\n</ul>\n<ul>\n<li>Background in Conversation Design or SDET roles.</li>\n</ul>\n<ul>\n<li>Experience leading team with direct reports.</li>\n</ul>\n<p style=\"margin-top:24px;font-size:13px;color:#666;\">XML job scraping automation by <a href=\"https://yubhub.co\">YubHub</a></p>","url":"https://yubhub.co/jobs/job_87f455b9-4bf","directApply":true,"hiringOrganization":{"@type":"Organization","name":"Cresta","sameAs":"https://www.cresta.ai/","logo":"https://logos.yubhub.co/cresta.ai.png"},"x-apply-url":"https://job-boards.greenhouse.io/cresta/jobs/5148813008","x-work-arrangement":"remote","x-experience-level":"senior","x-job-type":"full-time","x-salary-range":null,"x-skills-required":["Quality Engineering","Deployments","Technical QA","AI","SaaS","LLMs","RAG","Prompt Logic","Multi-Turn Conversational Flows"],"x-skills-preferred":["CCaaS","Telephony","STT/TTS","Conversation Design","SDET"],"datePosted":"2026-04-18T15:55:59.640Z","jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"United States (Remote)"}},"jobLocationType":"TELECOMMUTE","employmentType":"FULL_TIME","occupationalCategory":"Engineering","industry":"Technology","skills":"Quality Engineering, Deployments, Technical QA, AI, SaaS, LLMs, RAG, Prompt Logic, Multi-Turn Conversational Flows, CCaaS, Telephony, STT/TTS, Conversation Design, SDET"}]}