The Anti-Abuse team is the front line defending Replit's platform from exploitation. We detect and shut down phishing deployments, prevent cryptomining on free-tier infrastructure, stop LLM token farming, and keep bad actors from weaponizing the platform against our users.
\nThis is adversarial work: attackers adapt constantly, and we build the detection systems, heuristics, and automated responses that stay ahead of them.
\nWhat makes this role unique is the AI-native nature of Replit's platform. You'll work on problems that barely exist elsewhere: building guardrails for AI-generated code, detecting prompt injection attacks at scale, and using LLMs as a defensive tool against abuse.
\nIf you want hands-on experience applying AI to security problems, this is one of the few places you can do it in production with real attackers. You'll own problems end-to-end, from identifying emerging abuse patterns to shipping the systems that stop them at scale.
\nResponsibilities
\nRequirements
\nNice to Have
\nTools + Tech Stack for this role
\nXML job scraping automation by YubHub
","url":"https://yubhub.co/jobs/job_f2e93c37-5a0","directApply":true,"hiringOrganization":{"@type":"Organization","name":"Replit","sameAs":"https://replit.com/","logo":"https://logos.yubhub.co/replit.com.png"},"x-apply-url":"https://jobs.ashbyhq.com/replit/75e69146-a092-43a1-b1d6-023d433d3ae7","x-work-arrangement":"hybrid","x-experience-level":"staff","x-job-type":"Full time","x-salary-range":"$190K - $240K","x-skills-required":["security engineering","anti-abuse","trust & safety","fraud detection","Python","TypeScript","SQL","BigQuery","Hex","ML/LLM-based classifiers","prompt injection","jailbreaking","common attack patterns","phishing infrastructure","account takeover","credential stuffing","resource abuse"],"x-skills-preferred":["payment abuse","financial crime","device fingerprinting","IP reputation","email validation services","CI/CD security tooling","container security","Linux internals","cloud infrastructure","abuse reporting pipelines","trust & safety tooling","content moderation systems"],"datePosted":"2026-04-24T13:13:20.268Z","jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Foster City, CA (Hybrid) In office M,W,F"}},"employmentType":"FULL_TIME","occupationalCategory":"Engineering","industry":"Technology","skills":"security engineering, anti-abuse, trust & safety, fraud detection, Python, TypeScript, SQL, BigQuery, Hex, ML/LLM-based classifiers, prompt injection, jailbreaking, common attack patterns, phishing infrastructure, account takeover, credential stuffing, resource abuse, payment abuse, financial crime, device fingerprinting, IP reputation, email validation services, CI/CD security tooling, container security, Linux internals, cloud infrastructure, abuse reporting pipelines, trust & safety tooling, content moderation systems","baseSalary":{"@type":"MonetaryAmount","currency":"USD","value":{"@type":"QuantitativeValue","minValue":190000,"maxValue":240000,"unitText":"YEAR"}}},{"@context":"https://schema.org","@type":"JobPosting","identifier":{"@type":"PropertyValue","name":"YubHub","value":"job_fc4a0972-622"},"title":"Principal Product Manager, AI Model Security","description":"Microsoft Superintelligence team’s mission is to empower every person and every organization on the planet to achieve more.
\nAs employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
\nThis role is part of Microsoft AI’s Superintelligence Team. The MAIST is a startup-like team inside Microsoft AI, created to push the boundaries of AI toward Humanist Superintelligence , ultra-capable systems that remain controllable, safety-aligned, and anchored to human values.
\nOur mission is to create AI that amplifies human potential while ensuring humanity remains firmly in control. We aim to deliver breakthroughs that benefit society , advancing science, education, and global well-being.
\nWe are hiring a Product Manager to own AI model security , the discipline of making our frontier models resilient against adversarial attack and purpose-built for security practitioners.
\nThis role has a dual mandate: (1) harden our models against the full spectrum of LLM security threats , prompt injection, data exfiltration, jailbreaking, training data extraction, zero-day exploit generation, model poisoning, and agentic workflow exploitation , and (2) partner closely with Microsoft Security product teams (Azure Security, Security Copilot) to ensure our models deliver best-in-class capabilities for real-world security workflows.
\nResponsibilities:
\nOwn the model security roadmap: Define and prioritize the security hardening strategy for our frontier models across the full OWASP LLM threat surface , prompt injection (direct and indirect), data exfiltration, jailbreak resistance, system prompt leakage, training data extraction, and adversarial manipulation of agentic workflows.
\nDrive zero-day and exploit defense: Work with researchers to evaluate and mitigate the risk of models being used to generate zero-day exploits, malware, or novel attack vectors.
\nBuild and scale red-teaming frameworks: Design, run, and iterate adversarial testing programs , both automated and human-driven , to continuously probe model vulnerabilities.
\nEstablish metrics (e.g., jailbreak success rate, injection bypass rate, exfiltration resistance) and drive measurable improvement over time.
\nPartner with Microsoft Security product teams: Work closely with Azure Security and Security Copilot teams to translate their product requirements into model training priorities.
\nEnsure our models are purpose-built for threat detection, incident triage, vulnerability assessment, log analysis, and compliance reasoning.
\nDefine security-specific model evaluations: Build benchmark suites and evaluation frameworks that measure real-world security usefulness , not just academic performance.
\nDrive training data strategy to improve domain-specific model quality for security practitioners.
\nShape security policy and launch readiness: Establish clear security criteria for model launches.
\nOwn the security dimension of go/no-go decisions, with frameworks that balance capability, risk, and deployment context.
\nStay at the frontier: Track the rapidly evolving LLM security landscape , new attack techniques, emerging standards (OWASP, NIST AI RMF), regulatory requirements (EU AI Act), and academic research.
\nTranslate what you learn into actionable product priorities.
\nInfluence model training and architecture: Partner with researchers and engineers to embed security considerations into model training, fine-tuning, RLHF, and post-training safeguards.
\nQualifications:
\nBachelor’s Degree AND 5+ years experience in product management, security engineering, or software development OR equivalent experience
\nDemonstrated hands-on experience with AI/ML systems , you have personally built, evaluated, or shipped ML-powered products or security tools
\nDeep familiarity with LLM security threats: prompt injection, jailbreaking, data exfiltration, adversarial attacks on generative models , through professional experience, red-teaming, or security research
\nExperience defining product requirements and driving decisions in partnership with researchers or ML engineers
\nTrack record of building evaluation systems, security benchmarks, or adversarial testing frameworks , not just consuming them
\nAbility to operate autonomously, make decisions with incomplete information, and drive projects from ambiguity to shipped outcomes
\nPreferred Qualifications:
\nTechnical background in computer science, security, or AI/ML , a postgraduate degree is a plus but not required
\nExperience in offensive security, penetration testing, or red teaming , ideally applied to AI/ML systems
\nFamiliarity with security workflows and tooling (SIEM, SOAR, EDR, threat intelligence platforms) and how practitioners use them in production
\nUnderstanding of the model lifecycle (pre-training, fine-tuning, RLHF, deployment, monitoring) and where security interventions are most effective
\nExperience working with or within enterprise security organizations (e.g., Microsoft Security, CrowdStrike, Palo Alto Networks, or similar)
\nPublished research, blog posts, or public contributions in AI security, adversarial ML, or LLM red teaming
\nXML job scraping automation by YubHub
","url":"https://yubhub.co/jobs/job_fc4a0972-622","directApply":true,"hiringOrganization":{"@type":"Organization","name":"Microsoft AI","sameAs":"https://microsoft.ai","logo":"https://logos.yubhub.co/microsoft.ai.png"},"x-apply-url":"https://microsoft.ai/job/principal-product-manager-ai-model-security/","x-work-arrangement":null,"x-experience-level":null,"x-job-type":"full-time","x-salary-range":null,"x-skills-required":["AI/ML systems","LLM security threats","prompt injection","jailbreaking","data exfiltration","adversarial attacks on generative models","product requirements","security engineering","software development","evaluation systems","security benchmarks","adversarial testing frameworks","autonomous decision-making","project management","offensive security","penetration testing","red teaming","security workflows","tooling","SIEM","SOAR","EDR","threat intelligence platforms","model lifecycle","pre-training","fine-tuning","RLHF","deployment","monitoring"],"x-skills-preferred":[],"datePosted":"2026-04-24T12:15:26.485Z","jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Redmond"}},"employmentType":"FULL_TIME","occupationalCategory":"Engineering","industry":"Technology","skills":"AI/ML systems, LLM security threats, prompt injection, jailbreaking, data exfiltration, adversarial attacks on generative models, product requirements, security engineering, software development, evaluation systems, security benchmarks, adversarial testing frameworks, autonomous decision-making, project management, offensive security, penetration testing, red teaming, security workflows, tooling, SIEM, SOAR, EDR, threat intelligence platforms, model lifecycle, pre-training, fine-tuning, RLHF, deployment, monitoring"},{"@context":"https://schema.org","@type":"JobPosting","identifier":{"@type":"PropertyValue","name":"YubHub","value":"job_25c868eb-c32"},"title":"Principal Product Manager, AI Model Security","description":"Microsoft Superintelligence team’s mission is to empower every person and every organization on the planet to achieve more.
\nAs employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
\nThis role is part of Microsoft AI’s Superintelligence Team. The MAIST is a startup-like team inside Microsoft AI, created to push the boundaries of AI toward Humanist Superintelligence , ultra-capable systems that remain controllable, safety-aligned, and anchored to human values.
\nOur mission is to create AI that amplifies human potential while ensuring humanity remains firmly in control. We aim to deliver breakthroughs that benefit society , advancing science, education, and global well-being.
\nWe are hiring a Product Manager to own AI model security , the discipline of making our frontier models resilient against adversarial attack and purpose-built for security practitioners.
\nThis role has a dual mandate: (1) harden our models against the full spectrum of LLM security threats , prompt injection, data exfiltration, jailbreaking, training data extraction, zero-day exploit generation, model poisoning, and agentic workflow exploitation , and (2) partner closely with Microsoft Security product teams (Azure Security, Security Copilot) to ensure our models deliver best-in-class capabilities for real-world security workflows.
\nResponsibilities:
\nOwn the model security roadmap: Define and prioritize the security hardening strategy for our frontier models across the full OWASP LLM threat surface , prompt injection (direct and indirect), data exfiltration, jailbreak resistance, system prompt leakage, training data extraction, and adversarial manipulation of agentic workflows.
\nDrive zero-day and exploit defense: Work with researchers to evaluate and mitigate the risk of models being used to generate zero-day exploits, malware, or novel attack vectors.
\nBuild and scale red-teaming frameworks: Design, run, and iterate adversarial testing programs , both automated and human-driven , to continuously probe model vulnerabilities.
\nEstablish metrics (e.g., jailbreak success rate, injection bypass rate, exfiltration resistance) and drive measurable improvement over time.
\nPartner with Microsoft Security product teams: Work closely with Azure Security and Security Copilot teams to translate their product requirements into model training priorities.
\nEnsure our models are purpose-built for threat detection, incident triage, vulnerability assessment, log analysis, and compliance reasoning.
\nDefine security-specific model evaluations: Build benchmark suites and evaluation frameworks that measure real-world security usefulness , not just academic performance.
\nDrive training data strategy to improve domain-specific model quality for security practitioners.
\nShape security policy and launch readiness: Establish clear security criteria for model launches.
\nOwn the security dimension of go/no-go decisions, with frameworks that balance capability, risk, and deployment context.
\nStay at the frontier: Track the rapidly evolving LLM security landscape , new attack techniques, emerging standards (OWASP, NIST AI RMF), regulatory requirements (EU AI Act), and academic research.
\nTranslate what you learn into actionable product priorities.
\nInfluence model training and architecture: Partner with researchers and engineers to embed security considerations into model training, fine-tuning, RLHF, and post-training safeguards.
\nQualifications:
\nBachelor’s Degree AND 5+ years experience in product management, security engineering, or software development OR equivalent experience
\nDemonstrated hands-on experience with AI/ML systems , you have personally built, evaluated, or shipped ML-powered products or security tools
\nDeep familiarity with LLM security threats: prompt injection, jailbreaking, data exfiltration, adversarial attacks on generative models , through professional experience, red-teaming, or security research
\nExperience defining product requirements and driving decisions in partnership with researchers or ML engineers
\nTrack record of building evaluation systems, security benchmarks, or adversarial testing frameworks , not just consuming them
\nAbility to operate autonomously, make decisions with incomplete information, and drive projects from ambiguity to shipped outcomes
\nXML job scraping automation by YubHub
","url":"https://yubhub.co/jobs/job_25c868eb-c32","directApply":true,"hiringOrganization":{"@type":"Organization","name":"Microsoft","sameAs":"https://microsoft.ai","logo":"https://logos.yubhub.co/microsoft.ai.png"},"x-apply-url":"https://microsoft.ai/job/principal-product-manager-ai-model-security-2/","x-work-arrangement":"onsite","x-experience-level":"senior","x-job-type":"full-time","x-salary-range":null,"x-skills-required":["AI/ML systems","LLM security threats","prompt injection","jailbreaking","data exfiltration","adversarial attacks on generative models","product management","security engineering","software development","model training","fine-tuning","RLHF","post-training safeguards"],"x-skills-preferred":[],"datePosted":"2026-04-24T12:12:01.831Z","jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Mountain View"}},"employmentType":"FULL_TIME","occupationalCategory":"Engineering","industry":"Technology","skills":"AI/ML systems, LLM security threats, prompt injection, jailbreaking, data exfiltration, adversarial attacks on generative models, product management, security engineering, software development, model training, fine-tuning, RLHF, post-training safeguards"},{"@context":"https://schema.org","@type":"JobPosting","identifier":{"@type":"PropertyValue","name":"YubHub","value":"job_138b24e2-2bd"},"title":"Senior Software Engineer, Anti-Abuse & Security","description":"Rewrite this job ad in your own words, matching the tone of voice of the original. Reuse the same section headings from the original ad (e.g. if the ad says "Responsibilities", use that heading, not "What you'll do").
\nStart with an opening paragraph (no heading): what the role is, who the company is, why it matters. If the ad mentions salary, include it here.
\nRephrase bullet points in your own words while keeping the factual content. Combine related points where it makes sense.
\nFor benefits/perks: gather them from anywhere in the ad into one section. If the ad mentions nothing about benefits, omit a benefits section entirely.
\nDo not invent information that is not in the original ad.
\nAbout the role The Anti-Abuse team is the front line defending Replit's platform from exploitation. We detect and shut down phishing deployments, prevent cryptomining on free-tier infrastructure, stop LLM token farming, and keep bad actors from weaponizing the platform against our users. This is adversarial work: attackers adapt constantly, and we build the detection systems, heuristics, and automated responses that stay ahead of them.
\nWhat makes this role unique is the AI-native nature of Replit's platform. You'll work on problems that barely exist elsewhere: building guardrails for AI-generated code, detecting prompt injection attacks at scale, and using LLMs as a defensive tool against abuse. If you want hands-on experience applying AI to security problems, this is one of the few places you can do it in production with real attackers. You'll own problems end-to-end, from identifying emerging abuse patterns to shipping the systems that stop them at scale.
\nIn this role you will…
\nRequired skills and experience:
\nNice to have:
\nTools + Tech Stack for this role
\nThis role may _not_ be a fit if
\n_This is a full-time role that can be held from our Foster City, CA office. The role has an in-office requirement of Monday, Wednesday, and Friday._
\nFull-Time Employee Benefits Include: 💰 Competitive Salary & Equity 💹 401(k) Program with a 4% match ⚕️ Health, Dental, Vision and Life Insurance 🩼 Short Term and Long Term Disability 🚼 Paid Parental, Medical, Caregiver Leave 🚗 Commuter Benefits 📱 Monthly Wellness Stipend 🧑💻 Autonomous Work Environment 🖥 In Office Set-Up Reimbursement 🏝 Flexible Time Off (FTO) + Holidays 🚀 Quarterly Team Gatherings ☕ In Office Amenities
\nXML job scraping automation by YubHub
","url":"https://yubhub.co/jobs/job_138b24e2-2bd","directApply":true,"hiringOrganization":{"@type":"Organization","name":"Replit","sameAs":"https://jobs.ashbyhq.com","logo":"https://logos.yubhub.co/replit.com.png"},"x-apply-url":"https://jobs.ashbyhq.com/replit/5bdadf61-7955-46e8-8fdf-bd69818358b7","x-work-arrangement":"hybrid","x-experience-level":"senior","x-job-type":"full-time","x-salary-range":"$190K – $240K","x-skills-required":["security engineering","anti-abuse","trust & safety","fraud detection","Python","TypeScript","SQL","BigQuery","Hex","ML/LLM-based classifiers","prompt injection","jailbreaking","common attack patterns","phishing infrastructure","account takeover","credential stuffing","resource abuse"],"x-skills-preferred":["experience at a platform company","fraud detection","payment abuse","financial crime","device fingerprinting","IP reputation","email validation services","CI/CD security tooling","container security","Linux internals","cloud infrastructure"],"datePosted":"2026-03-07T15:19:04.069Z","jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Foster City, CA"}},"employmentType":"FULL_TIME","occupationalCategory":"Engineering","industry":"Technology","skills":"security engineering, anti-abuse, trust & safety, fraud detection, Python, TypeScript, SQL, BigQuery, Hex, ML/LLM-based classifiers, prompt injection, jailbreaking, common attack patterns, phishing infrastructure, account takeover, credential stuffing, resource abuse, experience at a platform company, fraud detection, payment abuse, financial crime, device fingerprinting, IP reputation, email validation services, CI/CD security tooling, container security, Linux internals, cloud infrastructure","baseSalary":{"@type":"MonetaryAmount","currency":"USD","value":{"@type":"QuantitativeValue","minValue":190000,"maxValue":240000,"unitText":"YEAR"}}}]}