Director, AI Automation Engineering

Important: Testing Harness and Infrastructure

• Own the infrastructure that continuously verifies whether autonomous agents are taking the actions we intend, before and after they reach production.
• Stage, test, and lifecycle-manage bots and agent workflows before they reach production.
• Establish baseline quality metrics and coverage standards for AI initiatives.
• Build the infrastructure to run automated research chains and agent evals at scale.

Evaluation Frameworks

• Create automated evaluation frameworks for LLM outputs: accuracy, safety, bias, and regression.
• Implement RAG evaluation, prompt regression testing, and golden-set benchmarking.
• Define and enforce quality thresholds that AI Engineers and builders must meet before ship.
• Partner with AI Researchers and Engineers on eval design for new model and agent patterns.

CI/CD and Release Quality

• Build the release-gating infrastructure that governs when an agent is trusted to act in production, including staged rollouts and rollback authority.
• Develop AI automation solutions to ensure that agentic coding delivers highly functioning software while maintaining quality across the product portfolio
• Integrate testing into GitHub Actions (or equivalent) for AI model and agent releases.
• Maintain existing test suites and keep them current as models, prompts, and tools change.

Cross-Team Quality Partnership

• Work with AI Solutions Architects on eval criteria for workflow automations they build in the field.
• Partner with Principal AI Product on launch-readiness and QA for AI projects across the org.
• Support product teams (Ascend, DocUpdate, Platform) on AI feature validation without owning their roadmaps.
• Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing.

Qualifications & Experience

• 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety
• Direct experience evaluating LLM outputs: RAG eval, prompt regression, safety and accuracy benchmarking.
• Experience building CI/CD pipelines with integrated test gates (GitHub Actions or similar).
• Comfortable working across AI platform infrastructure and product-facing AI features.
• Operates independently as a senior IC and communicates quality standards to engineers and non-engineers.
• Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus.
• Familiarity with LLM observability and eval tooling (LangSmith, Braintrust, custom harnesses, etc.) is a strong plus.
• Background in healthcare, pharma, or regulated environments where quality gates matter is a strong plus.
• Prior experience standing up a QA or automation function from scratch, with appetite to grow into leading a team.
• Proficiency in Python and standard testing frameworks (pytest or equivalent).

Compensation and Benefits

The base salary range for this role is $185,000 - $210,000

Where you land within the range will reflect your skills, experience, and location, while keeping team parity in mind and leaving room for future growth.

Important Notice

To protect candidates, please note that all communication from our team will come directly from an @impiricus.com email address. We do not engage third party recruiters to schedule interviews for this role. If you receive outreach from anyone outside of our domain claiming to represent Impiricus, please disregard and let us know by forwarding the message to careers@impiricus.com

Benefits

Impiricus focuses on taking care of our teammates’ professional and personal growth and well-being.

 Healthcare: Medical, dental, and vision coverage for you and your dependents + on-demand healthcare concierge
 HSA, FSA & DCFSA: Pre-tax savings options for healthcare and dependent care, with monthly employer contributions to HSA (if enrolled in a high-deductible plan)
 Coverage & Protection: 100% paid short- and long-term disability, plus life and AD&D insurance
 Flexible Time Off: Take the time you need with a flexible vacation policy — recharge your batteries your way
Parental Leave: Paid parental leave to spend time with your newborn, adopted, or foster child (available after 6 months)
 Your Work, Your Way: If you’re close to an office, we encourage spending some time in-person to collaborate and connect. If not, remote is always an option.
 401(k): Save for your future with tax advantages (and company match!)

Impiricus provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.