Principal AI Automation Engineer

Work Flexibility: Hybrid or Onsite

Principal Engineer – AI Quality & Evaluation Architecture
Role Overview
Vocera (Stryker) is seeking a Principal Engineer to own end-to-end AI quality across the lifecycle — data, models, prompts, evaluation, deployment, and monitoring. This role will define and scale reliable, measurable, production-grade AI systems across speech, NLP, and GenAI in healthcare.
Key Responsibilities
AI Quality Ownership

• Own AI quality across the full lifecycle

  
  
  

• Define SLAs, KPIs, release gates, and production readiness decisions

  
  
  

Evaluation & Reliability

• Build evaluation frameworks for ASR (WER, latency), NLP (intent/entity), and LLMs/RAG (hallucination, safety, groundedness)

  
  
  

• Develop benchmarking, regression pipelines, and golden datasets

  
  
  

• Drive adversarial testing, edge case handling, and failure analysis

  
  
  

AI Testing Platform

• Architect scalable evaluation platforms (offline, regression, A/B, shadow testing)

  
  
  

• Integrate with CI/CD and MLOps pipelines

  
  
  

• Implement monitoring, observability, and drift detection

  
  
  

Data Governance

• Define standards for data curation, annotation, and versioning

  
  
  

• Ensure reproducibility and feedback loops from production

  
  
  

• Maintain healthcare data compliance

  
  
  

MLOps & Continuous Quality

• Establish AI MLOps standards for evaluation, retraining, and deployment

  
  
  

• Enable continuous evaluation and performance monitoring at scale

  
  
  

Leadership

• Act as AI quality authority across the organization

  
  
  

• Mentor teams and align with product and business goals

  
  
  

Qualifications

• 12+ years in software/AI engineering; 5+ years in LLMs, NLP, RAG, or speech

  
  
  

• Experience building scalable AI evaluation frameworks

  
  
  

Expertise in:

• LLM evaluation (hallucination, safety, groundedness)

  
  
  

• Golden datasets, regression testing, adversarial testing

  
  
  

• Prompt validation, Python, data analysis, automation

  
  
  

• CI/CD, MLOps, distributed systems

  
  
  

Nice to Have

• RAG evaluation & retrieval benchmarking

  
  
  

• Speech/ASR evaluation

  
  
  

• Azure ML / OpenAI / AI Search

  
  
  

• Responsible AI & compliance