Senior Quality Engineer II (AI & Data)

Role Overview:

As a Senior Quality Engineer on the AI Team, you will be responsible for defining and executing the quality strategy for our AI-driven products. You will work at the intersection of data engineering and frontend application development, ensuring that our machine learning models are reliable, our data pipelines are accurate, and the final user experience is seamless.

Responsibilities:

• Data Pipeline Validation: Test the integrity of data ingestion and transformation processes to ensure models are trained on high-quality datasets.
• API & Model Testing: Conduct deep API testing using tools like APIDog to validate model responses, handling complex edge cases such as zero-value parameters or unauthorized access.
• Generative Model Output Accuracy: Define and implement evaluation frameworks to measure the quality of generative AI outputs — including relevance, factual accuracy, coherence, and hallucination rate — using techniques such as LLM-as-a-judge, human-in-the-loop scoring, and automated benchmark suites.
• AI Agent Response Validation: Design test strategies to assess the correctness and reliability of agentic AI responses, covering tool-call accuracy, multi-step reasoning integrity, goal completion rate, and graceful failure handling for out-of-scope or adversarial inputs.
• Security Testing: Proactively identify vulnerabilities like request interception and price manipulation to safeguard the system including prompt injection and jailbreak attempts specific to generative AI surfaces.
• UI Automation: Build and maintain scalable automation suites using low-code tools like mabl, aiming for high Mabl Test Coverage %.
• AI Tooling Advocacy: Champion the use of AI assistants like Jira Rovo for requirement analysis and test case generation to increase team velocity.
• Root Cause Analysis (RCA): Lead the RCA process for production bugs and developer-rejected tickets to continuously improve the "Definition of Done".