Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.
Job Overview:
As a QA Engineer in a AI Ops Business Pod, you will play a critical role in ensuring the quality, stability, and correctness of GenAI-powered agents and workflows. You will design and execute test plans, automate regression checks, and validate agent behavior under varied inputs and edge cases. This role requires an analytical mindset, understanding of AI workflows, and a keen eye for risk and detail in automated decisions.
Key Responsibilities
- Design and implement test strategies and plans for GenAI agents and integrated workflows
- Create and execute functional, integration, and regression test cases for agents built on Agent OS and ServiceNow
- Validate LLM-driven outputs, including prompt results, decision paths, and workflow accuracy
- Collaborate closely with AI Engineers, Pod Lead, and Domain SMEs to understand business logic and critical flows
- Build and maintain automated test scripts (where applicable) and simulate edge cases
- Participate in design and code reviews to provide early feedback on quality, completeness, and testability
- Own defect triage, ensure traceability between requirements and test cases, and manage bug tracking
- Validate agent readiness for deployment and provide go/no-go recommendations
- Document test results, quality metrics, and lessons learned for continuous improvement
Key skills:
- 5+ years of experience in QA/test engineering, preferably in enterprise or automation-heavy environments
- Experience testing AI/GenAI-powered applications, LLM workflows, or agentic systems is highly desirable
- Strong familiarity with API testing, test case design, and working in Agile POD-based teams
- Exposure to platforms like ServiceNow, LangChain, Agent OS (or similar workflow automation tools) a plus
- Working knowledge of test automation tools (e.g., Pytest, Postman, Selenium, or similar frameworks)
- Ability to think critically about test scenarios, edge cases, and quality risks
- Excellent documentation and communication skills
- Comfortable collaborating across engineering, product, and business teams
What You’ll Gain
- Opportunity to work on cutting-edge GenAI solutions in a large-scale enterprise automation program
- Exposure to proprietary AI agent platforms, enterprise workflows, and real-world AI governance
- A collaborative, fast-paced environment where quality is core to delivering business value
Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.
Candidate Privacy Policy
Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:
- What information we collect during our application and recruitment process and why we collect it;
- How we handle that information; and
- How to access and update that information.
Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.
