AI-Enabled QA Engineer

The AI-Enabled QA Engineer will test the tools supporting acquisitions, financial analysis, approval workflows, and marketing operations, ensuring they are accurate and production-ready.

This role will write automated checks, test edge cases, validate AI-generated outputs against source data, and help define what “production-ready” means for the organization. The AI-Enabled QA Engineer will work closely with the engineer, product manager, M&A and marketing leads, and the Head of IT.

Location

Fully Remote | Must overlap US Eastern business hours through 5:00 PM ET

Key Responsibilities

QA & Test Automation

• Build test automation using Playwright, Cypress, or Selenium for interfaces and pytest or equivalent for underlying systems.
• Design test cases from specifications using equivalence classes, boundaries, negative cases, and cases not covered by the original specification.
• Run tests through CI pipelines so checks execute automatically.
• Track and triage defects using Jira, Linear, or Azure DevOps.

• Write actionable bug reports that engineers can use without requiring additional meetings.

Data Validation & API Testing

• Use SQL to independently verify data through joins, aggregation, and record-level investigation.

• Validate data at scale through row counts, duplicate detection, orphaned records, referential integrity, and reconciliation of totals.
• Test APIs using Postman or code.

• Validate status codes, payload structures, authentication failures, and behavior on repeated identical requests.

AI Output & Document Testing

• Evaluate AI-generated output using a fixed set of examples.

• Measure precision and recall using historical cases.
• Run regression tests following prompt changes.
• Check AI-generated output for fabricated facts.
• Verify that fields extracted from financial statements can be traced back to their source documents.
• Test document extraction against scenarios such as scanned documents, rotated pages, and missing signatures.

• Verify target-analysis summaries against source financials so every number in an AI-generated summary is traceable to a document.

Workflow & Notification Testing

• Test the diligence data room against cases including missing documents, duplicate uploads, and rooms that appear complete but are not.

• Test notifications to ensure they trigger when a room stalls without generating unnecessary alerts.
• Test the approval step on the priority tracker to ensure nothing is submitted without review, including attempts to bypass the approval process.

• Write test cases for AI-generated outputs that every change must continue to pass.

Brand Asset Quality Assurance

• Check generated brand assets across eight to ten new brands.

• Verify that each generated asset contains the correct brand name, logo, and colours.

• Ensure assets do not contain branding belonging to another brand.

Production Readiness

• Develop the production-ready checklist with the Product Manager and Head of IT.

• Apply the production-ready checklist to the first tool.

Qualifications

Experience

• 3+ years of QA experience across both manual and automated testing.

• Experience testing data-heavy applications rather than only user interfaces.
• Experience with test automation in code.
• Experience evaluating AI-generated outputs methodically.
• Experience with data validation and API testing.
• Experience testing document-heavy or extraction-based systems is a plus.
• Experience checking creative or brand output for consistency is a plus.
• Healthcare data experience is a plus.
• Finance or accounting testing experience is a plus.
• Test automation experience within a Microsoft environment is a plus.

• Experience creating a definition of done that an organization adopted is a plus.

Skills

• Hands-on ability to use Playwright, Cypress, or Selenium for interface testing.

• Ability to use pytest or equivalent for automated testing.
• SQL skills for joins, aggregation, and independent data verification.
• Ability to test APIs using Postman or code.
• Strong test-case design skills covering equivalence classes, boundaries, negative cases, and overlooked scenarios.
• Ability to validate row counts, duplicates, orphaned records, referential integrity, and data reconciliation.
• Ability to evaluate AI output using golden sets, precision and recall, regression testing, and fabricated-fact checks.
• Ability to test document extraction and trace extracted information back to source documents.
• Ability to run automated tests through CI pipelines.
• Ability to track and triage defects using Jira, Linear, or Azure DevOps.
• Ability to write clear, actionable bug reports.
• Understanding that AI-generated outputs can fail confidently, plausibly, and inconsistently.
• Ability to document processes and testing standards.

• Performance testing, synthetic test-data generation, accessibility checks, and Power Platform testing are additional skills that are beneficial but not required.

What Success Looks Like

Within the first 90 days:

• The diligence data room is tested against realistic edge cases.

• Target-analysis summaries are verified against source financials.
• The notification system is tested for both required notifications and unnecessary alerts.
• The approval step on the priority tracker prevents unreviewed submissions.
• A fixed set of test cases is established for AI-generated output.
• Generated assets across eight to ten brands are checked for correct brand names, logos, and colours.

• A production-ready checklist is written with the Product Manager and Head of IT and applied to the first tool.

Opportunity

The AI-Enabled QA Engineer will work directly with the engineer, Product Manager, M&A and marketing leads, and Head of IT to establish testing standards for tools supporting acquisitions, financial analysis, approval workflows, and brand assets. The role combines manual and automated QA, with half of the work focused on building checks that can run automatically.

Application Process:

To be considered for this role these steps need to be followed:

• Fill in the application form

• Record a video showcasing your skill sets