Senior Test Automation Engineer (ML, LLM, GenAI)

  • Australia
  • Melbourne
  • Permanent
  • Attractive package + bonuses

We are seeking an experienced Senior Test Automation Engineer (ML, LLM, GenAI) to join our Technology Delivery team supporting a strategic AI-powered system.

This role plays a critical part in ensuring the quality, reliability, and performance of modern Machine Learning, Large Language Model (LLM), and Generative AI solutions. You will design and implement scalable automation frameworks across the full technology stack – from React-based user interfaces through to backend services, AI models, search platforms, and data orchestration pipelines.

You will lead testing innovation for AI-driven applications, validating contextual accuracy, factual correctness, and ongoing performance of LLM models while enabling continuous delivery of platform enhancements.

Key Responsibilities

  • Design and implement enterprise-grade test automation frameworks across UI, API, backend, and AI services

  • Develop automated testing strategies for ML, LLM, and Generative AI applications

  • Validate AI outputs for accuracy, relevance, and response quality across multiple models

  • Integrate automation into CI/CD pipelines to support continuous delivery

  • Build end-to-end regression and performance test suites

  • Monitor automation metrics and provide actionable quality insights

  • Lead test planning, coverage strategy, and risk-based testing activities

  • Mentor junior engineers and promote automation best practices

  • Collaborate closely with developers, product teams, and stakeholders

  • Drive continuous improvement of testing frameworks and DevOps practices

Technical Environment

  • Languages & Tools: Python, pytest, FastAPI

  • Automation: Playwright, Selenium, Appium, REST Assured

  • AI & Data: Azure OpenAI, AWS Bedrock, Databricks, Elasticsearch, Vector Search

  • Frontend & Backend: React / Next.js, APIs, distributed services

  • Cloud & DevOps: AWS, Azure, Git, CI/CD pipelines

Skills & Experience

  • Strong experience designing scalable test automation frameworks

  • Hands-on testing of AI, NLP, or LLM-based applications

  • Advanced Python automation development

  • Experience testing complex distributed systems, APIs, and data platforms

  • Deep understanding of Agile delivery and DevOps practices

  • Experience with cloud-based testing environments

  • Ability to produce meaningful quality metrics and reporting

What You’ll Bring

  • Technical leadership in automation and quality engineering

  • Strong analytical and problem-solving capability

  • Passion for emerging AI technologies and innovation

  • Ability to work autonomously while contributing to high-performing teams

Desirable Certifications

  • ISTQB Advanced Test Automation Engineering (CTAL-TAE)

  • AI/ML or Data Science certifications

  • Python or SQL certifications

Why Join Us?

This is an opportunity to work at the forefront of AI and Generative AI technology, shaping testing practices for next-generation intelligent platforms while delivering meaningful enterprise impact.

For more information or a confidential discussion about this role, please contact Ronald at ronald.tran@talentinternational.com

Apply now

Submit your details and attach your resume below. Hint: make sure all relevant experience is included in your CV and keep your message to the hiring team short and sweet - 2000 characters or less is perfect.