Software Development Engineer in Test - Agentic Automation

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test - Agentic Automation based in Canada.

This is an opportunity for an experienced SDET to help shape how AI agents are tested, evaluated, and trusted in production. You will join a platform engineering team working on a large-scale, cloud-based cybersecurity environment, building the automation that supports reliability, performance, and quality. Alongside traditional functional and integration testing, you will contribute to agentic test generation, AI-powered workflows, and eval-driven development. Your work will help ensure that AI systems behave accurately, safely, and consistently before reaching production. You will collaborate with engineers across the development lifecycle in a remote-first environment that values autonomy, experimentation, and continuous improvement.

• Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.

• Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.

• Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.

• Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.

• Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.

• Identify and mitigate unreliable AI outputs, including hallucinated assertions, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.

• Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.

• Test integrations across platform systems, services, data, applications, and customer-facing features.

• Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.

• Share technical knowledge and contribute to team-wide quality and evaluation standards.

Requirements:

• Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.

• At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.

• Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.

• Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.

• Experience testing web services and APIs, including REST, gRPC, and JSON.

• Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.

• A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.

• Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.

• Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.

Nice to have:

• End-to-end eval-driven development experience, including release gating, trajectory and tool-use evaluations, and calibrating LLM-as-judge systems against human review.

• Experience with prompt engineering and designing agent context or tools.

• Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.

• Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.

• Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.

• Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.

Benefits:

• Annual base salary ranging from CA$120,000 to CA$170,000, plus variable or incentive compensation, equity, and benefits.

• Market-leading compensation and equity award opportunities.

• Comprehensive physical and mental wellness programs.

• Competitive vacation allowance and holidays.

• Paid parental and adoption leave.

• Professional development opportunities available to employees at every level.

• Employee networks, local community groups, and volunteer opportunities.

• Remote work opportunities across eligible Canadian location