
Posted today
AI Evaluation & Quality Engineer
AI Summary
Designs evaluation strategies, metrics, datasets and regression tests for LLM agent workflows, runs red-teaming, and monitors production agent quality for product and engineering teams.
About this role
AI Evaluation & Quality Engineer
WeQuel is looking for an AI Evaluation & Quality Engineer for an assignment with one of our clients. You will make sure agentic workflows are measured honestly before and after they reach production – so the organisation knows what works, what does not, and why.
About the assignment
Workload: 100%
Location: Hybrid
Language: English
Your responsibilities
Design evaluation strategies and quality metrics for LLM agents (accuracy, task success, safety, cost, latency)
Build and maintain evaluation datasets, test cases and automated regression tests
Set up LLM-as-judge and human review processes where needed
Run red-teaming and robustness tests, including prompt injection scenarios
Monitor agent quality in production and feed findings back to development teams
Report results clearly to product owners and stakeholders
Requirements
At least 3 years of experience in test automation, QA engineering or data science
Hands-on experience evaluating LLM or ML solutions
Strong Python skills
Understanding of statistics and experiment design
Fluent English, written and spoken
Meritorious
Experience with evaluation and observability tools (e.g. Langfuse, LangSmith, Ragas, DeepEval, promptfoo)
Experience with CI/CD pipelines for ML or LLM applications
Knowledge of AI safety and the EU AI Act
Background in industrial or manufacturing environments
Swedish
Who you are
You are analytical, sceptical in a constructive way and thorough. You enjoy finding where things break, explain results in plain language, and help teams make better decisions based on evidence.
Applications are reviewed continuously – apply today!
Skills
Explore related jobs
More jobs at WeQuel
Similar Red-teaming jobs
- A
AI Observability EngineerAIG Employee Services, Inc. · GA-Atlanta - P
Principal Product Manager - AI Observability (Chronosphere)Palo Alto Networks, Inc. · Victoria, Australia - P
Manager, Software Engineering - AI Observability (Chronosphere)Palo Alto Networks, Inc. · Victoria, Australia - 9
Platform Engineer - AI Observability & Governance (OpenShift)94-1687665 Bank of America, National Association · Charlotte - A
Dynatrace Senior Consultant – AI ObservabilityArctiq · Brentwood, Tennessee - C
AI Observability Engineering Technical LeaderCisco Systems, Inc. · Milpitas, California
Jobs in Stockholm
- UVenture Capital Conference VolunteerUnited Media · Stockholm, Sweden
- UVC Event VolunteerUnited Media · Stockholm, Sweden
- UVC Conference VolunteerUnited Media · Stockholm, Sweden
- UPrivate Markets Event VolunteerUnited Media · Stockholm, Sweden
- UPrivate Markets Conference VolunteerUnited Media · Stockholm, Sweden
Browse these categories
Market data for this role
All reports →- SeriesRole reportsOne role family at a time: how many openings, what changed this week, who is hiring, what it pays.
- SeriesSalary reportsWhat employers publish in job postings, by level and workplace. Not self-reported pay.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.