ALPHA10X
All open roles
AIDoha, Qatar (on-site)

Senior AI Evals Engineer

We are hiring a Senior AI Evals Engineer to make the quality of our AI systems measurable.

Apply

About this role

ALPHA10X is an award-winning, AI-native Decision Intelligence platform built for private markets - a market projected to exceed $65 trillion in AUM by 2032, but one in which investment decisions remain constrained by fragmented data, subjective judgment and limited predictive infrastructure. At its core is ALPHAcodex, a governed reasoning system that combines data, inference, institutional memory and continuous learning to generate quantitative signals and probabilistic foresight. The platform is designed to improve underwriting and capital allocation over time: each decision becomes institutional memory, strengthening the intelligence applied to subsequent decisions and creating the potential for a compounding advantage in judgment, conviction and investment performance.

Founded in France, ALPHA10X has progressively shifted its center of gravity to the Middle East as the region has become increasingly important to global capital allocation. The company is establishing its global headquarters and principal operating hub in Doha, providing a strategic base for product development, commercial expansion and global growth.

ALPHA10X has raised $25 million in seed capital from prominent investors and academics and completed more than 70 engagements across 11 countries. The company is now entering its next phase of scale, strengthening leadership, accelerating R&D, building its Doha organization and expanding institutional market penetration ahead of a planned Series A financing.

We are hiring a Senior AI Evals Engineer to make the quality of our AI systems measurable. Our agents, retrieval layer, and scoring models inform real investment decisions, so “it looks good” is not good enough. Your job is to tell a real improvement apart from a plausible-looking one, and to give the team the tools to ship model, prompt, and retrieval changes on evidence rather than intuition.

Reporting to the Head of Engineering, you will own evaluation across the AI platform: the datasets, metrics, judges, and pipelines that tell us whether our systems are getting better, and why they fail when they do.

This is an individual contributor role. You will not manage a team; you will shape how the team works through the evaluation standards you set, the evidence you bring to decisions, and the rigor you model in every review.

The successful candidate will be someone who takes initiative, stays curious and has an unrelenting drive to push the boundaries of AI and software engineering. They will be adaptive and introspective: willing to learn, guide, lead and follow, with the low ego of someone for whom the outcome matters more than who gets the credit.

Key Responsibilities

  • Evaluation Strategy & Frameworks: Define how we judge quality for each AI system: agents, search, extraction, and scoring. Build the evaluation framework and harness the whole team uses, so benchmarking our systems is as routine as running a test suite.
  • Datasets & Ground Truth: Design and maintain evaluation datasets that reflect real usage: golden sets, adversarial and edge cases, and regression suites. Work with investment and domain experts to define what a “good answer” means compared to a merely plausible one.
  • Release Gating & Online Evaluation: Integrate evals into our delivery workflow and release assessment. Run tests and online monitoring of production and catch quality drifts early.
  • Cross-functional Collaboration: Work closely with AI engineers, data scientists, Product, and Solutions. Make quality legible to non-specialists through clear reports and dashboards and raise the evaluation bar across the company through collaboration and example.

What We Are Looking For

This role requires a combination of evaluation expertise, statistical rigor, and strong engineering. We are looking for someone who:

  • Has a Master’s degree (minimum) in computer science or engineering.
  • Has 7+ years in ML engineering, applied science, or software engineering with AI, including hands-on ownership of evaluation for LLM or ML systems in production.
  • Has deep, practical knowledge of offline and online evaluation (dataset design, human annotation and inter-rater agreement, LLM-as-judge design and validation, and A/B testing) and knows the limits and failure modes of each.
  • Has a solid grounding in experimental design, significance testing, confidence intervals, and sampling, enough to say when a difference is real and when it is noise.
  • Has a practical understanding of tool-using agents and prompt engineering, and of where each tends to fail.
  • Is an expert in Python, writes clean, testable code, and is comfortable building eval pipelines that run reliably at scale.
  • Has an ownership mindset: takes a problem end to end, including the unglamorous parts.
  • Communicates clearly and concisely and is comfortable working with researchers and engineers alike.
  • Brings pragmatic problem-solving and a bias for action in a fast-moving environment.
  • Is curious about business domains and quickly develops an instinct for what good looks like.
  • Mentors through example, making the people around them better without needing authority to do it.

Our Tech Stack

What you will work with day to day:

  • Python
  • Postgres
  • Elasticsearch
  • Azure Cloud

Apply by sending your CV to hr@alpha10x.com

Apply
Other open roles
All open roles