Instructure
Research & Development
SeniorAppliedDataScientist,RetrievalandSemanticSystems
Neural analysis suggests this role is
optimal for Senior candidates.
“Senior Applied Data Scientist, Retrieval and Semantic Systems at Instructure. Skills: Retrieval systems, Semantic systems, AI systems. Design production retrieval systems. Build production retrieval systems”
What You'll Achieve.
Prove relevance is good; Prove relevance stays good; Block bad changes from shipping
Industry & Context.
Root cause analysis; Troubleshooting
Background check, Identity verification
What They're Looking For.
Must Have
6+ years experience ML/AI systems, Owned retrieval system in production, Python skills, Experience building services and APIs, Grounding in embeddings, Grounding in ANN search, Grounding in retrieval and ranking systems, Experience designing indexing strategies, Experience designing refresh strategies, Experience with data quality controls, Experience with safe backfills, Define and run retrieval evaluation, Build gold sets, Choose relevance metrics, Analyze failures by slice, Prevent regressions, Tradeoff judgment
Nice to Have
Experience with hybrid retrieval, Experience with learning to rank, Experience with domain-specific reranking, Experience integrating graph-structured context, Experience integrating knowledge graphs, Experience with evaluation for LLM systems, Experience with observability for LLM systems, Experience with evaluation for retrieval systems, Experience with observability for retrieval systems, Experience with drift detection, Experience with failure analysis, Experience with regression prevention, Experience with AWS-native retrieval architectures, Experience with AWS-native indexing architectures, Experience in edtech, Experience in content modeling, Experience in curriculum modeling, Experience in skills modeling
What You'll Do.
Design production retrieval systems
Build production retrieval systems
Ship production retrieval systems
Own vector store selection
Own vector store operation
Build indexing pipelines
Operate indexing pipelines
Build refresh pipelines
Operate refresh pipelines
Implement semantic retrieval
Implement hybrid retrieval
Own retrieval evaluation
Defend core tradeoffs
Partner on deployment
Partner on observability
Partner on reliability
Partner on relevance requirements
How You'll Work.
Team & Collaboration
Product partners; Engineering partners; Research partners; AI Platform engineers; Infrastructure engineers
Full Job Description
At Instructure, we believe in the power of people to grow and succeed throughout their lives. Our goal is to amplify that power by creating intuitive products that simplify learning and personal development, facilitate meaningful relationships, and inspire people to go further in their education and careers. We do this by giving smart, creative, passionate people opportunities to create awesome. And that's where you come in: Our team builds AI-native capabilities, reusable AI systems, and shared infrastructure that power multiple products and workflows across the platform. We are looking for a Senior Applied Data Scientist to own retrieval and semantic systems end to end, as a core, reusable capability that multiple AI products depend on. You will own the full retrieval vertical: vector store selection and operation, indexing and refresh pipelines, semantic and hybrid retrieval, reranking, and the evaluation systems that prove relevance is good and stays good. You will own retrieval-specific architecture and its day-to-day operation, while our infrastructure owner provides the underlying cloud, cluster, and CI substrate and our AI Platform engineers provide the general MLOps and service scaffolding you build on. You will work closely with product, engineering, and research partners to turn advanced AI ideas into reliable product capabilities used at scale. Important note on scope: This is a deep individual-contributor specialist role. We are looking for someone who has owned a retrieval system in production, not someone who has only used a vector database in a prototype. Retrieval evaluation is central to this role: if you cannot measure relevance and catch regressions before they reach users, the system is not done. WHAT YOU'LL DO - Design, build, and ship production retrieval systems that power AI product capabilities across multiple products - Own vector store selection and operation, including scalability, latency, reliability, cost, and multi-tenant design - Bu
Applying for this Senior Applied Data Scientist, Retrieval and Semantic Systems role?
Most applicants get filtered before a human reads their resume. See if yours makes the cut.
How to Apply on Ashby
- Ashby is a fast modern ATS — most applications take under 3 minutes.
- The resume parser is strong; verify parsed experience dates and job titles.
- Custom screening questions are often scored algorithmically — answer completely.
- Location field affects geo-based screening; use your actual metro area.
ANONYMOUS · UNFILTERED
What do employees actually say about Instructure?
Real rants from real employees. Read before you apply.