R/reports/ai

This is a market for the applied AI/product engineer, not the ML researcher. The overwhelming pattern across ~45 postings: ship LLM-powered features end-to-end in production, with agents and evaluation as the two dominant themes. Concrete takeaways:

• Agents are the center of gravity. Agent harnesses, orchestration, tool-calling, multi-agent coordination, and guardrails appear in the majority of roles (Cursor, Decagon, Cohere North, Harvey, Hex, Lumos, Docker, Anthropic, Databricks). If you build one deep thing, build a production agent loop with tools and evals.
• Evaluation is the most under-supplied, most-demanded skill. Nearly every posting asks you to make "quality" measurable: offline eval datasets, LLM-as-judge, regression alerts, online A/B, precision/recall on agent output. Cursor, Harvey, Hippocratic, Mercor, Canva, Decagon, Databricks all treat evals as a first-class deliverable. This is the single highest-leverage gap to fill.
• The default stack is Python + TypeScript/React on Kubernetes/AWS. Most "AI engineer" roles are full-stack product roles wearing an LLM hat, not training jobs. Pure model-training roles are concentrated and gated (Cohere MTS, Hippocratic MLE, Faire) and ask for distributed training (JAX/XLA, PyTorch) plus publications.
• A distinct, deep specialist track pays a premium: retrieval/IR (Databricks, Redis) and inference serving (Databricks vLLM/SGLang). These want years of search or distributed-systems depth, not generalist AI-product skills.
• Domain framing is the differentiator, not the tooling. The same agent skills get sold into legal (Harvey), security (Semgrep, Cohere, Docker, Anthropic), finance (Cohere North), healthcare (Hippocratic), data analytics (Hex, ClickHouse, Mixpanel), and identity (Lumos). Pick a domain story; the underlying stack transfers.

What to do: get undeniably strong at "production agent + rigorous evals + RAG," show it full-stack (Python backend, React frontend), and attach it to one regulated/enterprise domain. Skip chasing model-training credentials unless targeting the few frontier-lab MLE roles.

Must have

• Python, TypeScript/JavaScript, React
• LLM APIs (OpenAI GPT-4/GPT-5, Claude, Gemini), prompt engineering, tool/function calling
• Kubernetes, AWS, Docker, CI/CD, Terraform, Helm
• RAG stacks, vector databases, agent/orchestration frameworks (LangChain, LangGraph)
• GraphQL/Apollo, Postgres, Redis, REST API design
• Go (backend-heavy and infra roles: Docker, Redis, Cribl, Lumos)
• Eval frameworks / LLM-as-judge, observability and monitoring

Nice to have

• MCP (Model Context Protocol), agent frameworks (CrewAI, AGNO, ADK)
• PyTorch, TensorFlow, JAX, XLA/MLIR, TF-Serving
• vLLM, SGLang, CUDA, ONNX Runtime, TensorFlow Lite, llama.cpp
• Rust, C/C++ (Redis search/inference, Applied Intuition embedded)
• FAISS, ScaNN, HNSW, Elasticsearch/OpenSearch/Lucene (BM25)
• SCIM, OAuth2, SAML, IDP/HRIS integrations (Lumos); Greenhouse, Workday, Rippling (Anthropic)
• GCP, Azure, Node.js, Flask, SQLAlchemy, Django, Streamlit, Gradio

Agents and orchestration

• Agent harness design - the agent loop, tool schemas, model routing, execution environment, multi-agent (router/planner/supervisor) patterns, subagents; the control plane that wraps the model (Cursor, Decagon, Hex, Harvey).
• Tool use and orchestration - tool/function calling, ReAct-style multi-step workflows, parallel tool calls, tool allowlists, structured outputs, failure handling and retries.
• Context and memory management - context-window budgeting, session state, long-running/multi-turn memory, what the model sees and why; caching strategies for latency.
• Guardrails and agent safety - safe dispatch, sandboxing, approvals, audit trails, access controls (RBAC), agent governance and auditability for enterprise data.

Evaluation and quality

• Eval system design - curated datasets, offline replay, scorers/judges, regression alerts, dashboards; turning ambiguous "quality" into concrete metrics (Cursor, Harvey, Mercor).
• LLM-as-judge and rubric eval - judge harnesses, rubric-based scoring, online quality metrics, narrative/intent-satisfaction beyond raw model metrics (Canva, Hippocratic).
• Failure analysis - debugging agent behavior, clustering failure modes, precision/recall, false-positive triage, "good/bad/degraded" session definitions (Cursor Bugbot, Decagon).
• Feedback loops - collecting and cleaning real usage signals, online experimentation/A/B at scale, human-in-the-loop labeling pipelines.
• Synthetic data - synthetic data generation, simulation environments, stress-testing models across scenarios (Mercor, Hippocratic).

Retrieval and search

• RAG pipelines - chunking, embeddings, hybrid vs vector search, re-ranking with cross-encoders, grounding to reduce hallucination, token-efficient context.
• Information retrieval - lexical (BM25) and dense retrieval (ANN), learning-to-rank, query rewriting/decomposition, intent classification, entity resolution (Databricks, Redis).
• Vector indexing - HNSW, IVF, quantization, ACL-aware/permissioned indexing, freshness, multi-tenancy, recall/latency trade-offs.
• Retrieval evaluation - nDCG, MRR, Recall@K, Precision@K; offline/online experimentation for relevance.
• Agentic retrieval - search planners, multi-hop/iterative retrieval, sufficiency/self-reflection checks, deciding what to fetch and verifying what came back (Databricks).

Inference and serving

• Low-latency serving - high-throughput CPU/GPU inference, routing, caching, intelligent autoscaling, token-based rate limiting, observability and SLAs (Databricks).
• On-device/edge inference - quantization, pruning, compilation, CPU/GPU/NPU profiling, thermal/memory/latency budgets (Applied Intuition).
• Model lifecycle (MLOps) - versioning, monitoring, lineage, deployment workflows, integrating models into production with evals.

Model adaptation and training

• Fine-tuning - supervised fine-tuning of (multi-modal) LLMs, embedding/reranker training, human-in-the-loop training (Faire, Databricks, Cohere).
• Distributed training - large-scale distributed GPU training and inference strategies (Cohere).
• Multimodal/voice - speech recognition, TTS, streaming/real-time architectures, turn-taking, transcription pipelines (Hippocratic, Decagon, Canva).
• Safety research - red-teaming, adversarial evals, reward hacking, scheming/deception, RLHF/RLVR (OpenAI, Hippocratic).

The 20% that covers 80% — learn these first
The fastest path to "qualified" for most of these postings is one tight cluster: a production agent loop (tools + orchestration + guardrails), a real eval harness around it, and a RAG pipeline feeding it — all shipped full-stack in Python + TypeScript/React on Kubernetes/AWS. This single project profile maps onto the majority of roles (Cursor, Hex, Cohere, Harvey, Lumos, Cribl, Docker, Decagon, Anthropic product roles).

• Agent harness + tool calling
• Eval datasets, LLM-as-judge, regression detection
• RAG with re-ranking and grounding
• Python backend + React/TypeScript frontend

Table stakes vs differentiators
Table stakes (expected everywhere, won't set you apart): calling LLM APIs, basic prompt engineering, Python, shipping full-stack on K8s/AWS. Genuine differentiators (frequently required, rarely held well): rigorous evaluation systems, agent reliability/failure-mode analysis, and deep retrieval/IR. Eval depth is the biggest leverage gap — nearly every posting wants it, few candidates can show it.

Go deep vs stay broad

• Go deep on exactly one: evals, agent orchestration, or retrieval/IR. These are where "one level deeper than peers" turns into an offer.
• Stay broad (awareness is enough): specific frameworks (LangChain vs LangGraph vs CrewAI are interchangeable — learn one), specific vector DBs, specific clouds.

Interchangeable vs non-negotiable

• Interchangeable (learn one): agent frameworks (LangChain/LangGraph/CrewAI/AGNO/ADK); vector indexes (FAISS/ScaNN/HNSW); search engines (Elasticsearch/OpenSearch); clouds (AWS/GCP/Azure); inference servers (vLLM/SGLang).
• Non-negotiable: Python; LLM API + prompting/tool-calling fluency; ability to define and run evals; for product roles, TypeScript/React.

Rising vs fading
Rising: agent harnesses, MCP, eval infrastructure, agentic/multi-hop retrieval, on-device LLMs, agent security/governance. Fading or niche: classic ML-from-scratch framing ("strong ML background") shows up mostly in the gated training roles; generalist full-stack with no AI story is no longer enough.

Unique to this domain vs transferable
Unique to AI eng: agent loops, context engineering, eval/LLM-as-judge, RAG grounding. Transferable in (your leverage): distributed systems and low-latency serving (Databricks), IR/search (Redis, Databricks), security engineering (Semgrep, Cohere, Docker, Anthropic). If you have one of these backgrounds, you can enter the AI market without "AI experience" — Databricks Foundation Model Serving explicitly says "no prior ML or AI experience is necessary."

Company-stage signals

• Frontier labs (Anthropic, OpenAI, Cohere): pay top ($300K–$485K), reward 0→1 ownership, MCP/Claude integration, safety/red-teaming, and (for MLE) publications + distributed training. Several require citizenship/clearance (Anthropic Public Sector, Cohere Sovereign).
• Growth/unicorn product (Cursor, Decagon, Harvey, Hex, Lumos, Mercor): reward shipping velocity, taste in agent/model behavior, and eval obsession over credentials. "We don't do programming interviews" (Hightouch) — they test product sense and system design.
• Mature/infra (Databricks, Redis, Cribl, Grafana, Docker): reward deep systems (retrieval, serving, Go/Rust/C++) and operational rigor (on-call, SLAs).

Lead with a shipped agent that has measured quality
The single strongest signal across these postings is "I shipped an LLM agent to real users and made its quality measurable." Put one such project at the top with concrete metrics.

• Strong: "Built a production code-review agent; cut false-positive comments 40% by adding an LLM-as-judge eval harness and regression gating across 5K+ PRs."
• This combo (agent + eval + measurable quality drop in errors) is an immediate yes at Cursor, Harvey, Decagon, Cohere, Cribl.

The "wow, that's deep" combo: retrieval or serving with hard numbers
Going one level deeper than prompt-and-ship sets you apart. Show retrieval or inference depth with latency/recall/cost trade-offs.

• "Built hybrid retrieval (BM25 + HNSW dense + cross-encoder rerank); lifted Recall@10 from 0.62 to 0.89 and held p95 latency under 200ms."
• "Cut LLM serving cost 35% via continuous batching and token-based rate limiting on GPU inference."

Name the eval methodology, not just "evals"
Recruiters and EMs are scanning for whether you actually know how to measure AI quality. Specifics read as senior.

• Junior: "Used AI to build a chatbot feature."
• Senior: "Defined offline eval set (300 curated cases) + LLM-as-judge rubric + online A/B; used nDCG and task-success rate to gate releases."

Concrete projects EMs are hoping to find (grounded in these postings)

• Multi-hop retrieval subagent that decides what to fetch, verifies sufficiency, and falls back when results are weak (Databricks) → "Built an agentic retrieval layer with sufficiency checks; reduced ungrounded answers 50%."
• Agent harness with model routing under cost/latency constraints (Cursor "Auto", Harvey AI Platform) → "Built model-routing layer choosing among 4 providers by task; cut cost 30% at equal quality."
• Eval/synthetic-data flywheel (Mercor, Hippocratic) → "Built synthetic data pipeline + automated benchmark that gated production model releases."
• Domain agent in a regulated vertical with audit trails/RBAC (Cohere North, Harvey, Lumos) → "Shipped finance/security agent with data isolation, approvals, and audit logging."

Junior vs senior framing of the same work

• Junior reads as: tool user, no measurement, no ownership. "Integrated the OpenAI API into our app using LangChain."
• Senior reads as: end-to-end ownership, trade-offs, measured impact. "Owned an LLM feature from prototype to production for 10K users; designed the eval harness, made the model/latency/cost trade-offs, and drove a 25% task-success improvement."

Red flags that make recruiters pass

• Tutorial-grade, no production, no metrics: "Built a RAG chatbot over my PDFs with LangChain" (with nothing shipped or measured).
• Buzzword stacking without depth: "Expert in RAG, agents, fine-tuning, LLMs" with no named techniques or numbers.
• Pure-backend or pure-ML-research framing for product roles that explicitly want full-stack and shipping (Cursor Bugbot, Anthropic product roles call this out directly).

Full-stack fluency is a quiet gate
Many "AI engineer" roles are full-stack product roles; an inability to move between React frontend, Python/Node backend, and the model layer is disqualifying (Cursor Bugbot, Hex, Cribl, ClickHouse, Cohere full-stack). Show at least one feature you owned across UI → backend → model.