About this role
Description
Build the AI agent that investigates production incidents for AWS customers. The AWS DevOps Agent reads the same telemetry a human operator would — logs, metrics, traces, topology — reasons about what went wrong, and delivers a written root-cause analysis in minutes. We're hiring an SDE II to work on the investigation agent: the core reasoning engine that turns an alarm into an answer.
This is not wrapper code over an LLM. You'll work on agent orchestration, tool design over MCP (Model Context Protocol), prompt engineering at scale, sandbox isolation for untrusted code execution, and evaluation systems that measure whether the agent actually got the diagnosis right. The system spans Rust, Python, and TypeScript services across a hundred packages, and ships to production across all commercial AWS regions.
Key job responsibilities
- Design and build agent reasoning systems that investigate production incidents end-to-end — from alarm to root-cause analysis — using large language models and a tool-calling architecture over MCP.
- Own and evolve the investigation agent's tool surface: the capabilities the agent uses to read CloudWatch, X-Ray, Kubernetes, CloudTrail, and service topology on a customer's behalf.
- Build and improve evaluation systems (offline scenarios with ground-truth grading, online trajectory evaluation of production runs) that measure agent quality and catch regressions before customers do.
- Work across the stack — Python agent runtime on Strands, Rust Data Plane services, TypeScript control plane and CDK infrastructure — in a system where a single investigation crosses half a dozen services.
- Design prompt and skill architectures that shape how the agent reasons without hard-coding investigation logic, balancing model flexibility with reliability.
- Collaborate with security, AppSec, and partner service teams on the trust boundaries that let an AI agent safely read customer telemetry at scale.
A day in the life
You might start the morning reviewing an eval regression — the agent's accuracy on a new scenario class dropped, and you're digging into the journal trace to understand what the model missed. After lunch you're pairing with a teammate on a new MCP tool that gives the agent access to deployment history, deciding what information to surface and what to withhold. By end of day you're reviewing a CR that changes how the agent handles multi-signal incidents where logs, metrics, and traces tell conflicting stories. The problems are ambiguous, the feedback loop is fast (run the agent, read the trace, grade the result), and the work ships to every AWS commercial region.
About the team
The AWS DevOps Agent team builds an AI agent that investigates incidents, reviews releases, and answers operational questions about running AWS environments. We own the full vertical: the agent runtime, the tool servers, the sandbox, the evaluation pipeline, and the data plane that records what every run did. Our system serves AWS customers across all commercial regions, and the investigation agent is the flagship — the thing that turns "an alarm fired" into "here's what happened and what to do about it." We're a team that cares about whether the agent is actually right, not just whether it sounds right, and we've built the eval infrastructure to tell the difference.
Basic Qualifications
- 3+ years of non-internship professional software development experience
- 2+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience
- 1+ years of software development engineer or related occupational experience
- 1+ years of designing and developing large-scale, multi-tiered, multi-threaded, embedded or distributed software applications, tools, systems, and services using: C#, C++, Java, or Perl experience
- 1+ years of Object Oriented Design experience
- Bachelor's degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field
- Experience programming with at least one software programming language
Preferred Qualifications
- 3+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience
- Bachelor's degree in computer science or equivalent
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits .
USA, WA, Seattle - 143,700.00 - 194,400.00 USD annually