About this role
Job Description:
Model development produces experiments, checkpoints, evaluation results and many versions of the same model. Keeping those records connected lets engineers reproduce results, compare approaches and move a trained model into use.
We are looking for a Senior Software Engineer to build the backend services that make those workflows reliable. You will connect training runs to their data, code and configuration, manage model artifacts and help machine-learning engineers spend less time reconstructing what happened in an experiment.
What you'll do:
- Build and maintain the software and configuration that let developers run GPU workloads on Kubernetes.
- Design quotas, scheduling priorities, permissions and isolation so multiple teams can share compute predictably.
- Automate deployment and upgrades of GPU services, including Run:ai, with infrastructure as code and tools such as Terraform and Helm.
- Investigate why jobs wait, fail or run slowly, and fix recurring scheduling, configuration, storage and networking problems.
- Build diagnostics that connect queue times, GPU utilization, memory requirements and training throughput to the developer’s experience.
- Benchmark representative workloads and recommend when to add GPU capacity, how much to add and which hardware fits the workload.
- Deliver repeatable installations and upgrades across cloud, on-premises and air-gapped environments.
Key Outcomes:
- Developers can get the GPU resources they need through understandable access and resource policies.
- Teams can diagnose slow or blocked training jobs and distinguish capacity shortages from configuration problems.
- GPU expansion decisions reflect measured demand and workload performance.
- Platform changes improve reliability without disrupting existing training workflows.
Required qualifications:
- Experience building and supporting production Kubernetes platforms, including GPU or distributed computing workloads.
- Strong software engineering skills in Python, Go or a comparable language.
- Experience automating infrastructure deployment, configuration and upgrades.
- Practical Linux, networking and storage troubleshooting skills.
- Experience with shared resource scheduling, access controls or workload isolation.
- Ability to work directly with machine-learning engineers, understand their workloads and turn recurring problems into maintained platform capabilities.
Preferred qualifications:
- Experience with Run:ai or comparable GPU scheduling and resource-management systems.
- Familiarity with distributed training and tools such as Ray or PyTorch.
- Experience delivering on-premises or air-gapped platforms, or contributing to open-source infrastructure.
Why Join Us
The AI Factory is the foundation for how Shield AI develops and deploys autonomy.
As a Senior Staff Engineer, you will help build the reference architecture used internally by our engineering teams and delivered to customers building their own AI capabilities. Your work will directly impact how quickly researchers can iterate, how efficiently compute resources are utilized, and how AI systems move from experimentation to operational deployment.
This is an opportunity to work at the intersection of modern AI, distributed systems, Kubernetes, and defense, building a platform that enables the next generation of autonomy systems while leveraging the best of the open-source AI ecosystem.
#LI-DM2
#LF
Full-time regular employee offer package:
Pay within range listed + Bonus + Benefits + Equity
Temporary employee offer package:
Pay within range listed above + temporary benefits package (applicable after 60 days of employment)
Salary compensation is influenced by a wide array of factors including but not limited to skill set, level of experience, licenses and certifications, and specific work location. All offers are contingent on a cleared background and possible reference check. Military fellows and part-time employees are not eligible for benefits. Please speak to your talent acquisition representative for more information.
###
Shield AI is proud to be an equal opportunity workplace and is an affirmative action employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender identity or Veteran status. If you have a disability or special need that requires accommodation, please let us know.