Production Systems Engineer, NPI

Meta is seeking a highly skilled and experienced Systems/Hardware engineer to join our Release to Production (RTP) team. The RTP team is responsible for the end-to-end hardware lifecycle of all Meta servers, including prototyping, pre-production hands-on system validation, hardware debugging, and stress testing. As a Systems/Production Systems Engineer, you will work closely with various teams, including HW/SW co-design teams, hardware designers, networking teams, system manufacturers, component vendors, capacity engineering, production engineering, production services, and data center operations teams to enable new systems that will be deployed in our production data centers.

Responsibilities

Drive and lead end-to-end system validation, and deployment for cutting-edge Compute and Storage hardware platforms, including CPUs, Flash, Hard Disk Drives and Data Center Processing Units (DPUs) in data center environments
Proactively create experiments and tooling to diagnose and root cause hardware, firmware, and software failures across compute, storage, and networking systems using telemetry, logs, and hands-on debugging
Design and implement robust system-level test plans, including functional, stress, and performance tests. Test plans may also include emerging security domains such as Confidential Computing
Drive component vendors and system manufacturers to resolve hardware and firmware issues identified during production operations
Interface with external vendors and internal teams to understand the system architecture and develop hardware fault management for various server products
Drive new platform enablement, hardware validation, tooling specifications, and integration, customer workload testing, and experiment creation to detect and diagnose hardware/firmware/software health issues
Leverage understanding of RAS (reliability, availability, serviceability) to improve error reporting and error handling mechanisms for better operation quality and cost/effectiveness
Analyze production failure data to identify systemic trends and drive corrective actions with internal teams and external vendors
Create and maintain dashboards and data visualizations to surface hardware health signals and track remediation progress across the fleet
Drive necessary discussions with external and internal teams on test specifications and methodologies to improve test quality continuously, both within Meta, and in the broader industry

Qualifications

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
Bachelor's degree in Computer Science, Computer Engineering, a relevant technical field, or equivalent practical experience
6+ years of work experience in one or more domains such as: ASIC development, compute (ARM, x86), AI-ML hardware/software, storage (HDD, TLC/QLC SSD, HBA), memory (HBM, DDR5, LPDDR5), network (NIC, DPU), server interconnect technologies, or similar
Knowledge of architecture and components on one of the following products: server/PC/Laptop
Development or debug experience in one or more areas: hardware fault management, error reporting, error handling on hardware products
Experience with Python, C/C++ and/or similar languages, within a Linux environment, for server system management, automation, version control, CI/CD, or similar
Excellent problem-solving skills, with the ability to troubleshoot complex technical issues
Experience communicating technical findings and recommendations to cross-functional teams, including hardware designers, vendors, and operations stakeholders
Experience working in a matrix organization 7+ years of experience with a subset of one of the following domains: Compute Systems, Storage Systems, Accelerated Compute Systems/HPC, Kernel/Firmware Development and/or testing, Post Silicon Bringup
Experience with x86 or ARM-based CPUs and their subsystems (e.g., memory, inter-chiplet communications, RAS/DFT, power management, security)
Working/functional knowledge of common bus protocols such as I2C, SPI, USB, LPDDR, and/or PCIe
Hands-on experience troubleshooting problems at the system level, crossing across multiple components, as well as hardware, firmware, and software boundaries
Familiarity with debugging tools for systems-on-chip (SoCs) - e.g., JTAG, GDB, DSTREAM, Trace32
Understanding of the hardware development process and how it pertains to the test strategy. Experience authoring test plans for complex chipsets for functional, stress, and performance testing
2+ years of experience scripting automation in Python or equivalent
Experienced in the integration of lab tools for automated workflows with large-scale deployments
Hands-on experience in managing/debugging Linux servers
Proficiency in continuous integration/continuous delivery tools