Senior Observability Engineer

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

Senior Observability Engineer

Location:

Eli Lilly, Hyderabad

Position Type:

Full-Time

Level:

P3

At Lilly, we believe in the talent of our workforce. One of the best ways to utilize and develop that talent is to use our existing workforce to fill new and/or open positions. If you are looking for a new position within Lilly, you can view and apply to open roles posted in the internal job posting system. You must meet the minimum qualifications outlined in the job description and have/obtain work authorization in the country the position is located in order to be considered. When applying internally for a position, your current supervisor will receive notification that you have applied to the position. We encourage employees to discuss the opportunity with their supervisor prior to applying.

Note: Roles are posted at the lowest level of a band; however, employees should search across all levels of the band to identify all opportunities. Employees hired on banded positions (e.g., P1–P3, R1–R2, B1–B3, etc.) transfer at their current level, despite the level indicated on the job posting.

Lilly is seeking a passionate and curious Senior Observability Engineer to help build and maintain monitoring solutions that improve system reliability and operational visibility. This role supports the implementation of observability tools such as Splunk Cloud , Splunk IT Service Intelligence (ITSI) , and OpenTelemetry , working closely with senior engineers and cross-functional teams to deliver actionable insights.

Key Responsibilities

Focus: Observability + ARE

This role is accountable for designing, implementing, and scaling Lilly's enterprise observability capabilities across applications, infrastructure, cloud platforms, and digital products. The role leverages Splunk, OpenTelemetry, automation, AI-powered operations, and Agentic Reliability Engineering (ARE) practices to improve visibility, reliability, operational efficiency, and autonomous support capabilities across the enterprise.

The primary focus is to ensure enterprise services are observable, actionable, resilient, and increasingly autonomous through modern monitoring, telemetry, automation, and AI-enabled operational workflows.

What You'll Do

Observability Engineering

- Engineer, administer, and enhance enterprise observability platforms including Splunk Cloud, Splunk Enterprise, Splunk ITSI, Infrastructure Monitoring, and OpenTelemetry.
- Design scalable collection of logs, metrics, traces, and events across cloud, infrastructure, and application domains.
- Develop Service Trees, KPIs, Entity Models, Glass Tables, Correlation Searches, and Service Health Dashboards.
- Expand observability coverage by onboarding enterprise applications, cloud services, and technology platforms.

Agentic Reliability Engineering (ARE)

- Design AI-enabled workflows for autonomous detection, diagnosis, triage, and remediation of operational issues.
- Partner with operations, platform engineering, and automation teams to reduce toil and improve service reliability.
- Contribute to Lilly's shift from reactive operations toward autonomous and intelligent operations.

Automation, Integration & AI Operations

- Develop REST APIs, integrations, and automation frameworks that improve observability operations.
- Integrate observability with ServiceNow, Event Management, Incident Management, and automation platforms.
- Use AI-powered tools to accelerate scripting, troubleshooting, incident analysis, diagnostics, and runbook development.
- Implement AIOps capabilities including alert correlation, noise reduction, anomaly detection, and intelligent incident triage.

Operational Excellence

- Perform root-cause investigations and reliability analysis for complex operational issues.
- Improve monitoring standards, telemetry quality, observability adoption, and operational maturity.
- Partner with cloud, application, infrastructure, and support teams to improve reliability and availability.
- Create and maintain technical documentation, standards, and operating procedures.

How You'll Succeed

Observability & Engineering Expertise

- Strong experience with Splunk Cloud, Splunk Enterprise, Splunk ITSI, and enterprise observability solutions.
- Deep understanding of logs, metrics, traces, OpenTelemetry, and telemetry architecture.
- Strong Python and/or Golang development skills with API integration experience.
- Ability to design scalable monitoring and service intelligence solutions aligned to business outcomes.

Automation & ARE

- Experience building automation that reduces manual effort and improves operational efficiency.
- Understanding of ARE principles, autonomous operations, and self-healing architectures.
- Ability to leverage AI-assisted engineering tools with proper governance, testing, and validation.

Collaboration & Delivery

- Ability to work across Operations, Platform Engineering, Cloud Engineering, Service Management, and Application Support.
- Strong problem-solving, communication, and stakeholder management skills.
- Ability to independently drive solution design, implementation, testing, and adoption.

What You Should Bring

- Bachelor's degree in Computer Science, Engineering, Information Technology, or related discipline.
- 12+ years of overall technology experience in observability and infrastructure, production support roles.
- 3-5 years of hands-on Splunk ITSI engineering experience.
- Experience with Splunk Cloud and/or Splunk Enterprise.
- 2-4 years of experience with observability, monitoring, or operational intelligence platforms.
- Strong knowledge of SPL, OpenTelemetry, Prometheus, logs, metrics, traces, and distributed tracing.
- Experience with AWS and/or Azure cloud services.
- Hands-on experience using AI coding and automation assistants (such as, but not limited to, Claude Code, GitHub Copilot, and OpenAI Codex) to accelerate scripting, troubleshooting, and infrastructure work

Preferred Qualifications

- Experience with Dynatrace, Datadog, New Relic, AppDynamics, Grafana, or similar platforms.
- Experience integrating observability platforms with ServiceNow or enterprise event management tools.
- Experience with Kubernetes, cloud-native platforms, autonomous remediation, and self-healing workflows.
- Splunk, Cloud, OpenTelemetry, or AI-related certifications.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( https://careers.lilly.com/us/en/workplace-accommodation ) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly