Site Reliability Engineer

About the role:

The Site Reliability Engineer will serve as the technical leader responsible for designing, implementing and operating enterprise observability platforms and standards. This role will drive the strategy for logging, metrics, distributed tracing, monitoring, alertin, and self-healing automation across application, infrastructure and AI-enabled environments. The role will establish visibility into AI and agentic workflows, including model interactions, retrieval and tool execution, performance, quality, cost and operational risk. This role will partner closely with development teams, platform engineers, AI platform teams and enterprise architects to ensure observability and traceability are built into deployment pipelines and solution architectures by design. This role will lead the evolution of monitoring maturity, improve operational resilience and enable faster detection, diagnosis and resolution of incidents.

What you'll do:

- Lead the design and implementation of enterprise observability standards across logging, metrics, tracing and alerting
- Own and evolve the observability platform strategy and roadmap, including Datadog and SolarWinds
- Define and implement instrumentation standards for applications, APIs and infrastructure to ensure consistent telemetry collection
- Partner with development teams to embed observability into CI/CD pipelines as a first-class deployment requirement
- Design and implement automated monitoring, alerting and self-healing patterns to improve service reliability and availability
- Develop reusable observability frameworks, libraries and deployment templates for engineering teams
- Establish monitoring-as-code, alerting-as-code and dashboard-as-code standards
- Design service-level indicators (SLIs), service-level objectives (SLOs) and error budgets in partnership with product and engineering teams
- Ensure monitoring coverage for cloud, on-prem, containerized and hybrid environments
- Implement distributed tracing and log correlation to enable end-to-end transaction visibility
- Define observability and tracing standards for AI-enabled applications, copilots and autonomous or agentic workflows, including model calls, prompts and responses, retrieval steps, tool invocations, latency, token consumption, cost and failure states
- Implement end-to-end AI tracing that correlates user requests across orchestration layers, models, retrieval-augmented generation (RAG), vector stores, APIs, tools and downstream services
- Establish monitoring and evaluation patterns for AI response quality, hallucination risk, model or prompt drift, guardrail violations, safety events and degraded user outcomes
- Define telemetry governance for AI workloads, including sensitive-data redaction, access controls, retention, sampling and secure handling of prompts, responses and trace data
- Serve as the escalation point for complex monitoring, performance and reliability issues
- Collaborate with enterprise and platform architects to ensure observability requirements are built into reference architectures
- Drive root cause analysis and reliability improvements following major incidents
- Develop platform governance standards for monitoring, alerting thresholds, and on-call best practices
- Ensure observability is tightly integrated with CMDB and Service Mapping for accurate service context and impact analysis
- Partner with DevOps and platform engineering teams to improve deployment reliability and pipeline automation
- Tune alerts continuously to reduce noise and improve signal quality
- Support capacity planning, performance engineering and load testing efforts
- Engage with Datadog, SolarWinds and ServiceNow vendors and user communities to stay ahead of platform capabilities
- Develop training and enablement materials for engineering teams on observability best practices
- Contribute to reliability engineering practices including chaos testing, fault injection and resiliency validation
- Drive continual improvement of incident response, post-incident reviews and operational readiness
- Integrate observability platforms with ServiceNow for incident, event, change and problem automation
- Defines observability policy aligned with CMDB governance, service modeling and configuration management standards

What you need:

To perform this job successfully, an individual must be able to perform each essential duty to satisfaction. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

- Education Bachelor’s degree in computer science, engineering or equivalent combination of education and experience required.

- Experience 1-3 years’ site reliability engineering, DevOps or platform engineering experience required
- Hands-on experience building and operating enterprise observability platforms
- Experience implementing observability and distributed tracing for AI-enabled applications, large language model (LLM) workflows or agentic systems preferred

- Skills & Abilities Ability to demonstrate success implementing monitoring, logging, and tracing at scale across complex environments
- Deep expertise with Datadog and SolarWinds monitoring platforms
- Strong experience designing observability for cloud, hybrid and on-prem environments (Azure, Kubernetes, VMs)
- Expertise in logging platforms, APM, distributed tracing and infrastructure monitoring
- Working knowledge of generative AI and LLM application architectures, including copilots, agents, orchestration frameworks, retrieval-augmented generation (RAG), vector databases, model gateways and tool-based integrations
- Experience with OpenTelemetry or comparable instrumentation standards and the ability to apply trace context across AI, application, API and infrastructure dependencies
- Understanding of AI observability concepts including prompt and model versioning, token and cost analysis, latency decomposition, quality evaluation, drift detection, guardrails and human feedback signals
- Ability to balance diagnostic trace depth with privacy, security, retention and cost requirements when collecting AI telemetry
- Strong proficiency in automation and scripting (Python, PowerShell)
- Experience implementing observability into CI/CD pipelines (GitHub Actions, Azure DevOps, GitLab)
- Strong understanding of infrastructure-as-code and configuration management (Terraform, Ansible, etc.)
- Experience with container and orchestration platforms (Docker, Kubernetes)
- Strong understanding of networking, operating systems and application architectures
- Experience with event-driven monitoring, anomaly detection and predictive alerting
- Excellent troubleshooting, root cause analysis and incident management skills
- Strong documentation, architecture diagramming and platform governance capabilities
- Ability to lead cross-functional reliability initiatives and influence engineering standards

Employment visa sponsorship is unavailable for this position. Applicants requiring employment visa sponsorship now or in the future (e.g., F-1 STEM OPT, H-1B, TN, J1 etc.) will not be considered.

About Us
Total Quality Logistics (TQL) is one of the largest freight brokerage firms in the nation. TQL connects customers with truckload freight that needs to be moved with quality carriers who have the capacity to move it.
As a company that operates 24/7/365, TQL manages work-life balance with sales support teams that assist with accounting, and after hours calls and specific needs. At TQL, the opportunities are endless which means that there is room for career advancement and the ability to write your own paycheck.
What’s your worth? Our open and transparent communication from management creates a successful work environment and custom career path for our employees. TQL is an industry-leader in the logistics industry with unlimited potential. Be a part of something big.
Total Quality Logistics is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, age, national origin, genetic information, disability or protected veteran status.
If you are unable to apply online due to a disability, contact recruiting at [email protected]