Senior Observability Software Engineer – Data Platform

Job Description

Senior Software Development Engineer (US Federal) | Workday

The Tone:
This is a full-time hybrid role at Workday, primarily based out of Reston, VA, USA, with client site presence in the DC, MD, and VA (DMV) area. Workday is a leading AI platform for managing people, money, and agents, shaping the future of work for teams. This role is crucial for developing and operating a large-scale distributed data platform specifically for U.S. Federal contracts, empowering real-time insights across Workday’s platforms, infrastructure, and applications. You will be responsible for managing Workday within air-gapped U.S. AWS cloud regions.

The TL;DR
• Role: Full Time
• Type: Hybrid
• Location: Hybrid – Reston, VA, USA
• Pay: $148200–$264000 USD yearly
• Team: Data Platform and Observability team, forming the core team responsible for managing Workday within air-gapped U.S. AWS cloud regions
• Mission: To design, build, and enhance a large-scale distributed data platform that fuels critically important Workday applications for U.S. Federal contracts.
• Tech Stack: Kubernetes, OpenStack, Bare Metal, Java, Python, Go, Prometheus Ecosystem, Elastic Search, Grafana, InfluxData stack, PromQL, InfluxQL, Helm, Linux, Ansible, Chef, Terraform

What You’ll Actually Do
• Design, build, and enhance critical Observability services, including Monitoring, Logging, Alerting, and Tracing.
• Develop data capture and collection services using pioneering technologies across diverse infrastructure environments.
• Construct Metrics ingestion pipelines, Alert definitions, and automation for Dashboard lifecycle management.
• Build and deploy infrastructure components in production environments, championing observability with a focus on data quality and availability.
• Manage and optimize Workday within air-gapped U.S. AWS cloud regions, collaborating with commercial engineering and Federal Cyber Defense teams for security and compliance.

The Must-Haves
• Background: BS/MS in Computer Science or a related technical field, with a strong software engineering background and a consistent track record of delivering high-quality products at scale.
• Experience: 10+ years of experience building high-performance data ingestion software, addressing concurrency, latency, and efficiency challenges. Must be a United States citizen and have the ability to obtain and maintain a U.S. government issued security clearance (TS/SCI w/CI Poly preferred).
• Skills: Exceptional coding skills in Java or Python (Go is a plus), solid understanding of high-performance data capture and collection systems, API design, reliable data delivery, Time-Series Databases, Log collection and management, Kubernetes, and Linux fluency.
• Bonus: Demonstrable experience with the Prometheus Ecosystem, Elastic Search, Grafana and InfluxData stack, PromQL and InfluxQL. Hands-on experience with Helm, Infrastructure Automation (Ansible, Chef, Terraform). Preferred DoD 8570/8140 compliant with at least IAT Level Il certification.