Principal Software Development Engineer (Service Platform & Orchestration)

Job Description

Principal Software Development Engineer- Go | Confidential Employer

The Tone:
This is a hybrid full-time role at Zscaler, requiring 3 days in office, focused on the Zero Trust Exchange department. Zscaler provides a SASE-based Zero Trust Exchange platform that secures customers from cyberattacks and data loss by connecting users, devices, and applications globally. This position is critical for engineering the management plane and reliability systems that govern ZIA’s massive fleet lifecycle, transforming legacy automation into a durable, stateful management plane. It involves building self-healing capabilities and AI-driven SRE practices for a global fleet of 100k+ instances.

The TL;DR
• Role: Full Time
• Location: Hybrid – US
• Pay: $185,000–$265,000 yearly
• Team: Zero Trust Exchange department, reporting to the Director, Software Engineering
• Mission: Engineer the management plane and reliability systems to govern ZIA’s fleet lifecycle at massive scale, transitioning to a durable, stateful management plane.
• Tech Stack: Go, Temporal, Cadence, PostgreSQL, REST/gRPC, LLMs, ML

What You’ll Actually Do
• Lead Development: Lead the hands-on development and migration to a workflow-as-code platform (Temporal), building replay-safe, idempotent workflows for deterministic global operations.
• Transform Infrastructure: Move the organization beyond scripted automation toward a robust management plane that treats the global fleet as a single, eventually-consistent distributed system.
• Implement AI/ML Services: Design and implement services that leverage LLMs and ML for intelligent signal correlation, automated triage, and self-correcting fleet operations.
• Develop Framework-Level Services: Develop framework-level services and internal APIs to ensure all new products are delivered “orchestration-ready” with reliability hooks built directly into the code.
• Build Deep Telemetry: Build deep telemetry (metrics, traces, and events) into the management plane so that every fleet-wide action is fully explainable, auditable, and replayable.

The Must-Haves
• Background: BS/MS in Computer Science or a related technical field with 12+ years of experience in hyperscale systems, with a deep understanding of unique failure modes and technical hurdles at massive scale.
• Experience: 12+ years designing and operating complex distributed systems, focusing on systemic challenges in concurrency, failure handling, and performance optimization; proven track record of developing Platform APIs (REST/gRPC) with strong guarantees for idempotency, verification, and safe rollout patterns.
• Skills: Mastery of backend systems languages with a strong preference for Go (or deep expertise in Java, C++, or Python with a demonstrated ability to rapidly adopt Go); foundational understanding of AI/ML technologies and experience leveraging them to optimize outcomes.
• Bonus: Proficiency with AI code-assistance tools (e.g., Cursor, Windsurf) to accelerate refactoring and system development; proficiency in PostgreSQL or other relational stores for high-scale, stateful management-plane services; direct experience building or operating systems with Temporal.io, Cadence, or similar workflow engines.