Principal Data Engineer – Data Architecture & Governance

Job Description

Principal Data Engineer | Optiver

The Tone:
This is a full-time role at Optiver, a leading market-making institution that constantly improves market health by injecting liquidity, providing accurate pricing, and acting as a stabilizing force across global exchanges. In this pivotal role, you will build a greenfield, governed data foundation on a modern stack, directly replacing fragmented, siloed, and manually managed data systems across critical business functions like Finance, People, Procurement, Tax, and Compliance. Your work is essential for establishing a single source of truth, enabling unified reporting, self-service analytics, and powering the company’s emerging AI-driven conversational analytics capabilities. This robust data foundation is not only vital for regulatory scrutiny but also for fostering deep trust in business data across the entire organization.

The TL;DR
• Role: Full Time
• Type: Full-time
• Location: In-person Australia
• Mission: Design and build a unified, governed data foundation and semantic layer to replace fragmented business data across multiple domains.
• Tech Stack: Databricks, dbt, SQL, PySpark, Unity Catalog, DataHub, Databricks Genie, Delta table

What You’ll Actually Do
• Data Architecture: Architect and own the end-to-end delivery of shared data products for Finance, People, Procurement, and Compliance.
• Semantic Layer: Build and evolve the unified semantic layer to power reporting, self-service, and AI-driven analytics, ensuring trustworthiness.
• Data Engineering: Design and build scalable data pipelines, data models, and governed data products using Databricks, dbt, SQL, and PySpark.
• Data Governance: Implement end-to-end governance capabilities, including Unity Catalog access controls, column-level security, classification, lineage, and quality standards.
• Metadata Management: Drive DataHub adoption by defining metadata standards, lineage, and ownership models to enhance data discoverability.

The Must-Haves
• Background: Senior-level professional with a track record of architectural ownership in designing and building end-to-end production data solutions. Possesses domain knowledge in Finance, People/HR, Procurement, or Compliance, understanding business logic to challenge requirements.
• Experience: 12+ years of experience designing and building production data solutions, including leading complex technical initiatives end-to-end. Deep experience with Databricks, Lakehouse architectures, or comparable modern data technologies, covering Delta table design, Unity Catalog governance, and compute trade-offs for BI/AI workloads.
• Skills: Expert-level SQL and data transformation skills, strong hands-on experience in dbt, PySpark, or both, with an ability to design data products for scale and write tests efficiently. Proven experience implementing governance at scale, including access control, PII handling, column-level security, data lineage, and data quality management in production environments.