Big Data Engineer, GCP Migration

Job Description

Big Data Engineer, GCP Migration | American Express

The Tone:
This is a full-time role at American Express, offering a flexible working model with hybrid, onsite, or virtual arrangements. American Express cultivates a culture of innovation and shared values, committed to backing its customers, communities, and colleagues. As part of Amex Tech, you will join talented teams, contributing to the Enterprise Data and AI Organization (EDAI). EDAI consolidates critical functions including data governance, AI engineering, generative AI enablement, and automation, aiming to transform products, operations, and customer experiences by leveraging data as a core driver of innovation. This group also spearheads R&D and supports Amex Digital Labs, translating research into business impact. In this role, you will be integral to migrating LUMI, American Express’s largest Big Data Platform designed for computationally and data-intensive processing, to Google Cloud Platform (GCP). This transition will simplify access to Amex’s Big Data Platform, enabling the development of world-class products.

The TL;DR
• Role: Full Time
• Type: Full-time
• Location: Flexible (hybrid, onsite or virtual)
• Pay: $103750–$174750 yearly
• Team: Member of an engineering or delivery and integration team, reporting to an Engineering Manager or Engineering Director.
• Mission: Design, develop, test, troubleshoot, and optimize solutions to simplify access to American Express’s Big Data Platform, particularly during its migration to Google Cloud Platform (GCP).
• Tech Stack: BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Dataflow, Cloud Pub/Sub, Cloud Storage, Google Composer, SQL, Git, Unix Shell scripting, Oracle PL/SQL.

What You’ll Actually Do
• Design & Implement: Design, evaluate, and implement scalable and efficient data architectures and software systems on GCP for customer-facing, business, and internal applications.
• Develop & Deploy Pipelines: Build, test, and deploy robust data pipelines utilizing GCP’s big data technologies such as BigQuery, Dataflow, Dataproc, and Pub/Sub, ensuring reliable, scalable, and performant data processing from various sources.
• Collaborate & Solve: Collaborate with diverse cross-functional teams to understand data requirements, develop solutions that meet business needs, and engage in problem-solving and debugging for complex data challenges.
• Optimize & Troubleshoot: Monitor system performance, proactively optimize data pipelines for efficiency, and troubleshoot and resolve any encountered issues within the data platform.
• Document: Create and maintain comprehensive documentation for all tools, architecture, processes, and solutions, supporting clear understanding and ongoing maintenance.

The Must-Haves
• Background: Experience with big data platforms and a foundational understanding of data architecture principles.
• Experience: Demonstrated experience developing in an agile environment and working knowledge of GCP services including Cloud Dataflow, Cloud Pub/Sub, BigQuery, Cloud Storage, and Google Composer.
• Skills: Proficient in SQL, Git and Git workflows, Unix Shell scripting, and Oracle PL/SQL programming.
• Bonus: Experience providing technical support for environment, software, and application-level issues, including platform engineering activities related to network and firewall.