Big Data Engineer, GCP Migration

Job Description

About the Role: Big Data Engineer, GCP Migration

hackajob is partnering with American Express to find exceptional professionals for this role. American Express offers a culture built on innovation, shared values, and a commitment to backing its customers, communities, and colleagues. As part of Team Amex, you’ll receive comprehensive support for your well-being, opportunities for skill development, leadership growth, and career advancement.

Joining Amex Tech means contributing to something significant. You’ll collaborate with talented tech teams, build a unique career, and be empowered by the latest technologies and a commitment to the open-source engineering community. Amex Tech thrives on its technology, culture, and colleagues.

The Enterprise Data and AI Organization (EDAI)

The EDAI organization consolidates data governance, strategy, engineering, and product teams with those responsible for AI engineering, generative AI enablement, and automation product and engineering. This group is crucial for leveraging data as a core driver of innovation and integrating AI capabilities to transform products, operations, and customer experiences.

EDAI also encompasses technology Research & Development, experimentation with emerging capabilities, and engineering support for Amex Digital Labs, ensuring research breakthroughs translate seamlessly into business impact.

Purpose of the Role: LUMI & GCP Migration

LUMI is American Express’s largest Big Data Platform, designed for computationally and/or data-intensive processing applications. It provides robust, cost-efficient capabilities for batch, online, or streaming data workloads.

As a Big Data Engineer, you will join a team of dedicated engineers working with exciting and upcoming technologies. The Cornerstone platform challenges engineers daily to build world-class products. You will be an integral part of a fast-paced Agile team, designing, developing, testing, troubleshooting, and optimizing solutions to simplify access to Amex’s Big Data Platform, particularly as the company embarks on its journey to move to the public cloud – Google Cloud Platform (GCP).

Focus

The role involves designing, developing, problem-solving, debugging, evaluating, modifying, deploying, and documenting software and systems for customer-facing applications, business applications, and/or internal end-user applications.

Organizational Context

This position is for a member of an engineering or delivery and integration team, reporting to an Engineering Manager or Engineering Director.

Responsibilities

• Implement scalable and efficient data architectures on GCP.
• Collaborate with cross-functional teams to understand data requirements and develop solutions that meet business needs.
• Data Pipeline Development:
  • Build, test, and deploy data pipelines to move, transform, and process data from various sources to GCP.
  • Ensure the reliability, scalability, and performance of data pipelines.
  • Utilize GCP’s big data technologies such as BigQuery, Dataflow, Dataproc, and Pub/Sub to implement effective data processing solutions.
• Monitor system performance and proactively optimize data pipelines for efficiency.
• Troubleshoot and resolve issues.
• Create and maintain comprehensive documentation for tools, architecture, processes, and solutions.

Qualifications

• Understanding of GCP services including Cloud Dataflow, Cloud Pub/Sub, BigQuery, Cloud Storage, Google Composer, etc.
• Strong SQL knowledge.
• Understanding of fundamentals of Git and Git workflows.
• Experience of working in an agile application development environment.
• Technical support to applications on troubleshooting Environment, software, and application-level issues.
• Write and test programs using Unix Shell scripting and Oracle PL/SQL programming.
• Experience of supporting platform engineering activities, including network and firewall.

Salary & Benefits

Salary Range: $103,750.00 to $174,750.00 annually + bonus + benefits. Your pay will be determined by your location, experience, and other job-related factors.

American Express offers comprehensive benefits to support your holistic well-being, including:

• Competitive base salaries
• Bonus incentives
• 6% Company Match on retirement savings plan
• Free financial coaching and financial well-being support
• Comprehensive medical, dental, vision, life insurance, and disability benefits
• Flexible working model with hybrid, onsite or virtual arrangements (Amex Flex)
• 20+ weeks paid parental leave for all parents
• Free access to global on-site wellness centers (depending on location)
• Free and confidential counseling support through our Healthy Minds program
• Career development and training opportunities

For a full list of Team Amex benefits, visit our Colleague Benefits Site.

Commitment to Diversity & Inclusion

American Express is an equal opportunity employer, making employment decisions without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other status protected by law. The company will consider all qualified applicants, including those with arrest or conviction records, in accordance with applicable state and local laws.

Employment eligibility to work with American Express in the U.S. is required as the company will not pursue visa sponsorship for these positions.