Python Cloud Data Engineer

About the role

IBM Consulting is seeking a Python Cloud Data Engineer to join its Data Platforms team on AWS. You will advise clients, design, develop, and maintain robust data engineering solutions that power both batch and real‑time analytics across hybrid cloud environments.

Key responsibilities

• Design, build, and operate batch and real‑time data pipelines using AWS services such as EMR, Glue, Glue Catalog, and Kinesis.
• Create and manage data layers on Redshift, Aurora, and DynamoDB, and perform migrations with AWS DMS.
• Develop pipelines for data warehouses and data lakes, leveraging Kinesis and Managed Streaming for Apache Kafka.
• Schedule, monitor, and optimize data services on the AWS platform.
• Apply open‑source tools like Apache Airflow, dbt, and Spark (Python or Scala) to enhance data workflows.

Required profile

• Bachelor’s degree in a relevant field.
• Proven experience with core AWS services (S3, Lambda, Glue, Aurora, Redshift, QuickSight).
• Strong Python programming skills for data processing and automation.
• Hands‑on experience designing, building, and managing ETL processes.

Required skills

• AWS EMR, AWS Glue, Glue Catalog, AWS Kinesis, AWS Redshift, AWS Aurora, AWS DynamoDB, AWS DMS, AWS S3, AWS Lambda, AWS QuickSight
• Python
• Apache Airflow
• dbt
• Apache Spark (Python/Scala)
• Managed Streaming for Apache Kafka