Databricks Unified Data Analytics Platform Engineer

About the role

We are looking for a Databricks Unified Data Analytics Platform Engineer to design, develop, and maintain end‑to‑end data solutions. The role focuses on building scalable ETL/ELT pipelines, ensuring data quality, and supporting advanced analytics and GenAI use cases.

Key responsibilities

• Develop high‑performance ETL/ELT pipelines using Databricks Delta Lake, Auto Loader, and Delta Live Tables (DLT).
• Write production‑grade Python and PySpark code for data transformation, validation, and PII masking.
• Configure Unity Catalog, set up catalogs, schemas, access controls, and audit logging.
• Implement ingestion patterns with checkpointing, schema evolution, and secure external locations.
• Build and monitor stakeholder‑facing SQL dashboards for KPI and pipeline health tracking.
• Prepare GenAI‑compatible datasets, manage vector embeddings, and use Feature Store with MLflow.
• Package pipelines with Databricks Asset Bundles and deploy via CI/CD pipelines (GitHub/GitLab).
• Troubleshoot, tune, and optimize jobs using Photon engine and serverless compute for cost efficiency.

Required profile

• Strong programming and debugging skills in Python.
• Hands‑on experience with PySpark for large‑scale data processing.
• Proficiency in at least one cloud platform (AWS, GCP, or Azure).
• Solid understanding of data modeling, data warehousing, and medallion architecture.
• Experience with cloud‑based data services, real‑time streaming, and serverless computing.

Required skills

• Python
• PySpark
• Databricks Delta Lake
• Databricks Auto Loader
• Delta Live Tables (DLT)
• Unity Catalog
• AWS, GCP, Azure
• SQL
• Power BI, Tableau, Looker
• GitHub, GitLab
• CI/CD pipelines
• Photon engine
• Vector Search
• MLflow
• Apache Airflow
• Informatica, Talend, Matillion, Fivetran
• dbt
• Kafka, Hadoop, Hive