Databricks Unified Data Analytics Platform Engineer

About the role

We are looking for a Databricks Unified Data Analytics Platform Engineer to design, develop, and maintain end‑to‑end data solutions. The role focuses on building scalable ETL/ELT pipelines, ensuring data quality, and supporting advanced analytics on the Databricks Lakehouse.

Key responsibilities

• Develop high‑quality, scalable ETL/ELT pipelines using Databricks technologies such as Delta Lake, Auto Loader, and Delta Live Tables.
• Write production‑grade Python code and PySpark transformations, including PII masking and validation logic.
• Configure and manage Unity Catalog, setting up catalogs, schemas, access controls, audit logging, and data masking.
• Implement ingestion patterns with Auto Loader, checkpointing, and schema evolution for structured and semi‑structured data.
• Build secure DLT pipelines with Bronze/Silver/Gold medallion layering and DLT Expectations.
• Integrate Power BI, Tableau, or Looker via optimized ODBC/JDBC connectors and Unity Catalog security.
• Create stakeholder‑facing SQL dashboards to monitor KPIs, pipeline health, and SLAs.
• Prepare GenAI‑compatible datasets, manage vector embeddings, and use Feature Store with MLflow.
• Package and deploy pipelines using Databricks Asset Bundles through CI/CD pipelines in GitHub or GitLab.
• Troubleshoot, tune, and optimize jobs using Photon engine and serverless compute for cost efficiency.

Required profile

• Strong programming and debugging skills in Python.
• Hands‑on experience with PySpark for data transformation and validation.
• Proficiency in at least one cloud platform (AWS, GCP, or Azure).
• Solid understanding of data modeling and data‑warehousing principles.

Required skills

• Python
• PySpark
• Databricks
• Delta Lake
• Auto Loader
• Delta Live Tables
• Unity Catalog
• SQL
• Power BI
• Tableau
• Looker
• ODBC
• JDBC
• GitHub
• GitLab
• CI/CD
• Photon engine
• AWS
• Azure
• GCP
• Kafka
• Hadoop
• Hive
• Apache Airflow
• Informatica
• Talend
• Matillion
• Fivetran
• dbt
• MLflow
• Databricks Vector Search
• Feature Store
• GenAI
• Embeddings