Databricks Data Engineer

About the role

We are looking for a Data Engineer with deep expertise in Databricks to design, develop, and maintain robust data pipelines. The role involves building scalable ETL/ELT solutions, ensuring data quality, and supporting analytics and machine‑learning workloads across cloud environments.

Key responsibilities

• Develop high‑quality, scalable ETL/ELT pipelines using Databricks Delta Lake, Auto Loader, and Delta Live Tables (DLT).
• Write production‑grade Python and PySpark code for data transformation, validation, and PII masking.
• Configure Unity Catalog, manage access controls, audit logging, and data masking for sensitive fields.
• Implement ingestion patterns with Auto Loader, checkpointing, and schema evolution for structured and semi‑structured data.
• Integrate Power BI, Tableau, or Looker with Databricks via optimized ODBC/JDBC connectors and Unity Catalog security.
• Build stakeholder‑facing SQL dashboards to monitor KPIs, pipeline health, and SLAs.
• Prepare GenAI‑compatible datasets, manage vector embeddings, and use Feature Store with MLflow.
• Package and deploy pipelines using Databricks Asset Bundles, CI/CD pipelines in GitHub or GitLab.
• Troubleshoot, tune, and optimize jobs with Photon engine and serverless compute for cost efficiency.

Required profile

• Minimum 2 years of hands‑on experience with cloud‑based data engineering services.
• Proficiency in at least one cloud platform (AWS, GCP, or Azure).
• Databricks Certified Professional or equivalent certification is a plus.
• Strong understanding of data modeling, data warehousing, and performance optimization techniques.

Required skills

• Python
• PySpark
• Databricks Delta Lake
• Auto Loader
• Delta Live Tables (DLT)
• Unity Catalog
• AWS, GCP, or Azure
• SQL
• Power BI / Tableau / Looker
• Photon engine
• CI/CD (GitHub, GitLab)
• Apache Airflow
• dbt
• Apache Spark
• Hadoop, Hive, Kafka
• MLflow
• Databricks Vector Search
• Feature Store
• ETL tools (Informatica, Talend, Matillion, Fivetran)