Databricks Senior Lead Consultant

About the role

The Databricks Senior Lead Consultant designs, builds and operates modern analytics and data platforms on the Databricks Lakehouse Platform. The role bridges business stakeholders, data teams and cloud architects to deliver secure, cost‑effective, enterprise‑ready solutions for BI, data engineering, data science and AI/ML workloads.

Key responsibilities

• Design and implement modern data platforms using Databricks Lakehouse.
• Build ingestion and transformation pipelines with Databricks Workflows, Apache Spark (PySpark / Spark SQL) and Delta Lake for batch, streaming and near‑real‑time processing.
• Define platform standards for data ingestion, modelling, quality and governance, ensuring alignment with cloud, security, identity and cost‑management requirements.
• Develop and optimise Spark pipelines and Delta Lake models to support analytics, data science and machine‑learning use cases.
• Implement monitoring, performance tuning, cluster optimisation and CI/CD for notebooks, jobs and data pipelines.
• Collaborate with analysts, data scientists and application teams, advise on Databricks capabilities and mentor junior members.

Required profile

• 10‑15+ years of data and analytics experience, with 7‑10+ years designing enterprise‑scale data platforms.
• Proven experience defining lakehouse architectures and data strategies.
• Strong stakeholder‑engagement and delivery skills.

Required skills

• Databricks Lakehouse Platform
• Apache Spark, PySpark, Spark SQL
• Delta Lake and data modelling
• Structured Streaming and Kafka
• Performance tuning and cluster optimisation
• Data security, governance and cost optimisation
• CI/CD pipelines for notebooks and jobs
• BI tools such as Power BI or Tableau for governed data consumption