Lead Data Engineer

Job summary

The Lead Data Engineer will blend hands-on development with technical leadership to enhance a Databricks-based data platform. This role involves collaborating with engineers, architects, product owners, and business stakeholders to convert business goals into scalable data solutions while guiding critical technical decisions.

Qualifications

• Extensive hands-on experience delivering production-grade solutions using Databricks
• Proficiency in Python, PySpark, and SQL, with skills in complex transformations and performance optimization
• Experience in designing Databricks Asset Bundles and managing deployments across multiple environments
• Strong knowledge of Unity Catalog, data governance, access control, and data management best practices
• Familiarity with CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools
• Ability to communicate architectural decisions clearly to both technical and non-technical stakeholders

Responsibilities

• Lead the design, development, and optimization of enterprise-scale data solutions on Databricks
• Design and implement scalable data ingestion frameworks for batch and streaming workloads
• Own and evolve Databricks Asset Bundles, deployment pipelines, and multi-environment configurations
• Drive data governance initiatives through Unity Catalog, including access management and metadata management
• Define and maintain engineering standards, including coding practices and data quality frameworks
• Mentor fellow data engineers and serve as the primary technical escalation point for complex challenges

Skills

• Experience with Databricks AI/BI Dashboards, Genie, or semantic layer design is advantageous
• Familiarity with Azure identity and secrets management, including Entra ID and Azure Key Vault
• Knowledge of GDPR and data confidentiality requirements in data engineering solutions

Education

• Not specified

Tools

• Databricks, Python, PySpark, SQL, GitHub Actions, Azure DevOps