Lead Data Engineer

Job summary

As a Lead Data Engineer, you will blend hands-on development with technical leadership to enhance a Databricks-based data platform. You will collaborate with engineers, architects, product owners, and business stakeholders to convert business goals into scalable data solutions while guiding technical decisions.

Qualifications

• Extensive hands-on experience delivering production-grade solutions using Databricks.
• Proficient in Python, PySpark, and SQL, including complex transformations and performance optimization.
• Experience designing Databricks Asset Bundles and managing deployments across multiple environments.
• Strong knowledge of Unity Catalog, data governance, access control, and data management best practices.
• Familiarity with CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools.
• Ability to communicate architectural decisions to both technical and non-technical stakeholders.

Responsibilities

• Lead the design, development, and optimization of enterprise-scale data solutions on Databricks.
• Design and implement scalable data ingestion frameworks for batch and streaming workloads.
• Own and evolve Databricks Asset Bundles, deployment pipelines, and multi-environment configurations.
• Drive data governance initiatives through Unity Catalog, including access management and metadata management.
• Define and maintain engineering standards, including coding practices and data quality frameworks.
• Mentor fellow data engineers and serve as the primary technical escalation point for complex challenges.

Skills

• Experience with Databricks AI/BI Dashboards, Genie, or semantic layer design is advantageous.
• Familiarity with Azure identity and secrets management, including Entra ID and Azure Key Vault.
• Knowledge of incorporating GDPR and data confidentiality requirements into data engineering solutions.

Education

• Relevant degree or equivalent experience in a related field is preferred.

Tools

• Databricks, Python, PySpark, SQL, GitHub Actions, Azure DevOps.