About this role

Data Specialist

Primary Skills

Databricks Engineer

Role Overview

We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform . The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL , with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.

Key Responsibilities

- Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark .

- Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.

- Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.

- Create, monitor, and optimize Delta Live Tables (DLT) pipelines.

- Implement scalable and efficient data ingestion processes using Auto Loader .

- Develop and manage real-time data processing solutions using Structured Streaming .

- Orchestrate, schedule, and monitor data workflows using Databricks Workflows .

- Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.

- Establish and enforce data governance, security, and access controls using Unity Catalog .

- Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.

- Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions.

Required Skills (Must Have)

- Databricks Platform
- Delta Lake

- Delta Live Tables (DLT)

- Unity Catalog

- Databricks Workflows

- PySpark and Apache Spark

- Structured Streaming

- Auto Loader

- SQL

- Lakehouse Data Modeling

- Strong understanding of data engineering best practices and scalable data architectures

Preferred Skills (Good to Have)

Azure Ecosystem

- Azure Data Factory (ADF)

- Azure Synapse Analytics

- Microsoft Purview

- Microsoft Fabric

AWS Ecosystem

- AWS Glue

- AWS Lambda

- AWS Step Functions

Data Engineering & Integration

- Apache Airflow

- DBT

- Fivetran

- Informatica

Streaming & Analytics

- Apache Kafka

- Power BI

Data Governance

- Collibra

- Alation

GCP

- BigQuery

Qualifications

- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.

- Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.

- Strong analytical, troubleshooting, and problem-solving capabilities.

- Experience working in agile and collaborative environments.

- Excellent communication and stakeholder management skills.

Preferred Candidate Profile

- Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.

- Strong understanding of data governance, security, and compliance frameworks.

- Experience delivering both batch and real-time data processing solutions.

- Ability to work independently while collaborating effectively across global teams.

Key Technologies

Specialization

- Databricks Engineering: Lead Data Engineer

Job requirements