About this role
Job Description: DataBricks – Technical Lead (Exp: 8 Yrs to 12 Yrs)
We are seeking a talented and motivated DataBricks expert to join our team as a Technical Lead.
Required Skills & Qualifications:
- 8 to 12 years of IT experience, with 3+ years focused on big data and Databricks.
- Strong hands-on experience with Databricks, PySpark, and Spark SQL.
- Experience with modern cloud data lakes (Azure ADLS, AWS S3).
- Hands-on experience with Azure Data Factory (ADF) or AWS Glue for orchestration.
- Strong experience with relational databases and SQL.
- Strong background in Agile delivery, CI/CD pipelines, and version control (Git).
- Experience with Delta Lake, Unity Catalog, and the medallion architecture.
- Experience with streaming data (Structured Streaming, Kafka, Event Hubs).
- Experience with Infrastructure-as-Code (Terraform).
- Experience in agile development processes using Jira and Confluence.
- Design scalable data engineering solutions and lead their delivery on Databricks.
- Understanding of the SDLC and Agile methodologies.
- Communication with the customer and producing the daily status report.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- Databricks Certified Data Engineer Professional certification.
- Microsoft Certified: Fabric Data Engineer Associate (DP-700) or AWS Certified Data Engineer - Associate.
- Should have good oral and written communication.
- Should be able to lead and mentor a team.
- Should be proactive and adaptive.
Key Responsibilities:
- Design scalable, high-performance data pipelines and Lakehouse workflows using Databricks and PySpark / Spark SQL.
- Build complex ETL/ELT pipelines on cloud platforms such as Azure or AWS.
- Guide development teams, perform code reviews, and enforce engineering best practices.
- Mentor engineers on Databricks, Spark, and data engineering design patterns.
- Work with product owners, business analysts, and architects to translate requirements into technical solutions.
- Monitor production workloads, fine-tune ETL/ELT performance, and ensure data reliability.