About this role
Key Responsibilities
- Design, build, test, maintain, and monitor scalable Lakeflow Spark Declarative Pipelines to ingest data from Event Hubs and transform data through all layers of medallion architecture.
- Requirements gathering.
- Writing documentation.
Qualifications & Requirements
Technical Skills:
- Proficiency in Python, SQL, PySpark, Spark SQL.
- Proficiency with Databricks and Azure.
- Proficiency with Unity Catalog.
- Hands-on experience with structured streaming and Event Hubs is a plus.
- Understanding and/or experience with Declarative Automation Bundles is a plus.
- Understanding and/or experience with Spark Declarative Pipelines is a plus.
- Understanding and/or experience with SDP-meta is a plus.
Soft Skills:
- Strong analytical mindset with the ability to troubleshoot complex issues.
- Attention to detail.
- Proactivity, taking ownership.
- Team spirit, communication skills.