About this role
Accountabilities
- Define and evolve data architectures using Databricks, ensuring pipeline optimization, processing efficiency, and effective resolution of complex technical challenges.
- Build, maintain, and optimize scalable, reliable data pipelines for data ingestion, transformation, and delivery.
- Design and implement Lakehouse architectures and Medallion structures (Bronze, Silver, and Gold).
- Develop batch and streaming data processing solutions, including Change Data Capture (CDC) and Structured Streaming.
- Build and orchestrate data ingestion and integration processes using Azure Data Factory (ADF), Auto Loader, and Lakeflow.
- Work with Apache Spark, Airflow, Kafka, dbt, SQL and NoSQL databases, and cloud data platforms.
- Implement data governance, security, lineage, and access control practices using Unity Catalog.
- Develop REST APIs, integrations, microservices, and automation solutions using Python.
- Establish CI/CD processes, version control, and automated deployments using Git and Databricks Workflows.
- Optimize Spark performance, compute resource allocation, and processing costs.
- Monitor and maintain production data pipelines, ensuring observability, reliability, and operational continuity.
- Act as a technical reference for the team, mentoring professionals and promoting engineering standards and best practices.
Requirements:
- Advanced expertise in Databricks, PySpark, Spark SQL, and Delta Lake.
- Strong experience with Delta Live Tables (DLT) / Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming.
- Hands-on knowledge of Liquid Clustering, Auto Loader, and Delta table optimization.
- Strong understanding of Unity Catalog, including data governance, security, lineage, and access control.
- Proven experience designing and implementing data architectures based on Lakehouse and Medallion principles (Bronze, Silver, and Gold).
- Practical experience with Azure Data Factory (ADF) for data orchestration, ingestion, and integration.
- Advanced Python programming skills and experience developing REST APIs, integrations, microservices, and automation.
- Advanced knowledge of Spark performance tuning, including Photon, partitioning, shuffle optimization, and execution plan analysis.
- Experience with Databricks Workflows, CI/CD, Git, and Declarative Automation Bundles.
- Ability to optimize workloads, size computing resources, and reduce processing costs.
- Experience with observability, monitoring, and production pipeline support.
- Ability to act as a technical leader, guide other professionals, and establish data engineering best practices.
- Relevant Databricks certifications:
Databricks Certified Data Engineer Associate.
- Databricks Certified Data Engineer Professional.
Benefits:
- Meal and/or food allowance.
- Health insurance and dental insurance.
- Pet health insurance.
- Home-office allowance for remote positions.
- Welcome kit during onboarding.
- Life insurance.
- Zenklub access, including psychological and nutritional support.
- TotalPass membership.
- Certification and continuous learning program with partners such as Adobe, Anthropic, AWS, Google, Microsoft, and Salesforce.
- Partnerships and discounts supporting professional development.
- Access to UOL content.
- Maternity kit.
- Extended maternity and paternity leave.
- Employee referral program.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1