Senior Data Engineer

Position Description

We are seeking a Senior Data Engineer to lead the design and implementation of end-to-end data solutions within a modern Azure-based cloud environment. This hands-on, technical role will partner closely with data scientists, analysts, and business stakeholders to ensure that data is accurate, accessible, and optimized for analytics, reporting, and operational needs. You will play a key role in building Data Lakehouse, scalable data pipelines and integrations while championing best practices, mentoring junior engineers, and leading strategic data initiatives across teams.

Key Responsibilities

- Architect and implement secure, scalable data pipelines using Azure Data Factory, Azure Functions, and Azure Data Lake Storage

- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks (PySpark/Spark SQL) to ingest, transform, and process large volumes of structured and unstructured data

- Build and manage data pipelines using Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure SQL Database

- Implement data quality, validation, and monitoring frameworks to ensure reliability and accuracy of data pipelines

- Collaborate cross-functionally with data scientists, analysts, and business stakeholders to understand data requirements and deliver fit-for-purpose data solutions

- Lead modernization and optimization efforts, improving pipeline performance, maintainability, and scalability

- Support CI/CD practices for data pipelines using tools such as Azure DevOps, Git, and Databricks Repos

- Detect and resolve data quality issues; implement automated audits and monitoring processes

- Troubleshoot and resolve production data pipeline issues, ensuring high availability and minimal downtime

- Act as a technical leader on schema design, performance tuning, and Azure data architecture

- Mentor and support junior engineers across data engineering, analytics, and BI teams

- Participate in and lead code reviews, promoting clean, well-documented, and testable code

- Stay current on trends in data engineering and cloud technologies, identifying opportunities to innovate

Minimum Requirements

- 8+ years of hands-on experience in data engineering, including designing and implementing enterprise-scale data solutions.

- 2+ years of experience developing and operating Azure cloud-native data platforms.

Critical Skills

- Expertise in MS SQL Server, Python (pandas, PySpark), Azure Data Factory, Azure Functions and Azure Data Lake Storage.

- Strong expertise with Azure Databricks, including Spark (PySpark/Scala), Delta Lake, and cluster/job optimization.

- Solid understanding and hands-on experience building Data Lakehouse architecture

- Experience working with a variety of file formats (e.g., CSV, JSON, XML, Parquet).

- Experience with version control (Git) and CI/CD pipelines for data engineering workflows

- Familiarity using REST APIs for data extraction and integration.

- Proven experience designing and implementing data solutions.

- Strong understanding of cloud architecture, data warehousing and modern data stack components.

Additional Skills & Qualifications

- Demonstrated ability to perform root cause analysis on data and processing issues

- Strong problem-solving skills with the ability to explain technical concepts to non-technical audiences

- A successful history of manipulating, processing and extracting value from large disparate datasets

- Experience with Big Data technologies such as Databricks, Spark, or Azure Synapse

- Knowledge of CI/CD workflows, version control, and agile development practices

- Familiarity with data governance, privacy, and compliance frameworks

- Experience with data warehousing, analytics tools, and BI platforms

- Familiarity with streaming data technologies (Azure Event Hubs, Kafka, Structured Streaming)

Education

- 4-year degree in computer science, engineering or other related IT field of study, or equivalent professional work experience

Physical Requirements

- General office demands

Prolonged periods of sitting at a desk and working on a computer.

- Frequent reaching, handling, and fine manipulation for using office equipment, filing, and managing paperwork.

- Manual dexterity sufficient to operate a keyboard, mouse, and other office tools.

- Occasional standing, walking, and bending.

- Ability to lift up to 10-20 pounds occasionally.

- Vision abilities required include close vision for computer work and reading documents.

- Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.