About this role
Key Responsibilities
- Design, develop, and support batch and incremental data ingestion pipelines into the data lake.
- Develop and optimize complex SQL queries, data transformations, and ETL workflows.
- Build, maintain, and monitor Airflow DAGs for orchestration and workflow automation.
- Analyze and troubleshoot data quality, data reconciliation, and pipeline execution issues.
- Perform end-to-end data lineage analysis and trace data from source systems to downstream consumers.
- Debug data discrepancies by investigating source systems, transformation logic, and target datasets.
- Support data migration and modernization initiatives from legacy platforms to cloud-based data platforms.
- Work closely with business stakeholders, architects, and upstream/downstream teams to understand data requirements.
- Validate data completeness, accuracy, and consistency across multiple systems.
- Participate in production support activities and resolve data-related incidents within SLA timelines.
- Develop reusable data engineering components and recommend process improvements.