Data Engineer

Role Overview

We are looking for a highly skilled Data Quality Engineer with strong Data Engineering expertise to ensure the accuracy, reliability, and scalability of enterprise data platforms. The ideal candidate will possess hands-on experience with Databricks, PySpark, Hadoop, Hive, and Cloud technologies, along with advanced SQL skills to validate data across large-scale data pipelines and Lakehouse architectures.

Required Experience

- 3+ years of experience in Data Engineering, Data Quality Engineering, or Data Testing.
- Hands-on experience with Databricks and PySpark.
- Strong experience in Hadoop ecosystem components such as Hive, HDFS, Spark, and related Apache technologies.
- Advanced SQL expertise for large-scale data validation and analysis.
- Experience working with Data Warehouses, Data Lakes, and Lakehouse architectures.
- Understanding of Star Schema, Snowflake Schema, and dimensional modeling.
- Experience with cloud platforms (Azure, AWS, or GCP).

Requirements

Key Responsibilities

Data Quality & Validation

- Perform end-to-end validation of data pipelines across ingestion, transformation, and consumption layers.
- Execute source-to-target reconciliation and data quality checks.
- Identify, investigate, and resolve data anomalies and inconsistencies.
- Define and implement data quality frameworks, metrics, and controls.

Data Engineering & Processing

- Develop and validate data pipelines using PySpark and Databricks.
- Work with large-scale datasets in Hadoop, Hive, and Lakehouse environments.
- Support ETL/ELT workflows and ensure data integrity throughout the data lifecycle.
- Optimize data processing jobs for performance and scalability.

SQL & Analytics

- Write advanced SQL queries for data profiling, reconciliation, and root cause analysis.
- Perform complex joins, window functions, CTEs, aggregations, and query optimization.
- Validate business rules and transformation logic against source systems.

Lakehouse & Cloud Platforms

- Validate and monitor data across Databricks Lakehouse architecture.
- Work with cloud platforms such as Azure, AWS, or GCP.
- Collaborate with Data Engineers, Architects, and Analysts to ensure reliable data delivery.

Defect & Incident Management

- Analyze production issues and conduct root cause analysis.
- Track and manage data defects through resolution.
- Implement proactive monitoring and automated quality checks.

Required Experience

- 6+ years of experience in Data Engineering, Data Quality Engineering, or Data Testing.
- Hands-on experience with Databricks and PySpark.
- Strong experience in Hadoop ecosystem components such as Hive, HDFS, Spark, and related Apache technologies.
- Advanced SQL expertise for large-scale data validation and analysis.
- Experience working with Data Warehouses, Data Lakes, and Lakehouse architectures.
- Understanding of Star Schema, Snowflake Schema, and dimensional modeling.
- Experience with cloud platforms (Azure, AWS, or GCP).

Preferred Skills

- Automated data testing frameworks.
- Data observability and monitoring tools.
- CI/CD implementation for data pipelines.
- Experience with Delta Lake, Unity Catalog, or similar technologies.
- Knowledge of Airflow, Kafka, or other Apache ecosystem tools.