Data Engineer

Job Summary

We are seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines and modern data platforms. The ideal candidate will have hands-on experience in data engineering, ETL/ELT processes, cloud technologies, big data frameworks, and data warehousing solutions. The role involves collaborating with data scientists, analysts, and business stakeholders to deliver reliable and high-quality data solutions.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines and data integration solutions.
- Build and optimize ETL/ELT workflows for processing large-scale datasets.
- Develop and manage enterprise data warehouses, data lakes, and lakehouse architectures.
- Integrate data from multiple structured and unstructured sources.
- Ensure data quality, governance, security, and compliance standards.
- Optimize data storage, retrieval, and processing performance.
- Collaborate with Data Scientists, BI Developers, Product Owners, and business teams.
- Monitor and troubleshoot data pipelines and production data issues.
- Implement data validation, lineage, and monitoring frameworks.
- Support real-time and batch data processing requirements.

---

Required Skills

Core Technical Skills

- Strong proficiency in SQL and database design principles.
- Hands-on experience with Python , PySpark, or Scala.
- Experience building ETL/ELT pipelines and data integration solutions.
- Strong understanding of Data Modeling (Star Schema, Snowflake Schema, Dimensional Modeling).
- Experience with Data Warehousing concepts and best practices.
- Knowledge of batch and real-time data processing frameworks.

Big Data Technologies

- Apache Spark
- Hadoop Ecosystem
- Databricks
- Apache Kafka
- Apache Airflow

Cloud Platforms

- Microsoft Azure (ADF, ADLS, Synapse, Fabric)
- AWS (S3, Glue, Redshift, EMR)
- Google Cloud Platform (BigQuery, Dataflow)

Databases

- SQL Server
- PostgreSQL
- Oracle
- MySQL
- MongoDB

---

Preferred Skills

- Experience with Microsoft Fabric and modern Data Lakehouse architecture.
- Hands-on experience using Azure Data Factory (ADF), Synapse Analytics, and Databricks.
- Experience with streaming technologies such as Kafka and Spark Streaming.
- Knowledge of data governance and cataloging tools.
- Experience with CI/CD implementation for data pipelines.
- Familiarity with Infrastructure as Code (Terraform).

---

Tools & Technologies

- Python
- SQL
- PySpark
- Databricks
- Apache Spark
- Kafka
- Airflow
- Azure Data Factory
- Azure Synapse Analytics
- Microsoft Fabric
- Git
- Azure DevOps
- Jenkins

---

Educational Qualification

- Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.

---

Good-to-Have

- Experience working in Retail, E-commerce, Banking, or Enterprise Data Platforms.
- Understanding of Machine Learning data pipelines.
- Knowledge of data governance, data security, and compliance frameworks.
- Strong analytical and problem-solving skills.
- Experience working in Agile/Scrum environments.