Senior-KDNI

Job Summary:

We are looking for a skilled Senior Data Engineer with around 4-6 years of experience to join our data engineering team. The ideal candidate should have strong hands-on experience in Snowflake , SQL, Python, PySpark, Spark Architecture and along with a solid understanding of data modeling and ETL pipeline development .
The candidate will be responsible for designing, developing, and maintaining scalable data pipelines, transforming and processing large datasets, and implementing data solutions using Snowflake and distributed data processing technologies. This role requires strong problem-solving, communication skills and the ability to work effectively with cross-functional teams.

Key Responsibilities:

- Design, develop, and maintain scalable ETL/ELT data pipelines using modern data engineering technologies.
- Develop complex and optimized SQL queries for data extraction, transformation, and processing.
- Build and maintain data processing solutions using Python, PySpark, and Apache Spark.
- Design, develop, and optimize data solutions using Snowflake.
- Design and implement scalable data models to support analytical and reporting requirements.
- Develop data applications and interactive solutions using Streamlit. Perform data transformation, cleansing, validation, and quality checks to ensure data accuracy and consistency.
- Identify and resolve performance bottlenecks across SQL queries, Spark jobs, Snowflake workloads, and ETL pipelines.
- Provide technical guidance and mentoring to junior and mid-level team members.
Take ownership of technical deliverables and ensure timely and high-quality delivery.
- Collaborate with business stakeholders, architects, developers, and other teams to understand requirements and translate them into scalable data solutions.
- Participate in technical discussions, design reviews, code reviews, and solution planning.
- Communicate technical concepts, project status, risks, and dependencies effectively to both technical and non-technical stakeholders.
- Follow engineering best practices for coding, testing, version control, documentation, and deployment.

Qualifications :

- Bachelor’s degree in computer science, Information Technology, Engineering, or a related field.
- Around 4-6 years of professional experience in Data Engineering or a related field.
- Mandatory: Strong hands-on experience with Snowflake and Snowpark including its data warehousing capabilities.
- Mandatory: Extensive hands-on experience with internal and external stages, file formats, data loading, Snowpipe, Streams, Tasks, stored procedures, and Snowflake security/access controls.
- Mandatory: Strong hands-on experience with Snowpark, particularly using Python/Snowpark for data processing and transformation within Snowflake.
- Mandatory: Strong understanding of Snowflake architecture, virtual warehouses, compute/storage separation, workload management, performance optimization, and cost optimization
- Mandatory: Familiarity with cloud storage technologies such as Amazon S3 or Azure Data Lake Storage.
- Mandatory: Strong hands-on experience with SQL.
- Mandatory: Strong programming experience in Python and hands-on experience with PySpark for large-scale data processing.
- Good understanding of data modeling concepts, including relational and dimensional modeling.
- Experience in designing and developing ETL/ELT pipelines.
- Strong understanding of data transformation, data integration, and data processing concepts.
- Good analytical and problem-solving skills.
- Ability to work independently as well as collaboratively in a team environment.

Nice to have:

- Experience with Apache Airflow or other workflow orchestration tools.
- Knowledge of DBT and modern ELT practices.
- Experience with Git and CI/CD practices.
- Experience working with large-volume datasets and distributed data processing.
- Snowflake SnowPro Core or Advanced certification.
- Experience with CI/CD pipelines and DevOps practices.
- Knowledge of data cataloging tools and metadata management.
- Exposure to BI tools like Power BI, Tableau or Looker.
- Excellent communication and stakeholder management skills.

Qualifications :

- Bachelor’s degree in computer science, Information Technology, Engineering, or a related field.
- Around 4-6 years of professional experience in Data Engineering or a related field.
- Mandatory: Strong hands-on experience with Snowflake and Snowpark including its data warehousing capabilities.
- Mandatory: Extensive hands-on experience with internal and external stages, file formats, data loading, Snowpipe, Streams, Tasks, stored procedures, and Snowflake security/access controls.
- Mandatory: Strong hands-on experience with Snowpark, particularly using Python/Snowpark for data processing and transformation within Snowflake.
- Mandatory: Strong understanding of Snowflake architecture, virtual warehouses, compute/storage separation, workload management, performance optimization, and cost optimization
- Mandatory: Familiarity with cloud storage technologies such as Amazon S3 or Azure Data Lake Storage.
- Mandatory: Strong hands-on experience with SQL.
- Mandatory: Strong programming experience in Python and hands-on experience with PySpark for large-scale data processing.
- Good understanding of data modeling concepts, including relational and dimensional modeling.
- Experience in designing and developing ETL/ELT pipelines.
- Strong understanding of data transformation, data integration, and data processing concepts.
- Good analytical and problem-solving skills.
- Ability to work independently as well as collaboratively in a team environment.