Data Engineer

Job Requirements

Role/ Job Title: Data Engineer(Devops)

Function/ Department: Data and Analytics

Job Purpose:

The AWS DevOps Engineer will work closely with the Data Engineering and Analytics teams to set up and manage daily business-processing pipelines. The role is responsible for building and maintaining a highly available AWS cloud platform that enables deeper analysis and reporting for the Data and Analytics function.

The role will work with the IDFC FIRST Bank Data Platform team across technologies including AWS Cloud, Jenkins, the Cloudera data stack, Apache Airflow, Python and shell scripting.

Roles & Responsibilities:

- Build, manage and automate the AWS cloud platform to support scalable and highly available data-processing workloads.
- Create and maintain end-to-end continuous integration and continuous delivery pipelines using Jenkins and other relevant tools.
- Manage source code through Git and support controlled production deployments.
- Work with Data Engineering and Analytics teams to set up, monitor and optimise daily business-processing pipelines.
- Support infrastructure management, operational controls and audit requirements for the cloud platform.
- Collaborate with development teams to identify and document high-impact business use cases and potential technical solutions.
- Develop automation scripts using shell scripting and Python.
- Prepare technical design documentation for projects, automation workflows and data pipelines.
- Contribute to the delivery and presentation of technical solutions to relevant stakeholders.
- Troubleshoot platform, deployment and pipeline issues and implement preventive improvements.

Education Qualification:

Graduation: Any Graduation in Engineering, Computer Science, Information Technology or a related discipline.

Professional Qualification/Certification: AWS certification or relevant DevOps/Cloud certification will be preferred.

Experience:

3 to 5 years of relevant experience in DevOps and cloud technologies, with hands-on exposure to AWS, Jenkins, continuous integration and continuous delivery, Git, Python and shell scripting.