Data Engineer III

Job Responsibilities:

- Utilize architectural and design skills for a variety of data-oriented tasks.

- Establish and maintain strong relationships with business stakeholders.

- Build highly reliable & scalable data systems with good data governance that provide complete, secure, accurate and timely data for analytics.

- Expertise in data & dimensional modeling with good business acumen in supporting new product features and services - including financial reporting.

- Assist development and analytics organization to meet its delivery goals with regards to data design, flow and infrastructure.

- Perform Data Modeling, Big data architecture design, support and implementation of OLTP, real time OLAP/data warehouse systems and ETL pipelines.

- Track/Build & monitor data/system KPI’s like storage, performance & usage

- Hands on administration of big data databases/compute engines (Redshift, EMR, Spark or similar) and optimization/tuning techniques

- Prepare required documentation as outlined by departmental standards.

- Identify, analyze and back track data problems as needed.

- Provide recommendations for application and system improvements.

- Mentor other team members

- Work with business users to understand business requirements, issues and business and/or client processes.

- Develop, test, and maintain high-performance of our data systems to meet the requirements of the business and/clients while adhering to departmental standards.

- Perform quality assurance testing for all work performed.

- Meet with agile teams as required to define and document application requirements.

- Follow project development, deployment process & maintain industry standards and best coding practices.

- Maintain security and organization of the company’s data.

- Provide off-hour support as assigned.

- Gather business requirements/features, estimate the tasks, design & develop scalable solutions.

- Work with manager to ascertain the company’s data requirements.

- Plan work to meet project deadlines, accommodate demands by development teams, set priorities and escalate issues appropriately.

-  Provide recommendations to development teams for data restructuring and performing complex maintenance.

- Deploy and manage changes in development, staging and production.

- Assist development and dev ops teams on SQL queries and tuning.

Required Skills & Experience:

- At least 7 years’ experience in data engineering, Data modeling, SQL, ETL, Big database, Data warehousing & basic-level administration of data infrastructure

- At least 7 years coding experience in Java, Python, R or other equivalent programming language

- At least 4 years big data experience

- At least 2 years’ experience as technical lead architecting data lake/data warehouse/lake house systems & mentoring team of Data Engineers / DBAs

- Proficient in a variety of big-data tools and technologies, including Hadoop, Spark, etc., handling different storage formats like Parquet, Delta/Apache Hudi/Apache Iceberg, & managing/working with different OLTP systems & data ingestion mechanisms.

- Experience building large scale data lakes & warehouse systems leveraging cloud environments including AWS technologies –S3, Redshift, Aurora/RDS, DMS, EC2, Glue spark, Cloud watch, & EMR

- Experience with data orchestration tools like Airflow

- Experience in building pipelines, ingesting data from heterogeneous systems onto data lake environments in real time

- Proficient in Python, SQL and PL-SQL, query tuning, optimization, ETL, ELT, and Data Warehousing

- Proficient in Business Intelligence (BI), analytic database and products, able to take business requirements and translate into database tables, pipeline design and tasks.

- General data-specific concepts, frameworks and processes

- Agile development practices

- Working within an SDLC

- Designing data warehouses including definition, structure, documentation, maintenance, long-range requirements, operational guidelines, and protection

- Data governance, meta data management, data security measures, retention & archival across data lake layers

Other Skills and Experience

- Cloud experience, such as AWS, GCP or Azure

- Building/Managing data lake & lake house architectures.

- Experience working on different data lake storage formats.

- Excellent oral and written communication

- Multi-tasking and managing multiple priorities.

- Working well in a team-oriented, collaborative environment with people from different disciplines and varying degrees of technical experience

- Working in an Agile team environment.

Behavioral Competencies:

- Cultivates Innovation

- Decision Quality

- Manages Complexity

- Drives Results

- Business Insight