Quantexa Data Engineer

Role Overview

We are seeking a talented and experienced Data Engineer with strong expertise in Quantexa, Hadoop, Scala, Apache Spark, Elasticsearch, OpenShift Container Platform (OCP), and DevOps practices .

The successful candidate will be responsible for designing, developing, optimizing, and maintaining scalable big data solutions using Apache Spark, Scala, Hadoop, and Elasticsearch . The role involves collaborating with cross-functional teams to build efficient data processing pipelines and search applications.

Knowledge and experience in the Compliance / AML domain will be an added advantage.

Key Responsibilities

- Design, develop, and implement Spark/Scala applications and data processing pipelines for large volumes of structured and unstructured data.
- Implement data transformation, aggregation, enrichment, and computation processes to support analytics and machine learning initiatives.
- Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions.
- Integrate Elasticsearch with Spark for efficient data indexing, querying, and retrieval.
- Implement transformations and aggregations using Spark RDDs, DataFrames, Datasets, and Spark SQL .
- Develop scalable, reliable, and fault-tolerant Spark applications following industry best practices and coding standards.
- Optimize and tune Spark jobs and Elasticsearch queries to improve performance, scalability, and resource utilization.
- Monitor job performance, identify bottlenecks, troubleshoot issues, and implement appropriate optimizations.
- Troubleshoot and resolve issues related to data processing, data quality, Spark performance, and Elasticsearch integration .
- Ensure data quality, consistency, accuracy, and integrity throughout the data processing lifecycle.
- Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration technologies.
- Optimize data engineering workflows for containerized environments and efficient resource utilization.
- Collaborate with DevOps teams to streamline deployments and implement CI/CD pipelines .
- Implement data governance, data lineage, and metadata management practices to ensure data accuracy, traceability, and compliance.
- Implement monitoring and logging mechanisms to ensure the health, availability, and performance of data infrastructure.
- Monitor and optimize end-to-end data pipeline performance and implement required enhancements.
- Document data engineering processes, workflows, architecture, and infrastructure configurations for knowledge sharing and future reference.

Requirements

- Bachelor’s or Master’s degree in Computer Science, Software Engineering, Information Technology, or a related field .
- Quantexa Certified Data Engineer / Data Architect with hands-on experience and strong proficiency in the Quantexa platform.
- Proven experience as a Data Engineer working with Hadoop, Spark, and large-scale data processing technologies.
- Strong proficiency in Scala and familiarity with functional programming concepts.
- In-depth understanding of Apache Spark architecture, RDDs, DataFrames, Datasets, and Spark SQL .
- Strong expertise in Hadoop ecosystem technologies , including HDFS, Hive, Pig, and related tools.
- Hands-on experience with Elasticsearch , including data indexing, search applications, data modeling, indexing strategies, and query optimization.
- Experience with OpenShift Container Platform (OCP) and Kubernetes-based container orchestration.
- Strong programming skills in Scala, Python, Java, and/or Spark .
- Good understanding of DevOps practices, CI/CD pipelines, and infrastructure automation .
- Experience with tools such as Docker, Jenkins, Ansible, and Bitbucket .
- Experience with distributed computing, parallel processing, and large-scale datasets .
- Strong experience in performance tuning and optimization of Spark applications and Elasticsearch queries.
- Experience with Git and collaborative software development workflows.
- Strong analytical and problem-solving skills with the ability to troubleshoot complex technical issues.
- Excellent communication and collaboration skills with the ability to work effectively with cross-functional teams.
- Experience with Grafana, Prometheus, and Splunk will be an added advantage.
- Exposure to cloud platforms such as AWS, Azure, or GCP and their data services will be a plus.
- Knowledge or experience in the Compliance / AML domain will be an added advantage.