Senior Data Engineer OpenSearch

Your primary responsibilities will include: Design and Develop OpenSearch Solutions: Design, implement, and maintain scalable OpenSearch clusters and search solutions for large-scale structured, unstructured, and vector-based data. Implement Vector Search: Develop and optimize vector search solutions using approximate nearest neighbor (ANN) techniques, with particular focus on HNSW and filtered vector search. Optimize Search Relevance: Analyze semantic and hybrid search quality and performance, including chunking strategies, embeddings, filtering, scoring, ranking, and retrieval strategies, and implement improvements based on search evaluation results. Optimize Queries and Search Performance: Profile slow or resource-intensive queries, identify bottlenecks, and rewrite queries to improve latency and throughput. Favor efficient filtering and caching strategies where appropriate over expensive scoring operations. Tune OpenSearch Clusters: Right-size shards and replicas, balance workloads across nodes, tune indexing and search performance, and identify appropriate data-node instance types based on workload characteristics and search patterns. Manage Index Lifecycle: Design and maintain appropriate index lifecycle strategies, including Index State Management (ISM)/ILM concepts, rollover, retention, shard management, and data tiering, to support sustained production performance. Monitor and Troubleshoot Production Platforms: Analyze cluster health, node utilization, JVM/memory pressure, CPU, disk usage, indexing throughput, search latency, and query behavior to identify and resolve production issues. Improve Platform Efficiency: Continuously optimize OpenSearch infrastructure for performance, scalability, reliability, and cost efficiency. As a Senior OpenSearch Engineer specializing in large-scale search and vector search platforms, you will design, develop, optimize, and operate high-performance OpenSearch solutions supporting semantic, hybrid, and traditional search use cases. The role requires strong hands-on experience with vector search, approximate nearest neighbor (ANN) algorithms, search relevance, query optimization, and OpenSearch cluster performance tuning. You will work across the full search lifecycle, from index and data modeling through relevance optimization, scaling, performance troubleshooting, and production operations. Strong hands-on experience with OpenSearch and/or Elasticsearch in production environments. Proven experience designing and operating large-scale OpenSearch clusters, preferably with 100M+ vectors and 200GB+ of data. Strong understanding of vector search and approximate nearest neighbor (ANN) algorithms. Practical experience with HNSW-based vector search and associated index configuration and performance tuning. Experience implementing and optimizing filtered vector search. Strong understanding of semantic search, hybrid search, relevance, ranking, scoring, and retrieval strategies. Experience evaluating and improving search relevance and search quality, including the impact of chunking strategies, embeddings, filtering, and scoring. Strong knowledge of OpenSearch query DSL and the ability to profile, analyze, and optimize slow or resource-intensive queries. Experience with query performance optimization, caching strategies, filtering, scoring, and reducing search latency. Strong understanding of sharding, replicas, shard sizing, node allocation, cluster topology, and workload balancing. Experience tuning OpenSearch clusters based on CPU, memory, JVM, disk I/O, indexing throughput, search throughput, and latency characteristics. Experience with index lifecycle management, rollover, retention, and index/data lifecycle strategies. Experience with OpenSearch Serverless and/or AWS-managed OpenSearch. Experience with AWS services such as Amazon OpenSearch Service, Amazon S3, AWS Lambda, AWS Glue, Amazon Bedrock, or related cloud services. Experience building RAG (Retrieval-Augmented Generation) solutions using OpenSearch as a retrieval/vector search layer. Experience working with embedding models, vector databases, semantic retrieval, and LLM-based search applications. Experience with hybrid search architectures, combining lexical/BM25 search with vector/semantic search. Knowledge of search relevance evaluation methodologies and experience defining search quality metrics. Experience with Python for search evaluation, performance analysis, automation, and data processing. Experience with OpenSearch Dashboards, monitoring, observability, and operational tooling. Experience with Terraform or other Infrastructure as Code technologies. Experience with CI/CD, Git-based development workflows, automated deployments, and DevOps practices. Experience designing search platforms for high availability, fault tolerance, and horizontal scalability. Experience working with large-scale AI/ML workloads and production GenAI applications. Romania Data & Analytics Hybrid Professional Bucharest, RO (0112) IBM Romania Srl