Software Architect - RAG Based Agentic AI Platform

At IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world. Navi is IBM/HashiCorp's agentic AI platform, spanning multiple product pillars (Assistant, Insight, GTM, with a fourth in development) and handling tens of thousands of queries per month for field, customer-facing, and go-to-market teams. We're looking for a Band 10 AI Architect to serve as the end-to-end technical co-owner of Navi's architecture — someone equally comfortable designing an intuitive user experience, architecting agentic reasoning flows, engineering the underlying data pipelines, and making platform-level infrastructure decisions. This is a full-stack AI architecture role: you won't hand off between specialists — you'll be expected to move fluently across every layer of the stack, from UI/UX through to deployment infrastructure, and to write production code yourself where needed. Product ownership across Navi is currently divided among a small group of architects, each with deep context in different areas of the platform. This role requires someone who can operate independently while also actively collaborating with those architects — brainstorming design options, reconciling different perspectives, and converging on the best overall solution rather than optimizing for their own area in isolation. What You'll be working on UI/UX Layer •      Architect intuitive, trustworthy interfaces for surfacing AI-generated answers, confidence signals, and source citations to non-technical business users •      Design interaction patterns that make agentic reasoning transparent (e.g., how confidence scoring, source grounding, and escalation paths are presented to end users) Agentic AI Layer •      Design and evolve multi-agent orchestration and reasoning flows (current stack includes Flowise and Langflow), including tool use, retrieval strategies, and prompt/system-message architecture •      Own the model-grounding strategy — ensuring answers are anchored in vetted, governed content sources rather than open-ended generation •      Design and refine confidence-scoring mechanisms, including evaluating and improving on current LLM-self-assessed scoring approaches •      Lead exploration and adoption of MCP (Model Context Protocol) to improve system portability and support broader enterprise integration •      Integrate agentic workflows with enterprise orchestration platforms (e.g., watsonx Orchestrate) Data Layer •      Architect data ingestion pipelines from diverse, mixed-reliability sources (formal validated designs, structured GTM/CRM data, informal internal knowledge) •      Design and improve retrieval infrastructure (vector search/RAG, e.g. Pinecone), including retrieval quality, chunking strategy, and reducing retrieval-driven answer variance •      Partner with data science on predictive modeling initiatives (e.g., churn prediction), including data readiness, feature pipelines, and drift monitoring •      Own data governance and content-scoring frameworks that rank source reliability and trigger low-confidence alerts to users Platform Layer •      Make infrastructure and re-platforming decisions as Navi scales across IBM business units, including cloud environment strategy (e.g., Azure to Cirrus transitions) •      Architect for reliability, determinism, and reproducibility in LLM-based systems (e.g., addressing answer inconsistency through temperature control, retrieval stability, and regression testing) •      Design for compliance, security, and governance requirements specific to enterprise AI deployment at IBM scale •      Evaluate and integrate new AI infrastructure/tooling as the ecosystem evolves What You'll Improve Beyond maintaining what exists, you're expected to actively advance Navi's current capabilities, including: •      Reducing answer inconsistency and improving reproducibility across agentic responses •      Strengthening confidence-scoring rigor beyond current LLM self-assessment approaches •      Improving text-to-SQL / structured-query reliability for analytics-style queries •      Advancing the re-platforming and MCP-adoption effort to support enterprise-wide governance and portability goals •      Building more robust content governance and feedback loops with content owners and subject-matter experts 20+ years of software/AI/data science engineering experience, including significant hands-on architecture experience Demonstrated ability to design and ship production AI applications end-to-end — not just prototype-level work Deep, practical expertise in LLM application architecture: RAG/retrieval systems, agentic/multi-agent frameworks, prompt and system-message design, and orchestration tooling Strong data engineering background: pipeline design, vector databases, data quality/governance frameworks Experience with cloud infrastructure and platform architecture at enterprise scale Proficiency across the full stack — able to write production code in UI, backend, and data/ML layers as needed, not purely a "whiteboard architect" Strong communication skills — able to represent architecture decisions to both engineering teams and executive stakeholders Proven ability to collaborate effectively with other architects on a shared platform — brainstorming openly, reconciling differing viewpoints, and building consensus around the best overall solution rather than working in isolation Experience with Flowise, Langflow, or comparable low-code/agentic orchestration tools Familiarity with Model Context Protocol (MCP) or similar interoperability standards Experience with enterprise orchestration platforms (e.g., watsonx Orchestrate) Background in applying AI to GTM, customer success, or sales analytics use cases Prior experience navigating large-enterprise governance, compliance, and re-platforming initiatives Canada Software Engineering Remote Professional Markham, CA (0026) IBM Canada Limited - IBM Canada Limitee