About this role
Role summary: The Data Engineer builds the data foundation that AI agents, scorecards, and dashboards run on: ingestion pipelines, connectors, metadata and search indexes, vector stores, data marts, and quality instrumentation.
Experience: 3 to 6 years in data engineering. Senior Data Engineer: 2+ years, including ownership of data foundation architecture and connector frameworks.
### Key responsibilities
- Design and build ingestion pipelines for structured data, unstructured content (documents, PDFs), and metadata.
- Build reusable connectors to enterprise systems, catalogs, content repositories, and third-party or licensed sources via APIs.
- Model and build raw-to-mart data layers that serve analytics and AI use cases.
- Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG.
- Set up and manage vector databases and knowledge repositories used by LLM agents.
- Implement data quality rules, profiling, scoring outputs, and exception handling.
- Register lineage and maintain source registries, version tracking, and refresh controls.
- Design data stores for signals, findings, audit trails, and user feedback loops.
- Apply security and governance controls: RBAC, PII/sensitivity flagging, and approved data handling.
- Support SIT/UAT data validation, defect fixes, and production release activities.
Required skills
- Strong Python and SQL; solid data modeling (dimensional and normalized).
- Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services.
- Pipeline orchestration and transformation (Spark, Airflow, ADF, dbt, or similar).
- API-based integration (REST), JSON handling, and incremental/CDC ingestion patterns.
- Data quality frameworks and testing practices for pipelines.
- Version control (Git) and CI/CD for data workloads.
Preferred skills
- RAG data preparation: chunking, embeddings, vector databases (Azure AI Search, pgvector, Pinecone, or similar).
- Unstructured content processing: text extraction, OCR, document parsing.
- Metadata management, data catalogs, ontologies, or knowledge graphs (for example Neptune or other graph databases).
- Experience supporting LLM or agentic applications with grounded, traceable data.
- Life sciences data exposure (commercial, medical, regulatory, or launch data) and regulated-data handling.
- Cloud certification (Azure Data Engineer, Databricks, AWS, or Snowflake).