About this role
We are seeking a Senior Data Engineer with AI/GenAI expertise to join the Data Platform (Foundry) team. This role will focus on building and optimizing enterprise-scale data pipelines while also enabling metadata-driven, AI-driven use cases, including copilots, LLM integrations, and intelligent data workflows using Databricks, Azure, ADF, Azure SQL, Astronomer/Airflow or Control M
Detailed Description:
- Design, build, and maintain data pipelines using Databricks, Spark, ADF, Advanced SQL/Python, and Azure services
- Develop and optimize bronze → silver → gold transformations
- Schedule and orchestrate via ADF, Airflow, Control M. Airflow experience would be a huge plus
- Enable AI/GenAI use cases across the platform, including:
Copilot integrations
- LLM-based workflows
- Intelligent data enrichment pipelines
- Work with Databricks GenAI capabilities .
- Integrate external AI platforms (e.g., Anthropic, Azure OpenAI ) into enterprise data workflows
- Ensure data reliability, performance, and operational stability
- Troubleshoot and resolve production issues with strong SLAs
- Support ingestion using tools such as:
Fivetran / ADF / Auto Loader
- Implement data quality, monitoring, and reconciliation frameworks
- Ability to read/write code in SQL, Python, other
- Collaborate across:
Data Engineering teams
- Data Platform engineering
- Business stakeholders
- Contribute to platform modernization and AI adoption strategy
- Follow governance best practices:
Code quality
- Deployment controls
- Audit and compliance
- Required Skills & Experience
- 8–10+ years in Data Engineering
- Strong experience with:
Databricks (Delta Lake, Spark)
- Azure Data Platform - ADF, Synapse
- SQL - Azure SQL, SQL Server, Teradata, DB2, Oracle
- Hands-on experience with AI/GenAI technologies , including:
Building or integrating copilot-style experiences
- Working with LLMs (e.g., Anthropic Claude, OpenAI models)
- Strong programming skills:
Python, SQL
- Orchestration using — Airflow or Astronomer, Control M. Preferably Airflow
- Experience building scalable ETL/ELT pipelines
- Familiarity with production support and incident management
#LI-KS1