AI Engineer

AI Developer – Azure Databricks, Python, Java, Microservices & AI (LLM/Copilot or Equivalent )

---

Job Summary

We are looking for a AI Developer with strong expertise in Python , Azure Databricks (ADB) , Delta Lake , and AI/LLM technologies to design and build Databricks jobs, reusable scripts, and cloud-native microservices . The role requires hands-on experience with Copilot agentic mode (or equivalent) , building LLM-driven AI use cases , and the ability to work independently by taking full ownership of deliverables without the need for micromanagement .

---

Roles and Responsibilities

Azure Databricks & Data Engineering

- Design, develop, and manage Azure Databricks jobs , workflows, and notebooks.
- Build reusable Python/PySpark scripts for data ingestion, transformation, and analytics.
- Implement Delta Lake–based lakehouse architectures with ACID transactions and versioning.
- Optimize Databricks workloads for performance, scalability, and cost efficiency .
- Implement batch and streaming pipelines using Databricks.

Microservices & Application Development

- Develop Python-based microservices to expose data and AI capabilities via APIs.
- Build and integrate RESTful services for data access, AI inference, and orchestration.
- Ensure secure, scalable, and resilient service design using cloud-native patterns.
- Drive development tasks independently, from design through deployment, with minimal supervision.

AI, LLMs & Copilot Agentic Mode

- Design and build LLM-powered AI use cases such as copilots, intelligent assistants, and automation agents.
- Implement Copilot agentic mode workflows, including tool calling, task planning, and multi-step reasoning.
- Develop prompt engineering strategies , system prompts, and guardrails for enterprise use.
- Integrate LLMs with Databricks, data lakes, and microservices for real-time and batch inference.
- Build RAG (Retrieval-Augmented Generation) pipelines using enterprise data.
- Evaluate, fine-tune, and monitor LLM performance, accuracy, and cost.

Architecture, Governance & Best Practices

- Contribute to solution architecture and technical design decisions.
- Ensure data security, governance, and compliance in data and AI solutions.
- Implement logging, monitoring, and observability for Databricks jobs and AI services.
- Perform code reviews and enforce engineering best practices.

Leadership & Collaboration

- Mentor peer developers and provide technical guidance when needed.
- Collaborate effectively with data scientists, architects, DevOps teams, and business stakeholders.
- Clearly communicate technical concepts, progress, and risks through excellent verbal and written communication skills .
- Take full ownership of assigned workstreams and drive them to completion without requiring micromanagement.

---

Required Technologies & Skills

Programming & Frameworks

- Python (advanced)
- PySpark / Spark SQL
- REST APIs and microservices frameworks (FastAPI, Flask, etc.)

Azure & Data Platforms

- Azure Databricks (ADB)
- Delta Lake
- Azure Data Lake Storage (ADLS Gen2)
- Azure cloud services and security fundamentals

AI / LLM & Copilot

- Hands-on experience with LLMs (Azure OpenAI, OpenAI, or similar)
- Experience building Copilot solutions and agentic workflows
- Prompt engineering and orchestration frameworks (LangChain, Semantic Kernel, etc.)
- Experience with RAG architectures , vector databases, and embeddings
- Understanding of AI safety, guardrails, and responsible AI practices

DevOps & Engineering

- CI/CD pipelines (Azure DevOps, GitHub Actions)
- Git version control
- Containerization (Docker preferred)
- Infrastructure-as-code concepts (Terraform/Bicep – preferred)

---

Preferred Qualifications

- 8 to 10 years of experience in Python, data engineering, or backend development
- Proven experience delivering production-grade AI/LLM solutions
- Experience in enterprise-scale cloud environments
- Exposure to streaming platforms (Kafka, Event Hubs – preferred)

---

Soft Skills

- Excellent verbal and written communication skills
- Strong sense of ownership and accountability
- Ability to work independently and proactively drive outcomes
- Strong problem-solving and analytical skills
- Collaborative mindset with leadership capabilities