Notice: Immediate

Role overview

We are hiring an Azure Databricks Engineer with about 3 years of experience to build the data and ML backbone behind our agentic AI applications. You will turn enterprise data into governed, retrieval-ready knowledge and help ship LLM agents to production on Databricks.

Key responsibilities

Data engineering for AI

• Build batch and streaming pipelines with PySpark, Delta Lake, Delta Live Tables and Auto Loader on the medallion (bronze/silver/gold) pattern.
• Ingest structured and unstructured enterprise data (ERP, SharePoint, PDFs, email, contracts) and prepare it for LLM use: parsing, chunking, metadata enrichment and embeddings.
• Govern data, models, functions and vector indexes in Unity Catalog with lineage, access control and PII handling.

RAG and agent development

• Build and tune retrieval pipelines on Mosaic AI Vector Search, including hybrid search, re-ranking and index sync from Delta tables.
• Develop tool-calling agents with the Mosaic AI Agent Framework, LangChain/LangGraph or similar, exposing Unity Catalog functions and SQL as agent tools.
• Integrate foundation models via Databricks Model Serving, Azure OpenAI and external model endpoints.
• Support multi-agent workflows for finance, procurement and operations use cases (e.g. invoice validation, quotation comparison, contract retrieval).

LLMOps and production

• Track, version and register agents and models with MLflow; run evaluations with Mosaic AI Agent Evaluation (groundedness, relevance, safety) and LLM-as-judge.
• Deploy agents to Model Serving endpoints; monitor latency, cost and quality through inference tables and tracing.
• Automate delivery with Databricks Asset Bundles, Git integration and Azure DevOps or GitHub Actions.
• Optimise cluster, job and serving cost; apply guardrails, prompt-injection protection and data-leak controls.

Collaboration

• Work with solution architects and business analysts to turn BRDs into data and agent designs.
• Expose agents to business users through apps, APIs, Copilot Studio or Power Platform integrations.
• Document designs, runbooks and evaluation results for client handover.

Required skills and qualifications

• Bachelor’s degree in Computer Science, Engineering or a related field.
• About 3 years in data or ML engineering, with at least 1.5 years on Azure Databricks in production.
• Strong Python and PySpark; solid SQL and Spark performance tuning.
• Hands-on with Delta Lake, Unity Catalog, Databricks Workflows and MLflow.
• At least one delivered GenAI project: RAG, embeddings, vector search or an LLM agent in production or pilot.
• Working knowledge of an agent framework (Mosaic AI Agent Framework, LangChain/LangGraph, Semantic Kernel or AutoGen) and of prompt engineering and tool/function calling.
• Familiarity with Azure services: ADLS Gen2, Azure Data Factory, Key Vault, Azure OpenAI, Entra ID.
• Git, CI/CD and basic infrastructure-as-code (Terraform or Bicep).
• Clear written and verbal communication; comfortable in client-facing discussions.

Good to have

• Certifications: Databricks Certified Data Engineer Associate, Databricks Generative AI Engineer Associate, or Microsoft DP-203 / AI-102.
• Azure AI Foundry, Azure AI Search or Document Intelligence experience.
• Exposure to Microsoft Power Platform or Copilot Studio integrations.
• Domain exposure to finance, procurement, construction or HSE processes.
• Knowledge of LLM evaluation, guardrails and Responsible AI practices.
• Experience with Model Context Protocol (MCP) or agent-to-agent orchestration patterns.