Job Description:
The Technical Lead Databricks Engineering will own the architecture, development, and operational excellence of the Databricks platforms, including integration with Microsoft Fabric, Purview, Semarchy (MDM), and Power BI. This role requires a blend of visionary leadership and hands-on engineering capability to design scalable, secure, and high-performing data ecosystems that power analytics, operational reporting, and data-driven innovation.
The ideal candidate will partner closely with business and technology leaders to enable end-to-end data delivery — from ingestion and mastering to visualization — while ensuring governance, compliance, and performance at scale.
Key Responsibilities:Strategic & Technical Leadership
• Define and execute the enterprise data platform strategy across Databricks, Microsoft Fabric, Semarchy, and Power BI.
• Align platform architecture with business priorities, data governance policies, and future scalability needs.
• Serve as the subject matter expert for platform capabilities, integration patterns, and best practices. Platform Architecture & Management
• Lead design, implementation, and optimization of the Databricks Lakehouse for advanced analytics, data science, and AI workloads.
• Support Power BI environment architecture, including catalog implementation and governance, semantic model design, performance optimization, and enterprise reporting integration.
Data Engineering & Delivery
• Direct the development of robust ingestion pipelines, data transformations, and orchestration workflows.
• Ensure data pipelines and models meet SLA requirements for performance, accuracy, and reliability.
• Implement CI/CD processes for data platform components, including version control and automated testing. Governance & Compliance
• Enforce security, privacy, and compliance standards across all platforms (e.g., CCPA, HIPAA, CFIUS, etc.).
• Establish and monitor platform usage policies, metadata management, and audit logging.
• Collaborate with the Data Governance team to maintain data cataloging and lineage tracking.
Technical Leadership & Development
• Foster cross-functional collaboration between engineering, analytics, governance, and business stakeholders.
• Develop training and upskilling plans to maximize platform adoption, federation, and team expertise. Innovation & Continuous Improvement
• Evaluate and recommend emerging tools, features, and architectural enhancements.
• Drive automation and efficiency improvements across data ingestion, mastering, and reporting workflows.
• Promote reusability, scalability, and reliability in all platform engineering work.
Qualifications:Education & Experience
• Bachelor’s or Master’s in Computer Science, Data Engineering, Information Systems, or related field.
• 10+ years of experience in data engineering and platform leadership roles.
• Proven hands-on expertise with Databricks, Microsoft Fabric, and Power BI in an enterprise setting.
• Experience designing and managing large-scale, multi-domain data architectures.
Desired Skills:• Expert knowledge of Databricks SQL, Delta Lake, Spark, and data pipeline orchestration.
• Deep understanding of Microsoft Fabric components (Data Factory, Data Warehouse, OneLake, Lakehouse).
• Strong Semarchy MDM skills, including data modeling, workflow design, matching rules, and stewardship processes.
• Advanced Power BI DAX, data modeling, and performance tuning skills.
• Proficiency in SQL, Python, and cloud data services (Azure preferred).
Leadership & Soft Skills
• Exceptional leadership, stakeholder management, and communication skills.
• Ability to translate business needs into technical solutions.
• Strong problem-solving and decision-making abilities.
Success/Performance Metrics:
• Platform uptime, scalability, and performance SLAs met.
• Reduction in data duplication, inconsistencies, and quality issues.
• Timely delivery of data pipelines and analytics solutions.
• Adoption and satisfaction rates of data platforms across the organization. • Team skill growth and retention.
