We are seeking a Data Engineering Manager to lead the development of scalable data pipelines, ETL workflows, and analytics solutions while driving the migration of on-premises Microsoft SQL Server databases to Azure. This role is responsible for ensuring efficient data ingestion, transformation, and storage while leveraging Apache Kafka for real-time data streaming and event-driven architectures.
The ideal candidate has strong expertise in Azure cloud data services, Microsoft SQL Server, and data engineering best practices, with experience leading high-performing teams and managing cloud migration projects
Role and responsibilities
Data Engineering & Pipeline Development
Design, build, and maintain scalable ETL and data pipelines using Azure Data Factory and Databricks.
Develop real-time and batch data processing workflows for analytics and reporting.
Ensure data integrity, security, and governance across all pipelines.
Implement event-driven architectures with Apache Kafka and Azure Event Hubs.
On-Premises to Azure Migration
Lead the migration of Microsoft SQL Server databases to Azure SQL Database and Synapse Analytics.
Define and implement Azure-based data architecture for performance, scalability, and cost efficiency.
Optimize data storage, indexing, and partitioning strategies in Azure.
Work closely with DBA, DevOps, and security teams to ensure a seamless migration with minimal downtime.
Data Architecture & Infrastructure
Oversee the development of a modern Azure data platform (data lake, data warehouse, and real-time processing).
Manage and optimize Kafka-based streaming architectures using Azure Event Hubs.
Implement data governance, security policies, and compliance frameworks (BNM RMiT, GDPR).
Monitor and enhance data performance, availability, and reliability.
Data Analytics & Business Intelligence Enablement
Work with data analysts and scientists to develop self-service analytics and reporting.
Design efficient data models to support business intelligence (BI) using Azure Synapse Analytics and Power BI.
Enable seamless integration between operational and analytical data sources.
Team Leadership & Collaboration
Lead and mentor a team of data engineers, fostering best practices in Azure data engineering.
Promote Agile methodologies, DevOps, and CI/CD pipelines for data workflows.
Collaborate with engineering, product, and business teams to define data objectives.
Performance Monitoring & Continuous Improvement
Monitor and troubleshoot Azure data pipelines, ETL processes, and Kafka streams.
Identify opportunities to automate and improve efficiency.
Stay up to date with Azure cloud, big data, and streaming technologies.