Data Engineering Manager
Apply on
Data Engineering Manager (DE) plays a critical role in supporting our Azure Data and Databricks under our GDP environment. This role manages DE projects and day to day activities focusing on build, design and deployment of new data pipelines and integrating between source, data warehouse and business visualization layers. This role is a heavily focus technical role, with sound experience and expertized in both these technologies with the ability to load, transform and extract data for business intelligence/analytical activities. Working closely with our business stakeholders, you will also be expected to participate in the ground up implementation of a full DE framework that places data at its core of the business. The role will require supporting and leading other functions like Data Visualization, Vendor Management and Data Architecture to ensure seamless delivery and accuracy of data to drive decision making. Effective cross-functional collaboration skills are desired in the ideal candidates.
Job Scope:
1.Data Engineering
• Translate business functional demands into technical requirements with appropriate timelines and costs related to DE activities.
• Provide DE design, solution options and recommendation that meet requirements and consistent with company’s best practice.
• Support any DE issues, incident that might impact data loads, data transformation and data visualization.
• Provide and Implement design solutions to integration data sources into our GDP layer thru Azure integration tools. • Provide and Implement Databricks tiered layers to support transformation from Bronze to Gold.
2.Engineering Manager
• Lead and manage a team of skill data engineers to support DE activities.
• Support vendors and resourcing pool as per size and complexity from different DE projects.
• Ensure DE governance and strategy are in place and embedded as part of data quality checks, accuracy of data transformation, standardization of business measures and dimensions and KPIs.
3. Project Management
• Working with vendors and suppliers to key DE and BI projects.
• Ensure DE timelines and project management activities are track and align with other data activities from different data teams.
• Support our business on ensuring timely delivery of BI insights, reports and dashboards thru data accuracy, data availability and sound data pipelines.
• Project Planning & Management - Defined requirements and scope, developed project specifications and work plans, determined risk/contingency plans, and directed project planning and execution (as well as training programs for stakeholders.
Job Requirements:
1.Data Engineering and Datawarehouse Architecture
Collaborate with stakeholders to define the business layer and data transformation layer of the Data Model that is consistent with easy access to data dimensions and schemas.
Review the current Data Engineering architecture and recommend changes as appropriate.
Ensuring the product/solution architecture is scalable and fit for purpose withing the GDP layer Data Engineering
2.Governance Solution Design
Leading in design, specification and selection of Data solutions with specific focus on functionality, data, security, integration, infrastructure and performance.
Must have sound understanding for all Big Data components & Data Warehousing concepts, preferably in Azure Databricks, Azure Data Factory, SQL and non-SQL technologies like Snowflake.
Analyse requirements and architect appropriate solutions, with detailed design documents.
3.Business Partnering
Strong understanding of Data Quality, Metadata Management, Master Data Management concepts.
Good understanding of privacy laws and regulations, especially on areas 1st party data.
Good understanding of financial laws around the Data Security, Compliance and other related regulations.
Able to work under tight deadlines and dynamic working environment.
Education:
You should have a degree in IT or Data Science, preferably you experience leading and managing a Data Team in a MNC organization.
Experience:
20 to 15+ years of Experience with one or more Big Data or Analytics Products and Services: Hadoop, Spark, Hive, Azure Databricks.
12+ years in Data Warehouse
12+ years in Data Modelling
10+ years in BI project management
Good knowledge of EPR, B2B , D2C and Sales systems and reporting.
Exposure to AI and Machine Learning.
Data Engineering – Data Factory / Databricks (PySpark, SQL)/ Pandas / Numpy / MXNet / TensorFlow).
Deployment - Azure Data Factory / Apache Airflow
AI & ML – CNN, Federated Learning, Poisoning attacks and defenses
Programming Languages – Python / SQL / Java
Experience in utilizing (Power BI, SAP BI/BW and COGNOS, SQL).
Data Engineering Manager (DE) plays a critical role in supporting our Azure Data and Databricks under our GDP environment. This role manages DE projects and day to day activities focusing on build, design and deployment of new data pipelines and integrating between source, data warehouse and business visualization layers. This role is a heavily focus technical role, with sound experience and expertized in both these technologies with the ability to load, transform and extract data for business intelligence/analytical activities. Working closely with our business stakeholders, you will also be expected to participate in the ground up implementation of a full DE framework that places data at its core of the business. The role will require supporting and leading other functions like Data Visualization, Vendor Management and Data Architecture to ensure seamless delivery and accuracy of data to drive decision making. Effective cross-functional collaboration skills are desired in the ideal candidates.
Job Scope:
1.Data Engineering
• Translate business functional demands into technical requirements with appropriate timelines and costs related to DE activities.
• Provide DE design, solution options and recommendation that meet requirements and consistent with company’s best practice.
• Support any DE issues, incident that might impact data loads, data transformation and data visualization.
• Provide and Implement design solutions to integration data sources into our GDP layer thru Azure integration tools. • Provide and Implement Databricks tiered layers to support transformation from Bronze to Gold.
2.Engineering Manager
• Lead and manage a team of skill data engineers to support DE activities.
• Support vendors and resourcing pool as per size and complexity from different DE projects.
• Ensure DE governance and strategy are in place and embedded as part of data quality checks, accuracy of data transformation, standardization of business measures and dimensions and KPIs.
3. Project Management
• Working with vendors and suppliers to key DE and BI projects.
• Ensure DE timelines and project management activities are track and align with other data activities from different data teams.
• Support our business on ensuring timely delivery of BI insights, reports and dashboards thru data accuracy, data availability and sound data pipelines.
• Project Planning & Management - Defined requirements and scope, developed project specifications and work plans, determined risk/contingency plans, and directed project planning and execution (as well as training programs for stakeholders.
Job Requirements:
1.Data Engineering and Datawarehouse Architecture
▪ Collaborate with stakeholders to define the business layer and data transformation layer of the Data Model that is consistent with easy access to data dimensions and schemas.
▪ Review the current Data Engineering architecture and recommend changes as appropriate.
▪ Ensuring the product/solution architecture is scalable and fit for purpose withing the GDP layer Data Engineering
2.Governance Solution Design
▪ Leading in design, specification and selection of Data solutions with specific focus on functionality, data, security, integration, infrastructure and performance.
▪ Must have sound understanding for all Big Data components & Data Warehousing concepts, preferably in Azure Databricks, Azure Data Factory, SQL and non-SQL technologies like Snowflake.
▪ Analyse requirements and architect appropriate solutions, with detailed design documents.
3.Business Partnering
▪ Strong understanding of Data Quality, Metadata Management, Master Data Management concepts.
▪ Good understanding of privacy laws and regulations, especially on areas 1st party data.
▪ Good understanding of financial laws around the Data Security, Compliance and other related regulations.
▪ Able to work under tight deadlines and dynamic working environment.
Education:
▪ You should have a degree in IT or Data Science, preferably you experience leading and managing a Data Team in a MNC organization.
Experience:
▪ 20 to 15+ years of Experience with one or more Big Data or Analytics Products and Services: Hadoop, Spark, Hive, Azure Databricks.
▪ 12+ years in Data Warehouse
▪ 12+ years in Data Modelling
▪ 10+ years in BI project management
▪ Good knowledge of EPR, B2B , D2C and Sales systems and reporting.
▪ Exposure to AI and Machine Learning.
▪ Data Engineering – Data Factory / Databricks (PySpark, SQL)/ Pandas / Numpy / MXNet / TensorFlow).
▪ Deployment - Azure Data Factory / Apache Airflow
▪ AI & ML – CNN, Federated Learning, Poisoning attacks and defenses
▪ Programming Languages – Python / SQL / Java
▪ Experience in utilizing (Power BI, SAP BI/BW and COGNOS, SQL).
