Data Engineer Team Lead

We have a great opportunity as a Data Engineer Team Lead for an international award-winning online trading platform. The company has presence in 15 countries with 20 offices worldwide. They employ over 1,300 employees consisting of over 80 nationalities.

You will be based at the Malaysian head office in Cyberjaya. This role is open for both Malaysians and foreigners provided you have the relevant experience. The relevant experience and knowledge are necessary as you will be required to sit for an assessment before proceeding for the interview.

Responsibilities :

As the Data Engineer Team Lead, you will play a crucial role in overseeing the design, development, and optimization of our data pipelines, ensuring seamless integration and availability of data across all systems. You will lead a talented team of data engineers and spearhead the execution of data strategies that are integral to supporting critical business decisions and operations. A significant responsibility is to develop a unified data pipeline that effectively serves both internal systems and external applications, ensuring secure and real-time data flow across all platforms.

Team Leadership: Lead and mentor a team of data engineers to design, build, and optimize scalable data pipelines for analytics, reporting, and operational needs.

Unified Data Pipeline Management: Create and manage a unified data pipeline that integrates multiple data sources, facilitating efficient access for both internal and external systems, including third-party tools and client-facing platforms.

Utilization of Low-Code/No-Code Tools: Leverage low-code/no-code integration tools such as Fivetran or RudderStack to accelerate data workflows and improve integration efficiency.

Cross-Functional Collaboration: Work closely with various teams to identify data needs and ensure smooth data flows between systems, enhancing high availability and integration.

Data Governance and Quality Assurance: Implement and manage data governance and quality standards, ensuring data consistency and reliability throughout the organization.

Real-Time and Batch Data Architecture: Architect and oversee the implementation of real-time and batch data pipelines using data streaming tools like Kafka or Pub/Sub, ensuring low-latency, high-performance data processing.

Technical Guidance and Support: Provide technical guidance to your team, aiding in troubleshooting complex data issues and optimizing performance and cost-effectiveness while adhering to best practices.

Continuous Improvement: Stay informed on the latest in data engineering technologies and best practices, especially concerning real-time processing and low-code/no-code integrations, to enhance our data infrastructure continually.

Stakeholder Coordination: Communicate with stakeholders to ensure timely delivery of data solutions that effectively meet business requirements.

Requirements :

Education: Bachelor's degree in Computer Science, Data Engineering, or a related field.

Experience: 10+ years in data engineering, with at least 3+ years in a management or leadership role.

Expertise in designing, building, and maintaining data pipelines and data warehouses.

Technical Skills :

Strong knowledge of cloud platforms (Google Cloud or AWS) and familiarity with tools like BigQuery, PostgreSQL, and Airflow.

Proficient in programming languages such as Python, with experience in ETL/ELT processes.

Hands-on experience with low-code/no-code data integration tools.

Deep understanding of real-time data processing and streaming technologies such as Kafka or Pub/Sub.

Understanding of Data Frameworks: Experience in building unified data pipelines for secure and consistent data flows, working with both real-time and batch processing systems.

Governance Knowledge: Strong understanding of data governance, security, and compliance best practices.

Communication Skills: Excellent communication abilities with a talent for explaining complex technical concepts to non-technical audiences and mentoring junior team members.

Job Type: Permanent

Pay: RM15,000.00 - RM20,000.00 per month

Benefits:

• Free parking
• Health insurance
• Maternity leave
• Opportunities for promotion
• Parental leave
• Professional development
• Work from home

Schedule:

• Monday to Friday

Application Question(s):

• Are you familiar with tools like Big Query, PostgreSQL or Airflow?
• Are you proficient in programming language such as Python?
• Do you have expertise in designing, building and maintaining data pipelines and data warehouses?
• Do you have knowledge of cloud platforms like Google Cloud or AWS?

Experience:

• leadership/management: 3 years (Required)
• data engineering: 10 years (Required)