Technical Functional Analyst (Data Engineer/Data Analyst)

Company Description

Virtual Calibre Group, a JAST Company, is listed on the Tokyo Stock Exchange Prime Market with operations across Japan, Malaysia, China, Thailand, Singapore, India & UAE. The company's revenue exceeded RM 1 billion in FY 2023. Virtual Calibre offers SAP, Data Engineering, and Software Engineering Consulting Services in the Asian and Middle East regions, catering to various industries with a flexible offshore delivery approach.

Role Description

This is a contract on-site role for a Technical Functional Analyst (Data Engineer/Data Analyst) at Virtual Calibre in WP. Kuala Lumpur. The role involves tasks such as functional analysis, defining functional requirements, applying analytical skills, conducting business analysis, and improving business processes.

Must-Have:

• Insurance or financial services experience (e.g., ACORD Messaging)
• Good working knowledge of Splunk and databases (e.g., DB2 and MySQL)
• Understand XML principles and can work with XML messages

Good-to-Have:

• Excellent analytic, investigation and technical problem-solving skills
• Excellent verbal and written communication skills, especially the ability to effectively communicate and engage with remote support teams
• Good practical workflow and scripting skills

Key criteria’s:

ETL*

medallion architecture; common data pipeline development using framework or meta data; how to handle data quality (lookup, check); data warehousing concepts ( star/snowflake; fact/dim, surrogate key / primary key / Slowly Changing Dimension (SCD) - SCD I&II difference, Normalization types ( 2NF vs 3 NF); how to identify Delta records/files

PySpark*

Spark architecture; create Data frame from collection of data; remove duplicate value from DF; PySpark vs TSQL difference including select, aggregate functions, union, limit, add new column into DF, filtering; Window functions ( lead, lag, string replace , substring index); JOINS, group by having count greater than/lower than; selecting data from multiple tables ; Case when with multiple conditions are ok; Performance optimization ( how to mitigate shuffle and data skew);

Hive*

managed vs external table ; create DDL for external table ; Change the settings within a Hive session; validate functions like Trim, Replace , concat , etc; How to establish JDBC connection ; What are the 3 primary complex Datatypes in Hive & difference

Azure Synapse*

azure Subscription/Service principal/ tenant ID; Azure Synapse Pipeline development, Azure Dedicated Pool SQL DDL/DML development , Azure Synapse Spark notebook development; ADLS,

Communication*

Crip & confident, Video without background animation, Validate pyspark or SQL exp using candidate notebook via screen sharing

Scheduler*

Oozie workflow for high level Orchestration & Scheduling , How/Where to create spark application and how to invoke on this, validate event-based triggers , re-run from failure point, notifications, and exception handling