Epicareer Might not Working Properly
Learn More

Data Engineer

Salary undisclosed

Checking job availability...

Original
Simplified

The Data Engineer role will be responsible for maintaining and further developing DigiHaul's data platform. This will include managing existing ETL processes, building new data pipelines, troubleshooting and resolving issues when required and supporting scaling our data infrastructure to support analytics and data science initiatives.

Main Accountabilities

  • Develop and Maintain Data Solutions: Support data lake and warehouse systems for efficient data collection, storage, processing, and transformation.
  • Assist in Building and Optimizing Data Pipelines: Help design and develop reliable ETL/ELT pipelines for automated data ingestion and reporting.
  • Monitor and Troubleshoot System Performance: Monitor cloud systems, identify issues, and resolve them with senior team support.
  • Support Data Infrastructure and Scalability: Assist in scaling and optimizing data infrastructure, refining data models, and ensuring performance best practices.
  • Participate in Learning and Development: Learn new tools and best practices, seeking guidance to improve skills and contribute to team goals.
  • Manage Workload and Updates: Prioritize tasks, meet deadlines, and provide clear updates during team meetings.
  • Collaborate and Communicate Effectively: Work with cross teams to deliver data solutions and provide progress updates.

Desirable Experiences and Skills

  • Minimum 5 years of relevant work experiences.
  • Bachelor's degree in Computer Science (or related field).
  • Minimum 3 years of experience with Azure Data Stack, Azure Functions, Azure Storage, AKS, and Snowflake for data storage, processing, and analytics.
  • Minimum 3 years of experience with SQL, Python or R for data manipulation, analysis, and scripting.
  • Minimum 3 years of hands-on experience in designing, building, and optimizing ETL and ELT pipelines using Airflow, Power BI, and APIs for automated data transformation and reporting.
  • Minimum 2 years of experience with Orchestration Airflow
  • Familiarity with GitHub for version control, and Slack for team communication and alerts.
  • Experience working with cloud platforms, managing data lakes, warehouses, and storage solutions, specifically within Azure and Snowflake environments.
  • Familiarity with AKS and real-time streaming technologies such as Apache Kafka would be plus.
  • Strong troubleshooting skills to identify and resolve issues in cloud-based systems and data pipelines.
  • Experience collaborating in cross-functional teams, using Notion for documentation and sprint planning, and ZOHO for ticket management.