Apply on
Micro Lending is an emerging business arena where the focus is on giving micro loan to Small Medium Enterprise (SME) Business owner; it aims to bring financial inclusion in a mass level as well contribute to the overall ecosystem of driving business and economies. The potential is vast for this industry with a mass demand ready while the last mile connectivity remains a key challenge in terms of unique market dynamics (e.g., Regulator, competition, credit trends) for each of the markets.
We are looking for an experienced Data Engineer who will take a key role on our team.
The ideal candidate must have knowledge in engineering best practices, data management fundamentals, data storage principles, and be current on recent advances in distributed systems as it pertains to data storage and computing. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. This is an opportunity to make a significant impact in a fast-paced, startup-like environment in a great company.
Responsibilities:
You will be responsible for designing, building, and maintaining an efficient, extensible, and scalable data pipelines for processing high-volume data. This position requires collaboration with DevOps engineers, machine learning engineers, product managers, and technical partner teams.
Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader. Assemble large, complex data sets that meet functional / non-functional business requirements. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies. Work with data and analytics experts to strive for greater functionality in our data systems. Assist in the integration of technical and application components to meet business requirements. Follow defined methodology and standards including preparation and maintenance of documentation for all stages of development work.
Qualifications:
- At least 3 years of experience building, operating and optimizing distributed, large-scale data storage and analysis solutions using Amazon Web Services (e.g., S3, EC2, EMR, Spark, Redis, DynamoDB, Athena, Glacier)
- Experience with object-oriented/object function scripting languages.
- Preferred: Python and Bash Shell.
- Experience in ETL pipelining and data warehousing and Proven record of accomplishment of processing unstructured and/or complex semi-structured data streams and repositories.
- Required Experience on file processing: CSV, Json, Excel, XML
- ETL Tools: Apache Nifi or Talend
- AWS services: Glue, S3, RDS
- Strong understanding of computer science concepts, object-oriented design principles
- Knowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations will be ab added advantage.
- Experience with data warehouse design, build, deploy along with building real-time stream-processing systems, preferably using open-source solutions
- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases including NoSQL databases.
- Preferred: MySQL, PostgreSQL, MS SQL, Presto
- Experience with big data tools: Spark, HDFS, AWS Redshift, AWS S3, AWS Glue, AWS Athena, etc.
Education:
- A bachelor's degree, or equivalent experience in Computer Science/Engineering/IT or similar field, including
- Expertise in fundamentals of data structures
- Expertise in fundamentals of algorithm design, problem solving, and complexity analysis
