Data Engineer (Hadoop)
Location: Singapore
Overview:
We are seeking a highly skilled Data Engineer with expertise in Hadoop to join our data engineering team. The ideal candidate will have a strong background in big data technologies, data processing frameworks, and a proven ability to handle large-scale data pipelines. This role requires a proactive individual who can design, implement, and maintain efficient data solutions that drive the company's data infrastructure forward.
Key Responsibilities:
- Design and Development:
- Develop and manage robust data pipelines and architectures to process and analyze large datasets.
- Utilize Hadoop technologies (HDFS, MapReduce, Hive, Pig, etc.) for efficient data storage and computation.
- Build scalable and reusable code and libraries for data processing.
- Data Integration and Management:
- Integrate diverse data sources and manage data flow.
- Work with relational and non-relational databases to ensure data consistency.
- Optimize Hadoop jobs to run efficiently and minimize costs.
- Collaboration:
- Work closely with data scientists, analysts, and business stakeholders to ensure data meets the required standards.
- Collaborate with other engineering teams to align on project goals and timelines.
- Performance Optimization:
- Monitor and troubleshoot data pipeline issues and optimize performance for faster processing.
- Implement best practices for data security and governance.
- Documentation and Reporting:
- Maintain detailed documentation of data pipelines, data structures, and processes.
- Provide regular status reports and data insights to the relevant teams.
Qualifications:
- Experience: [X+] years of experience in a data engineering role, with a focus on Hadoop-based technologies.
- Technical Skills:
- Proficiency in Hadoop ecosystem tools (HDFS, YARN, Hive, HBase, Pig, etc.).
- Hands-on experience with Apache Spark and Kafka is a plus.
- Strong skills in programming languages such as Java, Scala, or Python.
- Experience with data modeling and ETL/ELT processes.
- Education: Bachelor's or Master's degree in Computer Science, Information Technology, or related field.
The client for is for a government project.
The client prefers candidates with no Visa requirements.
