We are seeking a highly skilled and motivated data engineer to join our Data Infrastructure team. The ideal candidate will have a strong background in algorithm design, data structure, database warehousing ETL, big data technologies, cloud computing, distributed systems, data mining and machine learning, and will play a crucial role in enhancing and maintaining our data infrastructure.
Key Responsibilities:
- Design and implement scalable data pipelines for ingesting, processing, and storing large-scale datasets from various crypto exchanges.
- Collaborate with quantitative researchers to understand their data needs and customize data streams to facilitate feature engineering and model training.
- Utilize AWS services for data storage and processing, ensuring high availability and reliability of data services.
- Implement data validation and quality checks to ensure accuracy and integrity of the datasets provided to the quantitative research team.
- Monitor and optimize data workflows and queries for performance, scalability, and cost-effectiveness.
- Maintain clear documentation of data processes, architectures, and workflows to ensure knowledge sharing within the team.
Qualifications:
- Master's or PhD in Computer Science, Software Engineering, Data Science or a related field.
- Strong in algorithm design and data structure
- Strong understanding of database warehousing ETL, big data technologies, cloud computing, distributed systems
- 3+ years of experience as a data engineer or in a similar role, preferably in finance or crypto markets.
- Proficiency in Python and Rust.
- Experience with SQL and NoSQL databases (PostgreSQL, MongoDB, etc.).
- Familiarity with data mining and machine learning is a plus
- Excellent problem-solving skills and the ability to work under pressure.
- Strong communication skills and the ability to collaborate effectively with cross-functional teams.
