Position Description:
The Data Engineer is a software development role within the L3 squad of the Data Team in the JERAGM IT Department.
The L3 squad are responsible for designing, building, and testing complex data ingestion and processing pipelines. The L3 squad are complemented by the L2 squad which is responsible for platform administration, operational support, small change and release management.
JERAGM is investing in Data and Analytics to enable a data driven approach to commercial decision making, risk management and operations. As part of this, the Data Team is leading delivery of a modern data platform using Azure services and Databricks. The team is delivering new capabilities across the business, including:
- A central warehouse of trade, valuation, and risk data
- A central hub for sharing market fundamentals for commodity markets
- Pipelines for scraping, ingesting and analysing internal, public and licensed data
- Processes and tools for data governance and discovery
- A reporting tool for EOD P&L Attribution
- Analytics and visualisation tailored to JERAGM business user needs
The role will be part of a highly qualified team, primarily based in Singapore. The team operates an agile delivery framework in a fast-paced trading environment following DevOps principles and practices. The team believe in the mantra of acting fast, failing fast and learning fast. Every team member is hands-on and team members are encouraged to work closely with each other and end users. Everyone in the team is highly engaged and we continually share our thoughts and ideas in daily stand ups and weekly system design meetings. At the same time as collaborating as a team, we encourage our Engineers to plan and execute their own work so they can work autonomously and take ownership.The role will cover below areas:
Data Engineering (80%):
- Design and deliver data solutions using an agile, iterative approach based on Scrum and JERAGM's change management process.
- Collaborate with the Data Engineering Manager on technical architecture and design.
- Understand JERAGM's commercial data usage to identify system requirements with Business Analysts or users.
- Analyse and estimate IT changes, providing input on technical opportunities, constraints, and trade-offs.
- Create documentation and present to both technical and non-technical audiences.
- Handover to L2 team and provide third-line support for short periods after releases.
- Own their learning to remain a technical subject matter expert.
- Collaborate with the Data Engineering Manager, other Data Engineers, Lead BA, and X-team SMEs to deliver software to Production with minimal impact.
- Conduct detailed testing for development activities and demonstrate results according to the delivery methodology and coding standards.
- Create and productionize complex data pipelines with quick turnaround and high quality.
Support activity (20%):
- Assist the team with Production code deployment and data platform support.
- Support the L2 team in fixing Production bugs.
- Act as a Release Engineer in sprints as needed.
Qualifications
Experience Required:
Around 5 years experience in a similar role covering:
Technologies:
- Extensive cloud experience in Azure and expertise in modern cloud-based data architectures.
- Advanced coding experience in Python and SQL.
- Experience in Databricks or equivalent experience in Spark.
- Advanced knowledge of Databricks e.g. Unity Catalog, Delta Live Tables, Platform administration, know internal of Databricks e.g. Optimize, Vacuum, Z-order etc.
- In-depth knowledge of Azure services, including VNET, Key Vaults, Azure Data Factory, ADLS Gen2, Virtual Machines, App Services, Storage Accounts, and Azure Active Directory.
- Proficient in dimensional modelling (e.g., star and snowflake schema design).
- Experience orchestrating data pipelines using Azure Data Factory or Airflow.
- Knowledge of Python packages such as Pandas, Numpy, and Seaborn.
- Strong understanding of Big Data, MapReduce, Spark, and file formats like Parquet, Avro, and ORC.
- Familiarity with reporting tools such as Power BI or Tableau.
Software engineering and delivery
- Source code management e.g. Azure DevOps, Git
- Agile delivery methodologies such as SCRUM or Kanban
- Knowledge and work management tools (e.g., JIRA, Confluence)
- Certified in Data Engineering
Person Specification:
- Strong academic background with a degree, equivalent professional qualification, or experience.
- Analytical, flexible, and curious, open to new ideas.
- Committed to lifelong learning and engineering excellence.
- Highly organized with strong attention to detail.
- Adaptable, able to multi-task and work independently or in various team settings.
- Confident and effective communicator, comfortable contributing in meetings.
- Collaborative, Agile, trustworthy, and takes ownership with a getting things done attitude.
