Data & Machine Learning Engineer

The Data & Machine Learning Engineer plays a hybrid role that focuses on developing a modern Lakehouse platform, building data pipelines, and deploying machine learning models at scale. This position blends the responsibilities of a Data Engineer and Machine Learning Engineer to ensure robust data infrastructure and effective deployment of ML models. The ideal candidate will collaborate with cross-functional teams, working closely with Data Analysts, Business Analysts, and DevOps engineers to deliver reliable, scalable, and efficient data and ML solutions.

Key Responsibilities:

• Build, automate, and optimize ETL/ELT pipelines to ensure smooth data flow and availability across the Lakehouse platform.
• Partner with analysts to design scalable data storage solutions optimized for analytics and ML applications.
• Ensure data quality by implementing monitoring mechanisms to guarantee consistency, reliability, and compliance.
• Continuously monitor ML models, retrain and adjust them as necessary to maintain high performance.
• Design and implement CI/CD pipelines for seamless model deployment and ensure reliable delivery of models into production environments.
• Provide guidance and mentorship to junior engineers, supporting their growth and enhancing project delivery capabilities.

Skills & Experience:

• Strong teamwork skills, able to align data solutions with business needs, and communicate effectively across technical and non-technical teams.
• Expertise in analysing data-driven insights and translating them into technical solutions.
• Proficient in designing and implementing data pipelines using Databricks and Spark.
• Advanced skills in SQL and Python for querying, modelling, and transforming data efficiently.
• Expertise in deploying machine learning models and creating APIs for serving predictions.
• Skilled in processing large-scale datasets and supporting machine learning workflows on Databricks.
• Experience in building CI/CD pipelines for automated deployment, testing, and validation of data pipelines and ML models using GitHub and GitHub Actions.
• Proven ability to monitor ML model performance, track key metrics, and maintain accuracy.
• Experience with automating cloud infrastructure using tools like Terraform for consistent deployment and scaling.
• Basic knowledge of Power BI for visualizing model performance and metrics.

Qualifications Required:

• 5+ Years in Data Engineering: Solid experience building and maintaining data infrastructure, including best practices in data pipeline development.
• 3+ Years in Machine Learning Engineering: Demonstrated experience managing the lifecycle of ML models from deployment to monitoring and retraining.
• Business & Data Transformation: Proven track record in supporting business transformation and process improvements by leveraging data-driven solutions.
• Agile Methodologies: Comfortable working in Agile environments with the ability to adapt and respond quickly in fast-paced settings.