- Proficiency in SQL and Python as data processing languages
- Experience with Apache Spark or Databricks as data processing engines
- Strong experience with Lakehouse patterns and medallion architecture in data engineering
- Familiarity with Azure Data Services and other relevant data management tools including Microsoft Fabric, Synapse Analytics and Data Factory.
Data Engineer to work on implementing a new data platform for a new Annual Motor Insurance Product. The platform will be a Lakehouse environment using the medallion architecture in Microsoft Fabric. The candidate will be expected to work with the Director of IT and Business Intelligence Analysts to ingest motor insurance data from multiple sources, assist with design of Lakehouse and warehouse table schemas, manage and enhance data quality and ensure project complies with Data Governance requirements. This role is pivotal in delivering management information and reporting for new Annual insurance product and providing key data to Pricing, Underwriting and Fraud teams.
,[ Implement Medallion Architecture: Apply the medallion architecture to organise data layers (Bronze, Silver, Gold) within the Lakehouse, ensuring progressive enhancement of data quality., Design and Optimize Data Storage: Develop scalable and secure data storage solutions that align with the medallion architecture principles. Develop Data Processing Pipelines: Construct and maintain robust data pipelines for efficient data ingestion, transformation, and consolidation., Ensure Data Quality and Compliance: Uphold high standards of data governance, ensuring compliance with industry regulations and data privacy laws. Collaborate with Cross-Functional Teams: Work closely with stakeholders across the company to understand data requirements and deliver insights that drive strategic decisions.] Requirements: SQL, Python, Apache Spark, Databricks, Lakehouse Patterns, Medallion Architecture, Azure Data Services Tools: Agile, Scrum. Additionally: Sport subscription, Private healthcare, Flat structure, Small teams, International projects, Worksmile, LuxMED, Multisport, Integration events.
databricks Azure Data Factory Agile SQL Python Apache Spark Data Engineering Scrum Azure Data Services Medallion Architecture