Data Platform MS Fabric Databricks

About the job

Company Description

Talan is an international consulting and technology expertise group that supports and accelerates the digital transformation of its clients through the levers of technology, data, and innovation.

For over 20 years, Talan has been advising and supporting companies and public institutions in defining and implementing their transformation and innovation projects in Switzerland and internationally.

Present on 5 continents in 21 countries, the Group, certified Great Place To Work, has more than 7000 employees.

Equipped with a Research and Innovation Centre, Talan places innovation at the heart of its development and operates in areas of technological change such as Artificial Intelligence, Data Intelligence, Blockchain, to serve the growth of large groups and mid-sized companies in a committed and responsible approach.

By placing "Positive Innovation" at the core of its strategy, Talan is convinced that technology amplifies its impact on society as a whole by being at the service of humanity.

#TalanSuisse

Job Description

Mission Context

As part of the evolution of a Data platform within a banking environment, we are looking for an experienced Data Engineer with expertise in Databricks and Microsoft Fabric.

The mission mainly consists of extending the functional coverage of an existing Data platform by integrating new datasets according to already defined patterns and standards.

With the architecture and framework already in place, the objective is not to redefine the solution but to deploy new data flows respecting the existing models, quality rules, and CI/CD processes.

Main tasks

• Integrate new datasets within the existing Data platform.
• Configure new flows according to established patterns and standards.
• Develop and adapt processing using PySpark and SQL.
• Correctly define the modelling of new data: dimensions, facts, granularity, business keys, SCD1/SCD2, snapshots.
• Configure datasets through a declarative approach, notably via TOML.
• Validate data according to existing quality rules and contracts.
• Analyse and correct anomalies encountered during the integration of new sources.
• Deploy evolutions through CI/CD Azure DevOps pipelines.
• Diagnose and resolve failures related to quality gates and various promotion stages.
• Functionally understand source banking data to ensure their proper integration into the platform.

Qualifications

Data Engineering

• Strong experience with Databricks
• Mastery of Delta Lake
• Good knowledge of Unity Catalog
• Knowledge of Change Data Feed
• Very good level in PySpark
• Very good level in SQL

Data Modelling

Ability to determine for each new dataset:

• dimension vs fact;
• SCD Type 1 vs SCD Type 2;
• snapshot logic;
• granularity;
• natural business keys;
• transformation and validation rules.

CI/CD & DevOps

• Good mastery of Azure DevOps
• Experience with existing CI/CD pipelines
• Understanding of gate mechanisms and branch promotion
• Ability to diagnose deployment errors
• Exposure to Databricks Asset Bundles and/or Terraform is an advantage.

Banking Knowledge

Experience in the banking sector, ideally in Private Banking / Wealth Management, is essential.

Good understanding of data such as:

• positions;
• transactions;
• Assets under Management (AuM);
• Net New Assets (NNA);
• booking centres;
• client hierarchies.

The ability to functionally understand data from a Core Banking System is particularly important.

Appreciated Skills

• Experience with TOML or declarative/config-driven Data frameworks
• Experience with Auto CDC patterns
• SCD1 / SCD2
• Kafka
• Spark Declarative Pipelines
• Knowledge of Microsoft Fabric
• Experience in dimensional modelling
• Knowledge of the Kimball methodology
• Terraform

Additional Information