Migrating & Upskilling from SAS to Databricks

Our client is a leading financial services institution, well-established within the New Zealand banking sector.

They offer a range of banking and financial products, including personal and business banking, loans, mortgages, and investment solutions. As part of a strategic shift, they aimed to modernise their data infrastructure, improve operational efficiency, and elevate their analytics capabilities by adopting cloud-based solutions. However, growing data volumes and evolving analytics demands exposed the limitations of their on-premises SAS system, impacting performance and competitiveness.

The Challenge

Transitioning to a modern data infrastructure meant overcoming some significant obstacles tied to the limitations of their legacy SAS system:

Data Platform Incompatibility: Their risk modelling remained in SAS, while data had already moved to Snowflake. This lack of integration caused cumbersome data transfers, creating bottlenecks and inefficiencies in critical workflows.‍
Performance Constraints: SAS's inability to efficiently handle large datasets, particularly under the demands of simultaneous users, led to sluggish analysis times and hindered the team's ability to deliver timely insights.‍
Technological Lag: The fast-paced evolution of data science left SAS behind. The client needed access to modern machine learning and AI capabilities, where Python and open-source technologies had become the industry standard.‍
Talent and Resource Shortages: The scarcity of SAS expertise made it challenging to hire and retain skilled professionals. Shifting to Python promised a wider talent pool and a future-proof skill set for the team.‍
High Maintenance Costs: The expense of maintaining on-premises infrastructure and SAS licensing fees was draining resources. The client needed a more cost-effective, scalable solution that could dynamically adjust to their needs.
Collaboration Limitations: With a growing data science team, SAS's restricted collaboration features impeded efficient teamwork. The client required a collaborative, cloud-based platform to streamline project development and data sharing.

‍

Our Approach

To help the client modernise their risk modelling and maximise the benefits of a cloud-based infrastructure, we implemented a strategic, step-by-step solution:

Migration to Databricks and Snowflake: Risk scorecard models were transitioned to Databricks, leveraging its scalable processing power. Snowflake served as the integrated data storage solution, ensuring efficient handling of large datasets and seamless workflows.
Python Training and Upskilling: Tailored training enabled the client’s team to confidently transition from SAS to Python, equipping them with future-ready skills for building and managing models in Databricks.
Cloud Workflow Optimisation and EDA: Best practices for distributed computing and data management were introduced, alongside guidance on cleaning, exploring, and selecting key data features. This approach enhanced model accuracy and interpretability.
Ongoing Support and Adaptable Tools: A flexible scorecard template was created to meet future needs, and continuous support ensured a smooth transition while building team confidence in managing the new system.

The Outcome

The client was successfully transitioned to a modern, cloud-based infrastructure, eliminating performance bottlenecks and enabling efficient handling of large datasets. Python adoption and upskilling empowered their team with future-ready capabilities, while the collaborative platform streamlined workflows and reduced costs. These changes positioned the client for long-term growth and competitiveness in the financial sector.

‍