The University has embarked on a strategy to transition all its cumbersome legacy systems, including its entire business intelligence (BI) ecosystem, to the cloud by 2030. This central services project was no different. The solution required migrating 15,000 existing Wherescape RED ‘extract, transform, load’ (ETL) objects from the on-premise servers to a cloud-based data warehouse.

First, the University’s team established the guidelines, the architecture and how the data should be moved to the cloud. Snowflake on AWS was chosen for the data warehouse due to its scalability and cost-effectiveness across storage and computing resources.

Qrious worked with the University to implement the architecture and the data models. “The only people that we let play in our environment are very experienced. Qrious knows our environment very well,” says Indrajeet Singh, the University’s BI Services Manager.

Qrious automated the migration process for the majority of the ETL objects to the cloud, leveraging the advantages of Wherescape’s metadata management. Fivetran was used for the data extraction process. It was also important to support the extensive and intense parallel testing required to ensure precisely the same results were generated as for the existing ETL.

Another considerable undertaking was determining what data to migrate and what to leave behind. Over the years, the legacy system had accumulated a significant volume of data, much of which was not being used.

Additionally, integrating data from various systems posed a major challenge. “Until we established the new data feeds in the cloud, we were still somewhat constrained by the limitations of our legacy system,” says Moss.

The team was effectively building the new platform while maintaining the existing one and adding new imperatives as they arose. “It's a very dynamic environment,” says Singh. “Qrious made sure that changes to the existing platform were reflected in the new cloud solution.”

With the new system in the cloud, the ETL process completes in less than an hour, well down from the six hours it once took. Moreover, it operates with more up-to-the-minute data, which is available ahead of the start of the work day, and requires significantly less computing resources.

Should there be a delay, the catch-up is fast-paced. This is largely due to the flexibility of the new computing infrastructure, which empowers the ETL and MicroStrategy reports to process data much more quickly. All reports can be run daily without any noticeable lag.

This improvement ensures that critical information is available in a timely manner, enhancing decision-making processes and overall operational efficiency.

Importantly, with the adoption of the same platform and datasets, data-sharing practices have significantly improved. Now, different users can easily share data, fostering greater collaboration, transparency and efficiency across teams.

By adopting Snowflake, the University of Auckland is securing a strategic advantage for maximizing the future potential of its data. Snowflake's ongoing commitment to embedding Machine Learning (ML) and Artificial Intelligence (AI) tools directly within the platform positions UoA perfectly to leverage these cutting-edge technologies. This will empower UoA to streamline operations, make data-driven decisions, and unlock new possibilities as Snowflake's ML/AI capabilities continue to evolve.