CHALLENGE
Finding an efficient way to process data without resource competition
While Rakuten had moved from their initial SQL database in 2014 to an AtScale-powered Hadoop solution in 2018, this wasn’t sufficient, and they soon began to experience a resource crunch based on the sheer size of their database. Rakuten’s existing architecture meant that business users didn’t have the computing resources necessary to work with large datasets. This meant that users’ disk and processor usage reached critical levels, leading to competition between business units for hard disk access, memory, and CPU time.
The internal team was frustrated with the competition for resources, and the operational overhead and associated hardware and electricity costs also meant the solution was no longer cost-efficient. That, coupled with the continuous processing demands on storage infrastructure, forced Rakuten to consider new solutions for their data needs. They knew they needed more processing capability and flexibility to continue serving their customers effectively.
SOLUTION
Migrating to a cloud-based solution to control the complexity and cost of analytics
AtScale helped Rakuten transition their analytics to the cloud while still retaining all the analytical capabilities they had built, enabling them to deliver consistency for their team. AtScale insulated their Tableau-based reports and dashboards from changes in underlying raw data. While location and schemas changed, the AtScale model was untouched, allowing them to preserve their investment.
Once they had moved their data to Snowflake, AtScale was able to help Rakuten better optimize their costs by right-sizing cloud resource consumption based on real-time usage. AtScale also helped smooth out and mitigate user concurrency challenges without requiring additional compute resources and leveraged intelligent aggregates to accelerate query performance while keeping cloud costs down.
RESULTS
$10 million dollars in savings and better, faster data
By moving from an on-premises Hadoop data warehouse to Snowflake’s elastic, scalable resource model, Rakuten gained additional computing power. This enabled them to maintain responsiveness to queries during peak demand periods, while only paying for what they used. Now Rakuten can maintain flexibility to compensate for crunches, without paying excessively for operational or server overhead.
Because AtScale allowed Rakuten to move from Hadoop to Snowflake seamlessly, the team can now access their data faster and with more consistency. As high-demand queries were shifted to Snowflake, there was an immediate 30% drop in load on the on-premises computing cluster. And, with AtScale’s help, Rakuten managed to solve their concurrency issue as well: Rakuten ran a test of 10 terabytes of data with and without AtScale and discovered that AtScale ran the process 14x faster and at 4x less cost.
And, since AtScale acts as an abstraction layer for Tableau, the switch from where the data was stored did not lead to any interruptions in data access for Rakuten’s users. With the same data analytics tools in place before and after the migration, Rakuten didn’t need to retrain their team to adopt any new tools or working methods. The only thing they had to adapt to was receiving faster reports than before.