One of the key differentiators of the Acquia AgilOne CDP is the scalability of the platform. Scalability is one of AgilOne’s key architectural tenets, powered by our tech stack’s horizontal scalability and elasticity. This is evident from how large, high-demand enterprise brands deliver relevant and personalized customer experiences via the AgilOne CDP, processing hundreds of millions of customer records, billions of transactions and billions of events daily. One illustration of the AgilOne platform’s scalability is how easily we are able to add enormous volumes of historical data into the platform when onboarding each new enterprise client.
Another key illustration of the AgilOne platform’s scalability is our ability to handle and ingest big spikes of data - like we handled during this season’s Black Friday and Cyber Monday for our retail clients. In this blog post, we will focus on some interesting aspects from these data spikes.
The AgilOne real-time pipeline ingests real-time events, including web events (browse as well as checkout/transactional events) and email events. We saw huge spikes across brands in these events and transactions, and our platform successfully ingested these data spikes. This success is not just about data ingestion, but also about scalability in all the other modules like campaigns, analytics and machine learning, which all need to operate on these increased volume spikes. Supporting spikes like this is mission critical to our clients’ bottom lines.
Data points in web traffic volumes in 2019 Black Friday / Cyber Monday
- Multiple brands had spikes of greater than 20 times their regular web traffic volume on Black Friday and again on Cyber Monday; these peaks correlate to the times when they sent large campaigns. AgilOne’s CDP also ingested these spikes in email events, so the campaigns had double the impact on the data ingested into AgilOne.
- One brand had spikes of greater than 30,000 events per minute (compared to their regular volume of less than 1,000) - this was early morning on Black Friday.
- Another brand had sustained spikes over time resulting in greater than 1.3 Million events per hour, and greater than 25M events per day (which was on Black Friday).
- Another brand normally gets 20,000 calls per hour on average. But on Sunday, ahead of Cyber Monday, this brand reached peaks of ~430K calls per hour, i.e., greater than 20 times their average.
- Even some of our European clients saw about 20 times their normal volume consistently throughout the holiday weekend.
- In that week, across brands, we processed more than 350 million web events -- more calls in that week than an average month.
Data points from email traffic volumes
We saw spikes in email campaigns and the events generated by them, which are all ingested into the AgilOne platform via the extensive ESP connectors we have. Some interesting data points are:
- We ingested ~3.7 billion email events (which include email send, email open, email click) across all brands during the holiday month. This is 15% more than the email events in the previous month.
- We have consistently seen higher email click rates leading to better engagement during the holidays in 2018 and 2019. For example, in 2019, while 29% of the monthly email sends were in the holiday week, 32% of the monthly email clicks were during the holiday week thus showing greater engagement during holidays.
- We supported multiple intra-day spikes for web and email events on the days of Black Friday and Cyber Monday. We’ll dive into why this is significant in another blog.
Data points from transactional volumes
As expected with the increased web and email activity, this resulted in a spike in transactional data as well. We noticed some interesting data points in the transactional data ingestion:
- On Black Friday 2019, on average across brands, we ingested approximately six times our clients’ normal daily transactional volume across online and offline transactions.
- Some brands had greater than nine times their normal transactional volume on this day.
- On average, our clients saw 13% growth in transactional volume when compared to the same period last year. A majority of that growth came from digital sales channels driven by convenient features such as "Buy Online Pickup in Store."
- The percentage of digital transactions in the total transaction volume during the week of Black Friday & Cyber Monday was 23% in 2018, then jumped up to 30% in 2019.
Powering machine learning, campaigns and analytics
Once these data spikes are ingested through the AgilOne data pipeline, this data powers all the CDP modules, which illustrates the scalability in these modules as well. So, these data spikes directly translate to helping marketers achieve better results.
- AgilOne’s machine learning module generates greater than 1 billion predictions across brands, as highlighted in this case study. This was using this newly ingested data along with its spikes.
- AgilOne’s Metrics module powers dashboards and ad hoc reports to deliver insights, based on this increased event and transaction volume.
- AgilOne’s Actions module scales up to help marketers act on these newly ingested data spikes to deliver campaigns.