Who's Going Down This Year?

Three principles and four practices for avoiding the Black Friday crash

For most retailers, the fourth quarter of the year is where most sales happen. More specifically, the days and weeks right after Thanksgiving in the US can make or break a business. From “Black Friday” (the day after Thanksgiving) through “Cyber Monday” (the following Monday) to “Cyber Week” (the entire week after), this short period of intensive shopping is seen as the retail industry’s Super Bowl.

And that's especially true in the world of ecommerce. According to TechCrunch, US shoppers bought $6.95 billion worth of stuff online last Cyber Monday alone. And at the peak of Black Friday’s frenzy, shoppers were spending upwards of $1 million per minute online, which is why retailers spend their entire year preparing for this quarter. Promotions are planned. Emails are prepped. New products are added. Sites are updated, and merchandising is carefully mapped out.


Failure is expensive

Unfortunately, even with all that planning, site outages on these critical days are still common. And the resulting losses can be substantial: as much as eight percent of the day’s sales for each hour a site is down, according to some estimates.

A simple Google search of “Black Friday crashes 2017” shows some of the recent carnage. Last year, Macy’s and Lowe’s were in the crosshairs. A couple years prior, it was Target and Paypal. So even the biggest names aren’t immune, but why?

Retail—and ecommerce in particular—is tougher than it’s ever been. The customer journey has changed immensely over the past several years, and most ecommerce platforms simply weren’t built for the kind of complexity we’re seeing today. Being a large, established retailer or tech firm may not be an advantage, unless you’ve updated and adapted your systems to account for today’s increasingly mobile, and global, consumers.


"Ecommerce infrastructure should be instrumented for quick debugging and troubleshooting, so that engineers can spot a problem and fix it fast."

Who's at risk in 2018?

As much fun as it would be, I can’t give you an advance list of retail sites that are going to crash this year. (Sorry.) But I can tell you the key characteristics of systems that can better handle heavy loads and high traffic. After the Cyber Week smoke clears, the merchants left standing—and counting their cash—will be the ones whose ecommerce infrastructure was designed and built with these principles in mind.

Scalability: Infrastructure should be able to grow vertically (increasing the power of the machines) or horizontally (adding new machines) without loss of performance or reliability. Cloud services such as AWS provide this in the form of auto-scaling groups and containerized services and applications.

Elasticity: That’s the capacity of an infrastructure to grow horizontally in an automated way. The old-fashioned alternative to this is overprovisioning capacity, or paying for more server power than you actually need outside of peak traffic periods. Now, all major cloud computing solutions can provide elasticity.

Observability: Ecommerce infrastructure should be instrumented for quick debugging and troubleshooting, so that engineers can spot a problem and fix it fast. (No “black boxes!”) Robust systems include clear monitoring, logging, and alerting for key system events, in anticipation of problems.


"When systems are pushed to their limits, something will go wrong. You will need to make an update, fix a problem, and deploy new code."

Start doing these four things, now

If your ecommerce infrastructure doesn’t measure up, it’s frankly too late to do much about it this year. But as soon as you’re able to, my advice is to focus on these concrete actions you can take to at least make future holiday seasons less stressful.

1. Test, test, test.

The key to avoiding a crash is understanding what your platform can handle. If you send an email blast to your entire list, can your shop deal with the resulting spike in traffic? How about if your promotion gets syndicated on a major coupon site? In other words, how quickly can your systems scale when traffic and orders peak? Good load testing requires thoughtful setup and frequent re-testing. There are services (such as Blazemeter and LoadImpact) to do it for you if you don’t have the expertise in-house.

2. Build for speed.

Good testing should reveal your weak points. Is it your database? Do images load slowly? Site speed is the single most important feature. Apart from signaling that all might not be well with your infrastructure, a slow website is terrible for your business. Every fraction-of-a-second delay hurts conversions, and sites that take longer than a couple of seconds to load could chase more than half your shoppers away.

3. Have a backup plan.

If you do crash, how quickly can you come back online? How resilient and redundant are your systems? How ephemeral is your data? Have you practiced a full restore? If you are on a third-party commerce platform, do you know who to call 24/7/365? Do you know what your service provider’s SLAs actually cover? If you aren’t in control of your platform, you’d better know what type of help you can expect during these critical outages.

4. Get comfy with continuous updates.

Know this, accept it, and make it a strength instead of a weakness: When systems are pushed to their limits, something will go wrong. You will need to make an update, fix a problem, and deploy new code. Ecommerce giants like Amazon have built continuous integration and delivery into their daily engineering practice, to the point that they deploy new code every few seconds, without their sites ever going dark. This is the new reality, and we all need to adapt to it.


Now buckle up and enjoy the ride

A couple weeks from now, we’ll be reading post-mortem reports about this year’s devastating crunch-time crashes. Someone will go down. It’s been the reality every year. Here’s hoping it’s not you (fingers crossed). However, if you’ve been under-investing in the areas I described above, your time may be coming. Huddle up with your key technical decision-makers and review your infrastructure and engineering practices.

It may be too late for peace of mind this year, but hey, there’s always 2019.


Editor's note: This story was first published by Internet Retailer.

comments powered by Disqus