July 29, 2003
When Good QA Goes Bad
Adam Kalsey has a good write-up of some usability mistakes for an ecommerce site. Good stuff.
His write-up reminds me of the time we were testing the very first Borders.com Web site way back in 1998. The Quality Assurance department had a couple of credit card numbers they used to test the ecommerce pages. These cards were authorized so that the fraud check would allow them through, but the fulfillment (and actual shipping) was not supposed to happen.
One day, one of my peers in the QA department asked me to run some tests against the new ecommerce system to see if I could break it or find some holes. One of the first tests I came up with and ran was to find the most expensive item in the database. This happened to be a set of 50+ encyclopedias that retailed for $1395.99 each. Of course, you know I had to try it. Yup, I ordered 9999 of these suckers for a before-tax total of $13,958,104.05 just to see what would happen. Would the fraud department catch it?
Thinking nothing of it I went home, assuming my job for the day was done. The next day I was at work running more tests and my boss called me into his office. He said he'd gotten a call for the Fulfillment center in Tennessee asking if someone at the corporate office had placed an order for 10,000 sets of encyclopedias. I laughed out loud and explained to him that the QA team and I were just running some tests and that the order should never have made it to the Fulfillment center.
I can imagine a semi truck with pallets and pallets of encyclopedias pulling up to the building trying to deliver them. The only thing more funny would be the look on the CEO's face for a $14 million Mastercard bill.
Posted by Cameron Barrett at July 29, 2003 07:04 PMAah yes ... the good old days! Rick.com woulda beat your ass!
Posted by: Ben at July 29, 2003 07:06 PM
I'm curious to know how they would have fulfilled an order for 10,000 sets of encyclopdias. Did they have that many? Would they be drop-shipped? Would the poor UPS delivery man get some help in bringing them to your door?
In most ecommerce sites it makes sense to limit the quantity that someone can order to reasonable amounts. First of all, the fair probably couldn't fulfill an order for 1 billion tickets. Secondly, unless you're buying tickets for your family and the entire population of China, the input was probably a mistake.
Posted by: Adam Kalsey at July 30, 2003 02:49 AM
What happened was the person who received the order at the Fulfillment center in Tennessee realized it was abnormal, saw that the shipping address was the company's corproate office, traced it back to my boss, and made a call to find out what the order was all about.
There is no way the order would ever be filled since packing and shipping is still a manual process performed by humans, and someone would likely question such an order before trying to process it and fulfill it.
Most fulfillment centers, like the one Borders.com used, only stock 1-2 copies of about 20% of all the products in the database. The rest are ordered directly from the distributor(s), which are sent to the fulfillment center - which then proceeds to fulfill the order.
Adam, you are correct that it makes sense to limit the per-item quantity and I've seen some ecommerce sites do it correctly, with a cap at 99 items. I don't recall what we chose for the old Borders.com site but I think it may have 999 - or three digits.
Posted by: Cam at July 30, 2003 03:06 AM
That's nothin'. I did some QA work on an e-commerce site (which must remain nameless) where I started putting in negative values for the quantity.
Yup. Money was being creditied to the dummy account. That would have been a disaster if it had gone live.
Posted by: Scott at July 30, 2003 09:10 AM
Would have been worse if they credited the account AND shipped the goods.
Posted by: Adam Kalsey at July 30, 2003 11:20 AM
While fulfillment centers are staffed by humans and each order requires human hands to touch it, even in a highly automated (or overly automated) warehouse like the former WebVan or Amazon's, those humans aren't the most likely to care that much about the company. Orders that make little or no sense may still get shipped.
It's the job of the fulfillment system to implement business rules that prevent illogical orders from hitting the floor until the've been verified by a supervisor.
Posted by: Adam Kalsey at July 30, 2003 12:38 PM
