Stories from The Amazon.com : Tales from An Original Team Member

Infinite Loop 1

Most developers do their best work as they are falling asleep at night and the beta waves are washing gently across their brains. This is the time when they are obsessing on the days problems as they falling into a deep slumber. This was the case one night as I lay their thinking about our cascading purchasing system I designed.

Back in those days (late 90's), Amazon did not stock inventory; they placed orders only as they got them and then shipped them out immediately after fulfilling. I had designed a cascading purchasing system to fulfill customer orders based on their location to nearest Amazon warehouse (which apparently Jeff now insists on calling ‘fulfillment centers’) and the nearest distributors who could fulfill our orders.

The idea was that I bundled all customer orders based on their proximity to a warehouse and then tried to fulfill with the nearest set of distributors, then the next nearest, then the next nearest, etc. Basically it just went down a cascade of distributors based on their proximity to a warehouse trying to fulfill the customer orders. Those left unfulfilled were sent back into the queue the next day.

But something nagged at my brain as I drifted off to sleep. What did we do with backorders? We had thousands of books on backorder? Were they just looping indefinitely? Suddenly I jolted upright in bed. It was 2am. I called a company taxi (as we had an account with a local taxi service in those days that only a few of us knew about) and headed on down to warehouse #1 where my primary office was.

During the entire ride, I kept double checking the code in my head to make sure I was right because if I was, I would be waking up the entire company at 3am to tell them some pretty awful news about a few thousand customers… the exact number I wasn’t even sure of yet.

But I was nearly positive.

You see, the way the publishing system works is that before a book comes out, you can place orders for a book to gauge interest. This is known as a ‘backorder’. This helps a publisher to know how many to print OR to even go to press at all. And thats the kicker. Sometimes these books NEVER go to press. OR the book changes publishers and the original orders for the book under the other publisher never get fulfilled. AND the book remained listed in our system and continued to collect backorders if people searched for it and found it.

But we would have never known thise because back then, the distributor didn’t have api’s to delist the book from our system; once the book was in our system, we just sat their taking orders for it. People could be lining up placing orders and we would continue to take their money but that book would never be printed. And we didn’t know to remove it from our system. It just stayed their indefinitely. We had no way to SEE inventory that never went to print!

I finally made it to the warehouse and climb the stairs up to my office and logged in to my terminal to check the numbers. I ran the numbers and stared at my screen. I ran the numbers again to be sure I didn’t screw that up. Same number.

Over 60,000 customers had been looping in the cascade for the last 6 months trying to have backorders fulfilled.

We were not checking for canceled books nor were we refunding and cancelling orders; people orders were just looping forever and we continued to accept new orders.

I started making calls. I woke up my manager, who woke up her manager who woke up Jeff and the SNOC (development and network operations). They double checked all my numbers and the database and found I was 100% correct. They completely missed this.

Heads rolled. Customers were pissed. I was asked to take action and look into this but was shut out by developers who were blamed.

Since api’s didn’t really exist, my solution was to compare our catalog against publishers catalogs. It would have to be done manually at least once a month and potentially, we could digitize and resell the catalog back to publishers/distributors for use on the web or with electronic services.

In the long run, it was determined to merely check all orders against said lists. This did not stop future issues like this but did stop existing issues.

So the problem persisted for a long time and probably still persists to this day.

The real problem lies in Amazon’s inability or unwillingness to try to work with publishers. But at the same time, having worked with publishers for years, I see the flip side of the coin. Publishers are a notorious lot and often made working with them an impossible task.