A few days ago, I mustered the courage to clean out my closet. I had not thrown away old t-shirts from my life, regardless of their terrible condition, literal stains, holes, and overall wear. Some I will never wear in public again. Some I probably shouldn't wear personally.
I always suspected that Amazon, Microsoft, and Google had created a vast number of t-shirts, sweaters, hoodies, and coats since my time, but I didn't realize how many until I started sorting (30!).
Another realization was that despite my best intentions to remove my items, I couldn't bring myself to get rid of that shirt. Each had a story and a feeling attached, so they held sentimental value. Someday I will leave this world, and the person cleaning out my closet will look at them and shrug, tossing them into the donation pile or the trash. But until then, they are safe in a part of my closet.
How Some Dedicated Amazonians Saved Christmas
The Amazon.com site has thousands of critical moving parts. Most of them are taken for granted, and some people may not even realize they exist, but when they break, you will notice quite quickly.
As we approached Christmas in 2014, the wishlist feature was struggling. The wishlist is a neat little feature on retail sites that allows friends or family to love something, perhaps to buy for a birthday or Christmas. Naturally, those who don't naturally pick the right gift (like me) greatly appreciate a little hint. After 20 years of marriage, my wife gave up dropping subtle hints and extensively used the wishlist, so I personally loved this feature. And clearly, millions of other Amazon customers around the world felt the same.
The wishlist was a rather old, crufty service with some architectural bottlenecks. One of the engineers on the team predicted that given some numbers and traffic trends, it was likely to die a few weeks before Christmas. We could continue and risk a very public operational issue during the season. We could have completely disabled this feature until we figured things out, but that could have happened after Christmas, and millions of people had already added items to millions of wishlists.
Amazon, like others, is customer-obsessed. There were no options for engineers or leaders. So the decision was simple. We would rewrite, test, and deploy a completely new wishlist service capable of handling Amazon's scale in just four days. It was an ambitious goal.
I wasn't on the team and wasn't even the same SVP. However, for years I had been involved in the company's load and performance testing efforts, tools, and guidelines, so my name appeared as contribution. One of the prominent engineers in the org, Brad, asked if I could take some time to help. I innocently said, "When?" There was a pause. "Um, now?" I nearly choked on the coffee I was drinking. I underestimated the urgency. I quickly sorted it out with my manager and then headed to the war room on the other side of the Amazon campus.
A war room in the tech world is a physical or virtual group that quickly pulls together to handle emerging situations. Often, it is a virtual group responding to a well-known operational issue. In this case, it was a group of physical people gathered in a large room. When I walked in, boxes and arrows were drawn on every possible square inch of whiteboard, and there was an impressive collection of key engineers. We were no longer in Kansas. I felt awe at being in the same room as some of these people.
My specific job was to learn about the product and devise and implement testing strategies to see if the new version of the wishlist could improve the situation. And I did that for the next 24 hours. Being in a physical war room was a really neat experience because you could feel the energy, urgency, and excitement of the room, exchange ideas, and iterate very quickly. It was Thursday. We stayed late into the night, drinking copious amounts of coffee and soda. At 7 PM, the VP ordered pizza. By 9 PM, I was no longer sharp, so I went home but returned around 7 AM the next day.
I had the easy job. I finished the testing strategy and had a working suite ready to go by Friday afternoon. The engineers actually rewriting the system stayed over the weekend. That was a much harder job! I kept my phone on in case they had issues with the tests and needed my help. By Monday morning, the new wishlist was ready to be deployed. I ran all the tests, looked at many dashboards to evaluate how the system would scale, and gave the final sign-off for production deployment.
On Tuesday, Amazon.com launched a completely different wishlist service that provided valuable data to millions of people. It handled the load perfectly during the season. Perhaps none of our customers would have known how hard the dedicated Amazon people worked for an intense 4-5 days to save Christmas.
And we all got shirts to commemorate the experience!
When cleaning out my closet, I couldn't help but smile when I saw the wishlist shirt. No, I couldn't get rid of that shirt. It captured a precious and intense moment.
Up the River and Into the Mountains
Somewhere in 2014, I saw one of the directors I worked with wearing a cool raven. It had the "Up the River" logo on it. I wanted one! So I asked him how he got it. He smiled and said, "You have to earn it!"
I was intrigued because I love a challenge. Once a year, Amazon rents out a massive sewing resort called Sun Cardia, located in the Cascade Mountains, for about two hours from the Seattle headquarters. Every director nominates two engineers and submits a written justification for why they deserve to go. Attendees are generally expected to be top-tier (the best-performing engineers in the company) and are hand-selected. Since one of the goals was to expose hungry and promising junior engineers to more senior role models to accelerate their careers, levels didn't matter. The idea was to gather Amazon's most influential engineers, eat, drink, play, talk, and see what cool ideas cross-pollinate over 2-3 days in the middle of nowhere.
The river had just risen. So I could work all year to nominate the river. I wanted that raven!
The next 12 months were the most intense of my life, and I loved every moment. I had just been promoted to senior engineer and wanted more. I had great projects, great managers, and great directors, and I wasn't shy or embarrassed about wanting to be on his list for the river.
My heart soared when I saw in my inbox, "Congratulations! You have been selected for Up The River 2015!" I also realized that this came with responsibility and privilege. My director used one of his few spots to hand-pick me, and I didn't want to disappoint him.
Sun Cardia was a beautiful setting with mountains and forests. There would be 200 Amazonians from all levels, roles, tenures, and teams. Much of it was in an open space format, so anyone could get up on stage, throw out a topic, choose a time and place, and host a session. If you were interested in that topic, you would show up for the session and have a lively discussion with whoever else showed up. We shared meals together, exchanged ideas, made friends, drank copious amounts of coffee during the day, and even more whiskey after dark, and had epic and mischievous Nerf-gun battles to the dismay of the resort staff.
The river no longer exists. I think the company got too big. It's hard to quantify the impact of conferences (and training and networking), but I can say this: Up the River 2015 fostered several important relationships that have sustained throughout my career and was a definite turning point in my transition from senior engineer to principal engineer. And of course, I got a great raven out of it.
Do Amazon Vampires Exist?
The raven with the Amazon logo inspired by vampires is generally (especially around Halloween) pretty cool, but the fact that it has its own story is even more important to me.
Romania holds a special place in my heart. In 2010, I joined a team that had half its staff in Seattle and the other half in Iași, Romania. Thus began a lifetime of business travel. I traveled to IA six times in four years to visit my counterparts in Romania and was a manager of ten engineers for a while. My wife tagged along on my first trip. We rented a Dacia and drove through the Carpathians, exploring impossibly picturesque towns and castles that looked like they were set for a Dracula movie. The country is amazing. Another time, since the connection to Bucharest was through Istanbul, I spent a week backpacking through Turkey. Another time, I found out that Iași airport unexpectedly closed its doors as I was landing in Bucharest, so I improvised and took a train through western Moldova, having many debates in English with stern officials. I loved the people, loved the country, and looked forward to heading there every six months.
Before joining Amazon, I hadn't really done much business travel. Amazon opened my eyes to the world as I began to travel to every continent except Antarctica. And the first trip, the trip that started it all, was to Iași.
So when my Romanian colleagues decided to make a joke about the Amazon vampire shirt considering how many times people asked about Dracula, I was thrilled to wear it. And I still am today.
Paying It Forward
This is not just one long-sleeve shirt once a year.
In 2012, I was looking for ways to elevate my scope and responsibilities and get out of my comfort zone in as many ways as possible. Amazon had a company-wide weekly tech talk series called the Principal ("POA") conversations. There were about ~1000 people (in the room and remotely connected). Anyone who wanted to speak at POA was assigned a few key engineers as coaches who would give endless feedback on endless dry runs. These coaches had to ultimately approve the speaker as being worthy of the POA brand before they stepped on stage. Giving one of these talks was a badge of honor, and I signed up because I was very uncomfortable with public speaking.
If you successfully gave a POA talk, you received a shirt that you could proudly wear around the Amazon campus.
Fast forward to 2014, and I had been promoted to principal engineer. My experience as a POA speaker was very positive, and it was largely due to my POA coaches. These coaches patiently listened to my talks and insightfully suggested many improvements. I grew significantly from the experience. It was my turn to pay it forward and be on the other side. So I volunteered to be a POA coach.
I enjoyed having several fantastic conversations through the POA process and seeing the people I mentored in a few years. And guess what? This time I got a second shirt. This time it was the "Coach" shirt!
I love wearing this shirt. Both are badges of honor. But I love wearing them because they remind me that it is my duty to mentor the next generation of engineers as they grow up.