It’s been a year since my company sent everyone to work from home due to the pandemic, but I’ve been in the office, keeping the lights green and the wheels spinning.
I manage technology for part of a large American media company. In my division, I’m responsible for “everything that uses electricity.” That covers everything from laptops to servers to petabytes of storage to an entire application suite of tools that allow the creation of high-end graphics, audio, and video all connected with a fast ethernet network. Oh, and I worry about powering all these things 24/7 as well.
While many businesses can operate within a web/cloud environment, professional video post-production still requires powerful workstations, significant amounts of server-class hardware to render/convert video, and enormous amounts of high-speed storage.
Someday, we’ll get to doing this all in the cloud, but today, it’s just not practical. Pushing petabytes into AWS is a non-trivial task.
As the COVID situation worsened in February 2020, we started discussing how we might work from home in earnest, but the challenges were daunting. Many staff with workstations didn’t have laptops. Our capacity for remote editing was at a proof of concept stage and didn’t scale to hundreds of users. Getting our staff to understand the basics of how to use basics like VPNs and video meetings was trying, especially with them frustrated in having to learn a new way to work.
I was lucky that we had started a few “business continuity” proof of concept projects in the year before COVID hit. We built out a cluster of virtual workstations (usually called ‘VMs’) that directly connected to our post-production systems. By using high speed remote control software, this gave us the performance we needed from outside the company to operate. We also had worked with a third-party vendor to basically create and use the same technical set-up, but externally hosted.
The first week of March, I was away from the office, in Iceland, remotely trying to work with my team on planning what we’d need if a shutdown occurred. Personally, I wasn’t even sure if I was going to be able to get home, or whether I’d be quarantined somewhere. I had about two weeks worth of medicine with me, just in case. It was a tumultuous and uncertain time.
We developed a plan that might work, but it really hadn’t been done before at the scale we were discussing. The stark realization of what we needed was gut wrenching. Things grind slowly in big companies and we had to move fast.
I got back to Los Angeles on March 6th. No quarantine. Back in the office, the team worked to nail down the exact details of a plan we could execute. Six days later, on March 12th, the company told everyone that could, go work from home. The game was afoot.
That afternoon, I found myself in the office of the President, with the CFO, explaining the plan and the hundreds of thousands of dollars it would cost to build out what was needed and get laptops in the hands of everyone. Best case, we were months away from bringing more remote editing capacity online. Worst case, it simply wouldn’t work.
There was a strongly-worded argument that occured in that room as to whether the shutdown was going to be over in few weeks or not. The kind of intense scene you see in movies. I was mainly quiet, just giving numbers and yes or no answers while the debate raged loudly. In the end they told me to do “whatever is needed” and I placed the orders that night. This was the answer I wanted, but the responsibility was now placed firmly on my team and me.
Looking back, that conversation looks crazy, but at the time the uncertainty was off the charts, and no one really knew how hard COVID would hit the entire world and change almost everything about how people work.
In the office, most of the staff disappeared, but the core post-production team stayed to keep the “factory” working. We pushed the two remote projects into service, each limited to about 20 people working at a time.
I sent most of the engineering team to work from home. My rationale was that the two of us on site would probably get COVID at some point and be out for the count. Plans were made on who would be the next to come into the office if we fell sick. My family was super stressed at this point, as the virus seemed out of control and I was still going into work. I remember verifying if my life insurance was still in good standing, as I felt as if catching the virus was a matter of “when” not “if”.
The pressure in this moment was almost overwhelming. People were now at home, most with no way to work. Everyone feared widespread layoffs. My small team was tasked with finding a way to put all of these people back to work, and quickly. New hardware was months away and we had few tools to help.
After another week, the shutdown hammer came down in Los Angeles County and almost everyone was sent home leaving a small crew of about a half-dozen in a building that normally holds 500+.
That was a stressful time, trying to get laptops delivered and explaining how to use them. Keeping the backend running with my engineering team mainly at home and inventing remote practices on the fly. Planning the expansion of our VM systems. Racking, cabling, and powering hardware under pandemic conditions. Explaining to frustrated execs that it was not business as usual with the usual flow of video out the door. Trying to calmly explain to non-technical people about wifi, VPNs, and authentication over the phone. Trying to explain CDNs, network topology, latency, jitter, wifi channel collisions to non-technical staff that insist that “Netflix works fine, so why can’t I edit” tested our patience. Repeatedly telling people to mute their mics on video calls became a meme.
Soon, we had to start rotating time shifts to use the remote systems so more people could work. From 6AM to Midnight, we had three different shifts of people swapping in and out of the machines. Poor internet at home meant poor performance of remote editing. Frustration was high.
The next couple months were difficult, developing new workflows to utilize post-production staff at home, getting into the rhythm of video meetings, getting the needed laptops and home setups out to people, dealing with software licensing issues, and all the while trying to build out our hardware systems to handle the needed load. Pandemic safe processes were being developed on the fly by the supply chain of vendors, shippers, mail rooms, and on-site teams just trying to keep the wheels turning.
We tried to make the workplace as safe as we could, scrounging for cleaning supplies, masks, gloves, sanitizer, and other supplies while the store shelves of America were empty. We opened every cabinet in the building looking for hidden resources we needed. it felt like a zombie disaster film, as we celebrated finding a set of Clorox wipes under a desk.
Soon the Corporate groups started taking the on-site conditions seriously and mysteriously a plethora of stickers started appearing all over the building floors. Stickers to tell us which way to walk in the hallways, which sinks to use, and other “helpful” instructions that seemed quite out of sync with the reality of life in an empty building.
To destress, at times we would walk the halls, looking for office plants to water. Trying to find some normalcy in a time of chaos. A respite from the anxiety flooding into our inboxes and swamping us in Slack. I wish we had focused on this earlier as the dead plants still remain in offices, reminding us of our failure to save them.
The realization that this work at home situation would not be going away in weeks or months started to take hold with the staff. This is when I started getting asked to retrieve personal items.
Everyone was a little different on what they wanted sent to them. For some it was notebooks and printed material. Many wanted some of the family photos from their desk. We sent out yoga mats, pencil sets, personal computer mice, vintage computers, throw pillows, etc. We viewed it as doing what we could to make them feel better about the situation.
Outside my office, I keep a small basket of candy for the few people in the office. We started adding a small bag of candy to deliveries when we could to provide a little extra to those stuck at home.
By mid-summer we had upgraded enough software and installed better remote software that everyone could work simultaneously. Performance was good, not great, with some people having frustrating problems we simply couldn’t fix for them. The internet service providers are under tremendous load and some areas simply have bad service, that my team can’t fix for them.
We started to get into a rhythm, as the ‘crisis’ needs became fewer and we dealt with more mundane issues, mainly replacing laptops due to various predictable fates, being dropped and having coffee spilled into them.
Often, we have to enter offices to reboot or update computers, as many things still cannot be managed remotely. These moments sit with me. At someone’s desk, we still see the notes from the week before shutdown. It’s as if they just took the day off, nothing changed.
Looking around the room, you see the space they created, filled with personality and things special only to them. I have no context for most things at people’s desks. While waiting for the computer to reboot, I look at the objects and imagine why they are important. A concert ticket stub from the 90s. A plastic party mug from a bar in Florida. A set of Hot Wheels cars. A light saber. A lego box set. Offices of people that have left the company, still filled with personal items that have never been retrieved. Booze. Just so much booze in people’s offices, waiting for a Friday afternoon get-togethers that aren’t going to happen anytime soon.
The holidays were strange. Zoom meetings just don’t compare to cookie exchanges and white elephant gift exchanges. Traditions matter to people, and without them, marking time is hard.
As we approach a year in the empty offices, most of our day to day operations have transitioned and we are well out of crisis mode. Still not operating at Beforetimes capabilities or ease, but we get the job done.
Most days, there are only five or six people in the entire building, each in our own space, adhered to the safety rules. Masks at all times when we have to talk in person. There are “COVID” patrols in the halls that occasionally look for us breaking the rules.
These days, I bring my lunch to work and eat outside, alone, on a balcony meant for dozens, away from the ding of email and the braap of Slack. At some point, people will be back and I’ll have to share the space.
I’m eager for a vacation, but there’s nowhere to go. I’m anxious to get the vaccine. I fear getting COVID after a year of avoiding it, when the end is in sight. The weight of holding so much together for so many people presses on me. I just need to bear it for a little while more, trying to avoid falling to pieces when the next crisis appears on my plate.
But, as my father would say, “pressure makes diamonds”.
After this, I’ll be able to cut glass with my fingertips.
