Large mission-critical software deployment efforts are often compared to the task of changing engines on a jetliner during a flight.
After spending five years working on the replacement of the PBS traffic and scheduling system, the general consensus here is that it actually goes way beyond the jetliner analogy. In the next three installments of this column, I will try to impart some of the trials and tribulations associated with this enormous effort, which is finally coming to its conclusion.
As we move from a tape-based environment to what essentially is a file-driven digital media workflow, our transformation is more akin to going from ocean-liner trips to Star Trek-type teleportation than a mere in-flight engine change. But let's begin at the beginning.
As with any other such process, we went through a detailed requirements phase. We painstakingly examined every aspect of our operation, identifying all the pathways through which programs arrived at PBS, and all the tasks necessary to finally put those programs on the air. It quickly became obvious that there was no such thing as a standard process. Our entire workflow, not unlike what I heard from my colleagues at other network-level broadcasters, was composed of an interminable sequence of one-off exceptions that only by the grace of immensely dedicated people throughout all levels of the enterprise came together as an on-air schedule with a minimum of discrepancies.
In addition, as we embraced the possibilities of digital multicasting and explored high-definition distribution, what had been an intricate ballet of complex distribution rights issues, invariably late content, tape-driven technical impairments and last-minute additions or modifications became a constant struggle to ensure error-free performance across a burgeoning number of schedules.
Now it would be easy to see this as an indictment of the people or the organizations involved in all aspects of this process, but what all of us in broadcasting have experienced is the initial quake and the substantial number of severe aftershocks associated with the migration to digital environments after a very long stint in a world dominated by tape. As our systems and workflows evolved throughout the years, they became more and more complex to accommodate all the new business requirements without having the luxury to stop, re-examine and optimize our processes.
TIME TO START DIGGING
So it was time to start digging out.
As part of this requirements analysis phase, it became obvious that we needed to think not only about accommodating our ongoing business requirements, but also how to handle all the issues associated with a brave new media world with myriad distribution methodologies, even more complex rights management and "fly-by-wire" metadata-driven business processes.
Supply chain management became an imperative and, as you can easily imagine, this was a process that did not easily lend itself to the disciplined rigors necessary in a manufacturing environment. We continually unearthed variations on the processing methods across series, episodes, near-live and live programs. Just when we thought that we had found the last iteration, along came yet another nuance that necessitated another trajectory correction.
Eventually, it was time to move to the next phase. Armed with an encyclical of our requirements, it was time to do a market analysis and find out if there was a product that, off-the-shelf, could fulfill a substantial percentage of our requirements. One of the rules-of-thumb in software systems procurement and development is that if you need to customize more than 30 to 35 percent of an existing software package, you are probably better off writing one from scratch.
This phenomenon is caused by the triple-headed monster of the initial costs and risks associated with a large number of customizations, the typical maintenance costs charged by large software vendors and above all, the future costs associated with upgrading customizations whenever the underlying package goes through another release cycle.
Much to our chagrin, after scouring the market and looking at packages from U. S., European and Middle Eastern manufacturers, we came (at the time) to the unassailable conclusion that if we wanted to satisfy the majority of our business requirements, we would have to develop this system.
And so began a four-year odyssey of iterative divide-and-conquer development cycles that little by little chipped away at our requirements while allowing us to continue to fly the plane toward its final destination.
To summarize, we had taken off, we knew our destination, we knew that the plane we had was not going to get us there, and we knew would have to create one ourselves. So it was time to decide which part of the aircraft to remodel first. We settled on the gasoline intake systems, and I will tell you about that in the next installment.
Count on IT!
