No SSH Here: Ticketmaster’s Fervent Take on Containerization, Integration, and Change
We frequently talk about the innovations of the new stack — containerization, parallelization, resilience — as though they were inevitable progressions in the development of all information technology. And we typically characterize them as principles of a developer-led revolution that is remaking not only the architecture of software but the culture of the IT workplace.
But are we observing the cause or the effect? A panel discussion during the Tuesday late afternoon keynotes at the KubeCon conference in Berlin brought to light a few real-world use cases that tilt our revolutionary argument on its ear. In particular, Justin Dean, the senior vice president for platforms at live event access provider Ticketmaster [pictured above right], discussed his firm’s top-down approach to the imposition of containerization as a policy for his development teams.
It was a direction that no one could have expected for the conversation: a story of senior executives making the decision to execute a total transition to containerization, and of developers who, from their vantage point, can’t change their old coding habits soon enough.
The Fast and the Furious
“A couple of years ago, just like every other industry, we started realizing speed is the name of the game,” said Dean. “We needed to get faster, or we were going to start losing chunks of our business to competitors. We needed to be able to deliver software faster than small companies. Being a large company, it’s really, really, really hard to be nimble, and roll out new products and features to compete with a startup or two people in a garage.”
Dean didn’t say the word, the same way students at Hogwarts are never heard saying, “Ralph Fiennes,” but the competition he was referring to was StubHub. It’s the ticket reselling arm of eBay, and by many estimates, the world’s largest secondary market for event tickets. After eBay purchased StubHub in 2007, it began rebuilding its e-commerce platform. Though it took a few years, StubHub eventually gained the world’s attention, including with a then-novel approach to applying analytics to combatting fraud in 2012.
StubHub’s software developers were making names for themselves in the open source space, and still, today, maintain several projects on GitHub. In 2013, one StubHub developer named Akshay Vyas co-presented a session at a mobile Web developers’ conference, entitled, “Trends in Developing Mobile Web Applications.” Vyas made waves, and onlookers took notes. Today, Vyas is Docker Inc.’s head of UI engineering.
According to a Bloomberg Businessweek article printed in early 2015, StubHub resold some $3 billion in concert and sporting event tickets in the U.S., in a market whose annual valuation that same year had been projected at only $5 billion. Then in February 2016, StubHub announced the development of a complete, ground-up ticket sales management platform, involving a new and reimagined mobile app. It was fighting Ticketmaster on all fronts, from cyberspace to the courtroom.
Ticketmaster needed a completely revamped, reinvigorated platform. And it needed it yesterday.
Transformers
“A decade ago, we built a lot of amazing tools for the company,” explained Ticketmaster’s Dean. “We had a private cloud, and NFS to everything. We had our own configuration management system that we built, that does all kinds of sweet things for you… It made it really, really easy for [developers] to get tied into a million-and-one dependencies.”
These developers’ livelihoods seemed to have been ensured by the existence of a homegrown toolchain upon which everything in the business depended. The only way their comfortable, entitled perch would be threatened would be if Ticketmaster’s leaders invoked the nuclear option: repeal and replace the existing platform, instantly. That’s what happened. Not only was the entire platform to be chucked, but the remaining fragments of the developers’ pool were told to remain small and Agile-like. Each was given responsibility for its own, granular code project — and responsibility meant total ownership.
At one point, Ticketmaster was maintaining 150 concurrent continuous integration pipelines.
In the midst of all that, Ticketmaster chose Docker containers, and subsequently, the Kubernetes container orchestration engine for bringing order to chaos. But then the execs made a decision to deploy its applications on a public cloud — and not for the reasons you normally think of.
Dean said the adoption of public cloud was meant “to force every team to really address all the underpinnings of their software.” He told KubeCon attendees that the decision to move away from the comfortable, self-built nest that developers had built for themselves, into this new world of containerization, was decided wholly at the top and driven from the top down.
He admitted that this was a challenge, for the whole company. “All these things, you have to figure out yourself,” Dean said Ticketmaster developers were told.
“So the barrier to entry was pretty steep. When you sprinkle in that it’s an all-new language — all these technologies were fairly new, if you’re not really paying attention. If you’re on a development team, and you’re writing business software now, it’s like, ‘Oh, now I’ve got to learn all this lingo, and figure out how I’m gonna do this!’ From the end user perspective, there’s a pretty big hurdle of things you have to know, to be able to successfully take a legacy application and get it into a container without all the constraints, and then get it into a build pipeline and move it along.”
Grosse Point Blank
Transitioning projects from a private dev/test environment to a public production environment, he said, forces developers to adhere to one method. Without the promise of that single method, he went on, Ticketmaster would not have mustered the funds to invest in this new platform.
It’s an argument you might not hear, even from your own organization, but it may actually be true: Deployment to a public cloud forces dev teams to regularize their processes. In a private space, where developers can often have free reign over workflows and processes, multiple teams can concoct limitless permutations of how to go about their jobs. At one point, Dean told the audience, Ticketmaster was maintaining 150 concurrent continuous integration pipelines.
Narrowing down those pipelines to a more comfortable few dozen or so meant Ticketmaster’s execs had to stop caring about certain details, in order to intensify their focus on those details that they believed yielded results.
“How you build your software is not really the concern of this particular movement, at the moment,” said Dean. “It sort of assumes that you’re using whatever you’re using to get it to a certain point. In order to get entry into our clusters in our world, we’re trying to standardize on going through GitLab CI and forcing everything through a very strict manner. It’s got to be in a Docker container, it’s got to be in GitLab CI. Now with Helm, we’re trying to get more in line with standard templating of delivery.
“But it’s still — I don’t know — not super-mature, I would say. There’s a lot of teams doing whatever ‘du jour’ that they use,” he continued. “I would say there’s a bit of work there for us to get to a point where we’re true continuous delivery, continuous deployment, and what not. But the only way to get entry into this world is if you’re coming from a GitLab CI world. So at least we’re using that as our, kind of, choke point.”
The development workflow Justin Dean drew for the KubeCon audience involved narrowing developers’ options as much as possible, repeating words including “standard,” “strict,” and “force,” and invoking that surprising phrase from the realm of assembly line engineering — “choke point” — to drive home his argument that Kubernetes could make life easier for enterprise IT shops by narrowing their developers’ deployment options to one. However, you make your code fit through that choke point, do it.
“One of the biggest things you can do to get your container world right is, force that ‘no-access’ to it thing,” he said at one point. By “no-access,” he meant eliminating SSH logins into active containers, thus enforcing their process isolation as they were originally designed. Developers’ natural instincts, he said, are to make quick changes here and there, or test things on their own whim. “When you eliminate that, you force a lot of pre-thinking and a lot of maturity for your tech stack,” he said.
Eliminating the option for peeking into a container and seeing what’s going on, he went on, compels dev shops to utilize synthetic monitoring tools, time-series data, and more sophisticated alerts, making containers smarter so that they can take care of themselves.
Blue Thunder
Ticketmaster’s tale tells of an urgent, survival-of-the-business campaign to transition the mindset of its developers from a proprietary paradise to an open source bazaar. That campaign is far from finished. Should other large enterprises take their cues from Ticketmaster? Or might there be a danger in imposing an all-or-nothing transitional mindset upon their developers?
The New Stack put those questions to two of IBM’s own leading developers.
“If you could get support for that at first within your enterprise, then good. I don’t disagree, necessarily, with that viewpoint,” responded IBM Distinguished Software Engineer Andre Tost, a 28-year veteran of the company. “I just don’t think it’s realistic, especially for the enterprise we talk to. There is just no money in, ‘Let’s keep one kind of viewpoint,’ which means we would need to take literally hundreds of existing enterprise applications, and refactoring them and re-shifting them into containers.”
Tost and colleague Kyle Schlosser, who serves the Office of the Hybrid Cloud CTO, are long-time containerization advocates who have helped customers with transitioning existing n-tier software to containerized environments. One method they’ve recommended involves actually containerizing the middleware that the code from these legacy applications expects to see — a method that implies a much more gradual, multi-stage approach to a container exodus.
“The danger I see is that it’s done in a kind of half-baked way, and the expectation is that you’re going to gain all the benefits from it. But because you simply do a ‘lift-and-shift,’ you still need to manage them like you managed them before, because they’re not really meant to run in a microservices style.”
But that’s not the course Ticketmaster chose: It wants microservices and scalability and parallelism, and to heck with legacy. Perhaps it avoided the danger Tost foresaw.
“From all the clients we’ve talked to, we’ve seen a lot who’ve chosen, rather than to shift the workload to a container-based system,” said Schlosser, “take the approach of providing the right type of integration, so we can interact between those systems. In a lot of cases, that means exposing the APIs from existing systems, so they can be consumed from microservices — use that as a way of bridging those two domains.” (The New Stack will present more from IBM’s Schlosser and Tost in an upcoming feature on containerizing middleware.)
But as Justin Dean told the story, co-existence was not an option. Ticketmaster was competing with a platform for which the lack of legacy was its greatest asset. If you think about it (and you don’t have to think for very long), this situation could be applied to the leading firm in almost every industry on the planet. Adopting Ticketmaster’s radical approach — expediting the cutting of the cord, and forcing standardization through the establishment of choke points in the workflow — will be an option that these leaders will consider.
We like to portray open source as your choice. We haven’t talked much about it when it’s someone else’s choice.
⚡️ “CNCF Project Updates” from #Kubecon #CloudNativeCon https://t.co/FwH4Ozo5Xd
— The New Stack (@thenewstack) March 28, 2017
The Cloud Native Computing Foundation is a sponsor of The New Stack.
