Interview with Brian Stevens

Hi. I’m Don Marti and I’m here with Brian Stevens, Vice President of Engineering and CTO at Red Hat. Welcome to the podcast, Brian.

Hi, Don, nice to talk with you today.

Can you tell me a little bit about Red Hat’s plans on real-time and how some of the upstream kernel work fits into that, including Ingo Molar’s work?

Well, the real-time work started, you know, exactly through Ingo Molar several years ago. Originally it was some companies outside of Red Hat that were advancing real-time to bring it to their customers. Ingo joined that development effort with just an interest in making real-time a mainstream part of the Linux kernel. He saw performance improvements in audio and scheduling and whatnot. His work continues. We’ve since rounded out, and he's actually taken over the maintenance of that real-time kernel tree, and we’ve hired a couple of other developers that round out his team to actually make his life easier.

We've reached the point now that a lot of it's merged in Andrew and Linus's tree, but some of the biggest patches to fall are still around, like making the system fully preemptible. So, there’s still some major work to do in terms of merging it lock, stock and barrel into Linux, but we actually have found a product model that actually allows us, in the interim of that full-on merge to actually bring the capability to customers. So, we're going to be making a subscription offering around the real-time capability, so that customers that actually are seeing performance and a low latency advantage to running real-time can do so.

What kind of Red Hat customers do you find are asking for real-time?

Initially it's customers where outliers in terms of response time cost them real dollars, so a lot of the initial target customers are those in financial services that are running trading, market match networks that, you know, any -- you know, not any predictable performance and that actually have a fast performance, you know, translates into real dollars and then, you can imagine that in the Army and Federal services and whatnot, that’s there’s also the use of real-time in sort of conventional systems. So, a lot of the early pilot work has been with both those verticals.

So, in the financial markets, a lot of those customers are picking up nickels in front of a steamroller is the expression they use.

Yeah, exactly.

And what kind of real-time metrics do they look for?

Most of it is driven through a messaging layer. So, what we've found is that when we start to have a real-time conversation, most of our work is foreshadowed before we even talk to customers because they're well aware of where the Linux kernel community is going with real-time technology and Ingo's work. The affinity with Red Hat is well known, so the relationship with customers has happened long before we have a product to offer. When we have those conversations, what we often find is that a messaging system is part of the equation. So, you’re always thinking about grid level computing and distributed systems. The real-time performance of a system in isolation doesn’t really matter. It’s the response time and predictable performance all the way through the network. So, the typical workloads that we’ve seen so far have been message queuing systems.

Would that include the Advanced Message Queuing Protocol that Goldman Sachs is contributing to, along with Red Hat and some other companies?

Well, it doesn’t yet. So, the typical workloads that we're driving and characterizing with our customers today are largely around environments like Tibco and MQSeries, messaging solutions that they have in production so that they can consume the real-time work very quickly. But there is an interesting intersection there. The AMQP work originated about a year and a half ago from Red Hat where we found that observing those same products, that the performance gains that those products have, looking at it from a bottoms up, looking at it from a Linux kernel networking perspective, Linux on X86 was scaling faster than the messaging systems were.

So, that was really what the origination was around trying to use the open source innovation model that we believe in so much on the messaging problem. And so, the AMQP project originated through our survey of what technology, you know would be the best starting point for an advanced messaging system and the initial code contribution came from JP Morgan Chase and Red Hat built a relationship with JP Morgan Chase and we open-sourced the technology for them creating an Apache community around the -- and a working group trying to standardize the protocol.

And I think I remember reading that JP Morgan Chase already has an AMQP project in production. Is that right?

That’s correct. So, they’ve got an implementation in production because they just saw the problem. Many of these customers have multiple message queuing systems. They’re all trying to interface their applications together. Different ones solve different problems. They’ve got inoperability issues between the messaging platforms where they’re trying to bridge them and they’re in things like ESB’s that are trying to make systems interoperate that should, by nature, just be open. So, they've already been a very successful deployment. It’s had a huge impact on their business and they thought that through the work with Red Hat that they could scale that to another level.

Under the auspices of Apache, is that a Java-based project that's the implementation or do they have other languages implemented by now?

The original work was Java based and Red Hat has developed a C++ broker as well. So, you’ll actually be able to have a cross-platform broker that can run on Windows or Linux, and you’ll have a C broker that will really optimize any performance around. We’re even trying to drive a lot of the messaging in native Linux IO protocols. So, we’ll support both and then, with the API’s to developers being JMS, C, Ruby, Python, pick your poison.

So, whatever style of programming that development team is used to for a messaging protocol, they can write something similar and have it communicate over AMQP to any AMQP implementation out there.

Absolutely. So, that’s the point. You don’t have to be in Java just to be able to use this. There are existing standards that allow interoperability with legacy messaging queuing systems and then, you just picked up on the salient point. It's through trying to standardize the wire protocol that we can interface into even proprietary products implementing AMQP. We thought that was important to get ubiquity around a messaging platform.

Having just a Red Hat-led open source implementation that had to be dropped on the heterogeneous systems we didn’t think was going to have mass adoption. So, we do expect other vendors to be doing proprietary implementations that we interoperate with. In the end of the day that it brings the intersection with real-time that now we can have that conversation of quality of service and breakthrough performance all the way through the network layer as well, which we couldn’t do, and manage from a quality of service perspective, outside of the real-time kernel.

In the AMQP FAQ they sort of compare the semantics of it to how email works. You make a call to your AMQP implementation and and when it comes back, you're guaranteed that that AMQP message, like an email message, has made it through to whoever the recipient is. From the point of view from somebody who is using AMQP, what kind of performance measurements are they going to want to make on that system and what kind of results do they expect to see for those measurements?

You know, we're letting performance drive us hard so there are standards around some of these messaging APIs already. So, we know how they interface to applications, but we think performance is going to be a huge driver. That's why we started the project. So, right now, we've established a goal of a million messages per second. Whether we hit that by the time we GA remains to be seen, but we're well on track to hitting scalability levels that are five times further than any other implementation in the industry.

When you take a tightly coupled software stack like that, where you've got messaging middleware on top of a kernel and you try to virtualize it, how does putting a hypervisor or some other virtualization technology in the middle affect the kind of metrics that you can get out of the whole stack?

That's what's cool, what I like about Red Hat. When we do a lot of performance work, part of the solution will be grid and queuing work also built on top of messaging. We’re actually interfacing those into virtualized environments. So, we have a project underway that actually is making use of AMQP, itself, to build an orchestration system for our virtualized environment. So, think of sort of the existing virtualized environment and the toolings around it that are trying to manage workloads running on disparate systems and how do you actually manage eventing and monitoring and understand what’s happening where.

So, we're looking at actually using the AMQP system, itself, to actually service and drive an event-based delivery management system to an automation platform. So, a lot of our interest in messaging wasn't just to create a compelling messaging product as much as it was to also use that in next generation systems management.

So, does that mean a future generation of Red Hat Enterprise Linux just includes messaging or does messaging continue to be a separate product?

Right now, we're looking at an additive subscription that actually has a tight integration between our real-time work and the AMQP work.

One other area that large IT customers are asking about is energy efficiency. When you're talking about a system that has very short latency requirements and at the same time, you also have a pressure to try to put things into a deep sleep state in the kernel, how do you resolve that kind of conflict and how do you handle, say a big financial customer that wants to have a green data center and at the same time, quickly handle whatever the next big stock market event is?

There is definitely, at this stage of technology on power management, I wouldn’t say conflicting but a tough set of technologies and goals to intersect. You know, a lot of the early power management work in power states, being able to take advantage of those in the Linux kernel and then, still have efficient and reconciled timers has been difficult. So, I think we’re through a lot of those hurdles with AMD and Intel, but so often we’ve found in the past that you’ve actually had to, unfortunately, turn off power management if you wanted consistent timers.

So, I look at it as sort of the power management innovation that’s happening where the chipset manufacturers, their direct involvement with where the Linux kernel is going for native support, and then, sort of combining virtualization so that you can actually manage utilization. So, you can actually start to virtualize your servers. Now, all of a sudden, you can actually do best fit algorithms so there’s no reason to have a system powered up as well. So, instead of having the system just powered down in between cycles and have to understand when you're spiking, you can actually do compression algorithms to move workload on demand to the right system, understanding that the performance hasn’t dropped and actually, wholesale powering down extra systems in the grid. And virtualization is what really gives you that because it gives you the ability to separate the application from the box.

One of the new offerings out of Red Hat is a deployment model where you can use one of your existing Red Hat Enterprise entitlements on a virtual machine in Amazon's EC2 cloud. What kind of customers are interested in doing something like that, or is it something that’s being driven from the Red Hat side and you aren’t sure yet what kind of customers might want to do it?

You know, from our perspective, we work with Amazon’s IT department on a daily basis so we have a strong relationship with the company. They have a large commitment to open source. I think they are actually leading the marketplace and actually providing Linux on demand services to customers. We just want to, if customers choose and need full support services and need an ISV ecosystem, then we want to make that available.

So, we're seeing early stage companies that are using it for web hosting. We're seeing existing companies that might have some batch processing that had different hours that's non critical that they want to farm out. When you start to look at the numbers, the cost numbers, it's really a model that everybody wins, that Amazon wins, Red Hat wins and the customer wins.

What else can we expect to see from new Red Hat offerings? Should we just watch Fedora or where else can we learn about new stuff coming down the pipe?

Everything that we have under development is happening in public. I think that we stand unique in that and that the media and our customers are usually well aware of technology that’s going to be coming from Red Hat long before we productize it. So, Fedora 8, just made available in the last week, had 54,000 downloads and installs that we can even measure in the first four days, a vibrant development community around next generation technology whether that be KVM or appliances or spins or network manager improvements. So, Fedora is absolutely the place to watch the OS evolve.

Are we going to see a KVM based virtualization infrastructure in the next Red Hat Enterprise?

We haven’t made that decision yet so, right now, you know, all of our focus is around Fedora and RHEL5 so we haven’t made any product decisions on RHEL6 at this time.

Well, thanks for being on the podcast, Brian. Is there anything else you’d like to add?

No, Don, just thanks for your time.

Podcast version (15:33)