Managing the Unique Demands of Virtualization
Few technologies have changed IT as completely as virtualization. Instead of being an incremental improvement over previous approaches, virtualization fundamentally changed how we think about IT, became the de facto default for new deployments, and spurred decades of innovation. While powerful, virtualization places unique demands on the data center.
Optimizing Resource Utilization
Prior to the advent of virtualization a physical server could run only one Operating System Environment (OSE). If one wanted to run multiple workloads on that physical server, one had to install those applications into the same OSE.
Without the use of containers – which have only recently become usable by most organizations – running multiple workloads on a single OSE presented multiple problems. These workloads could – and did – regularly interfere with one another. Additionally, compromise of one workload led to compromise of all workloads. These, and other issues meant that server utilization pre-virtualization hovered around 15%.
Virtualization changed all of this by allowing servers to run multiple OSEs. Each workload could be wrapped in their own operating system, and if one should happen to crash it wouldn’t crash all the others. What’s more, virtualization allowed multiple different OSEs to be run on a single physical server: one server could host Windows, Linux, BSD and more, each with completely independent workloads.
Virtualization’s ability to run multiple OSEs on a single host led to greatly increased server utilization. Organizations saw vastly improved return on their capital IT investments.
In order for virtualization to be maximally effective, virtualization hosts must join into clusters, which each cluster having access to some form of shared storage. With shared storage, virtualization can enable functionality such as High Availability (HA), Fault Tolerance (FT) and Live Migration (LM). These technologies dramatically increase workload uptime and availability during maintenance cycles or in the event of hardware failures.
Unfortunately, the adoption of virtualization led to administrator sprawl. IT departments often needed separate administrators (or entire teams) for networking, storage, physical hosts, OSEs, applications and more.
This administrator sprawl had some major downsides. Apart from increased personnel cost, administrator sprawl frequently made problem resolution more difficult, as each team pointed to another as the source of the problem. In addition to responsibility conflict, provisioning of new workloads and time to complete change requests lengthened as each team needed to complete their tasks and sign off before the next could engage.
As virtualization become more common and more capable it absorbed not only more of the existing workloads within organizations, but birthed entirely new workload types that were not previously possible. Stateless Virtual Desktop Infrastructure (VDI) and burstable workloads are the canonical examples of workload classes enabled by modern virtualization.
These new workload classes required their own administrators. Early implementations of these workloads also required entirely separate infrastructure, with dedicated virtualization hosts, storage and so forth. Virtualization’s impressive array of features enabled entirely new capabilities for organizations, but these capabilities were leading resource islands, once more raising the spectre of underutilization of IT assets.
As the number of infrastructure components supporting virtualization grew, so too did the number of vendors providing those infrastructure components. The number of management interfaces administrators had to interact with to accomplish a single task grew proportionately.
A decade after x86 virtualization became mainstream, technology organizations were faced with having to tackle separate interfaces for networking, storage, physical hosts, OSEs, applications and so forth. Worse, many organizations had to handle multiple interfaces and multiple vendors for each of these layers, in addition to new vendors and new management interfaces for emerging workload categories such as VDI.
A decade after x86 virtualization became mainstream, technology organizations were faced with having to tackle separate interfaces for networking, storage, physical hosts, OSEs, applications and so forth.
All of this complicated issue resolution. In addition to administrators within an organization passing the buck from one team to the next, the vendors involved often blamed one another. This was compounded by traditional IT solutions being difficult to update, upgrade, scale up and scale out. Data center refreshes became expensive “forklift upgrade” nightmares, with hard cutovers between radically different infrastructure designs that could have months or even years of repercussions elsewhere in the organization.
Along came Hyperconverged Infrastructure (HCI). The basic goal of HCI is simple: instead of having an external, centralized storage array to provide shared storage customers would put storage into each of the virtualization nodes themselves. A layer of software would run on each virtualization node that would join all the storage from all nodes in a cluster together, providing shared storage without an additional third-party device.
While the concept seems simple, it was spectacularly difficult to achieve. To accomplish its goal, HCI has to not only do away with external storage arrays, but it has to provide an adequate storage solution for any and all workloads.
All of this has to be done in as automated a fashion as possible so that HCI’s storage can be managed by virtualization administrators. In one fell swoop, HCI eliminates the need for dedicated storage administrators, reduces the number of vendors and management interfaces required and begins the process of flattening data center design.
The net result of this is that data center administrators can start moving up the stack, away from the hardware and closer to the applications. With HCI, organizations can divert spending from keeping the lights on towards other priorities, such as automation and business process modernization.
The Resiliency Problem
Before virtualization, storage lived inside individual servers. In order to allow a server to survive the failure of and individual disk, multiple disks were made to operate together in a Redundant Array of Independent Disks (RAID). This provided an important measure of resiliency to workloads that were increasingly becoming mission critical to the organizations of the day.
Eventually, organizations sought to make servers resilient to more failures than just storage. HA solved this problem. HA is the ability of a virtualization cluster with shared storage to respond to the loss of an entire host by restarting workloads on another host in the cluster.
As uptime became more and more vital to organizations of all sizes HA became more than just a nice-to-have feature, it became a table stakes capability. Unfortunately, the requirement of shared storage for HA – and other increasingly vital virtualization features – to work made life difficult for data center administrators.
The I/O Blender
In order for a computer to process a workload’s data it must be able to load and store that data. If a computer cannot load or store the data quickly enough to feed the processor then system utilization will be low. Storage – specifically secondary storage – is a classic data center bottleneck. When virtualization is discussed, it is this secondary storage – hard drives and SSDs, not RAM – that is serving in the role of shared storage amongst nodes in a cluster.
A single workload on a single server using the storage resources of its own, dedicated RAID array is able to make relatively large sequential requests for data. Hard drives are good at sequential data access. They’re absolutely awful at random access.
If you have two workloads attempting to access the same RAID array then you increase the likelihood that any of those nice, efficient sequential accesses by one workload will be broken up by requests from the other workload. The amount of time the array spends performing random access increases. This is known as the I/O blender effect.
Centralized Storage Arrays
Centralized storage existed before virtualization. For organizations uninterested in providing each server with its own RAID array, a centralized storage array was used. This would provide each physical server with its own Logical Unit Number (LUN). A LUN can be thought of as a dedicated slice of the centralized storage array.
While pre-virtualization centralized storage experienced I/O blender issues, they were mitigated by the realities of pre-virtualization IT management. Before virtualization, each workload needed its own physical server. In addition, pre-allocated LUNs – potentially with dedicated drives in the array – could be combined with Quality of Service (QoS) policies and other optimizations to ensure critical workloads had enough storage throughput at low enough latencies to operate.
Even in the days before virtualization, balancing, monitoring and managing all of this was required specialist knowledge. After virtualization individual servers could hold dozens or even hundreds of workloads. A single LUN would regularly be used for more than one workload and individual arrays were being asked to handle many times the load that they would have been before.
Virtualization made creating, managing and maintaining workloads easy, but this came at the cost of making storage infinitely more complicated. Storage management had to move up the stack, away from the array. QoS policies needed to be applied to individual VMs.
Storage administrators managing virtualized environment now had to manage their storage arrays, the virtualization management software and quite possibly several layers of third party monitoring, auditing and QoS policy software as well.
For a while, centralized arrays kept up with demand. As organizations added more and more VMs array vendors would add more and more shelves of disks. If the controllers became overwhelmed, a project would be undertaken to split the array. Unfortunately, this resulted in its own inefficiency: in order to meet the performance requirements of virtualization far more disks were employed that was required on a strictly capacity basis.
The Biggest, Baddest Blender
Just as the I/O blender issue was reaching a crisis point, Solid State Drives (SSDs) burst onto the scene. While they were – and remain – far more expensive than traditional magnetic hard drives on a capacity basis, they wreck magnetic disks as regards performance.
Random I/O doesn’t affect SSDs in the same way that it does hard drives. While the I/O blender effect is devastating to hard drive arrays, it simply wouldn’t affect an all-flash array. SSDs alone couldn’t save the centralized storage array, however. The cost of all-flash was simply too high.
While the I/O blender effect is devastating to hard drive arrays, it simply wouldn’t affect an all-flash array.
Consider for a moment a sample of the various workloads storage administrators have to design for:
Structured data (Databases): Databases themselves tend to feature predominantly random I/O while database log files tend to be highly sequential. Both are critical the operation of the database, but constitute two entirely different problems to solve. Databases tend to load as much of themselves into RAM as possible, meaning that their storage interactions are largely writes.
Unstructured data (file and object storage): Unstructured data access tends to be highly random. Large numbers of users are served from a single storage point. Unstructured data is, however, heavily read-intensive. The overwhelming majority of unstructured data is accessed less than 15 times and then never accessed again.
Enterprise Applications: Large scale enterprise applications – for example Microsoft Exchange or Oracle Peoplesoft – each have their own unique access patterns. Each will be a different mixture of ransom and sequential I/O, with a different ratio of reads and writes. Many will back onto a database for at least part of their workloads, and some will produce multiple layers of logs, or large quantities of unstructured data.
VDI: As one of the canonical examples of entirely new workloads enabled by virtualization it should come as no surprise that VDI brings its own completely unique storage requirements. VDI demands are remarkably bursty; hundreds of VDI instances could cheerfully perform standard day-to-day operations off of a single SSD, but in certain configurations a monthly update cycle could overwhelm even a top of the line all-flash storage array.
Traditionally, storage administrators have traditionally had to figure out how to balance the demands of all of these different workloads whilst paying attention to the characteristics of the underlying storage. Not all I/Os are equal. Read requests, for example, are always going to be faster than writes.
Writes simply require more steps than reads, especially where data efficiency such as deduplication or compression are involved. Writes need to be acknowledged, and different media handle writes in different ways. Writing to an SSD, for example, requires reading all data in the block where the write is going to go, erasing the block and then writing the new block, which may contain data you wanted to write as well as data that was previous occupying part of the block.
Reads may also be coming out of a RAM cache, and thus not impact storage at all. Similarly, technologies such as server side caching might be put into play. This puts an SSD into each node to act as a cache tier, which is a complicated and expensive attempt to mitigate the access problems of addressing slow centralized storage arrays.
For those that couldn’t afford the raw power of an all-flash array an entire decade of increasingly complex Rube Goldbergian attempts to build storage solutions resistant to the I/O blender emerged. Every single one of them, however, retained the critical flaw of being difficult to manage, especially when the number of workloads under management was absolutely exploding.
In a world where enterprises had so many workloads under management that they could only be safely addressed using complicated scripting and automation something on the storage side had to give.
Storage arrays themselves evolved. Three major evolutions occurred essentially simultaneously. Hybrid arrays became possible; one could put a combination of SSDs and hard drives into a single array and software smarts inside the array would determine where to put the data. Frequently accessed data would be stored on SSDs and infrequently accessed data would live on slower magnetic disks. In and of itself this was a revolution.
Alongside the development of hybrid storage was the introduction of VM-aware storage. These were storage solutions that didn’t look at storage in terms of LUNs, but instead integrated with hypervisors to view and manage storage on a per virtual machine basis. Again; this was revolutionary. It made storage significantly easier to manage. Arguably, it made storage possible to manage at modern data center scale.
The third major evolution of storage was Software Defined Storage (SDS). SDS extends the basic logic of hybrid storage from using software to tier hot blocks towards attempting to automate as many storage tasks as possible. Most SDS consists of generic, whitebox servers filled with various hard drives and SSDs joined together into a single storage cluster. One of the key goals of SDS is scalability: if you need more capacity, or more performance, you simply add more storage into the cluster.
HCI is ultimate evolution of all the above storage trials and tribulations, and can be thought of as SDS with a ruthless focus on efficiency.
From one point of view, HCI is nothing more than a specific class of SDS. While SDS is still typically just a slightly less awful way to build a centralized storage array, HCI looks at SDS and says “why don’t we put the hypervisor on the storage nodes and use the spare capacity of those nodes to run workloads”?
HCI does more than simply bring storage and compute together. That storage is integrated into the virtualization management interface and made functionally invisible. An entire IT administrator specialty dedicated to determining how to place, balance and manage storage is replaced by a series of algorithms. HCI can use all hard disk, all SSD or a combination of both to build the most efficient possible storage with what is made available. No knob twiddling required.
HCI can use all hard disk, all SSD or a combination of both to build the most efficient possible storage with what is made available. No knob twiddling required.
Because of this, HCI can run multiple workload types on a single cluster. An organization wanting to combine VDI with enterprise applications, and run them both alongside databases is no longer flirting with disaster. That sort of mixed workload operation is just par for the course with HCI.
Another key to HCI is scalability. Traditional storage arrays would literally have to be moved in and out of data centers with forklifts during refreshes. Migrations from the old array to the new were long, complicated, miserable affairs. HCI clusters start as small as three nodes and scale up on an as-needed basis. Migrating from older nodes to newer ones can be done without taking any workloads down and is a simple, painless process.
Hyperquality of Service
One of the key advantages of modern hyperconvergence is that QoS is integrated directly into the solution. While much of the configuration required to optimize storage is handled automatically by various algorithms, modern HCI solutions offer administrators the opportunity to override the algorithms in specific circumstances.
By default, HCI works to ensure that all workloads get the best possible performance from available cluster resources. In the majority of use cases this is the right approach, however, exceptions do exist.
An HCI cluster that has been designed to meet requirements 80% of the time will still experience demand greater than its ability to deliver 20% of the time. The cost of specifying the cluster to meet that last 20% of demand may be prohibitive, especially if there is an alternative approach.
If specific workloads are configured to be “more equal” than others, then during times of resource contention they will be assured the resources they require while other, less important tasks are starved of resources. In many cases, this is a perfectly acceptable and even desirable approach; most organizations have background or batch tasks that do not perform operations whose output is required in real time. With simple QoS controls virtualization administrators can manage these priorities without needing specialist knowledge or dedicated storage administrators.
Some of how this QoS is managed lies in how modern HCI solutions manage their storage. Advanced storage techniques such as wide striping allow HCI clusters to make use of all storage resources across the whole cluster, sharing the storage load equally instead of concentrating it on one node with a hyperactive workload. In addition, in modern HCI solutions SSDs capacity is added to the total capacity of the cluster – an approach known as subvirtual tiering – and not treated merely as a caching tier.
Evolution in Action
HCI is the result of decades of ruthless focus on efficiency. Every time the solution to a performance or capacity problem in the data center resulted in inefficiency the market evolved. Vendors responded with new technologies and new approached.
With HCI scaling and migrating are both as easy as adding nodes to the cluster. Workloads are made resilient without compromising the ability to place multiple workloads on a single node. Enterprise data services, once punitively expensive, are now a standard part of the offering.
Multiple management interfaces are collapsed into one and multiple layers of infrastructure are automated. HCI takes what was once an entire specialty of IT and eliminates it from view, codified in adaptive algorithms.
Giving virtual administrators granular control of workload performance at the VM level without requiring specialist knowledge is the purpose of HCI. IT infrastructure has come a long way.