Expertise

Custom VMWare Solutions

Custom software solutions using VMWare virtualisation technology can deliver significant business benefits.

Blueberry Consultants has strong expertise in the development of custom solutions based on VMWare technology.

Introduction to VMWare

Virtualisation technologies like VMWare are now a strong feature of corporate IT departments, allowing a reduction in the number of individual servers and easier server management. At the same time, this technology has also opened up new possibilities -for example, companies engaged in software development can use VMWare to support testing on many different PC platforms.

But virtualisation systems like VMWare are by definition quite technical, and there is often a requirement to create an additional domain-specific management interface for end-users. Blueberry has developed such interfaces for two customers in the UK and the Netherlands.

VMWare also allows system designers to build redundant or high-availability (HA) systems by allowing a VM to be run from more than one server, depending on hardware availability. Blueberry has implemented a Linux-based server redundancy system to do just this for one customer.

Case Study I - Custom VM Management UI

VMWare and other virtualisation systems provide strong web-based management UIs, which allow administrators to start, stop and suspend individual VMs. However, in applications such as software testing, the end-users often require a more tailored management interface.

Blueberry Customer ABC Ltd undertakes software localisation and localisation testing for large software development companies. Delivering localisation testing requires that ABC has access to test machines with all possible combinations of Windows OS version and language. With perhaps 4 current Windows versions and 30 languages, this means they need to have up to 120 variants available. In practice, users also need to install additional software on their test machines, and retain the test machine for the duration of the project. Because of this a template system is required: firstly 120 templates are created in all possible OS combinations. When a new project starts, an end user specifies which combinations are required, and the system makes copies of the virtual machines for the project. An engineer can then install the software under test on each VM. End users (testers) then access the VMs remotely using Terminal Services (RDP) or VNC.

To deliver this functionality, Blueberry built a custom VMWare management system for Rubric. Based on the free edition of VMWare, the web-based management system provides an interface that is structured in terms that ABC’s users understand - such as projects and languages. The management system also enforces business rules: end users are not allowed to see or interact with the template machines, and some level of load-balancing is introduced – if a VM is not used for 5 minutes, it is automatically suspended.

The ABC custom management system was implemented using a number of technologies. The web-based management interface was built in ASP.NET and SQL Server. On each VM host, a special script was written to provide a web-services wrapper for VMWare’s own perl API. Custom code was added to automatically generate NAT mappings for new VMs, so that end-users could transparently connect.

This system has been used successfully on a number of high-profile testing projects, and is currently managing three large VM servers.

Case Study II - Custom VM Management UI

Blueberry customer, UPC Broadband, based in the Netherlands, includes a software test team responsible for testing internal software systems. These systems normally include multiple database and application servers. With multiple versions of these systems deployed, and many different systems in development, the test team needed a way to manage testing resources. They were already using VMWare as a platform to host test systems.

For UPC, Blueberry created a custom management UI similar to the ABC system, but with a different focus. UPC needed to manage systems at the “cluster” level – they wanted to be able to start, stop and suspend a group of VMs that together made up a test setup for a particular application. They also had specific requirements for IP address allocation: because their systems often used load balancing, some of the servers within a cluster needed to have fixed IPs during a test. The UPC system used two main VMWare servers, and one control server running the management UI.

Case Study III - Custom VMWare Server Redundancy Systems

Blueberry customer ABC wanted to upgrade their application server hosting facility, and also achieve a greater level of redundancy after experiencing problems with server stability.

At the time, VMWare did not have the high-availability solutions that it has today, so Blueberry investigated whether there was some way to use VMWare to achieve server redundancy.

The solution adopted was to use two Linux host machines, and run the Linux DRDB network replicator software. DRBD allows a partition on one Linux machine to be continually replicated to a second machine. Every disk write that takes place on the primary server is also written to the disk on the secondary server.

Blueberry’s approach was to setup a VM partition on the primary server, which is then replicated by DRDB to the secondary server, which has exactly the same configuration. In the event of a hardware failure on the primary server, the VM images could simply be restarted on the secondary. The virtual machines would lose any data in memory, and they would effectively behave as if they had suffered a hard reboot.

In addition, we configured a night-time backup of the complete VM. This was achieved by suspending the VM on the primary and flushing the disk. DRDB was then disconnected temporarily and the backup image on the secondary server was mounted and copied to a backup file. Then DRDB was reconnected. The effect of all this is to make a nightly copy of a suspended VM image (so data in memory is also included) on the secondary server with only 2 minutes downtime of each VM.

This system has now been deployed for many years, and has run well during this time. At one point, a hardware failure on the primary server required the VMs to be started on the secondary, and the system worked as designed. The customer was pleased.

However, we did find some limitations of this approach. DRDB’s requirement for a dedicated partition made disk space management very time consuming, as changing the partition size meant copying 100GB of VMs to a different place. On our systems, which were using standard SATA drives to reduce cost, DRDB did have a significant performance impact.

In addition, technology does not stand still, and VMWare themselves have solutions that deliver high availability for VM images. However, VMWare’s approach seems to require that all VM images be hosted on a common SAN – which is both expensive and introduces a new single point of failure (the SAN).

Conclusions

Blueberry has strong skills in VMWare API programming, Linux system configuration and networking programming, and .NET Web Application development. Combined, these skills allow us to deliver custom VMWare Solutions which can achieve significant business benefit.

In particular, we are able to develop custom VMWare management interfaces which allow VMWare based software test systems to be delivered to end-users in familiar terminology and functionality.