Custom Backup Solutions

As data volumes grow and systems become more complex, off-the-shelf backup tools often fall short of expectations.

Blueberry Consultants has strong expertise in the design of custom backup solutions for Windows and Linux servers. We can create backup systems that are tailored to your infrastructure and optimised to suit your data.

Introduction

IT systems are now so complex and data volumes so large that achieving reliable backup and redundancy can be a big challenge for many companies. Data volumes are increasing by 10-20% per year; users are more mobile, and the number of different systems used within organisations is increasing. At the same time, there is increasing awareness of the need for good data security and for disaster-recovery provision.

Blueberry Consultants' team includes strong expertise in Linux system configuration, Cisco firewall administration, SQL Server management, VMware administration and Windows development. This wide range of skills is concentrated in a tight-knit team, allowing us to work with any customer infrastructure and design effective backup strategies for even the most complex systems.

Backup Scenarios

Of course, many large organisations spend considerable amounts of money on backup tools. In many cases, these products and systems work perfectly well. In what situations might Blueberry’s skills deliver business benefit?

Scenario	Explanation
Optimisation of backup strategy to suit the data using custom compression and incremental transfer approaches.	Many standard backup systems are quite simple in approach – they effectively just copy files. But many file sets contain redundancy which can be exploited to dramatically reduce backup sizes and time to copy.
Handling special cases – e.g. backup of extremely large files over unreliable links.	Again, standard products may fail to handle extreme cases. We can design solutions to handle all possible cases.
Off-site backup or replication of databases.	Conventional backup strategies for databases tend to make daily backups. For MS SQL server, we can use transaction log shipping to achieve offsite backup with 15-minute resolution.
Integration of virtualisation technologies with backup systems.	Many companies are moving some of their servers onto virtualised platforms, like VMWare. This provides a strong opportunity to achieve backup at the level of the virtual machine, thus eliminating human error in deciding which files to copy.
Special requirements can often be met at low cost using off-the-shelf free tools.	Using only free Linux tools, it’s possible to create a simple backup system that gives users easy access to versions of their files from previous days – without needing to make separate copies.

Backup vs Redundancy

When discussing backup solutions with customers we often find there is a confusion between data backup and system redundancy. This isn’t particularly surprising, because the same technology may be used to deliver both – but we find it important to make a clear difference.

Backup is obviously about making copies of data at specific points in time – and ideally keeping multiple historic copies. System redundancy is about establishing a second copy of a system that can take over if there is a failure of the first system. Backup can deliver some level of redundancy – if you have a complete backup of a server, then you might be able to bring a spare server into service within a reasonable time. And redundancy can deliver a basic level of backup, because the redundant system typically has a full backup of the primary system. But neither are good solutions.

Tools and Techniques

Technology	Details
Rsync	A very commonly used tool for incremental backup of files across the WAN. Most commonly used on Linux, but will work on Windows.
Duplicity	A more specific Linux file backup tool built on top of Rsync technologies. Has superior handling of network faults, and can use less CPU.
MD5 / SHA checksums	Modern checksum algorithms are used extensively in backup and file transfer to identify files and confirm successful transfer. We have access to optimised libraries that perform these calculations particularly quickly.
BB FTE (File Transfer Engine)	A Blueberry-developed library that supports reliable block-based transfer of binary files over HTTP/S, with per-block checksums and strong resume capabilities. FTE is superior to file transports such as FTP and plain HTTP because it detects block level errors and retries at the block level.
MS SQL Server Transaction Log Shipping	This is an MS SQL Server feature which allows efficient continuous backup of SQL server databases. With correct configuration, SQL server will write out a file containing all the changes to a database every 15 minutes. This file can be sent to an offsite server and restored. The backup sever does not require an MS SQL licence.
VMWare	A popular virtualisation tool. VMWare allows multiple virtual PCs to run on a single host machine. The relevance for backup is that the virtual machines can be suspended and then copied to a backup server.
DRBD	A Linux technology used to allow reliable replication of disk volumes across a LAN. We’ve used this to establish auto-failover for VMWare servers.

Case Study – ABC Ltd

ABC Ltd has a hetrogenous collection of Windows / Linux servers located at a central data center, including a number of systems running VMWare, and some SQL Servers. The company needed to demonstrate to clients that they had a disaster recovery plan in place, and that all key data was backed up offsite.

Blueberry designed a backup plan based on three different data types used within the ABC network – conventional user files, SQL databases and large virtual machine images. A single new server was installed at a remote location, and configured wtih a Cisco firewall and a dedicated 24mbps DSL line. For the most important user files, a daily rsync job was used to replicate the files securely over SSH. For the SQL databases, transaction log shipping was used in conjunction with Blueberry’s FTE system to replicate the databases to a pararell SQL server running on the backup system. For the large VM images, duplicity was configured to run on a slow incremental cycle, establishing backups of 200gb of VM images on a rotating montly basis.