A well-designed and tested corporate data backup solution is more important than you think. Most anyone responsible for maintaining a computer system has some type of backup configured. Hopefully, they understand your system well enough to back up the data correctly and have the experience to ensure the backup solution will capture everything necessary to recover from the myriad of potential failures.
Telmar has extensive experience with all manner of corporate computer systems, and we have witnessed most types of failure. Security breaches that result in lost data, hardware failure that renders storage unusable, or user errors are all too common; more importantly, all can be remedied by restoring the data from a backup.
The risk of an incident is undeniable: one study from Positive Technologies demonstrated that competent hackers could gain unauthorized access to the corporate network 93% of the time (https://www.ptsecurity.com/ww-en/analytics/pentests-2021-attack-scenarios/). Considering most attacks ultimately result in some form of data loss – be it deleting critical data or purging the logs documenting the attack in progress, there is little doubt your corporate data backup solution will be called into action.
A properly-tested backup solution is essential for business continuity.
WHAT ARE YOU BACKING UP?
There are a multitude of different applications and technologies in use today. Examples include simple file servers that house the corporate-wide file share, Active Directory domain controllers, database servers, users’ laptops, and specialized application servers. Each of these requires a specific approach when performing the backup to ensure the data on the backup media is usable.
Fortunately, most backup solutions on the market support each of these scenarios, so there are fewer details to worry about. The backup vendor likely includes options and settings to handle various workloads:
- Active Directory, possibly even with a granular object-level restore
- Awareness of Microsoft SQL Server’s sophisticated transaction log functionality
- VSS integration to backup open, locked files
- Cloud solutions, such as Exchange or SharePoint
Even with such help from the vendor, it is critical to understand what is needed for the various recovery scenarios. For example, did someone delete an Excel file? Do you need to restore your database to a point in time three hours ago? Did you fall victim to a ransomware attack? Or did the building with your servers burn to the ground? Each of these examples requires a different approach to recovery, which means each scenario might require different things to be backed up.
How Do I Learn What to Back Up?
A common myth among technology workers is that the default settings in the backup software are sufficient. As a result, they believe that a full backup of the virtual machine will satisfy any requirement. But, unfortunately, plenty of situations are not effectively covered by a full backup. You can get to the data, but it might not be efficient. Work with the vendor to understand the options available. Keeping in mind the idea of Trust, but verify (https://en.wikipedia.org/wiki/Trust,_but_verify), take the time to test recovery under various conditions.
Not All Data Is the Same
Backing up standard workloads like file servers or Active Directory domain controllers is straightforward in any major backup tool. There will be plenty of documentation about configuration and best practices; take the time to review and implement them.
Complex or less common applications require a bit more care. Plenty of flat-file database systems are still in use today; standard backup techniques are not always sufficient for such a system. The laptop in the corner that manages all of the key fobs in the building needs to be recoverable.
In all cases, start with the vendor that supports the application to understand best practices. Consult documentation related to the core technologies. Contact your backup vendor. Whatever you do, make sure you have worked to understand the key components of the software for recovery purposes.
Where Should I keep my backups?
A good backup solution will store the backed-up data in a location and format with at least the following attributes:
- Close to the source server with high-speed access to facilitate quick recovery
- Far from the source server to ensure a physical disaster does not destroy the backup along with the original data
- The backup can be used as a virtual machine without fully restoring it, facilitating quick recovery
- The backup must be locked down in immutable storage, ensuring ransomware cannot encrypt it
- The backup needs to retain historical versions as indicated in company policy
Conflicting Requirements?
Notice that these are conflicting requirements. It is not possible to store a backup both “close” and “far” from the source server. Using an immutable, read-only copy of a backup as a virtual machine without restoring it should not be easy, as that archived copy should be difficult to access. Very old backups that are maintained for archival purposes should be stored on cheap storage that is slow to access.
The lesson here is that your backups should be stored in multiple locations.
An example scenario for a small organization might be:
- Keep replica copies of key VM’s in the same environment as the production VM’s; providing for very rapid recovery in the event of a software issue in the VM itself.
- Keep backup copies of VM’s onsite, but on different storage and computer hardware; do not store backups on the SAN that houses the associated VM.
- Archive the backup copies to an offsite server. Those copies should be encrypted, and the offsite server should not be in the same security domain as the onsite servers. Gaining access to the offsite server should require separate credentials. If the software supports it, make use of any available immutability functionality.
Substantial Risk
If you engage a company to store any portion of your data, read the Agreement carefully. Understand the risk you are assuming by using their services. Consider engaging two different companies and storing data at both.
This example is for illustrative purposes only. It allows for rapid recovery via the local replicas, can survive storage hardware failure via the onsite backups stored on alternate hardware, and provides protection for both physical disaster and ransomware attacks via the offsite backups. It does not cover other appropriate scenarios for larger organizations but is a starting point.
Is Your Data Backup Usable?
Now that you have poured a lot of effort into the backup solution, testing your backups is critical. I will repeat that: you must test your backup solution. Period.
I cannot count the number of conversations I have had with technology specialists that have never performed a restore in their production environment. When I ask a few questions to prompt them to do some testing, the response is normally frustration or disinterest. It is sad to me and unacceptable to their company because there is too much at risk.
Plan for Success
Start by considering each scenario you are working to protect against, and make a testing plan. Building burns down? Check! Mary deletes the depreciation Excel file? Check! Ransomware encrypts the engineering server? Check! Windows Update renders the server unbootable? Check!
For each scenario, consider what the recovery would look like. Also, consider possible variations of the recovery. For example, if you need to recover the Excel file, you could potentially fetch it from a server replica or the offsite archive.
Execute the plan, Regularly and Consistently
With this information, make a test restore plan and execute it on a scheduled basis. Telmar has a standard process we go through in our environment every quarter. We have adopted this process as it has identified countless issues over the years and is directed at finding the most common failure points.
Did You Know?
Configuration errors are very common in backup systems. For example, a new server isn’t added to the job, new software requires different configuration settings, or something happened to the service account that a job runs under. Continual monitoring and testing is the only way to mitigate this risk.
Every quarter, we perform the following tests on Telmar’s production environment:
- Point-in-time recovery of two production SQL Server databases
- Compare the list of servers in the backup jobs to the list of servers in the virtual environment, ensuring nothing is missed
- Recover key files and database backups from our Veeam backups of multiple servers, selecting a historical date at random. The source of these tests is also selected randomly, ensuring that we check our replicas, onsite backups, and offsite backups.
- Test recovery of an entire virtual machine from a backup
- Test recovery of an entire virtual machine from a replica
- Failover to a replica, and failback
We run through this plan each quarter and correct any issues that are identified during testing. As a result, several configuration issues have been identified over the years; the time is never wasted.
What might be missing?
It is important to have an open mind when considering backups and recovery testing. If you forget to back up whatever it is, there is no way you can recover it later!
Talk to people who understand the company’s technology and consider other things that might have been missed. I also find it helpful to talk to end-users; frequently, I discover something unexpected but critical in those conversations.
Spending time evaluating this process is important to ensure its success. Nobody wants to be in the hot seat when you cannot recover from a failure. I have been in those meetings, and it is ugly. Take the time needed to create a reliable solution.
Unusual Tip
End users are a wealth of useful information. It can be a struggle to sort through less-relevant items, but there is frequently a little gem that shows up in the conversation. Take the time to talk to your users!
How Can Telmar Help?
Telmar has extensive experience managing computer and database systems. One of our unique service offerings is our ongoing Data Maintenance Plan. We tailor your plan to your specific needs, ensuring that critical tasks, including test restores, are performed regularly. Typically, a finance or operations executive engages us to check up on the internal or external IT Support group, providing certainty that their data is being protected. We specialize in bridging the gap between the technical and operational groups in a company, removing communication barriers, and promoting change that benefits the business as a whole.
Providing a Level of Comfort
Our goal is to verify that every reasonable effort has been made to configure backups properly and test the results regularly. As outside contractors, we are not bound by the same political constraints as internal resources. Instead, we act independently, from the perspective of the business as a whole, ensuring that personal motives do not interfere with the continuity of your business.
We accomplish this by working with end-users to understand what data is important to them, and with technology workers to understand how the data is housed. We use all of this to create a plan, and we validate the execution of that plan monthly or quarterly, ensuring the recoverability of the system.
The result is a level of comfort that critical data can be recovered in the event of a disaster.
Our passion for data protection is undeniable.
If you would like help ensuring that your backups are properly designed and being tested regularly, please Contact Us.
NEED HELP MAKING YOUR DYNAMICS GP ENVIRONMENT SIMPLER TO MANAGE?
Learn how the Telmar Team can help!
Telmar Computer Solutions, Inc. is providing the information in this publication as a courtesy, free of charge, to share information with the general public. Nothing in this publication is intended to be, nor should be, interpreted as advice or consultation on any matter, including taxes, accounting, business management, system configuration or operation, or any other similar topic. Please engage the services of an appropriate professional for assistance in these areas. Telmar Computer Solutions, Inc. is not responsible for any loss, including downtime, lost files/data, rework, or other expenses sustained by you, your company, or any person who relies on this publication.
0 Comments