Hello All
Iâm writing to you today to provide you with an update about a recent data loss event that affected a small percentage of our VPS customers. This extremely rare event has created a significant service disruption for the customers affected, and we are responding to it with the utmost in urgency.
On Saturday, August 28, 2010, around 11am US EST, several customers began to report they were unable to access their websites hosted on their HostV VPS. It soon became clear that the issue only affected few nodes (internally labeled nodes 21-24), which house a very small percentage of our total customer base. Given the unique characteristics of the outage, we initially suspected a hardware failure was responsible for data being inaccessible. On Sunday, we emailed affected clients letting them know we were prepared to recreate their accounts and issue a service refund for the last three months.
Since we maintain several layers of backups (including RAID protected disks and R1Soft-powered CDP servers), we were prepared to immediately restore data for our customers. Although our Terms of Service agreement explicitly states that customers are to maintain their own backups, we also offer our backup services as a convenience, and we were prepared to utilize those assets.
During the day on Sunday, we continued extensively investigating the incident. After attempting to access backups, we noticed that not only had all VPS accounts on the affected nodes been terminated through the Virtuozzo Web Portal, but additionally, our primary backup control server had been wiped clean. Whatâs more, all of the relevant log files had also been destroyed. It became clear this outage was caused by a malicious criminal act, designed to cause immediate service disruption for our customers. The actor in this situation was somebody who maliciously gained access and clearly understood our infrastructure design.
HostV maintains industry-standard security practices. Each of our servers is patched nightly through Ksplice Uptrack, which ensures all known security vulnerabilities are patched as soon as patches are released. Additionally, we strictly control access to our servers, even among our staff, and regularly patch and update servers as needed.
At this time, we are contacting law enforcement to request assistance. We have strong suspicions on the origin of this attack, which we do not believe was orchestrated by a current staff member. But, the damage may be done, even if we are able to prosecute the perpetrators.
Most importantly, we are working around-the-clock to help our affected customers. We still do have a full backup node available, but since our primary R1Soft control server was wiped, we have been unable to restore the data. We are working with R1Soft and other data recovery consultants in the hope we can restore this data, but at this time we have not been able to do so. Restoring this data is our number one priority at this point in time.
For our customers who are able to restore from a personal backup, we have re-created their VPS on a different node, and have provided service credits. We are also working to refund the last three months of hosting services for all affected customers per request via billing along with assistance in restoring any backup data customers have re-uploaded.
It is our pledge to continue to update you on the facts as this situation unfolds. We have taken additional, undisclosed security measures to ensure an attack like this cannot happen again.
If you have any questions, comments, or concerns, please contact us. You may also private message me here directly with further questions or for callback.
Thank you again for your understanding and patience.
Sincerely,
â¨John Xie
Founder & CEO
HostV
