Last night was not a good night.
The main rack of servers hosting Scubbly.com were wiped out. The web server, and the backup drives – all of them were corrupted, and despite our efforts late into the night, they could not be restored. Massive tracts of the disk space were destroyed, and there was nothing we could do but wipe them clean, repair the ones that could be repaired, and rebuild Scubbly from source that was *luckily* still stored on one of our laptops.
So the code was saved. That’s good. In fact what you’re seeing today is a beta version of Scubbly that we were planning to launch in the next few days – a new design, and some nice enhancements to the cart checkout process.
But, the data. Oh the data… the main databases are backed up daily onto large storage drives, and we all figured that was a failsafe. I mean how often do multiple drives fail at once? Apparently one bad night is all it takes.
The latest complete data backup we had off-the-rack was from June 1, also lucky that someone had copied it off the racks and put it on a pair of DVDs… so we restored that. In addition to that DVD backup, we also have individual log files from every transaction since mid-October. There is a 3 month span from June to October for which we seem to have no surviving records at all.
From a forensic accounting point of view, we can also retrieve records from PayPal for all transactions.
What does this mean?
your balance
A lot of tedious, manual accounting work for us, that’s what. From those log files, we can recreate every seller’s balance as of the last payout – January 1 2012 – we’ll create a “reconcile” entry in every seller’s ledger, and we’ll work forward from there. It’s easiest to recreate things from January 1, since all the sellers who got their payout will have been at a $0 balance – a convenient starting point. Sellers who had a balance below $50 will have their accounting recreated from the original PayPal transaction logs.
your files
Unfortunately, there is another problem. Sellers who uploaded new products in the latter half of 2012 have their binaries stored in the secure distribution cloud: our rock-solid impenetrable fortress of data housed at several locations around the world. But the records of those products in our databases is what really matters – without those pointers connecting a “product” to a “bitstream”, they’re orphaned out there on the storage cloud and we don’t know what they are. Shucks for building the cloud with that kind of security, eh.
But we can recreate those products from the transactions logs. So if a product has sold some time between mid-October and last night, then we can recreate them. It will take many days and we’ll likely need to hire some extra help to do it… but it’ll get done.
If your product was uploaded after June 1, and it hasn’t had a single sale since October, then I’m afraid it’s likely lost for good.
Scubbly is OK today
We got the site back up and working again this morning at approximately 5am. Everything is hooked up, and you can use Scubbly today to sell your files. As far as we know, everything is OK… if you notice anything that isn’t working? Please let us know ASAP.
deadline: Feb 1
The ultimate deadline for all this repair work is the February 1 payout. By then, all accounts need to be reconciled and accounted so the balances are accurate and the payout goes to those who have earned it.
I’d like to emphasize that this wasn’t a security breach. No personal data was leaked, and this wasn’t the result of an external attack. This was an IT catastrophe that – needless to say – we’re going to be taking steps to ensure never happens again
