Payout for March 1, 2015

The biggest payout ever! Today Scubbly paid out for sales made before the 60-day clearance period, which means we’re finally getting paid for all those Christmas purchases. Hooray!

Worth noting: After being announced two months ago – and then the one free month with no storage fees – today is the first time that the new storage fee policy was applied. Each seller who has more than 60MB of files in their space will be dinged with a few pennies per hundred megabytes. For most of you it will make no difference at all, for some you might see that the charge was a few cents more or less than usual.

One consequence of the policy change is that right now there are hundreds of accounts that have a negative balance, for the first time ever. These are “dormant” stores set up by people who uploaded a pile of little files, never did any marketing, have not made any sales, have no profit, and were – for the first time today – charged $0.03 or $0.06 this month for their storage. As a result, all those accounts are showing a balance that is a few cents “in the red”.

To those who are currently holding a negative balance – don’t fret! Scubbly is not going to make you pay back a few cents; it’s not worth it. Concentrate on marketing your product, and make some sales. The profit from even a single sale is usually more than enough to cover a few pennies of storage cost.

Accounts that maintain a negative balance for a long time will be handled on a case-by-case basis. If an account has no active products and no profit, but is accumulating storage fees, we’ll contact the seller to discuss the situation. If a seller doesn’t respond to repeated attempts to contact them, we might shut down the account and delete the files.

Congratulations to all the sellers who earned a payout today. Keep up the good work!

New Feature: Sales reported by Country


A great feature was added yesterday. This one was a fantastic suggestion by seller RightBooth, who has a pretty neat software product for setting up a photo booth system. RightBooth is preparing to submit taxes, and discovered that they need to know the country for each buyer, because VAT is applied differently at different rates depending where the buyer lives.

So, as of yesterday, the Country Of Residence – if it is known – is now reported as part of every sale confirmation email. Also, there’s a new report under Account > Stats > Sales, which can show you a summary of the sales you’ve made broken down by country.

On that note, you should already be aware that Scubbly does collect or remit taxes for your store. We have sellers all over the world, and buyers from all over the globe too. The complexity of figuring out taxes that apply between one country and another is an insanely complex problem that would add significant complexity to the software, so we’ve never tackled it. Plus, for the vast majority of Scubbly sellers – who might earn a few hundred or perhaps thousand dollars per year – the income generated is low enough that taxation is irrelevant.

So it is the seller’s responsibility to determine if their Scubbly income is taxable, adjust their prices accordingly so that they can pay their taxes to maintain a legal business in their own jurisdiction. Scubbly tries to provide the reports and data necessary to do that with relative ease. If you find that it’s difficult to get the information you need from the system, feel free to contact us and tell us about it! RightBooth did, and we quickly enhanced the system to deliver the needed data.


Important Announcement: changes to oversize fee calculation

Hello sellers,

The monthly payout just went out, but there were no oversize fees charged. For this month only,  we’re not charging for oversized storage – because we’re in the midst of implementing and communicating a change to the formula we use to calculate the fees.

The previous formula we used was: “$0.03 per 100MB or part thereof in excess of 60MB, per month or part thereof”. This was applied per product – and only to products that had a downloadable file that was greater than 60MB.

What we discovered was that the formula is OK – it adequately covers costs of file storage – but the way we were applying it was unfair. Along with many other changes that happened in early 2014, sellers were given the ability to attach the same files to multiple products, and bundle files in different configurations. The side-effect of that change (and our formula) was that the same large file might incur the oversize fee multiple times.

Applying the fee to individual products also meant that a seller with a single 1GB product would be charged a fee, but someone with a hundred 10MB files would be charged nothing, even though they were both consuming the same amount of storage space.

The new calculation uses the same formula as before, but we’re applying the formula to the sum of all a seller’s files – not to each file individually, nor to each product. That means that sellers that have thousands of small files are no longer getting a free ride – Scubbly will charge the cumulative size of all their storage used. Sellers who bundle the same files into multiple products will be charged only once for that file’s storage.

This change is going into effect exactly a month from today, on March 1, 2015. The wording on the fees page will meanwhile be updated to describe the fee accurately, and an email is going to be sent to all sellers who will be significantly affected by the change.

Ian Ring


Improving fault tolerance, and disaster recovery measures.

Hi Folks,

As promised in December, I’ve figured out why the crash happened in December, why the server didn’t just restart automatically when it went down, and what will be implemented to prevent future incidents of the same sort. I’ll keep the technical details succinct.

The crash was caused by a series of unfortunate events: a memory leak, a password protection on reboot, and the unusual situation that everyone who had the password for rebooting was away on vacation and couldn’t be reached.

There’s an image library at Scubbly that is used to generate thumbnail images of the pictures you upload. That library is known to gobble up RAM and not release it. Over time, the loss of RAM will cause “swapping” to the disk. Swapping is bad. The cure for swapping is a reboot of the web server. It happens rarely, but when it does there’s hardly any service interruption at all. And there’s an automatic process in place with our web host that watches for swapping to trigger a reboot. All that stuff is set up and it’s good.

The failure can be blamed on password protection on the reboot process. Scubbly has authentication measures protecting pretty much everything from being screwed with. So when the web host rebooted the machine that is hosting Scubbly, the system required a password – literally typed in by a human logged in to the machine – before it would restart the web service.

Only one person has that password. That was me, on vacation with my family, at a resort with no wifi where no email could reach me for over a week. As we disembarked in Toronto after a week of sun and mojitos, I switched my phone out of Airplane Mode, that’s when the hundreds of downtime critical alert messages flooded in and I learned that had been in a coma for over three days. All I had to do is go home and type in my password to restart Apache. It took only a few seconds to fix the problem, but it was waiting for several days because of my own unavailability.

If there had been no password impediment (allowing the automatic reboot process to succeed) or if there was another person who could have intervened, then the interruption of service would have been completely avoided.

So. Here are some steps to prevent similar problems in the future. The first will be done immediately:
1) We will use a different authentication signing method to secure our web service, so it doesn’t require a human to be there when rebooting. This way our automated disaster recovery procedures will actually work without intervention. I’ll have that fixed some time in February.

The next two will happen some day, but not immediately:
2) Add some redundancy to the service by starting up a second server with a load balancer. Then if one machine goes kaput with RAM issues, the service will still be available running on other machines.
3) Some day Scubbly will invest in “managed hosting”. Instead of running on self-serviced machines maintained by me, the site will be managed by a hired team of professional geeks with 24/7 customer assistance and immediate incident management. Hosting like that is expensive, but it is where Scubbly is headed.
The impediment for those last two is budgetary – Scubbly will be able to do them when the service is earning enough to cover the increased costs.

That’s all. Once again, please accept my apologies for letting Scubbly flatline like it did.

Ian Ring



New Feature: videos!

Sellers will notice that yesterday a new feature was added: you can embed a YouTube video onto the product detail page.

Many types of products are difficult to describe using words alone. Being able to embed a video on the product detail page means you can deliver a preview to your customers so they can hear and see the product before buying. Video is also a powerful medium for persuasive marketing.

Scubbly has chosen YouTube as our video streaming service. Quite an obvious choice, really; they’re ubiquitous and reliable and have an excellent embedding feature that works on all devices and platforms. To embed video on Scubbly, first you must create a YouTube account, upload the video there, and then get its ID. Go to the new “Video” tab in the product editing tool, enter that ID and save the product.

For an example of a product using video as a Preview, look at this one:



December 13 – 17: Unexpected interruption of service

There was a long and unexpected interruption of service this past week. The interruption was caused by a total system crash of the machine that runs the website, and it occurred at a very unfortunate time, when no person was available to reboot the server for several days. The machine that serves was merely off, and the automated processes that are supposed to turn it back on didn’t do it properly.

Scubbly service was unavailable for approximately 3 days, 11 hours; from Saturday, December 13, 2014 19:43:30 (UTC) until it was manually rebooted at Wednesday, December 17, 2014 7:20:00 (UTC).

During this time, anyone who might have visited the website for any reason would have received either: 1) a timeout attempting to contact the Scubbly server, or possibly 2) a “service unavailable” error.

To sellers and customers, I apologize for the service interruption. We are investigating the reason for the crash, and for the failure to restart the service immediately following the crash. At this time we have no reason to suspect any foul play, and there was no damage to any historical data. All systems are working fine now.

Obviously, 3 days of service interruption is terrible. We want Scubbly to be working as close to 100% of the time as possible. Once we have identified why the crash happened, we will be using that knowledge to design a more fault-tolerant system, or to put processes in place to mitigate or manage “disaster recovery” situations more expediently; this could involve changes to Scubbly in ways that we can’t currently predict; any changes that will affect sellers or customers will be communicated here in the Scubbly blog.

With sincere apologies,
Ian Ring


Scubbly on POODLE: all is well

For those of you who keep up on security news.

Within a day of the security alert being publicized, we disabled SSL v3 on all our servers. Scubbly is not vulnerable to the POODLE hack. All is well.

Cheers ~

Design changes

We’re doing a little cosmetic tweaking back here… please forgive the dust.

That light blue and gray background wasn’t doing it anymore. We’ve bleached the entire site to a pure white background, in preparation for some nicer treatments of the navigation icons.

Incidentally, we’re also tinkering with a new logo design. Why not shake things up a little, right? There are a whack of new features coming down the pike, so it’s as good a time as any to do a little rebranding and aesthetic touchups on the UI.


Site is back up, after some brief downtime was down for a while this morning, from a little after midnight until 9:15am (Eastern Time Zone), a downtime of about 9 hours.

The hosting company that serves Scubbly has not yet produced an incident report explaining the outage, which reportedly affected many other customers, not just Scubbly. I can attest that the entire service was off.

During the outage, everything related to Scubbly was off – including new sales, widgets, and downloading.

So far as I can tell, everything’s back and I’ve restarted all the web services, and everything’s OK. Here’s hoping it stays up. I’ll comment here to update this story with news as it becomes available upstream.



Announcing the Scubbly API – and New Widgets

Wow! Hard to believe that after over a year of work, the Scubbly API is finally being released. This has been a HUGE project, which started in the summer of 2013, continued over the winter and spring, involved a giant release of a totally revamped technical architecture in the spring of 2014, enabled major UI improvements throughout the summer of 2014, and now the underlying API is ready with a public-facing endpoint for autumn of 2014.

This API is the engine that makes Scubbly work. This isn’t just a limited toy box of little utilities that we grudgingly allow you to fiddle with. This API is the back-end software that does the heavy lifting for the actual website. With only a few exceptions, everything you can do on the public website can also be done via the API. With this API, you could indeed build your own fully-featured app. Not surprisingly, now that the API exists, building more Scubbly apps is something we want to do — On top of this powerful architecture we want to build new web applications, mobile apps, plugins, and integrations. Now, Scubbly isn’t just a website where you can sell downloads; it’s a platform.

Now, bear in mind that this is a BETA release of a product that is still under quite heavy development. It is possible that the endpoints, actions, parameters and syntax might change without notice. We’ll almost certainly be adding new actions, and refining the existing ones to make the entire API as consistent and coherent as possible. If you want to start using the API, please contact us first! Let us know what you’re planning to build, so we can open a channel of communication and collaborate to make sure those parts of the API interface you’ll be using are coherent, perfect, and stable.

You can find the API at Account > Integrations > API.

On top of the new API, there’s also an experimental set of new widgets! The widgets are actually a set of custom HTML5 markup tags that let you embed dynamic, functional Scubbly assets on your own website. Now, on your own third-party site, you can embed Scubbly’s product details, cart, and checkout functionality, for seamless integration between your marketing materials and your Scubbly store. The technical voodoo that makes them work is pretty impressive, too. Now these widgets are VERY experimental, some of them are a little buggy (for instance when you specify different languages or currencies) — so use them with caution. Try ‘em first.

Go admire the new widgets at Account > Integrations > Widgets

There’s a lot of activity going on & the platform is growing, and getting better all the time. We haven’t yet finished some of the stuff we started in the summer, like the changes to the storage fees. No worries – we’ll keep the fees as they are for a while longer. There are some wicked new features coming soon that will knock your socks off…