I should have known better honestly.  I’ve had one strike with cloud billing catching me by surprise and I’m not sure why I’m shocked it happened again.  This time, however, I thought I really did plan it out, pay attention to things and asked what I thought were the right questions.  Unfortunately I didn’t get the full answers.  This time I was stung by my shiny new SQL Azure service choice.

UPDATE 12-APR-2012: Based on comments I've received I feel the need to clarify that I'm not bashing Azure or cloud services in general here.  I don't think anywhere I indicated Azure was a crap product or that I hated it at all.  In fact, I indicated I was completely happy with the service offering.  My frustration was *only* with the fact that the pricing was unclear to me based on how I researched it...that is all and nothing more.  As many have pointed out, cloud services like Azure are extremely important in the marketplace and the ability to scale real-time with minimal effort is an exceptional feature.  *FOR ME* I currently don't have those needs so I couldn't justify the charges beyond what I had planned...that is all, nothing more.  My experience with SQL Azure was a positive one as a product.  Quick setup, familiar tools to manage, worry-free database management, great admin interface and a reliable data storage solution.  My architecture, however, just didn't prove ideal currently with my site not being in Azure as well.  When VM roles come out of beta I will be sure to evaluate moving sites there and plan better.

A while back I heard about the change in price for some Windows Azure services and the one that piqued my interest was the SQL Azure.  At the time it hit me right as I needed to move around some of my hosting aspects of my site.  The lure of the $5/month SQL Azure database (as long as it was < 100MB) was appealing to me.  The SQL server aspect of my site has always been a management headache for me as I don’t want to have to worry about growing logs, etc.

Stung by marketing

I followed the announcements to the http://www.windowsazure.com site and read the descriptions of the services.  I was immediately convinced of the value and heck, it was a service from my company so why shouldn’t I give it a try and support it?  When I began to set it up, however, there were questions being asked during setup and I started to get concerned.  I asked around about if this $5 fee was really the only fee.  I didn’t want to get surprises by things like compute time.  Perhaps I wasn’t asking specific enough questions, but all answers I got was that signs pointed to yes, that would be my only fee.

NOTE: As of this writing yes I am a Microsoft employee, but this is my own opinion and I realize that peoples’ expectations and results vary.  This is only my experience.  I’m not only an employee but also a customer of Microsoft services and in this instance a full paying customer.  No internal benefits are used in my personal Azure hosting accounts.

Yesterday I learned that wasn’t the case.  I received my first Azure billing statement and it was way more than I expected.  Yes my $5 database was there as expected, but also was suddenly “Data Transfer” charges of $55.

Trying to make sense of billing

I immediately tried to make sense of this billing.  I immediately remembered that I had created a storage account as well for a quick test and perhaps I forgot to disable/delete that service.  I logged into the management portal and saw that my storage account was properly deleted and nowhere to be seen.  But how to make sense of these charges from the past week then?  Luckily Azure provides detail usage download data so I grabbed that.  The CSV file I download did indeed provide some detail…perhaps too much as some of it I couldn’t discern, namely the one piece that I had hoped would help me: Resource ID.  This ID was a GUID that I thought pointed to a service that I used.  It did not, or at least that GUID was nowhere to be seen on my Azure management portal.

I contacted the billing support immediately to help.  I was able to talk with a human fairly quickly which was a plus.  The gentleman explained to me that I had a lot of outgoing data leaving the Azure data centers and that was the source of the costs.  He asked if I knew if anything was connecting to my SQL Azure instance externally.  Well, duh, yes it was my site!  He went on to explain that this constitutes “Data Transfer” and I’m billed at a per GB rate for any data that leaves the Azure data center. 

I took a deep breath and asked where this was documented in my SQL Azure sign-up process.  We walked through the site together and he agreed that it wasn’t clear.  After being put on hold for a while, I was assured I would receive a credit for the misunderstanding.  Unfortunately for Azure, the damage was done and they lost a customer.

Where the failure occurred

For me the failure was twofold: me for not fully understanding terms and Azure for not fully explaining them in context.  I say “in context” because that was the key piece that was missing in my registration of my account.  Let me explain the flow I took (as I sent this same piece of internal feedback today as well) as a customer once I heard the announcement about the SQL Azure pricing changes:

• I received notice of updated SQL Azure pricing
• I visited the site http://www.windowsazure.com for more information
• I clicked the top-level “PRICING” link provided as that was my fear
• I was presented with a fancy graphical calculator.  I moved the slider up to 100MB and confirmed the pricing on the side (no asterisks or anything)
• I notice a “Learn more about pricing, billing and metering” link underneath the calculator and click it to learn more
• I’m presented with a section of 10 different options all presented at the same level giving the appearance as unique services.
• I choose the Database one and again read through and confirm the charge for the 100MB database option.
• I click the “More about databases” link to double-verify and am presented with another detailed description of the billing

Not once during that process was context provided.  Not at any of the steps above (3 different pricing screens) was there context that additional fees could also apply to any given service.  Data transfer, in fact, doesn’t even describe itself very well.  As I was assured in asking folks involved in Azure about my concern on pricing, this “Data Transfer” wasn’t brought up at all.  I’m not sure why at all it is listed along side services and almost presented as a separate service as it appears all Azure services are subject to data transfer fees.  This is not made clear during sign up nor marketing of the pricing for each service.  SQL Azure should clearly state that the fees are database *plus* any additional fees resulting from data transfer.  Heck Amazon does this with S3 which also makes it so confusing to anticipate the cost of billing there as well…but at least it is presented that I need to factor that into my calculation.

I’m to blame, so why am I whining

I said I’m to blame as well for not understanding better what I’m getting into.  It is unfortunate because I really did like the service and felt an assurance of more reliability with my database then I had before.  The management portal was great and the uptime and log management was something I didn’t have to think about anymore. 

So why, you might ask, am I complaining about a service fee for something that was providing me value? 

NOTE: You may ask why I didn’t just move my site within Azure as well so that no data would be leaving the data centers.  This is a fair question, but unfortunately my site won’t run on any Azure hosting services and additionally I manage a few sites on a single server so it is cost prohibitive to have multiple Azure hosting instances for me right now.

Well it is simple.  I’m not made of money.  This blog has no accounting department or annual budget and such, I have to be smart about even the smallest cost.  I already have sunk costs into the server that hosts this site as well as a few others.  A $5/month database fee was nothing and justifiable easily with the value I was getting and the minor additional cost.  $50 (and growing) just wasn’t justifiable to me.  It was already at the same cost as my dedicated server and just no longer made sense for my scenario here.  In this instance I’m the “little guy” and need to think like one.  Perhaps cloud services are not for me.

Summary

So what did I learn?  Well, I really need to understand bandwidth and transfer data better for the sites I have.  Unfortunately this isn’t totally predictable for me and as such if I can’t predict the cost then it isn’t something that I should be using.  If you are considering these types of services regardless of if they are from Azure or Amazon (or whomever) you need to really plan out not only the service but how it will be used.  Don’t be lured by those shiny cost calculators that let you use sliders and show you awesome pricing but don’t help you estimate (or alert you) to that some of those sliders should be linked together.

I think Azure (and other similar services) have real customer value…there is no doubt in that.  For me, however, it just isn’t the time right now.  The services, based on my configuration needs, just don’t make sense.  Had I had a clearer picture of this when signing up, I wouldn’t have been in this situation of frustration.  Choose your services wisely and understand your total usage of them.  For me it currently doesn’t make sense and I’m moving back to a SQL Express account on my server.  Yes I’ll have to manage it a bit more, but my costs will be known and predictable.

Hope this helps.

Here’s how it started…

Lisa (my wife) [shouting from office into the kitchen]: Tim, what’s this Amazon charge for $193?
Me [thinking what I may have purchased and not remembered]: Um, don’t know…let me look.

I then logged into my Amazon account to see what order I may have forgotten.  Surely I didn’t order $200 worth of MP3…that’s ridiculous.  Sure enough nothing was there.  Immediately I’m thinking fraud.  I start freaking out, getting mad, figuring out my revenge scheme on the scammer, etc.

Then it hit me: Amazon Web Services account.

The Culprit

Sure enough I logged in and my January 2010 billing account was $193 and change.  Yikes.  Well, I could let the (what has been averaging) $30 or so charge slide under the family CFO radar for a while…but this $193 charge…the chief auditor herself caught that one.

So I panicked.  I needed to figure out where/what the spike was.  I logged into the Amazon Web Services management console (I only use the S3/CloudFront storage in their services right no) to see what was going on.  I see ‘Usage Reports’ and click.  I’m met with essentially a bunch of useless data really.  No offense to Amazon, but really the usage reports weren’t really helpful at all.  First, they gave me a Resource ID which I thought would represent the URI I was looking for.  Nope, Resource ID == Bucket.  And they didn’t even put the bucket name in the report!

For some perspective, here’s essentially what I’m used to – here’s my December 2009 billing statement details:

Anyhow, after some hunting it was obvious that I wasn’t going to figure out what bucket objects/unique URIs were causing my spike.  This was primarily because I didn’t have logging turned on at all on my buckets.  I had in the past but really didn’t think I needed it so I turned it off.

I was wrong – go now and enable logging.

While I was searching for a solution to understand my traffic, I was curious for where my traffic was.  Like I said, I’d been averaging (actually *peaking*) at about a $30 charge for the S3 hosting.

NOTE: I use S3 for all my image/screenshot/sample code file hosting.  I’ve invested in S3 for a long time and built my blogging workflow around it with building tools like S3 Browser for Windows Live Writer.

What was interesting was my most usage of my CloudFront data was coming from Hong Kong.  Compare to above the December 2009 billing to this January 2010 billing:

Yeah, that was my reaction too.  I went from roughly 40GB of transfer bandwidth to over 960GB in one month.  I suspected I knew what happened, but needed to confirm before I changed things. 

Implementing Logging for Statistics

The problem was that I didn’t have logging enabled and I was pretty much stuck.  I needed to get some data from the logs before being for sure.  I quickly found S3Stat and it appears to be the de-facto reporting for Amazon S3 log files.  I signed up for the free trial and generated a new access key to give them.

NOTE: They have a ‘manual’ option which means a lot more work.  I simply generated a NEW S3 access key for this specific purpose.  That way I didn’t have to give them my golden key I’ve been using in other places and can shut this off at any time without issue to my other workflows.

24 hours later, I had some reports.  Wicked cool reports.  Here’s a list of what I’m currently looking at:

• Total hits, total files, total kbytes
• Hits/files per hour/day
• Hourly stats
• Top 30 URIs
• Top URIs by kbytes used
• Top referrers (find out who’s using your bits without you knowing)
• User agents

Here’s a quick snapshot of one:

Wow…honestly…THIS is what I was expecting when I see “usage” data reports.  S3Stat is awesome and you should use that now.  Yes, I’m buttering up to them…but they have a great tool here for $5/month if you are a heavy Amazon S3/CloudFront user.  Amazon frankly should just buy them and integrate this into their management console.  You can see other examples of their report outputs on their site at http://www.s3stat.com. 

What I also found out is that the tool I use for my desktop usage of S3/CloudFront (outside of my blogger workfow and S3Browser) has S3Stat integration built in!  I use CloudBerry’s S3 Explorer Pro for managing my S3 content.  It’s awesome and you should look at it.  When I look at the logging features in CloudBerry I see this:

And after enabling the logging, within CloudBerry I can view the log data within the tool:

Summary

Wow, this is incredibly helpful and insightful data.  I now know who/how/when my cloud storage data is being used in various ways I can see the data.  S3Stat immediately showed me incredible value within less than 24 hours of enabling it.  I know can confirm the culprit of the burst of usage and plan accordingly.

Now, to be clear I’m not complaining about the cost of cloud storage.  That has been clear to me from the beginning.  Nothing is hidden and I’m not an idiot for not understanding it.  What I did not account for was the popularity of some files…and then the ones that just happened to be the largest.  I could not have personally thought I’d see a 920GB spike in one month of usage…but now I know…and have to alter some plans. 

Hopefully this is helpful for some who are just exploring cloud storage solutions/services.  Make sure you have instrumentation and logging capabilities turned on so you can identify and tune your situations.  For me, S3Stat and CloudBerry are winners for my personal usages.  If you are an Amazon S3 customer, I recommend looking at S3Stat and turning on logging immediately!