Subscribe to RSS
Archive for the ‘cloud computing’ Category:
Wow am I happy now that I didn’t deploy serious apps on Google App Engine
First released in 2008, Google App Engine (GAE or AppEngine) was Google’s first attempt to compete with Amazon Web Services in providing cloud computing platform services for developers. In earlier posts, I took some heat for concluding that Google App Engine was not ready for “serious” applications, even when it was “free”.
Recently, Google announced shocking new pricing for appengine that has its users reeling. In short, the new pricing means:
- “Free” quotas have been drastically reduced
- Pricing of paid apps increased significantly
- SLA and operational support available for a premium
Google has provided a tool so customers can compare their current bills versus expected billis under the new pricing and customers report anywhere from 3x to 30x price increases, leaving many scrambling for alternatives.
Two of the most common complaints from customers are lack of notice and the uncertainty of the pricing (lack of control over costs).
In terms of cost control, the only way to know how much your costs are, is to ask Google, after you have already incurred those costs (and built and deployed your app). It’s impossible to map users or usage directly to cost. Google’s pricing scheme is as inscrutible as the worst telephone company billing.
The pricing was originally planned to take effect in September, which only gave customers a few weeks to react. Google has provided optimization guidelines for customers to try to reduce their costs, but given the short notice, customers simply do not have time to make major changes to their apps. Companies already had their development resources planned out. They aren’t sitting around waiting for Google to throw a wrench at them. And it’s not clear how much further optimization will really save you anyway since a lot of apps have already received cost-cutting optimizations.
To me, I think this goes a long way to confirm some of my concerns about Google as a cloud platform vendor and as an enterprise vendor in general. A lot of people think anything Google touches is golden (especially Google, just ask them), but I think this shows how they still just don’t get it when it comes to providing commercial grade services. I have asked before, regarding many Google products, whether Google was serious this time. This is the risk to me of doing any business with Google. All these other non-search products are simply “tests” for them. A few billion here, a few billion there, throw it out and see what sticks. The problem is, if you latch on to one of these products and then it becomes critical to your business, you just never know when Google might, on a whim, go in a different direction, hanging you out to dry.
And that appears to be how a lot of customers feel about this move by Google, such as expressed in this post on the mailing list:
App Engine is finished, here’s why
What has always been the biggest concern about App Engine? Lock-in. You’re at the mercy of Google. Sure there’s TyphoonAE etc… but really those are not alternatives.
What does Google go ahead and do? They do exactly what their critics said they would do and what us GAE adopters hoped like hell they would never do, screw us over.
App Engine is finished not because we’re all going to move off to EC2, but because people who are considering using App Engine will see exactly what has gone on here with the pricing, think about the lock-in argument against GAE, and decide not to use GAE. There will be a drop off in new apps, and eventually Google is going to see GAE isn’t really panning out and pull the 3 year plug.
Thankfully, I don’t operate any services on GAE with high costs, but even as it is, I feel ripped-off for my investment in AppEngine. I do run some services on it, some of which I would rather not have to shut down, so I might have to move those elsewhere. And there are some apps I will simply shut down because they are not worth the trouble to port elsewhere. Some of those apps were potentially interesting and gathering users – in that sense, I’m glad this move by Google is happening now, before these apps got big enough to have to now decide what to do with them.
What’s worse for me though, is simply all the time invested in learning AppEngine. What a waste of time that appears to have been. As one developer says:
The biggest complaint is that when my friends and peers objected to App Engine, its strange requirements and its potential lock in, they were right and I am a fucking naive idiot. And I really don’t like to be proven a naive idiot. I put my faith in Google’s engineers and they have utterly destroyed my credibility. THIS more than anything is the cost to me.
Still searching for a decent domain registrar – Mydomain.com and 1and1.com both suck in their own ways
Out of habit as much as anything, I had been using Mydomain.com for years for all my domains (I have dozens – don’t ask). They offer a decent set of features for the money, including email forwarding and DNS management. I have had various issues and outages over time with them and have been frustrated with them at times. In particular, the lack of control in the DNS administration and delays in updating their DNS servers had caused me to stop using the Mydomain.com DNS service for any domains hosting critical commercial services years ago, even for domains still registered with Mydomain.com.
But there is a lot of inertia in switching registrars. Transferring domains is a hassle. And how would I know things would be any better with a different service? However, after the major DNS outage that Mydomain.com had in July, where their servers were 100% down hard for many hours, I was finally pushed to take action. I started searching for another registrar and planned to move domains as they expire off of MyDomain.com.
Long story short, I moved some domains to 1and1.com based on various recommendations. What I can say is, I don’t think they’re really any better. Other than retaliation for Mydomain.com outages, I don’t think I’m better off at 1and1.com.
For Mydomain.com, the big problems are as noted above. First, they just have random outages far too often. If you use their DNS, their updates are really slow, where DNS changes can take days to appear on their own primary servers. And their admin interface is limited. You cannot create advanced kinds of records, like SRV, AAAA etc.
In terms of services and features, 1and1.com is even worse than mydomain.com. They provide even less management of DNS. They have a single setting for “set your IP address” and that’s it. As far as I can tell, this sets an A record for example.com and www.example.com and that’s all. Unless there’s a hidden setting on their admin interface that I havent found yet, you cannot manage DNS at all with 1and1.com.
And neither company provides customer support that is worth anything. Their support is pretty much a complete waste of time and resources. In most cases you dont get an answer at all beyond the canned one to email requests, as if no one even read the message. And phone support is always a hair-pulling experience of never ending circular discussions that go nowhere. Essentially the support staff is under-trained, incompetent, and not empowered to fix anything anyway.
I use a fantastic DNS cloud service that I cannot recommend highly enough: nettica.com (Note I’m not affiliated with them in any way other than as a customer and I don’t get referral credits). They offer great DNS service at reasonable prices, with full advanced administration and instant updates. Nettica staff are technically competent and very responsive. Nettica also offers domain registration, but unfortunately they are three to five times as costly as services like mydomain.com and 1and1.com, especially if you use any of the extra cost add-ons like email-forwarding that usually come free with other providers.
In the end, moving domains from mydomain.com to 1and1.com may provide me some satisfaction in not giving mydomain.com any more of my money and therefore not supporting their bad behavior but it probably isn’t really giving me any better service or saving me any money. I suppose the answer is to use a cheap service like 1and1.com for nothing other than domain registration and use other providers for everything else, like Nettica for DNS, perhaps Google for email, and your favorite cloud server or dedicated servers for your hosting.
If you have a domain registrar that you really like, please let me know who they are, what else you use them for if anything beyond registrations, and what’s good and bad about them.
Litetext for rendering text into a bitmap on Google App Engine Java
While developing Quick Bit Notes, I realized that the Google App Engine “sandbox” for Java doesn’t support AWT or other graphics Java classes, so I put together a simple “old-school” text rendering class that’s compatible with App Engine, called litetext.
This small package (less than 1000 lines of Java code) was developed to provide a small and simple package for rendering text into an image (bitmap). It was developed for Google App Engine use where AWT and BufferedImage et al do not exist. This small utility can be used to render text on App Engine within the constraints of the JRE Class White List. This is derived from the “pbmtext” utility from NETpbm Copyright (C) 1991 by Jef Poskanzer.
These are crude black-and-white bitmap fonts – no anti-aliasing. A small number of BDF fonts in fixed-sizes are bundled into the package: Courier, Lucida-Bold-Italic, Lucida-Medium, Lucida-Medium-Italic, LucidaBright-DemiBold, LucidaTypewriter, LucidaTypewriter-Bold, and a default serif font.
The following code snippet demonstrates use of litetext on App Engine:
byte[] bmp_data = fm.doRender(inputText, fontname);
ImagesService imagesService = ImagesServiceFactory.getImagesService();
Image bmpImage = ImagesServiceFactory.makeImage(bmp_data);
Transform flipit = ImagesServiceFactory.makeVerticalFlip();
Image newImage = imagesService.applyTransform(flipit, bmpImage, com.google.appengine.api.images.ImagesService.OutputEncoding.JPEG);
You can grab it via SVN at http://code.google.com/p/litetext/
There are docs on the project site for adding fonts and an example demo servlet is included. See the Live Demo.
Use this command to check out the latest project source code anonymously over HTTP:
svn checkout http://litetext.googlecode.com/svn/trunk/ litetext-read-only
Twitmart ported OFF of Google App Engine
I discussed a few weeks ago how I had ported the Twitmart.org site to Google App Engine for Java as an experiment. This was an excellent learning experience. Unfortunately, the performance on App Engine was simply unacceptable, so now the site is back off of App Engine and running on one of my own servers again.
You can actually compare the difference side-by-side by comparing the performance on a native Java web server (Jetty) platform vs. the exact same code running on App Engine:
Java Jetty –> http://twitmart.org
Google App Engine –> http://twitmart.appspot.com
This shows pretty clearly that the performance issues aren’t Twitter (most of the time) nor are they in my Java webapp implementation. The same code is running on both platforms. Nor is it hardware – the machine running Twitmart.org is a very modest server that is hosting a large number of other sites.
The showstopper problem for Twitmart on App Engine is that Twitmart.org is a Twitter mashup using the the Twitter API. It turns out that Twitter API calls from App Engine fail frequently. I’m not sure why. It should not be due to throttling as the app uses authentication but on the other hand it could possibly be that Twitter is overwhelmed by API calls from the App Engine platform and therefore throttles the entire Google infrastructure. I think the most likely answer is just that App Engine has problems making API calls (HTTP requests) in general, because I’ve seen this same problem to other APIs besides Twitter from App Engine, such as the issues we had with the Taglets API.
Regardless, it’s bad for App Engine and Twitter Mashups. It means Google’s platform is not useful for Twitter mashups or any kind of mashups. That sucks and makes the platform much less useful in general.
Beyond that showstopper issue for Twitmart.org (and all Twitter mashups considering using Google App Engine), is the performance / consistency in general. For no clear reason, about 20% of requests just take a ridiculous amount of time on App Engine, even to do the simplest thing. E.g. a url that simply renders a template, that usually takes sub 100ms, will about 20% of the time, take 3-5 seconds or more. That’s simply too high of a “near-fail” rate for a “serious” application.
I hope that Google does address these issues and start to take serving “important” applications seriously. The idea of cloud computing in general, and Google’s “App Engine” approach specifically, is really cool. It’s very attractive to have someone else taking care of the servers, the network, patches, etc. However, there is a certain performance bar the service has to meet too, and unfortunately, Google isn’t making it yet, even for the price of “free”.
Experimenting with Google AppEngine for Java
A while back, I noted that Google had announced that they would be supporting Java on their AppEngine cloud computing platform. I finally got around to working on a significant AppEngine for Java project (something beyond “hello world” or the demo “Guestbook” app).
Working with my friend and colleague, and serious Java guru, Mark Petrovic we decided that a good “Goldilocks” candidate, that was neither too big, nor too trivial, was the experimental service Twitmart.org, a classifieds marketplace mashup, using Twitter APIs.
We decided that “porting” an existing web-based application rather than inventing a new one made more sense because this way we really didn’t have to think a lot about the design or functional specifications. We already had them. We just had to think about how to implement to those specifications on a Java-based web application platform, and specifically, the Google AppEngine for Java platform.
With the Twitmart application, the first thing to address was that the site uses Restful-style urls (as opposed to a fully Restful architecture, for you Rest weenies). This introduces a number of issues. The AppEngine for Java platform implements the venerable, but now decade-old, Java Servlet API. Servlets don’t naturally support the clean urls that are essential in modern web applications.
Bad => /post.jsp?type=area&hash=forsale&postid=hKqo1
Good => /post/area/forsale/hKqo1
One way to go about this is to do it the old fashioned hard-coded way and manually parse urls and forward to servlets for the action. We considered doing so, as a last resort, but it would be painful and potentially a maintenance nightmare.
We had hoped to use Jersey but found that it was not quite there yet in terms of compatibility with Google AppAngine, although it looks like there is progress and we will see at least a subset of Jersey supported on AppEngine at some point. For this project, we decided that the outstanding issues with Jersey on AppEngine were more than we wanted to deal with. Since Twitmart is a web site rather than a web service API, perhaps Jersey isn’t quite the right platform anyway.
Regardless, we went with Restlet, which does support AppEngine in their latest unstable releases. Going with Restlet meant that we also needed a compatible template engine to replace JSP. We went with Freemarker which is supported as a Restlet extension.
There are many Java libraries for accessing Twitter APIs. We went with Twitter4J which appears to be the most AppEngine friendly (and it’s a nice, clean API too).
With all of these building blocks in place, the port of Twitmart to AppEngine for Java was mostly a matter of grinding it out. Here’s a bunch of stuff we learned in the process:
Restlet, Freemarker, and Twitter4J work on AppEngine
The Twitmart application doesn’t exercise every part of any of these tools, but the basics clearly work. That’s nice to know. We will use these again on future projects.
Servlets and Restlets can co-exist
A few of the operations that Twitmart does were better suited to traditional servlets. With some effort, we probably could have made them work as Restlets, but it was much quicker and easier to make them servlets, so that’s what we did. And it works. Servlet urls are directed to servlet classes in the usual way in the web.xml config while everything else is passed through the Restlet adapter/router class.
The AppEngine Text class overcomes the 500-character limit of Strings in the datastore
The datastore will only accept Strings of up to 500 characters. To store larger text, one must use blobs. AppEngine provides a Text class that makes this pretty easy – but it does make your application more AppEngine specific. A Text object cannot be viewed in the datastore viewer (admin panel) and they cannot be indexed or queried.
@Persistent(defaultFetchGroup = "true")
private Text desc;It’s not that hard to put arbitrary Java objects into the datastore
The Twitmart application accepts images uploaded by users to display with classified ad postings. Since AppEngine has no writeable file system, these “image files” must be placed into (and retrieved from) the datastore. This turns out to be pretty easy, as long as the size of the data conforms to AppEngine’s limits – e.g. 1MB per datastore entity (see http://code.google.com/appengine/docs/java/datastore/overview.html)
@Persistent(serialized = "true")
private ImageFile imgfile1;Objects in the datastore have unexpecetd dependencies
I don’t know if Java bytecode ends up in the datastore or not, but it kind of feels like it does. When we rearranged some classes, we discovered that datastore records using those classes failed with ClassNotFound errors. I guess we should have expected this, but it’s something to keep in mind – once something is written to the datastore, never move the class.
As I’ve noted before, the AppEngine datastore is not an RDBMS, despite providing a query capability that might make you think otherwise at first glance. The AppEngine datastore is based on the BigTable storage system. BigTable follows a very different philosophy than traditional RDBMS and, as a result, imposes many important restrictions. Knowing this from prior work on Taglets.org and other projects on AppEngine for Python, I knew about these restrictions and, in fact, the Twitmart application doesn’t make any queries at all; it always retrieves objects (records) by key and never via query.
Another lesson from our experience with the Python AppEngine was “be prepared to move off AppEngine” (for any number of reasons). This means trying to structure your application such that moving to a standard platform will not require a complete re-write. One place to start is to avoid any com.google.appengine.* imports. In the case of this Twitmart app, the only place we do that is for the special com.google.appengine.api.datastore.Text class (in order to store text strings larger than 500 characters). This is isolated to a couple of Java classes in this application, but in a larger app, or if such things found their way into too many places, it would be best to write abstraction libraries of some kind to make it easier to separate out AppEngine dependencies later.
The Twitmart app, as written, should run on any standard Java web application platform, such as Jetty, pretty easily. However, that said, while using JDO as the data management interface works, in theory, as a way to make your application work on AppEngine and on a standard platform, it might not turn out to be the best way, or the most common technique for managing data on standard web application platforms, in practice. In many cases, you would probably want to add an abstraction layer around your data models to make it easier to swap in different low-level APIs. I think in general, the AppEngine for Java platform makes this easier and more natural than the AppEngine for Python platform, where it takes a lot more effort to keep your app standard-platform ready.
The Twitmart port is not done. There are few functions left to implement and a few things still to fix in terms of better exception handling and related cleanup and enhancement, but the primary functionality is operational. In practice, even with light usage, it is slower on AppEngine than it was on my own server, but I guess that’s the price you pay for “free” – so much for scaleable. I guess the theory is, on AppEngine it will show similarly poor performance for a dozen users as it does for 100,000 users. 
Summary
If I develop more AppEngine apps, I’m going to use the Java AppEngine for sure over the Python AppEngine. Unless there is some really compelling reason (like a specific library I want to use) I don’t see me writing another App Engine app in Python ever again.
While I still don’t think Google AppEngine is ready for anything too important yet, if it ever will be, having a Java version is a big step in expanding the possible uses of Google’s cloud computing platform, IMHO, and I will almost certainly deploy more experimental applications like Twitmart using AppEngine for Java in the future.
References:
- Google App Engine for Java
- Restlet
- Restlet edition for Google App Engine
- Freemarker
- Freemarker extension for Restlet
- Twitter4J
- Twitmart.org
UPDATE October 16, 2009: The performance on App Engine was so poor that we had to move the Twitmart.org site off of App Engine and back to one of our own servers. Details here.
Clouds forming?
I couldn’t resist. Sorry about that. From Dave Rosenberg Cloud interoperability on the horizon?
I spent the first half of this week in Las Vegas at a nontech trade show, and missed both VMworld and the Red Hat Summit. However, watching and reading from afar, I noticed two major themes in discussion around both cloud computing and virtualization: cloud interoperability and the lack of application management tools.
Red Hat announced an open-source project called Deltacloud that looks interesting.
Amazon’s virtual private clouds should attract enterprises to the technology which will create a market for management products and services.
I think interoperability will be key to explosive adoption of virtualization at large – and it’s getting closer every day.
Debates rage on about what exactly “cloud computing” is – but fight it all you want, I think it’s pretty clear it’s here to stay, probably in a number of different forms. It’s probably time to accept that and start figuring out how to make the most of it.
Google App Engine adds Cron and Java
Google today announced that they will be supporting Java on their GAE cloud computing platform. I signed up for the beta, but don’t know when/if I’ll be approved.
I also saw that the platform also now includes a cron mechanism to run scheduled “jobs”, see: http://code.google.com/appengine/docs/python/config/cron.html
This could change things. It will be interesting to see how (if?) Amazon responds.
