Subscribe to RSS
Archive for the ‘twitter’ Category:
An Answer for Twitter OAuth-pacalypse
For your smaller Twitter API projects, bash scripts etc, we have launched SuperTweet.net in case you don’t get OAuth implemented by the time Basic Auth goes away June 30, 2010. It’s a Twitter proxy – you use Basic Auth to talk to the proxy, and it uses OAuth to talk to Twitter.
SuperTweet.net Access Credentials
For example, to send a tweet, use the http://api.supertweet.net/1/statuses/update.format such as:
curl -u user:password -d "status=playing with cURL and the SuperTweet.net API" http://api.supertweet.net/1/statuses/update.xml
The password shown in the example above is never your real Twitter password, but a separate password you set up just for use with the SuperTweet.net API – As with Twitter OAuth, you can revoke, change, or disable that password without any impact to your real Twitter password or Twitter account. Also, you can deauthorize the SuperTweet.net API application itself on Twitter.com if you think it’s being bad, again, without affecting your real Twitter password or other Twitter applications.
Learn more: http://www.supertweet.net/
Is it time to drop the Twit, Tweet, etc. from your App Name?
There has been a storm of sorts this past week surrounding Twitter, their developers, and the “ecosystem”. It started with some comments from VC Fred Wilson followed by comments from Twitter CEO Ev Williams in a New York Times interview, and culminating in Twitter buying Atebits and thereby creating an official iPhone app: Tweetie.
This has, of course, created quite a stir in the Twitter development community, with accusations that Twitter is “eating their young” and pushing third-party developers out of the market.
These recent events have really brought home for developers the risk that when building on top of someone else’s platform, they could decide to compete with you head-on. Of course we’ve seen this many times in the past, most notably with Microsoft, but also with Google, Cisco, and many others. Those seeking funding for Twitter mashups often struggle with this push-back from potential investors. Perhaps now, that will become even worse and it will be even harder to get funding for a Twitter-based product or service.
That’s not good news for Twitter. While their platform is popular, with a claimed 50,000 developers, it’s still relatively small, compared to Facebook with 500,000 or Apple’s 185,000 apps in the app store. Furthermore, both Facebook and Apple’s App Store have spawned lucrative breakouts like Zynga, Tapulous and PopCap, while Twitter’s ecosystem has yet to produce a similar hit in terms of revenue, to say nothing of profit. Even among the most popular Twitter apps with lots of users, there are plenty of companies burning cash, but few producing any.
So, is it time to drop the “twit” or “tweet” from your brand name and become less intricately linked and integrated with Twitter and the Twitter platform? Or is this a perfect time to show loyalty to Twitter in the hope that they reward you for that loyalty? To Twit, or not to Twit. That, is the question. What do you think?
Twitmart ported OFF of Google App Engine
I discussed a few weeks ago how I had ported the Twitmart.org site to Google App Engine for Java as an experiment. This was an excellent learning experience. Unfortunately, the performance on App Engine was simply unacceptable, so now the site is back off of App Engine and running on one of my own servers again.
You can actually compare the difference side-by-side by comparing the performance on a native Java web server (Jetty) platform vs. the exact same code running on App Engine:
Java Jetty –> http://twitmart.org
Google App Engine –> http://twitmart.appspot.com
This shows pretty clearly that the performance issues aren’t Twitter (most of the time) nor are they in my Java webapp implementation. The same code is running on both platforms. Nor is it hardware – the machine running Twitmart.org is a very modest server that is hosting a large number of other sites.
The showstopper problem for Twitmart on App Engine is that Twitmart.org is a Twitter mashup using the the Twitter API. It turns out that Twitter API calls from App Engine fail frequently. I’m not sure why. It should not be due to throttling as the app uses authentication but on the other hand it could possibly be that Twitter is overwhelmed by API calls from the App Engine platform and therefore throttles the entire Google infrastructure. I think the most likely answer is just that App Engine has problems making API calls (HTTP requests) in general, because I’ve seen this same problem to other APIs besides Twitter from App Engine, such as the issues we had with the Taglets API.
Regardless, it’s bad for App Engine and Twitter Mashups. It means Google’s platform is not useful for Twitter mashups or any kind of mashups. That sucks and makes the platform much less useful in general.
Beyond that showstopper issue for Twitmart.org (and all Twitter mashups considering using Google App Engine), is the performance / consistency in general. For no clear reason, about 20% of requests just take a ridiculous amount of time on App Engine, even to do the simplest thing. E.g. a url that simply renders a template, that usually takes sub 100ms, will about 20% of the time, take 3-5 seconds or more. That’s simply too high of a “near-fail” rate for a “serious” application.
I hope that Google does address these issues and start to take serving “important” applications seriously. The idea of cloud computing in general, and Google’s “App Engine” approach specifically, is really cool. It’s very attractive to have someone else taking care of the servers, the network, patches, etc. However, there is a certain performance bar the service has to meet too, and unfortunately, Google isn’t making it yet, even for the price of “free”.
Experimenting with Google AppEngine for Java
A while back, I noted that Google had announced that they would be supporting Java on their AppEngine cloud computing platform. I finally got around to working on a significant AppEngine for Java project (something beyond “hello world” or the demo “Guestbook” app).
Working with my friend and colleague, and serious Java guru, Mark Petrovic we decided that a good “Goldilocks” candidate, that was neither too big, nor too trivial, was the experimental service Twitmart.org, a classifieds marketplace mashup, using Twitter APIs.
We decided that “porting” an existing web-based application rather than inventing a new one made more sense because this way we really didn’t have to think a lot about the design or functional specifications. We already had them. We just had to think about how to implement to those specifications on a Java-based web application platform, and specifically, the Google AppEngine for Java platform.
With the Twitmart application, the first thing to address was that the site uses Restful-style urls (as opposed to a fully Restful architecture, for you Rest weenies). This introduces a number of issues. The AppEngine for Java platform implements the venerable, but now decade-old, Java Servlet API. Servlets don’t naturally support the clean urls that are essential in modern web applications.
Bad => /post.jsp?type=area&hash=forsale&postid=hKqo1
Good => /post/area/forsale/hKqo1
One way to go about this is to do it the old fashioned hard-coded way and manually parse urls and forward to servlets for the action. We considered doing so, as a last resort, but it would be painful and potentially a maintenance nightmare.
We had hoped to use Jersey but found that it was not quite there yet in terms of compatibility with Google AppAngine, although it looks like there is progress and we will see at least a subset of Jersey supported on AppEngine at some point. For this project, we decided that the outstanding issues with Jersey on AppEngine were more than we wanted to deal with. Since Twitmart is a web site rather than a web service API, perhaps Jersey isn’t quite the right platform anyway.
Regardless, we went with Restlet, which does support AppEngine in their latest unstable releases. Going with Restlet meant that we also needed a compatible template engine to replace JSP. We went with Freemarker which is supported as a Restlet extension.
There are many Java libraries for accessing Twitter APIs. We went with Twitter4J which appears to be the most AppEngine friendly (and it’s a nice, clean API too).
With all of these building blocks in place, the port of Twitmart to AppEngine for Java was mostly a matter of grinding it out. Here’s a bunch of stuff we learned in the process:
Restlet, Freemarker, and Twitter4J work on AppEngine
The Twitmart application doesn’t exercise every part of any of these tools, but the basics clearly work. That’s nice to know. We will use these again on future projects.
Servlets and Restlets can co-exist
A few of the operations that Twitmart does were better suited to traditional servlets. With some effort, we probably could have made them work as Restlets, but it was much quicker and easier to make them servlets, so that’s what we did. And it works. Servlet urls are directed to servlet classes in the usual way in the web.xml config while everything else is passed through the Restlet adapter/router class.
The AppEngine Text class overcomes the 500-character limit of Strings in the datastore
The datastore will only accept Strings of up to 500 characters. To store larger text, one must use blobs. AppEngine provides a Text class that makes this pretty easy – but it does make your application more AppEngine specific. A Text object cannot be viewed in the datastore viewer (admin panel) and they cannot be indexed or queried.
@Persistent(defaultFetchGroup = "true")
private Text desc;It’s not that hard to put arbitrary Java objects into the datastore
The Twitmart application accepts images uploaded by users to display with classified ad postings. Since AppEngine has no writeable file system, these “image files” must be placed into (and retrieved from) the datastore. This turns out to be pretty easy, as long as the size of the data conforms to AppEngine’s limits – e.g. 1MB per datastore entity (see http://code.google.com/appengine/docs/java/datastore/overview.html)
@Persistent(serialized = "true")
private ImageFile imgfile1;Objects in the datastore have unexpecetd dependencies
I don’t know if Java bytecode ends up in the datastore or not, but it kind of feels like it does. When we rearranged some classes, we discovered that datastore records using those classes failed with ClassNotFound errors. I guess we should have expected this, but it’s something to keep in mind – once something is written to the datastore, never move the class.
As I’ve noted before, the AppEngine datastore is not an RDBMS, despite providing a query capability that might make you think otherwise at first glance. The AppEngine datastore is based on the BigTable storage system. BigTable follows a very different philosophy than traditional RDBMS and, as a result, imposes many important restrictions. Knowing this from prior work on Taglets.org and other projects on AppEngine for Python, I knew about these restrictions and, in fact, the Twitmart application doesn’t make any queries at all; it always retrieves objects (records) by key and never via query.
Another lesson from our experience with the Python AppEngine was “be prepared to move off AppEngine” (for any number of reasons). This means trying to structure your application such that moving to a standard platform will not require a complete re-write. One place to start is to avoid any com.google.appengine.* imports. In the case of this Twitmart app, the only place we do that is for the special com.google.appengine.api.datastore.Text class (in order to store text strings larger than 500 characters). This is isolated to a couple of Java classes in this application, but in a larger app, or if such things found their way into too many places, it would be best to write abstraction libraries of some kind to make it easier to separate out AppEngine dependencies later.
The Twitmart app, as written, should run on any standard Java web application platform, such as Jetty, pretty easily. However, that said, while using JDO as the data management interface works, in theory, as a way to make your application work on AppEngine and on a standard platform, it might not turn out to be the best way, or the most common technique for managing data on standard web application platforms, in practice. In many cases, you would probably want to add an abstraction layer around your data models to make it easier to swap in different low-level APIs. I think in general, the AppEngine for Java platform makes this easier and more natural than the AppEngine for Python platform, where it takes a lot more effort to keep your app standard-platform ready.
The Twitmart port is not done. There are few functions left to implement and a few things still to fix in terms of better exception handling and related cleanup and enhancement, but the primary functionality is operational. In practice, even with light usage, it is slower on AppEngine than it was on my own server, but I guess that’s the price you pay for “free” – so much for scaleable. I guess the theory is, on AppEngine it will show similarly poor performance for a dozen users as it does for 100,000 users. 
Summary
If I develop more AppEngine apps, I’m going to use the Java AppEngine for sure over the Python AppEngine. Unless there is some really compelling reason (like a specific library I want to use) I don’t see me writing another App Engine app in Python ever again.
While I still don’t think Google AppEngine is ready for anything too important yet, if it ever will be, having a Java version is a big step in expanding the possible uses of Google’s cloud computing platform, IMHO, and I will almost certainly deploy more experimental applications like Twitmart using AppEngine for Java in the future.
References:
- Google App Engine for Java
- Restlet
- Restlet edition for Google App Engine
- Freemarker
- Freemarker extension for Restlet
- Twitter4J
- Twitmart.org
UPDATE October 16, 2009: The performance on App Engine was so poor that we had to move the Twitmart.org site off of App Engine and back to one of our own servers. Details here.
Twitter Spam and GO0GLE-WIZARD.com
Who is running Twitter and where are the engineers?
The GO0GLE-WIZARD.com spam thing is frustrating for two reasons:
- It means Twitter usernames and password were stolen from somewhere. That’s what Twitter needs to focus on rather than suspending the accounts of those that got their IDs stolen.
- Second, this was a simple enough attack that it would not have been hard for Twitter to put in a fix right away.
When this spam ramped up, it was obvious it was spam. It was also obvious a bot somewhere was using the Twitter API to send the spam tweets. These tweets were easy to recognize and all pointed to the same URL. It would have been easy for Twitter to catch them at the API level and traceback to the IP of the offending site and perhaps see who was really sending the spam (not the owners of the Twitter accounts).
But no, the spam continues and Twitter’s response is to suspend accounts of legitimate users. Lame!
I really want to know the offending site that stole the passwords and gave them to GO0GLE-WIZARD.com – that’s the important thing. They need to be tracked down and punished.
It sounds like Twitter needs to re-think it’s technical leadership.
Another Twitter password stealing spamming scam
Here we go again. A site called GO0GLE-WIZARD.com is actively posting bogus (forged) tweets on behalf of thousands of compromised Twitter accounts. There’s a few differences from the similar twittercut episode a few weeks ago. In the case of GO0GLE-WIZARD.com, it’s not yet obvious where the site is getting the compromised account credentials. Presumably, the users have logged in to some web site or service on the net using their Twitter username/password and that site is operated by the same people as GO0GLE-WIZARD.com or somehow made a deal with them to share compromised accounts. Another possibility is the GO0GLE-WIZARD.com folks compromised some other site that had thousands of Twitter IDs and passwords on file. The whois record for the domain says the info is “protected” , meaning hidden behind a proxy, in this case some outfit called “WhoisGuard”.
I’ll post updates if we find out who is really behind this and what site they’re getting the stolen Twitter credentials from. In the mean time, don’t believe any tweets that suggest your friends are making money from this scam website.
UPDATE 7/30/2009: Twitter’s spam-monitoring account sent a tweet a few hours ago warning anyone who had entered their log-in information into a website called TwitViewer to change their password. Perhaps this is the culprit of the stolen Twitter accounts.
UPDATE 7/31/2009: TwitViewer is apparently a scam too, but perhaps not related to GO0GLE-WIZARD.com. Some people are seeing a relation to Foursquare, an iPhone Twitter app, but some dispute that too. So I don’t think we know for sure yet where GO0GLE-WIZARD.com is getting the stolen Twitter IDs and passwords.
UPDATE 7/31/2009: What’s worse here is accounts are getting suspended by Twitter even if they delete the spam and change their password. Come on Twitter, get a clue.
BEWARE: twittercut is a password stealing SCAM!
A quick search on twitter will show this site is bogus: http://search.twitter.com/search?q=twittercut
DO NOT sign up there (or even visit the site). If you did sign up, change your Twitter password asap.
Here’s what the home page to the site says:
Welcome to TwitterCut.com.
TwitterCut.com is the best place for you to grow your twitter network and gain a ton of followers. We recommend giving it a shot, it’s free and will help you get the followers you need. This system is brand new, so the quicker you get involved the better it will be, fill out the form below and get started right away…
It looks like they stole some Twitter graphics there. I note that the Home, About etc. links are all no-ops.
The WHOIS record, gives the following:
Registrant:
Jordan EMbry
1646 thompson drive
owensboro, Kentucky 42302
United States
Domain Name: TWITTERCUT.COM
Created on: 21-May-09
Expires on: 21-May-10
Last Updated on: 21-May-09
Administrative Contact:
EMbry, Jordan jembry13@gmail.com
I have not found a white pages listing for an Embry at the above address. The domain is registered with GODADDY.COM, INC.
May or may not be the same person as:
- Jordan Embry – Bowling Green, KY | Facebook
- Jordan Embry – LinkedIn
- Jordan Embry (JordanEmbry) on Twitter (account suspended)
- Another Jordan Embry (ijordan89) on Twitter (also in KY)
Twitter #fixreplies and “intelligent” networks
A few weeks ago, I said “Twitter is a “Stupid Network” (the good kind)”. It turns out I was wrong.
Twitter created the #fixreplies problem (see here and here) for themselves when they decided to treat these tweets specially from the start, breaking the end-to-end principle.
These are tweets by people I follow – they are public tweets, not direct messages. They are in the public timeline. If I follow that person, they should be in my timeline like any other tweet from that person (whose tweets I follow), regardless of the content of that message.
I follow this person. That means I follow what they say, the updates they post. If Twitter never treated a post that starts with @ special in the first place, we wouldn’t be having this discussion. In that case, if I wanted to filter such tweets, I could do that in my client, at the “edge”.
Now Twitter has pulled those tweets out and so I can’t “filter” them – I have to go looking for them.
This is the kind of mess you get yourself in when you go around breaking the end-to-end principle all will-nilly.
#fixreplies hack – shows what Tweets you’re missing
UPDATE May 18 2009: fixed some bugs, removed Twitter API rate limit issues, and simplified usage to one simple HTML file
I put together a little hack that shows the tweets you’re no longer getting.
It’s pretty hacky and requires some manual labor to use, but some people may find it entertaining. I decided to do it all with Javascript so it requires nothing more than your own browser. Also, that way I can provide all the code and let people see exactly how it works and perform whatever beautification they may want, etc.
Basically, it produces output like the following, showing the @reply tweets you didn’t see in your Twitter stream becaue you don’t follow the person the message is directed to. In my case, it showed me 63 interesting tweets tonight. The output is something like this:
Each entry is shown with a link to the original Tweet on Twitter so you can click to get the full details of the tweet.
How to use it
It is all in one HTML file: whatumissed.html.
You can copy and paste the above into a local file on your computer and open in your browser, or simply execute the version at mrblog.org directly (and “View Source” to see how it works):
http://mrblog.org/whatumissed.html
- It should be pretty easy to turn off the diagnostic data if you like by using
visibility: hiddenfor the “statusmsg” div in the html/script. - There is no little CSS or other styling here – if you hack the code with cool CSS styling, please post back to me so others can benefit too.
- If you make other hacks to this code, or have other suggestions for me, please let me know.
Twitter #fixreplies fiasco is telling
I’ve seen many people try to explain (or argue with someone) about why Twitter matters, what they use it for, why it’s cool. These conversations are often interesting in how they demonstrate that even people that love Twitter often can’t explain it to anybody. Sometimes I think these people aren’t even sure themselves what they get out of Twitter, or why they find it compelling.
And I think the #fixreplies fiasco shows that even the Twitter brain trust may not understand the value of their own service very well. We certainly know they don’t seem to have any idea how to make money with it.
For those that haven’t caught wind of this yet, recently Twitter stopped displaying public tweets that are deemed to be “replies” between your friends and someone who you do not follow. In the past, such tweets would be seen as sort of one side of a conversation. If you follow @Fred for instance, you might have seen something like the following in your Twitter stream:
Fred: @Mary I liked your idea about http:…
In this case, let’s say you don’t know who @Mary is. Since you like Fred, you might want to know who this Mary character is and what she’s about. A tweet like the above was a perfect opportunity to be exposed to new people that your friends engaged with. In other words, it was a great way to discover new potential friends. it was also a great way to see what your friends are talking about – what topics were so interesting to them that they were compelled to respond. So they helped us stay more in touch with our friends, both in terms of who else they talk to and what they like talking about.
In addition to the tweet itself, it might also have said something like “in reply to Mary” which is a link to the original tweet by Mary that Bob is responding to.
By removing these tweets from my stream, Twitter has instantly made itself less valuable as a social tool. On Facebook, I can see when a friend comments on one of their friends’ items, even if that second person is not my friend. Friends of Friends is a major component of any social network and part of what separates a “social” site from just another web site.
Twitter argued that they had to do this for technical reasons. That makes no sense to me. What’s really special about this tweet by my friend Fred, as far as Twitter is concerned? Why are they treating it specially? In terms of the relationship to my stream, as a follower of Fred, how is it different than other tweets by Fred? Of course it’s not. So much so, that people have talked about simply inventing a new format for replies and not using @Mary anymore so that the message will appear in the stream. UPDATE: Twitter claims the “technical problem” is having to check every account’s preference every time a reply was posted was causing a huge strain on their database – so they removed the option and gave everyone the “most common” choice.
It also represents another step toward violating the end-to-end prinicipal , something that has gone toward greatly increasing the value of the Twitter network and separating Twitter from others. As I have said, dropping the end-to-end advantage is a big deal:
This end-to-end model permits myriad uses of the underlying network and fosters incredible innovation.
…
At first look, Twitter becoming more “intelligent” about the contents of tweets may appear to be a good thing. But it is a slipery slope. Once they start adding meaning to the 140 characters, the network becomes more restrictive. The more Twitter remains a “stupid network” the better, IMHO.
I think that fact that even Twitter themselves don’t understand what makes their service valuable tells us something. I hope they figure it out and don’t destoy the special thing they have created.
Update: an interesting counter-perspective: my thoughts on #fixreplies and the ‘dot fix by @HelloKit.
Also, see the whatumissed hack for a way to see these missed replies again: http://mrblog.org/whatumissed.html
