Developing using 3rd Party REST Servers

I have found it  imperative to create a suite of integration/functional tests for each 3rd party REST site you use in your app. (Facebook, Flickr, Bebo, etc) These should ideally represent the set of API functions that your app is using for each API. You should create test users and data for the sites. Then, when you app goes wonky, you should fire off these tests BEFORE you begin wasting your time looking for a bug in your code.

Ideally, the 3rd party sites would provide status pages that continually hit their own API’s and show the problems to us and them. Put some RSS feeds on that page, and you have a nice alert system. But none of them bother to provide that.

Ultimately, I want to add these tests into a continuous build system, like Hudson, so that I can get alerted within a short timespan when one of the services I depend on goes down. In addition, I’m thinking this should trigger an automatic update to my web app. Probably a warning on my site and support forums  that states that features using site XYZ are not behaving properly as of XPM on a given date.  Even better (though more risky?), if the failure shutdown functionality depending on that failed REST server until it passed my integration tests again. This would also show a message to  users and also alert the developers.

At a bare minimum though, setup your own tests, and be ready to run them at a moments notice. It will save you a lot of grief and wasted debugging.

Posted in Tips | Leave a comment

When IntelliJ goes bad

Recently, my IntelliJ 8.01 install stopped recognizing annotations w/ syntax highlighting. Then, it wouldn’t commit to subversion unless you removed the ‘optimize imports’ feature. Then, the debugger stopped working for incoming web requests, but it worked fine for debugging unit tests.

I’m on Mac OS 10.5.x . I reinstalled the IntelliJ application package, but that did nothing. Next, I renamed the IntelliJ 8 cache folder, so Intellij would create a new one the next time it started. This solved all the problems.

This folder on a Mac is located here:

/Users/[your user name]/Library/Caches/IntelliJIDEA80

/Users/[your user name]/Library/Caches/IntelliJIDEA70

Posted in Tips | Leave a comment

iphone mail – Fail

Wrote my longest ever email on the iphone mail program last night. It went into power save mode a couple times while composing it, and then I sent it. The message is gone. Not in Inbox, Drafts or Sent items. No error message either. Awesome. That sucked, Apple!

Rule Number Fucking 1: Don’t lose the user’s data!

Rule Number  2: Don’t lose the user’s data!

Rule Number 3: Find a text editor iphone app that can integrate with mail when it’s time to send. (Since there’s no cut and paste still…<sigh>)

Posted in Rants | Leave a comment

(micro)Scaling A Legacy Web App

I’ve got a client running survey software, Sawtooth WinCati, that was originally written in the 80’s for phone interviewing. The latest version has an ASP web app attached to the original software. Now, it’s hard for a small business to scale a call center because of the costs of labor, space, hardware, phone lines, etc. But a web survey should be able to handle a large load  w/ cheap hardware and relatively cheap internet access available these days.

I pitched to run the software on EC2, but the client wanted a dedicated machine and didn’t trust a virtual instance running in a huge Amazon data center. (Also, see the problems w/ transferring from EC2 small to medium instances I blogged about earlier. We’d need to pay licensing for another server installation to have it setup for small and medium/large instances. Amazon, you’re killing me with this decision!)

Anyway, the survey company tech support couldn’t tell me any stats about scaling for the web. They guessed it could handle 50 concurrent users before having serious problems. I called back and got one of their programmers and we went over the architecture and the bottlenecks. It seems this software uses a FoxPro database. I wasn’t in the software world when FoxPro was popular, but there’s a reason it’s gone. Each ASP session making a write request to the DB would need a full file lock to make the write. i.e. It uses database locking instead of table or row locking. ugh!  This is the bottleneck everyone sees. Bandwidth and CPU are pretty negligible, though I have seen this software chew throw memory due to ASP session leaks from poor programming and testing. They said that one day they will move to a client/server db, but obviously that doesn’t help now.

So, I asked the obvious question. If we loaded up the survey into two instances on the same box, then would that double throughput? They hadn’t considered this approach before for some reason. However, they did have a function to merge datasets from two identical questionaires. Perfect!

Essentially, I’m making multiple masters and synchronizing them manually once a day before data is converted in a report. Due to the nature of this app (Non-conflicting writes. Reads from tables that don’t change during a survey.), this is really easy to scale. In fact, if this were using MySQL, it would be very fast with a single db since no queries should be attempting to access the same row at the same time for writes. This is assuming the DB design done by Sawtooth is decent.

Anyway, I’ve got Jmeter loaded on ec2 and hitting some sample survey at the highest rate expected for a very successful project. Putting each duplicate survey instance into it’s own IIS6 Application pool and own database file appears to scale well so far. The separate db file is most important, but I experimented w/ separate app pools, and got improved performance. Plus, this should help mitigate the impact of the session leaks. I still have to test it out on a full size survey on the production server I starting setting up on Friday, but I feel  confident we can scale now.

We send out emails w links to the survey, so I’m just splitting up those emails to point to N survey instances, and using Jmeter to figure out how many instances I’ll need to handle the max load.

Now, as I write this, I’m thinking that a load balancer would be nice. One that checks the open sessoins per app pool and keeps everyone balanced. That way we only distribute a single URL to the 32K survey takers. Less management of urls, etc. Hmmmm. Well, there is one requirement that actually preclude a simple load balancer like this. However, the requirement is kind of unique, so future projects could probably benefit from LB. I haven’t looked into LB options, but may in the future.  I’m thinking I could set all this up on EC2 and offer a scalability service to any company requring this software for their web surveys. (Apparently, it’s entrenched in a lot of companies around the world.)


More manual managment of data and creation of multiple URLs, so more room for error.

The merging of the data will be a pain sometimes.


Successful execution of the project for the client.


I checked out the Win2k3 builtin Load Balance Service. It appears to only work at the IP level to distribute among different machines. I want to distribute load on one machine across multiple application pools.

I suppose I could write a simple ASP landing page to round robin among all the app pools. Not sure how much effort it takes to collect info from each app pool about active sessions, request queue length, etc. (Basically all the info the NLB is supposed to use for me automatically.) However, these are all identically sized surveys, so in spirit of KISS, the round robin would probably do the trick well enough as a simple load balancer.

Wrote the simple load balancer for each instance. It handles a Jmeter deluge well.

Posted in Tips | 1 Comment


I have to load test only survey software for a client. Ran into some bugs w/ Jmeter Proxy feature, but once I got it going…well, I love this tool! It’s been around long enough that most features and statistics I need have already been implemented.

One note of caution, It fails playback w/ SocketConnection exceptions when run under jre 1.6_10. I downgraded to 1.5.0_16 and it played fine. Running it on an ec2 instance so that bandwidth can match the T1. EC2 only give 1.5Mbps upstream from the instance, but that’s fine in my  case. If I needed more there are Jmeter EC2 slaves already setup in the EC2 repository. And at 2-3$ a day for running a small instance, this is a great solution.

Posted in Uncategorized | 1 Comment

Selenium Grid on EC2

I configured the demo for Selenium Grid on EC2. It ran fine as a grid on my local box, but had some issues running over EC2. Mainly the browsers were just hanging instead of executing their scripts. I got quick feedback from Philippe Hanrigou, who is managing the grid project. Unfortunately, I don’t need to use this tool more now, but great to know it’s there in the future.

I’d really like to see a preconfigured windows selenium grid ec2 instance in the repository since IE is the most used browser still. Currently, only linux is available.

Posted in Uncategorized | 2 Comments

Amazon EC2

Got going w/ Amazon EC2 instances last week. ElasticFox certainly made for a nice gui admin tool. Few bugs, but overall good. The only thing that kills me is Amazon’s decision to not offer a 64bit small instance (or a medium  32 bit instance w/ 4GB of ram.). I know they are doing it to get people to pay more money for the more expensive 64 bit instances. The 32 bit are supposed to be there to test out your work first. However, if I customize my bundle a lot in 32bit mode and then need more performance offered in the 64 bit instances, I am screwed. I have to do all the effort of setting up a brand new 64bit instance, including installing and configuring all the necessary software. This really defeats the purpose of being able to test ideas cheaply in 32bit small instances.

Small instance: $90/month for purely ec2 payment. (not storage, bandwith, io, etc)

Medium instance: $360/month

That’s a factor of 4. Not much for a business, but a lot of money for a solo programmer experimenting. Also, it can turn into a lot for a small business if the solution involves clustering.

Overall, a good experience getting started.

Posted in Uncategorized | 1 Comment

Dead body found in Google Maps!

Click here to check it out live on Google Maps! (No this isn’t a Rick Roll )

Can’t believe someone managed to sneak this in the production version… 🙂

Google maps hit and run

Google maps hit and run

Posted in Uncategorized | Leave a comment

Building a better nextbus iphone site

Nextbus sends my shiny new 3g iphone to their lame mobile site. The iphone can handle the superior ajax interface, so why redirect me and  impede my experience?

So, like I any code dev, I started monkeying with fixing it. So far, I’ve just proxied it using ruby’s hpricot gem on dreamhost. Have plans to generate Nextbus’ sms alert’s for your right on the page for each stop, so you can enjoy a latte without constantly checking to see how many minutes away your bus is.

Unfortunately for this project, I started riding my bicycle 5 days a week to work, and my displeasure with nextbus on the iphone has waned for now. (Plus, I’m pretty sure they aren’t stoked that I’m scraping their site 🙂

You can check out the alpha here:

Posted in Uncategorized | 1 Comment

Is Linux ready to be your desktop OS in 2008?


Not for lack of trying, but I’m looking into installing XenServer Express (as soon as I upgrade to a VT supported cpu), and running Ubuntu side by side with Windows while sharing data between the two with NFS. I had vmware 2 RC1 running under Ubuntu. Put XP on there, but sound won’t work. Our house is musically networked with Itunes and it’s kind of a bitch not to be able to tap into that in my room. Plus Vmware 2.0 won’t share a drive directly into the virtual box, so Itunes is trying to pull all the music off over the wifi network. This just crushes out internal network. When you consider all the time I’ve spent getting the janky pre-release of vmware running, it’s just not worth it anymore. And yes, It’s a little crazy to install a new cpu just to run xenserver.

For now, I think I’ll just enjoy my macbook pro and iphone. 🙂

Posted in Rants | Leave a comment