The Moth

In my quest to get some feedback I setup 3 surveys for my subscribers with identical questions on three different sites: surveymonkey, zoomerang and questionpro (I also signed up and test drove for a bit wufoo but it wouldn't let me setup the questions the way I wanted them so I gave up). I thought I'd share my learnings here so you can use the knowledge if you setup a survey for your own site/blog or even for your team at work or your customers. At the bottom I also have some generic tips.

Remember my disclaimer.

COMPARISON
All 3 vendors offer a free version of their paid for services. The main limitation of the free service on all sites is that they only allow 100 responses per survey.

In all cases you sign up by providing your email and then you receive the login info. Surveymonkey sent me one text-only email with login details and that was the last I heard from them. The other 2 sent me html emails and also decided to sing me up to their newsletter (which I promptly unsubscribed from). Surveymonkey gives you a URL that points directly to the survey; the other two force a landing page (where they have their adverts) before allowing the survey taker to proceed.

Zoomerang has a funky easy to use user interface and with minimal clicks you can setup the survey and ditto for browsing the results (both overall and individual). There are 2 reasons I will not be using their free service again. Not only you cannot download the reports but they will only hold them on their servers for 10 days since the launch time of the report. More importantly, they will not notify you when the 100 allowed responses are complete and instead let people fill in your survey beyond that and if you want to see the additional completes you have to pay (I noticed this at some point and had to manually close the survey)!

QuestionPro has a horrible unintuitive user interface IMO. In addition, it takes far too many clicks and slow response times to setup the survey and ditto for reading the reports. Its login timeout is also too short so it added additional clicks to my user experience every time I refreshed the reports page. The main advantage it has is that even the free version allows you to get csv/xls reports to your own machine, whereas the other 2 will not let you do that unless you pay. Even generating the report involved making a few clicks and then receiving an email with an attachment (instead of just right click->save as). Beware that they will not notify you when the response limit reaches 100 and if there are more than 100 responses they will not show you *any* reports until you upgrade!

Surveymonkey just works. Best experience for setting up, quickest to use for browsing reports, keeps you logged in with no apparent timeouts and smoother experience altogether both for me and for the people taking the survey. None of the issues I mentioned in the previous two paragraphs.

TIPS
To conclude, here are some random tips if you run a survey. This is based on my vast experience of doing this...twice ;-)

1.Keep the number of questions short; definitely make them fit on a single short page. Any more than that and users will abandon the survey the minute they see it. Budget 20 seconds per question and then gauge how long you think someone will spend overall there before giving up – this should guide you to the total # of questions.

2.Write questions in such a way so the user can just select the answer (i.e. Yes/NO or multiple choice). Again, if you force me to have to do a lot of typing, I will probably not take the survey. Make the last choice "Other" where applicable and offer an optional textbox for additional comment.

3.Following on from the previous point, try and make the multiple choice answers exclusive. Allowing multiple responses per question (like I did) means having to spend a lot more time analysing the results as the percentages alone do not show the full picture.

4.When you think you wrote a good question, visualise how the results will be interpreted. If you cannot think of how you would interpret the results or what action you'd take based on the results then the question is wrong. For example, I asked a question in my survey that was just to satisfy my curiosity and in hindsight it was not actionable or interpretable so I should have left it out.

5.The more factual-based questions you have, the faster the user can move through them. The more opinion-based questions that you have the longer it takes to fill in and the longer it will take you to analyze. Just something to bear in mind.

6.Assuming that you managed to write questions where the user didn't have to use the keyboard, make the last question open ended. Make it last, so you don't receive feedback that is covered in other questions. Make it open ended because the verbatim feedback will likely be more important than the rest of the questions.

7.Never use multiple different survey sites for the same set of questions. Aggregating the results from 3 different engines for consolidation is a bit of a time consuming nightmare.