Hi Charlie Balch, all, sorry for the delayed reply. Was away for a few days... Congratulations to keeping your software product open and in development! Below some thoughts on a few issues. At 9:44 Uhr -0500 5.7.2006, Charlie Balch wrote:
Thanks for giving me some additional direction for my research. My reading of the web survey literature suggested that order effects were best controled by leaving the order fixed for all participants and accepting that there is an order bias that can be controlled for.
It can only be controlled for by varying order in "orderly ways" (how else would you know if and how much your results would be biased by order?). Essentially, there are two ways of doing this: balancing and randomization. For example, two items can be ordered in two ways, A first or B first. By balancing these orders, i.e. making sure that half of the participants get order AB and the other half BA, we effectively cancel out any order effects. (Of course, this method has limits: some contents can only be displayed in a particular order or there may be so-called asymmetrical transfer.) Randomizing the order of items spares us from creating all possible orders and has about the same effect (with a certain remaining low probability for influences of order).
I'm going to look some more into order bias and methodologies to control for it. One of the reasons that I wrote my own survey system was that I wanted to be able to respond to best practices and explore "new" possibilities such as my image item types (I know they have been around on paper but I haven't seen them on web surveys). It would be trivial to add the code to randomize order of items and/or item groups. I could record both the response and the order of the items responded to but it seems easier to me to just accept the bias -- my head hurts when I think about the size of output for those exponential possibilities. Very few would have the skill to properly analyze it. This brings up another concern of mine.
In both cases, balancing and randomization, you wouldn't need to analyze it, because you eliminated order effects by design. Recording the actual orders would make sense, though, in case there are unexpected patterns in the results.
How much should the software protect the user from stupid mistakes? We agree that there are at least some occasions where order does matter. Every little bit of software feature creap creates a more cumbersome interface and the possibility that the option will be used incorrectly.
I completely agree and see this as a general problem with many softwares.
I suspect that you would agree that reseachers are reluctant to learn complex interfaces. I shudder to think of giving lots of options that might be randomlly selected without thought of the consequences.
Personally, I believe these decisions should be based on knowledge about the intended audience, usability considerations, and in a good educational sense. Check out WEXTOR, our (Web) experiment generator, at http://psych-wextor.unizh.ch/wextor/en/ , where we have tried to achieve an optimal balance for a dual audience: researchers and students. WEXTOR automatically avoids many pitfalls naïve researchers step into on the Internet, and it implements techniques that have shown to be useful in Internet-based research.
Once I get the dissertation behind me, I intend to rewrite BIRAT into a more portable environment such as PHP and MYSQL. Perhaps it should be another discussion but I was interested in your thoughts about free platforms.
Imho this is the way to go, as long as they ar useable. Proprietary software is always a risk.
I also look forward to the day when all software is free. Even so, portability versus usability are a concern. My experience is that ASP and the associated Access DB are very common in academic servers and a "no brainer" to install.
Access has been notoriously vulnerable.
PHP/MySQL applications often make up for their lack of cost and improved efficiency by their difficulty to install. What good is a publically available application if most folks can't figure out how install it?
A good reason to run it on a Mac, where PHP comes pre-installed and MySQL is easily added (e.g. via MAMP).
Of course BIRAT also uses other languages like HTML, CSS, Javascript and such as well. We live in interesting programming times.
Just in case the above has left you in less than serene, you might want to look at another of my free applications: http://serenesound.com. I've been amazed at how it has been adopted by the medical community.
Finally, any chance you can send me the content of the papers/book you mention below? I'd love to read them but am traveling and thus away from my school library.
Some of it is on http://www.psychologie.unizh.ch/sowi/team/reips/publikationsliste.html Let me know if you need any of the other chapters. Best --u