U.-D. Sorry for my delayed reply - these are fascinating topics and I feel very fortunate to have the conversation with one of the leading researchers in this field. I have read all your papers that I could find. They were brilliant! I'm pleased to note that I'd already cited some of them in my dissertation and have added more as a result of my recent reading. Order I think you would agree that order can only be changed when the item options are nominal. Even with that restriction, I'm still not agreeing that randomizing the options is useful. As I understand it, there is no strong pattern to pattern responses as some participants will tend to answer at the top, others at the middle, and some at the bottom of the options when they are motivated to complete the survey at all. I can see some benefit of testing for such patterns and discarding responses that appear to be following a pattern. I don't see that randomizing order helps to control for such pattern responses. I also appreciate that participants will often respond to what they perceive as the first "correct" answer. Randomizing the item order would certainly change the results when a significant number of participants answer in this way. ***Does a change in results indicate more accurate results?*** It seems to me that an instrument that is susceptible to this sort of error is inheritably neither reliable or valid. How can you trust your data if the participants are not considering the items? What do you think about randomizing the items? I don't like the idea as I think commonality of experience is essential. WEXTOR My respect for the WEXTOR website! I created an account on WEXTOR and ran through a sample experiment. WEXTOR is a wonderful tool for prototyping experiments that I wish I'd know about when I took a few of my graduate statistical courses. Perhaps BIRAT will be a companion software product one day. Another interesting companion product would be branching system that helped researchers select appropriate statistics. A few questions about the target population and the type of data collected would make this relatively easy. http://psych-wextor.unizh.ch/wextor/en/ Access I'm not sure that I would call Access vulnerable any more than other databases are to most common attacks such as SQL injection. I certainly agree that Access is not as robust and bad code would allow Denial Of Service attacks. Fortunately the Access SQL logic and tables are easily ported to other databases. I'm sure that it has a lot to do with my first programming experiences but I find the ASP/Access combination easier for rapid prototyping. I also find the resulting product capable of handling reasonably robust demands. As noted before, I do intend to rewrite in PHP/MySQL. I'm just going to touch on the Macintosh/PC discussion. Macs are great and overpriced. If you really want to use open source at the right price point, run a 'nix flavor on PC hardware. Charlie Balch Professor CIS Arizona Western College (New hire) Doctoral Candidate LSU (Dissertation hell) Nice Guy (Just ask my wife) http://charlie.balch.org -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of Ulf-Dietrich Reips Sent: Tuesday, July 11, 2006 8:51 AM To: air-l@listserv.aoir.org Subject: Re: [Air-l] Opinions about Web Survey Software Hi Charlie Balch, all, sorry for the delayed reply. Was away for a few days... Congratulations to keeping your software product open and in development! Below some thoughts on a few issues. At 9:44 Uhr -0500 5.7.2006, Charlie Balch wrote:
Thanks for giving me some additional direction for my research. My reading of the web survey literature suggested that order effects were best controled by leaving the order fixed for all participants and accepting that there is an order bias that can be controlled for.
It can only be controlled for by varying order in "orderly ways" (how else would you know if and how much your results would be biased by order?). Essentially, there are two ways of doing this: balancing and randomization. For example, two items can be ordered in two ways, A first or B first. By balancing these orders, i.e. making sure that half of the participants get order AB and the other half BA, we effectively cancel out any order effects. (Of course, this method has limits: some contents can only be displayed in a particular order or there may be so-called asymmetrical transfer.) Randomizing the order of items spares us from creating all possible orders and has about the same effect (with a certain remaining low probability for influences of order).
I'm going to look some more into order bias and methodologies to control for it. One of the reasons that I wrote my own survey system was that I wanted to be able to respond to best practices and explore "new" possibilities such as my image item types (I know they have been around on paper but I haven't seen them on web surveys). It would be trivial to add the code to randomize order of items and/or item groups. I could record both the response and the order of the items responded to but it seems easier to me to just accept the bias -- my head hurts when I think about the size of output for those exponential possibilities. Very few would have the skill to properly analyze it. This brings up another concern of mine.
In both cases, balancing and randomization, you wouldn't need to analyze it, because you eliminated order effects by design. Recording the actual orders would make sense, though, in case there are unexpected patterns in the results.
How much should the software protect the user from stupid mistakes? We agree that there are at least some occasions where order does matter. Every little bit of software feature creap creates a more cumbersome interface and the possibility that the option will be used incorrectly.
I completely agree and see this as a general problem with many softwares.
I suspect that you would agree that reseachers are reluctant to learn complex interfaces. I shudder to think of giving lots of options that might be randomlly selected without thought of the consequences.
Personally, I believe these decisions should be based on knowledge about the intended audience, usability considerations, and in a good educational sense. Check out WEXTOR, our (Web) experiment generator, at http://psych-wextor.unizh.ch/wextor/en/ , where we have tried to achieve an optimal balance for a dual audience: researchers and students. WEXTOR automatically avoids many pitfalls naïve researchers step into on the Internet, and it implements techniques that have shown to be useful in Internet-based research.
Once I get the dissertation behind me, I intend to rewrite BIRAT into a more portable environment such as PHP and MYSQL. Perhaps it should be another discussion but I was interested in your thoughts about free platforms.
Imho this is the way to go, as long as they ar useable. Proprietary software is always a risk.
I also look forward to the day when all software is free. Even so, portability versus usability are a concern. My experience is that ASP and the associated Access DB are very common in academic servers and a "no brainer" to install.
Access has been notoriously vulnerable.
PHP/MySQL applications often make up for their lack of cost and improved efficiency by their difficulty to install. What good is a publically available application if most folks can't figure out how install it?
A good reason to run it on a Mac, where PHP comes pre-installed and MySQL is easily added (e.g. via MAMP).
Of course BIRAT also uses other languages like HTML, CSS, Javascript and such as well. We live in interesting programming times.
Just in case the above has left you in less than serene, you might want to look at another of my free applications: http://serenesound.com. I've been amazed at how it has been adopted by the medical community.
Finally, any chance you can send me the content of the papers/book you mention below? I'd love to read them but am traveling and thus away from
my
school library.
Some of it is on http://www.psychologie.unizh.ch/sowi/team/reips/publikationsliste.html Let me know if you need any of the other chapters. Best --u _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/