Fascinating information Dr. Reips! http://birat.net is a proof of concept project associated with my dissertation on best practices in Internet research. While the system is stable, effective, and in world-wide use, I do not consider it complete. Given the changing nature of the net, BIRAT probably never will be complete. I intend for BIRAT to evolve as possibilities evolve and persons like you identify best practices. Thanks for giving me some additional direction for my research. My reading of the web survey literature suggested that order effects were best controled by leaving the order fixed for all participants and accepting that there is an order bias that can be controlled for. I'm going to look some more into order bias and methodologies to control for it. One of the reasons that I wrote my own survey system was that I wanted to be able to respond to best practices and explore "new" possibilities such as my image item types (I know they have been around on paper but I haven't seen them on web surveys). It would be trivial to add the code to randomize order of items and/or item groups. I could record both the response and the order of the items responded to but it seems easier to me to just accept the bias -- my head hurts when I think about the size of output for those exponential possibilities. Very few would have the skill to properly analyze it. This brings up another concern of mine. How much should the software protect the user from stupid mistakes? We agree that there are at least some occasions where order does matter. Every little bit of software feature creap creates a more cumbersome interface and the possibility that the option will be used incorrectly. I suspect that you would agree that reseachers are reluctant to learn complex interfaces. I shudder to think of giving lots of options that might be randomlly selected without thought of the consequences. Once I get the dissertation behind me, I intend to rewrite BIRAT into a more portable environment such as PHP and MYSQL. Perhaps it should be another discussion but I was interested in your thoughts about free platforms. I also look forward to the day when all software is free. Even so, portability versus usability are a concern. My experience is that ASP and the associated Access DB are very common in academic servers and a "no brainer" to install. PHP/MySQL applications often make up for their lack of cost and improved efficiency by their difficulty to install. What good is a publically available application if most folks can't figure out how install it? Of course BIRAT also uses other languages like HTML, CSS, Javascript and such as well. We live in interesting programming times. Just in case the above has left you in less than serene, you might want to look at another of my free applications: http://serenesound.com. I've been amazed at how it has been adopted by the medical community. Finally, any chance you can send me the content of the papers/book you mention below? I'd love to read them but am traveling and thus away from my school library. Charlie Balch LSU Doctoral Candidate http://charlie.balch.org -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of Ulf-Dietrich Reips Sent: Wednesday, July 05, 2006 8:26 AM To: air-l@listserv.aoir.org Subject: Re: [Air-l] Any opinions about Opinio web survey software Hi Charlie, all, At 6:21 Uhr -0500 5.7.2006, Charlie Balch wrote:
I would be very interested in seeing your results.
No problem, please just send me a reminder if you don't see our posting here before the end of the year. Btw., the first presentation of the results will be at the Web data collection workshop in Dubrovnik this fall: http://pdw2006.internet-research.info/
I'm also interested in why you think item randomization is important. I'm aware that there is some bias towards answering areas in web surveys. I'm also aware of the argument that any changes to a survey at the participant level create different environments and thus make the data questionable.
Admittedly, for some surveys and applications item randomization is not important or even harmful (e.g. with validated measures in personality research). However, there is a vast literature about order effects and context effects that cleary indicates vulnerability of survey results to fixed orders. The best solution to get rid of these problems is item randomization. As an illustration I would like to point you to a study two of my students and I reported in 2001 in Dimensions of Internet Science (http://www.psychologie.unizh.ch/sowi/reips/dis/). Changing the order of just 2 items made a difference of about 100 minutes (!) in reported television consumption per week (an effect of context and social desirability). Also, the order of groups of items influenced dropout behavior and data quality in the Web experiment.
By the way, http://birat.net is free including the source in ASP, but only runs on Windows Servers, and does not provide item randomization.
Thank you for the pointer (I also saw your earlier post and took a look at the system). A great initiative, but you may want to reconsider the platform restriction and set of features. In particular, I am afraid (or rather I am happy) the Internet will render most platform-dependent systems obsolete within the foreseeable future for a number of reasons you'll find below in an excerpt from a recent article. So better switch strategies ;-) Cheers, --u P.S. I liked "Dissertation Hell" as the building specification in your sig *grin* Excerpt from Reips, U.-D., & Lengler, R. (2005). The Web Experiment List: A Web service for the recruitment of participants and archiving of Internet-based experiments. Behavior Research Methods, 37, 287-292. http://homepage.mac.com/maculfy/filechute/BSC515.pdf "A number of tools have been developed for Internetbased experimenting that form a general framework of reference for the methodology. These tools can be grouped into two general classes of "software": programs and Web services. Programs follow the traditional format. They need to be installed on a computer and run locally. The working of the program depends on the computer's configuration, which may vary considerably over time (as other software is installed) and from user to user. Different types of operating systems may not allow a user to install the software at all. Upgrades and updates may be necessary. However, the user is in control of the service and independent of a connection to the Internet. An example of a tool for Internet-based experimenting (in this case for Web-based decision-making experiments) of the program type is WebDIP (Schulte-Mecklenbeck & Neun, 2005). Web services, on the other hand, run on a server that is connected to the Internet. Users access it via a Web browser and can use it only while they are connected to the Internet. Because the functionality of Web browsers is less dependent on the operating system (sometimes they are even referred to as being platform independent), all who access a Web service are likely to see and experience almost the same interface (but see, e.g., Dillman & Bowker, 2001, for browser-related problems in Internet-based research). Web services spare the user from upgrading and updating, since this is done by the Web service administrators at the server. Nothing is installed on the user's computer, saving space and time." (p. 287) -- PD Dr. Ulf-Dietrich Reips President, Society for Computers in Psychology (http://scip.ws) Editor, International Journal of Internet Science (http://www.ijis.net) Universität Zürich Psychologisches Institut Rämistr. 62 8001 Zürich, Switzerland iScience portal (http://psych-iscience.unizh.ch/) _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/