Hi Charlie, all, At 6:21 Uhr -0500 5.7.2006, Charlie Balch wrote:
I would be very interested in seeing your results.
No problem, please just send me a reminder if you don't see our posting here before the end of the year. Btw., the first presentation of the results will be at the Web data collection workshop in Dubrovnik this fall: http://pdw2006.internet-research.info/
I'm also interested in why you think item randomization is important. I'm aware that there is some bias towards answering areas in web surveys. I'm also aware of the argument that any changes to a survey at the participant level create different environments and thus make the data questionable.
Admittedly, for some surveys and applications item randomization is not important or even harmful (e.g. with validated measures in personality research). However, there is a vast literature about order effects and context effects that cleary indicates vulnerability of survey results to fixed orders. The best solution to get rid of these problems is item randomization. As an illustration I would like to point you to a study two of my students and I reported in 2001 in Dimensions of Internet Science (http://www.psychologie.unizh.ch/sowi/reips/dis/). Changing the order of just 2 items made a difference of about 100 minutes (!) in reported television consumption per week (an effect of context and social desirability). Also, the order of groups of items influenced dropout behavior and data quality in the Web experiment.
By the way, http://birat.net is free including the source in ASP, but only runs on Windows Servers, and does not provide item randomization.
Thank you for the pointer (I also saw your earlier post and took a look at the system). A great initiative, but you may want to reconsider the platform restriction and set of features. In particular, I am afraid (or rather I am happy) the Internet will render most platform-dependent systems obsolete within the foreseeable future for a number of reasons you'll find below in an excerpt from a recent article. So better switch strategies ;-) Cheers, --u P.S. I liked "Dissertation Hell" as the building specification in your sig *grin* Excerpt from Reips, U.-D., & Lengler, R. (2005). The Web Experiment List: A Web service for the recruitment of participants and archiving of Internet-based experiments. Behavior Research Methods, 37, 287-292. http://homepage.mac.com/maculfy/filechute/BSC515.pdf "A number of tools have been developed for Internetbased experimenting that form a general framework of reference for the methodology. These tools can be grouped into two general classes of "software": programs and Web services. Programs follow the traditional format. They need to be installed on a computer and run locally. The working of the program depends on the computer's configuration, which may vary considerably over time (as other software is installed) and from user to user. Different types of operating systems may not allow a user to install the software at all. Upgrades and updates may be necessary. However, the user is in control of the service and independent of a connection to the Internet. An example of a tool for Internet-based experimenting (in this case for Web-based decision-making experiments) of the program type is WebDIP (Schulte-Mecklenbeck & Neun, 2005). Web services, on the other hand, run on a server that is connected to the Internet. Users access it via a Web browser and can use it only while they are connected to the Internet. Because the functionality of Web browsers is less dependent on the operating system (sometimes they are even referred to as being platform independent), all who access a Web service are likely to see and experience almost the same interface (but see, e.g., Dillman & Bowker, 2001, for browser-related problems in Internet-based research). Web services spare the user from upgrading and updating, since this is done by the Web service administrators at the server. Nothing is installed on the user's computer, saving space and time." (p. 287) -- PD Dr. Ulf-Dietrich Reips President, Society for Computers in Psychology (http://scip.ws) Editor, International Journal of Internet Science (http://www.ijis.net) Universität Zürich Psychologisches Institut Rämistr. 62 8001 Zürich, Switzerland iScience portal (http://psych-iscience.unizh.ch/)