Denise, You might want to look at a free survey system that I wrote: http://birat.net My PHD is on best practices in Internet Survey Design and BIRAT reflects the research. BIRAT's web site is not a slick but the content of the survey system is solid and even has a few features that Opinio does not such as the ability to track user responses and image type items. BIRAT also provides export to Excel from which you can get to SPSS. Opinio does have some features that BIRAT does not as well. You can use BIRAT at the website or I'll send a copy of the code you can install on your own servers. Best regards, Charlie Balch In Dissertation Hell, Instructional Technology Lousiana State University http://charlie.balch.org -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of Denise N. Rall Sent: Monday, July 03, 2006 9:18 PM To: air-l@listserv.aoir.org Cc: ssankaran@scu.edu.au Subject: [Air-l] Any opinions about Opinio web survey software Dear Aoir-ers - We are currently evaluating Opinio (used to be SurveyMonkey) for our university. So far it seems to have that 'easy interface' but a few glitches moving the results to Excel and then to SPSS/PC - which is what we use for survey evaluation. Ok, any stories good and Especially Bad would be appreciated. Cheers, Denise Denise N. Rall, PhD thesis in revision, School of Environ. Science, Southern Cross University, Lismore NSW 2480 AUSTRALIA Tuesdays: Room T2.17, +61 (0)2 6620 3577 or Mobile 0427 245 497 http://www.scu.edu.au/schools/rsm/staff/pages/drall/index.html Virtual member, Cybermetrics Group, University of Wolverhampton, UK http://cybermetrics.wlv.ac.uk/index.html _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/

Hi Denise, all, in my research group we are currently undertaking an evaluation of all Web surveying software we are aware of. These are about 400 different products. We are half way through the evaluation. In the first step we boiled down the large number of tools to about 30, using three criteria that seemed most important to us: freeware or low price, platform independence, availability of item randomization. Surveymonkey (now Opinio) is one of the 30, and independent from this evaluation we have heard many positive reports from users. Also, one of the leading Web surveying experts, Dr. Michael Bosnjak, recommends it with convincing arguments. We will continue with our evaluation by applying further selection criteria and evaluate and experimentally test about 10 tools for use by laypeople and experts in Web surveying. And then let you know what we found :-) Best --u -- PD Dr. Ulf-Dietrich Reips President, Society for Computers in Psychology (http://scip.ws) Editor, International Journal of Internet Science (http://www.ijis.net) Universität Zürich Psychologisches Institut Rämistr. 62 8001 Zürich, Switzerland iScience portal (http://psych-iscience.unizh.ch/)

Hi Charlie, all, At 6:21 Uhr -0500 5.7.2006, Charlie Balch wrote:

I would be very interested in seeing your results.

No problem, please just send me a reminder if you don't see our posting here before the end of the year. Btw., the first presentation of the results will be at the Web data collection workshop in Dubrovnik this fall: http://pdw2006.internet-research.info/

I'm also interested in why you think item randomization is important. I'm aware that there is some bias towards answering areas in web surveys. I'm also aware of the argument that any changes to a survey at the participant level create different environments and thus make the data questionable.

Admittedly, for some surveys and applications item randomization is not important or even harmful (e.g. with validated measures in personality research). However, there is a vast literature about order effects and context effects that cleary indicates vulnerability of survey results to fixed orders. The best solution to get rid of these problems is item randomization. As an illustration I would like to point you to a study two of my students and I reported in 2001 in Dimensions of Internet Science (http://www.psychologie.unizh.ch/sowi/reips/dis/). Changing the order of just 2 items made a difference of about 100 minutes (!) in reported television consumption per week (an effect of context and social desirability). Also, the order of groups of items influenced dropout behavior and data quality in the Web experiment.

By the way, http://birat.net is free including the source in ASP, but only runs on Windows Servers, and does not provide item randomization.

Thank you for the pointer (I also saw your earlier post and took a look at the system). A great initiative, but you may want to reconsider the platform restriction and set of features. In particular, I am afraid (or rather I am happy) the Internet will render most platform-dependent systems obsolete within the foreseeable future for a number of reasons you'll find below in an excerpt from a recent article. So better switch strategies ;-) Cheers, --u P.S. I liked "Dissertation Hell" as the building specification in your sig *grin* Excerpt from Reips, U.-D., & Lengler, R. (2005). The Web Experiment List: A Web service for the recruitment of participants and archiving of Internet-based experiments. Behavior Research Methods, 37, 287-292. http://homepage.mac.com/maculfy/filechute/BSC515.pdf "A number of tools have been developed for Internetbased experimenting that form a general framework of reference for the methodology. These tools can be grouped into two general classes of "software": programs and Web services. Programs follow the traditional format. They need to be installed on a computer and run locally. The working of the program depends on the computer's configuration, which may vary considerably over time (as other software is installed) and from user to user. Different types of operating systems may not allow a user to install the software at all. Upgrades and updates may be necessary. However, the user is in control of the service and independent of a connection to the Internet. An example of a tool for Internet-based experimenting (in this case for Web-based decision-making experiments) of the program type is WebDIP (Schulte-Mecklenbeck & Neun, 2005). Web services, on the other hand, run on a server that is connected to the Internet. Users access it via a Web browser and can use it only while they are connected to the Internet. Because the functionality of Web browsers is less dependent on the operating system (sometimes they are even referred to as being platform independent), all who access a Web service are likely to see and experience almost the same interface (but see, e.g., Dillman & Bowker, 2001, for browser-related problems in Internet-based research). Web services spare the user from upgrading and updating, since this is done by the Web service administrators at the server. Nothing is installed on the user's computer, saving space and time." (p. 287) -- PD Dr. Ulf-Dietrich Reips President, Society for Computers in Psychology (http://scip.ws) Editor, International Journal of Internet Science (http://www.ijis.net) Universität Zürich Psychologisches Institut Rämistr. 62 8001 Zürich, Switzerland iScience portal (http://psych-iscience.unizh.ch/)

Hi Rowin, all, At 15:01 Uhr +0100 5.7.2006, Rowin Young wrote:

I would also be very interested in seeing the results of your research and am looking forward to them being announced.

I'll be happy to send them.

I also wondered how many (if any) of the tools you've looked at use the IMS Question and Test Interoperability specification (http://www.imsglobal.org/question/index.html), an open standard mainly being used in eassessment but which can also be used for surveys.

We are aware of the IMS standards (and have used an earlier version in our own e-learning development, see e.g. http://www.methpsy.unizh.ch/mesosworld/archive/standards/STD-QTI.html and http://www.methpsy.unizh.ch/mesosworld/archive/leitfaden/ulf/assessmentgen.h... ), but the IMS has a number of impracticalities and inconsistencies. In particular, none of the research on Web survey and Web experiment methodology has found its way into the standards (e.g., certain types of errors happen more easily with some response types or rendering types than others, so it is better to offer the latter in a tool). Also, the psychology and reality of Web surveying is not appropriately reflected in the standards. For example, the frequency of how often a question type is needed in creating Web surveys ought to be reflected in a tool's usability and selection of question types, but the standards are indifferent in this respect. Furthermore, response-type and rendering may theoretically ideal to be handled independently (as aimed for in the standards), but this is not the psychological reality for responders to and creators of questionnaires. They experience and prefer to handle both as a conglomerate. We can discuss further details off-list, if you like. However, we will consider the IMS in terms of the total picture of question types, layout types and response options etc. available in tools. Best --u -- PD Dr. Ulf-Dietrich Reips President, Society for Computers in Psychology (http://scip.ws) Editor, International Journal of Internet Science (http://www.ijis.net) Universität Zürich Psychologisches Institut Rämistr. 62 8001 Zürich, Switzerland iScience portal (http://psych-iscience.unizh.ch/)

Fascinating information Dr. Reips! http://birat.net is a proof of concept project associated with my dissertation on best practices in Internet research. While the system is stable, effective, and in world-wide use, I do not consider it complete. Given the changing nature of the net, BIRAT probably never will be complete. I intend for BIRAT to evolve as possibilities evolve and persons like you identify best practices. Thanks for giving me some additional direction for my research. My reading of the web survey literature suggested that order effects were best controled by leaving the order fixed for all participants and accepting that there is an order bias that can be controlled for. I'm going to look some more into order bias and methodologies to control for it. One of the reasons that I wrote my own survey system was that I wanted to be able to respond to best practices and explore "new" possibilities such as my image item types (I know they have been around on paper but I haven't seen them on web surveys). It would be trivial to add the code to randomize order of items and/or item groups. I could record both the response and the order of the items responded to but it seems easier to me to just accept the bias -- my head hurts when I think about the size of output for those exponential possibilities. Very few would have the skill to properly analyze it. This brings up another concern of mine. How much should the software protect the user from stupid mistakes? We agree that there are at least some occasions where order does matter. Every little bit of software feature creap creates a more cumbersome interface and the possibility that the option will be used incorrectly. I suspect that you would agree that reseachers are reluctant to learn complex interfaces. I shudder to think of giving lots of options that might be randomlly selected without thought of the consequences. Once I get the dissertation behind me, I intend to rewrite BIRAT into a more portable environment such as PHP and MYSQL. Perhaps it should be another discussion but I was interested in your thoughts about free platforms. I also look forward to the day when all software is free. Even so, portability versus usability are a concern. My experience is that ASP and the associated Access DB are very common in academic servers and a "no brainer" to install. PHP/MySQL applications often make up for their lack of cost and improved efficiency by their difficulty to install. What good is a publically available application if most folks can't figure out how install it? Of course BIRAT also uses other languages like HTML, CSS, Javascript and such as well. We live in interesting programming times. Just in case the above has left you in less than serene, you might want to look at another of my free applications: http://serenesound.com. I've been amazed at how it has been adopted by the medical community. Finally, any chance you can send me the content of the papers/book you mention below? I'd love to read them but am traveling and thus away from my school library. Charlie Balch LSU Doctoral Candidate http://charlie.balch.org -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of Ulf-Dietrich Reips Sent: Wednesday, July 05, 2006 8:26 AM To: air-l@listserv.aoir.org Subject: Re: [Air-l] Any opinions about Opinio web survey software Hi Charlie, all, At 6:21 Uhr -0500 5.7.2006, Charlie Balch wrote:

I would be very interested in seeing your results.

No problem, please just send me a reminder if you don't see our posting here before the end of the year. Btw., the first presentation of the results will be at the Web data collection workshop in Dubrovnik this fall: http://pdw2006.internet-research.info/

I'm also interested in why you think item randomization is important. I'm aware that there is some bias towards answering areas in web surveys. I'm also aware of the argument that any changes to a survey at the participant level create different environments and thus make the data questionable.

Admittedly, for some surveys and applications item randomization is not important or even harmful (e.g. with validated measures in personality research). However, there is a vast literature about order effects and context effects that cleary indicates vulnerability of survey results to fixed orders. The best solution to get rid of these problems is item randomization. As an illustration I would like to point you to a study two of my students and I reported in 2001 in Dimensions of Internet Science (http://www.psychologie.unizh.ch/sowi/reips/dis/). Changing the order of just 2 items made a difference of about 100 minutes (!) in reported television consumption per week (an effect of context and social desirability). Also, the order of groups of items influenced dropout behavior and data quality in the Web experiment.

By the way, http://birat.net is free including the source in ASP, but only runs on Windows Servers, and does not provide item randomization.

Thank you for the pointer (I also saw your earlier post and took a look at the system). A great initiative, but you may want to reconsider the platform restriction and set of features. In particular, I am afraid (or rather I am happy) the Internet will render most platform-dependent systems obsolete within the foreseeable future for a number of reasons you'll find below in an excerpt from a recent article. So better switch strategies ;-) Cheers, --u P.S. I liked "Dissertation Hell" as the building specification in your sig *grin* Excerpt from Reips, U.-D., & Lengler, R. (2005). The Web Experiment List: A Web service for the recruitment of participants and archiving of Internet-based experiments. Behavior Research Methods, 37, 287-292. http://homepage.mac.com/maculfy/filechute/BSC515.pdf "A number of tools have been developed for Internetbased experimenting that form a general framework of reference for the methodology. These tools can be grouped into two general classes of "software": programs and Web services. Programs follow the traditional format. They need to be installed on a computer and run locally. The working of the program depends on the computer's configuration, which may vary considerably over time (as other software is installed) and from user to user. Different types of operating systems may not allow a user to install the software at all. Upgrades and updates may be necessary. However, the user is in control of the service and independent of a connection to the Internet. An example of a tool for Internet-based experimenting (in this case for Web-based decision-making experiments) of the program type is WebDIP (Schulte-Mecklenbeck & Neun, 2005). Web services, on the other hand, run on a server that is connected to the Internet. Users access it via a Web browser and can use it only while they are connected to the Internet. Because the functionality of Web browsers is less dependent on the operating system (sometimes they are even referred to as being platform independent), all who access a Web service are likely to see and experience almost the same interface (but see, e.g., Dillman & Bowker, 2001, for browser-related problems in Internet-based research). Web services spare the user from upgrading and updating, since this is done by the Web service administrators at the server. Nothing is installed on the user's computer, saving space and time." (p. 287) -- PD Dr. Ulf-Dietrich Reips President, Society for Computers in Psychology (http://scip.ws) Editor, International Journal of Internet Science (http://www.ijis.net) Universität Zürich Psychologisches Institut Rämistr. 62 8001 Zürich, Switzerland iScience portal (http://psych-iscience.unizh.ch/) _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/

U.-D. Sorry for my delayed reply - these are fascinating topics and I feel very fortunate to have the conversation with one of the leading researchers in this field. I have read all your papers that I could find. They were brilliant! I'm pleased to note that I'd already cited some of them in my dissertation and have added more as a result of my recent reading. Order I think you would agree that order can only be changed when the item options are nominal. Even with that restriction, I'm still not agreeing that randomizing the options is useful. As I understand it, there is no strong pattern to pattern responses as some participants will tend to answer at the top, others at the middle, and some at the bottom of the options when they are motivated to complete the survey at all. I can see some benefit of testing for such patterns and discarding responses that appear to be following a pattern. I don't see that randomizing order helps to control for such pattern responses. I also appreciate that participants will often respond to what they perceive as the first "correct" answer. Randomizing the item order would certainly change the results when a significant number of participants answer in this way. ***Does a change in results indicate more accurate results?*** It seems to me that an instrument that is susceptible to this sort of error is inheritably neither reliable or valid. How can you trust your data if the participants are not considering the items? What do you think about randomizing the items? I don't like the idea as I think commonality of experience is essential. WEXTOR My respect for the WEXTOR website! I created an account on WEXTOR and ran through a sample experiment. WEXTOR is a wonderful tool for prototyping experiments that I wish I'd know about when I took a few of my graduate statistical courses. Perhaps BIRAT will be a companion software product one day. Another interesting companion product would be branching system that helped researchers select appropriate statistics. A few questions about the target population and the type of data collected would make this relatively easy. http://psych-wextor.unizh.ch/wextor/en/ Access I'm not sure that I would call Access vulnerable any more than other databases are to most common attacks such as SQL injection. I certainly agree that Access is not as robust and bad code would allow Denial Of Service attacks. Fortunately the Access SQL logic and tables are easily ported to other databases. I'm sure that it has a lot to do with my first programming experiences but I find the ASP/Access combination easier for rapid prototyping. I also find the resulting product capable of handling reasonably robust demands. As noted before, I do intend to rewrite in PHP/MySQL. I'm just going to touch on the Macintosh/PC discussion. Macs are great and overpriced. If you really want to use open source at the right price point, run a 'nix flavor on PC hardware. Charlie Balch Professor CIS Arizona Western College (New hire) Doctoral Candidate LSU (Dissertation hell) Nice Guy (Just ask my wife) http://charlie.balch.org -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of Ulf-Dietrich Reips Sent: Tuesday, July 11, 2006 8:51 AM To: air-l@listserv.aoir.org Subject: Re: [Air-l] Opinions about Web Survey Software Hi Charlie Balch, all, sorry for the delayed reply. Was away for a few days... Congratulations to keeping your software product open and in development! Below some thoughts on a few issues. At 9:44 Uhr -0500 5.7.2006, Charlie Balch wrote:

Thanks for giving me some additional direction for my research. My reading of the web survey literature suggested that order effects were best controled by leaving the order fixed for all participants and accepting that there is an order bias that can be controlled for.

It can only be controlled for by varying order in "orderly ways" (how else would you know if and how much your results would be biased by order?). Essentially, there are two ways of doing this: balancing and randomization. For example, two items can be ordered in two ways, A first or B first. By balancing these orders, i.e. making sure that half of the participants get order AB and the other half BA, we effectively cancel out any order effects. (Of course, this method has limits: some contents can only be displayed in a particular order or there may be so-called asymmetrical transfer.) Randomizing the order of items spares us from creating all possible orders and has about the same effect (with a certain remaining low probability for influences of order).

I'm going to look some more into order bias and methodologies to control for it. One of the reasons that I wrote my own survey system was that I wanted to be able to respond to best practices and explore "new" possibilities such as my image item types (I know they have been around on paper but I haven't seen them on web surveys). It would be trivial to add the code to randomize order of items and/or item groups. I could record both the response and the order of the items responded to but it seems easier to me to just accept the bias -- my head hurts when I think about the size of output for those exponential possibilities. Very few would have the skill to properly analyze it. This brings up another concern of mine.

In both cases, balancing and randomization, you wouldn't need to analyze it, because you eliminated order effects by design. Recording the actual orders would make sense, though, in case there are unexpected patterns in the results.

How much should the software protect the user from stupid mistakes? We agree that there are at least some occasions where order does matter. Every little bit of software feature creap creates a more cumbersome interface and the possibility that the option will be used incorrectly.

I completely agree and see this as a general problem with many softwares.

I suspect that you would agree that reseachers are reluctant to learn complex interfaces. I shudder to think of giving lots of options that might be randomlly selected without thought of the consequences.

Personally, I believe these decisions should be based on knowledge about the intended audience, usability considerations, and in a good educational sense. Check out WEXTOR, our (Web) experiment generator, at http://psych-wextor.unizh.ch/wextor/en/ , where we have tried to achieve an optimal balance for a dual audience: researchers and students. WEXTOR automatically avoids many pitfalls naïve researchers step into on the Internet, and it implements techniques that have shown to be useful in Internet-based research.

Once I get the dissertation behind me, I intend to rewrite BIRAT into a more portable environment such as PHP and MYSQL. Perhaps it should be another discussion but I was interested in your thoughts about free platforms.

Imho this is the way to go, as long as they ar useable. Proprietary software is always a risk.

I also look forward to the day when all software is free. Even so, portability versus usability are a concern. My experience is that ASP and the associated Access DB are very common in academic servers and a "no brainer" to install.

Access has been notoriously vulnerable.

PHP/MySQL applications often make up for their lack of cost and improved efficiency by their difficulty to install. What good is a publically available application if most folks can't figure out how install it?

A good reason to run it on a Mac, where PHP comes pre-installed and MySQL is easily added (e.g. via MAMP).

Of course BIRAT also uses other languages like HTML, CSS, Javascript and such as well. We live in interesting programming times.

Just in case the above has left you in less than serene, you might want to look at another of my free applications: http://serenesound.com. I've been amazed at how it has been adopted by the medical community.

Finally, any chance you can send me the content of the papers/book you mention below? I'd love to read them but am traveling and thus away from

my

school library.

Some of it is on http://www.psychologie.unizh.ch/sowi/team/reips/publikationsliste.html Let me know if you need any of the other chapters. Best --u _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/

Elijah, I've read good things about postgresql (http://www.postgresql.org/about/). I don't know why postgresql is not more popular. Portability is going to be a major concern when I do my next rewrite. While postgresql runs on many platforms, it does not seem to have the installed base of MySQL. Running with multiple DBs, languages, etc. will require exponential complexity for my BIRAT (http://biart.net) statistical code. If BIRAT becomes popular enough, I'll do it or perhaps others will join in this open source project. Charlie -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of elw@stderr.org Sent: Tuesday, July 25, 2006 9:06 AM To: air-l@listserv.aoir.org Subject: Re: [Air-l] Opinions about Web Survey Software (ordereffects/WEXTOR/Access)

Access I'm not sure that I would call Access vulnerable any more than other databases are to most common attacks such as SQL injection. I certainly agree that Access is not as robust and bad code would allow Denial Of Service attacks. Fortunately the Access SQL logic and tables are easily ported to other databases. I'm sure that it has a lot to do with my first programming experiences but I find the ASP/Access combination easier for rapid prototyping. I also find the resulting product capable of handling reasonably robust demands. As noted before, I do intend to rewrite in PHP/MySQL.

As someone who ends up supporting other people who want to use this kind of software... please, please, please make sure that your kit can run with a postgresql backend in addition (or, perhaps, in preference) to mysql. MySQL is *terrible*, as relational databases go. thanks, --elijah _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/

I think the "problem" with postgresql is that it is an enterprise level system. Enterprise level systems require a lot of expertise to install, maintain, and program. On the other hand they are more stable and efficient. I found an interesting read at http://docs.moodle.org/en/Arguments_in_favour_of_PostgreSQL - I'm convinced. Hopefully I'll not have to write too many different SQL statements. Charlie -----Original Message----- From: air-l-bounces@listserv.aoir.org [mailto:air-l-bounces@listserv.aoir.org] On Behalf Of elw@stderr.org Sent: Tuesday, July 25, 2006 12:56 PM To: air-l@listserv.aoir.org Subject: Re: [Air-l] : postgresql and mysql

Charlie Balch: I've read good things about postgresql (http://www.postgresql.org/about/). I don't know why postgresql is not more popular. Portability is going to be a major concern when I do my next rewrite. While postgresql runs on many platforms, it does not seem to have the installed base of MySQL.

mysql is 'easy' to pick up, but lacks many of the features that are actually useful for writing dependable software. it is very easy to write completely non-portable, inefficient, sloppy bad software with mysql - and never be able to move to some other backend without extreme pain. postgresql is a little further up the learning curve, but there's quite a bit more power exposed to the programmer/dba. and you're a lot closer to "standards-compliant" - which makes your support folks very happy. --elijah _______________________________________________ The air-l@listserv.aoir.org mailing list is provided by the Association of Internet Researchers http://aoir.org Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org Join the Association of Internet Researchers: http://www.aoir.org/