I'm still looking at the Suggestion Box Problem and finding ways citizen science can capture the breadth and depth of ideas available from the public. There are so many ideas of varying quality and in vastly different areas it seems impossible to sort through them all. But I'm determined to find some models that can.
Beginning my research I stumbled upon a working paper published by Matthew J. Salganik and Karen E.C. Levy titled "Wiki Surveys: Open and Quantifiable Social Data Collection" (arXiv:1202.0500v1). In it they compare different methods social scientists use to collect data from the public, and then go one step further to describe a new web site (www.allourideas.org) to implement their ideas and prove the value of "Wiki Surveys". This seems like a great starting point.
I've highlighted a few key points from the article and offer some of my own thoughts on it's applicability to citizen science. But I also recommend reading the full working paper yourself, available for free here.
The authors begin by looking at the problems with different ways of collecting data.
While surveys allow researchers to quantify large amounts of information quickly and at a reasonable cost, they are routinely criticized for being "top-down" and rigid; that is, the survey questions and possible responses are formulated before data collection begins, meaning that surveys generally are not open to novel or unexpected information from respondents. In contrast, interviews allow new information to "bubble up" directly from respondents, but are slow, expensive, and difficult to quantify...Advances in computing technology now enable a hybrid approach that combines the quantifiability of a survey and the openness of an interview; we call this new class of data collection tools wiki surveys.
The authors also describe the trade-offs involved in designing a collection instrument:
The primary advantage of closed questions is that responses can be handled with relative ease: answers can be assigned values, fed into statistical software, and employed in quantitative analysis. There processes are relatively straightforward, fast, and inexpensive, making closed questions an efficient choice for large-scale social science surveys. In contrast, responses to open-ended questions are more complicated for researchers to reliably code and quantify...In some cases, however, open methods may provide insights that closed methods cannot because they are receptive to new information that was unanticipated by the researcher...Because respondents have a strong tendency to confine their responses to the answer choices offered (Krosnick, 1999; Schuman, 2008), researchers who construct all the possible answer choices necessarily constrain what can be learned. This is unfortunate because unanticipated information is often the most valuable for research.
Looking at our own field, citizen science seems to rely much more heavily on the first type of collection type, closed questions, and must less on open-ended questions. In fact many of the most popular citizen science programs involve public participation by collecting defined sets of data or answering specific sets of questions. We see this in tools such as eBird, Nature's Notebook, and the many other survey instruments that ask people to collect and report wildlife data that can be . There are also specific advantages to closed questions in citizen science, since often the public does not know what to look for or how to report it, since they are not highly trained in that field. In these cases the structure of closed questioning helps them organize their thoughts and narrow their focus. It also allows data to be transmitted electronically and stored as structured data items, helping analysis and accommodating mobile applications. So for many projects this can be a highly successful approach. But it still means many things may be missed.
Traditional surveys attempt to collect a fixed amount of information from each respondent; respondents who want to contribute less than one questionnaire's worth of information are considered problematic and respondents who want to contribute more are prohibited from doing so. This contrasts sharply with successful information aggregation projects on the Internet, which collect as much or as little information as each respondent is willing to provide. Such a structure typically results in highly unequal levels of contribution: when contributors are plotted in rank order, the distributions tend to show a small amount of heavy contributors -- the "fat head" -- and a large number of light contributors -- the "long tail" (Anderson, 2006; Wilkinson, 2008).
The Zooniverse projects have had success combining types of questions. Most of their projects break down large scientific tasks into smaller pieces and walking participants through a series of questions to describe a piece of data (picture, light curve, whale song, etc.) in a structured manner. But they also encourage people to identify "unknown" or "interesting" aspects of each picture they are reviewing, and they also have discussion forums that allow users to discuss interesting items not captured by the project tools. This is how they stumbled on Hanny's Voorwerp though there are also other examples. Interestingly, the paper authors discuss this type of approach in their background:
[Yochai] Benkler (The Wealth of Networks: How Social Production Transforms Markets and Freedom; 2006) notes that successful information aggregation systems are typically composed of granular, modular tasks. That is, in successful systems, large problems can be broken down into smaller "chunks" which require low individual investment of time and effort (granularity) and these "chunks" can be independently completed by many individuals before being flexibly integrated into a larger whole (modularity).
In developing their model the authors propose three traits that any successful survey instrument needs to have to capture information from a wide number of people and perspectives. Although these are applied to the Wiki Surveys they may also have wide use for our citizen science "Suggestion Box" problem and will be a useful tool in evaluating other potential approaches. The proposed key traits are:
- Adaptive: They should be continually optimized to elicit the most useful information for estimating the parameters of interest, given what is already known.
- Greedy: They should capture as much or as little information as a respondent is willing to provide.
- Collaborative: They should be open to new information contributed directly by respondents that may not have been anticipated by the researcher, as often happens during an interview. Crucially, unlike a traditional "other" box in a survey, this new information would then be presented to future respondents for evaluation. In this way, a wiki survey bears some resemblance to a focus group in which participants can respond to the contributions of others.
For example, let's say you want to find out what the most popular color is. As a researcher you seed the survey with Red, Orange, Yellow, Green, Blue, Indigo, and Violet. When a user logs on they are asked "Which of these two is your favorite color?" and they are forced to choose between Red and Indigo. Then Orange and Blue. Then Red and Orange. The questions continue and slowly the system can rank responses based on how people respond. Additionally, someone could add their own favorite, Maroon. The system will add now include Maroon as part of the randomly-generated pairs, so some users will be asked to choose between Indigo and Maroon, or Yellow and Maroon.
The AllOurIdeas site has actually hosted many surveys since the paper was first published, with over 2.5 million votes cast over 1,500 surveys. There were also over 60,000 new ideas submitted that the individual survey designers had not initially included and presumably never thought of. This includes two specific surveys, one for the New York City Mayor's Office on creating a greener, greater city and one for the international Organization for Economic Cooperation and Development. They noted a couple of interesting findings.
One is that "...wiki surveys are best suited for situations in which there is a single predetermined question." In some ways this fits the needs of citizen science projects since many currently exist with well-defined questions. But it doesn't hit our biggest problem...capturing chance discoveries and unexpected results. Some of the most powerful discoveries have been made accidentally while a scientist was looking at a different problem, or made a "mistake" in an experiment. Just think of the discovery of Penicillin or invention of the microwave...none of these was planned for but they became very important. It also doesn't capture theoretical discoveries or insights, just data. So the model does require continued development.
Additionally, it was discovered that:
...in both wiki surveys, many of the highest scoring ideas were uploaded by users. For PlaNYC, 8 of the top 10 ideas were uploaded by users, as were 7 of the top 10 for the OECD. ..There seem to be two general classes of uploaded ideas that score well: novel information -- that is, new ideas that were not anticipated by the wiki survey creators -- and alternative framings -- that is, new and "stickier" ways of expressing existing ideas.
All of this has given me some great ideas on how to use wiki surveys for my own research on citizen science topics. But I'm getting ahead of myself; there is still a lot more research to do first. Hopefully this has given you some insights too, and let me know if there are other papers you find that are also interesting. I'm still working through some more but would welcome any additional ideas.
Just let me know in the comments below.
With surveys it becomes easy to understand all the necessary steps that need to be taken in order for the improvements. Surveys are also done to get thoughts, opinions from the participants. Today online surveys are also done by using online survey tools. It is easy to get the data at one go in an organized manner with the use of online software tools. Apart from these surveys can be local, among a community or global. There are also survey communities where people can give their responses. I think it is an easy way to know about a particular product or service.
ReplyDeleteI just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon.A fantastic presentation. Very open and informative.You have beautifully presented your thought in this blog post. Good Online Surveys Software
ReplyDeleteThere was no wide passage there, although had Franklin's men made it so far, they might have stumbled upon the Bellot Strait.http://bestpaidsurveyssites.net
ReplyDeleteIn fact many of the most popular citizen science programs involve public participation by collecting defined sets of data or answering specific sets of questions.
ReplyDeleteI called the dealer from the number recorded on the site and consented to meet him in his apartment suite. Following a 20 minute taxi ride I was remaining before a standout amongst the most excellent apartment suites I had ever observed. check cashing
ReplyDeletegiven article is very helpful and very useful for my admin, and pardon me permission to share articles here hopefully helped :
ReplyDeleteCara menyembuhkan lambung bengkak
Obat keputihan alami
Penyebab usus buntu dan pengobatannya
Survey is the best way to identify the problem and also easily find the solution based on doing several analyse. Nowadays lots of online pharmacy and other shopping sites doing many survey to provide a best products to their customers.
ReplyDeleteWe are also doing a survey on the problems faced by the people during the shopping from the online pharmacy. People are more intelligent nowadays and they are choosing internet platform to get medical needs at low price. We are also in the same industry and our online medical shop providing FDA approved medications.
ReplyDeleteNICE Artikel
ReplyDeletebandar togel
Agen togel
togel online
togel online terpercaya
buku mimpi togel
bandar togel terpercaya
main togel
Mari gabung bersama kami di Domino Halo
ReplyDelete* Bonus Referal 20%
* Bonus T.O mingguan sebesar 0.5%
* Minimal deposit RP.20.000
* Jackpot tercepat
DOMINOHALO - Agen Poker Terpercaya | Texas Poker | Poker Online | DominoQQ Online | BandarQ Online | Agen Poker Terpercaya Se-Indonesia
Mari gabung bersama kami di Domino Halo
ReplyDelete* Bonus Referal 20%
* Bonus T.O mingguan sebesar 0.5%
* Minimal deposit RP.20.000
* Jackpot tercepat
DOMINOHALO - Agen Poker Terpercaya | Texas Poker | Poker Online | DominoQQ Online | BandarQ Online | Agen Poker Terpercaya Se-Indonesia
Thank you for sharing! Glad to find.
ReplyDeleteemail login
Traditional surveys attempt to collect a fixed amount of information from each respondent; respondents who want to contribute less than one questionnaire’s worth of information are considered problematic, and respondents who want to contribute more are prohibited from doing so.
ReplyDeleteBefore purchasing any product from online pharmacy we should ensure about the quality and prices. So, survey is important for all; because it will help us to know more about the shop.
ReplyDelete
ReplyDeleteشركة نقل عفش
اهم شركات مكافحة حشرات بالخبر كذلك معرض اهم شركة مكافحة حشرات بالدمام والخبر والجبيل والخبر والاحساء والقطيف كذلك شركة رش حشرات بالدمام ومكافحة الحشرات بالخبر
شركة مكافحة حشرات بالدمام
شركة تنظيف خزانات بجدة الجوهرة من افضل شركات تنظيف الخزانات بجدة حيث ان تنظيف خزانات بجدة يحتاج الى مهارة فى كيفية غسيل وتنظيف الخزانات الكبيرة والصغيرة بجدة على ايدى متخصصين فى تنظيف الخزانات بجدة
شركة تنظيف خزانات بجدة
شركة كشف تسربات المياه بالدمام
شركة نقل عفش واثاث
ReplyDeleteشركة نقل عفش بالرياض وجدة والدمام والخبر والجبيل اولقطيف والاحساء والرياض وجدة ومكة المدينة المنورة والخرج والطائف وخميس مشيط وبجدة افضل شركة نقل عفش بجدة نعرضها مجموعة الفا لنقل العفش بمكة والخرج والقصيم والطائف وتبوك وخميس مشيط ونجران وجيزان وبريدة والمدينة المنورة وينبع افضل شركات نقل الاثاث بالجبيل والطائف وخميس مشيط وبريدة وعنيزو وابها ونجران المدينة وينبع تبوك والقصيم الخرج حفر الباطن والظهران
شركة نقل عفش بجدة
شركة نقل عفش بالمدينة المنورة
شركة نقل اثاث بالرياض
شركة نقل عفش بالدمام
ReplyDeleteشركة نقل عفش بالطائف
شركة نقل عفش بمكة
شركة نقل عفش بينبع
شركة نقل عفش بالخرج
شركة نقل عفش ببريدة
ReplyDeleteشركة نقل عفش بخميس مشيط
شركة نقل عفش بالقصيم
شركة نقل عفش بتبوك
شركة نقل عفش بابها
شركة نقل عفش بنجران
شركة نقل عفش بحائل
This comment has been removed by the author.
ReplyDeleteNice post. Thanks for sharing this post.
ReplyDeletebuy vardenafil generic online
nice information,
ReplyDeletebuy generic viagra online usa
Keeping your things safe and secure is always extremely important and the best way to do that is with a lockable storage box. Read on and find out why.plastic corrugated sheet manufacturers
ReplyDeleteFajartoto is the Bandar Togel Terpercaya, Online Togel Agent, Trusted Online Togel & Situs Casino Online that has been proven to be a trusted place to place online lottery betting numbers for online lottery lovers and Judi Casino Online. Fajartoto as the Trusted Online Bandar Togel, Online Togel Agent and Trusted Togel Site that provides 9 official online lottery markets such as SGP pools, lottery pools, lottery pools, canadia pools, togel saigon pools, Siamese pools, khmer pools , lottery israel pools and toto macanese 4D pools. Please come to visit my website Syair Togel Hongkong and Prediksi Togel Hongkong
ReplyDeleteThis is very interesting content! I have thoroughly enjoyed reading your points and have come to the conclusion that you are right about many of them. You are great. Customer survey
ReplyDeleteTelephone-based interviews are often influenced by the Cassandra phenomenon. Face-to-face interviews are expensive and can be led by the interviewer.surveys that pay well
ReplyDeletegreat post.
ReplyDeleteLatest Sad Shayari
Suggestion box and survey will helps to know several problem reasons and also it supports to reach into great success.
ReplyDelete
ReplyDeleteescort service job description in kolkata
male escort service in kolkata
call girl numbers kolkata
call girl job vacancies in kolkata
call girl service in kolkata
independent call girls in kolkata
call girl hayyat regency kolkata
kolkata escort real
Escorts services kolkata
Advertise your job for fast, easy and effective recruitment. Reach thousands of candidates actively looking for a new job today
ReplyDeleteif you want more just look here "ceogem"
CALLL GIRL kolkata
ReplyDeleteFEMALE CALL GIRLS KOLKATA
FEMALE CALL GIRLS
SEXY CALL GIRLS SERVICE IN KOLKATA
ESCORTS SERVICE IN KOLKATA
KOLKATA FEMALE ESCORT SERVICE
TOP ESCORTS SERVICE KOLKATA
BEST KOLKATA ESCORTS SERVICE
CALL GIRLS SERVICE
ESCORTS SERVICE
ESCORT SERVICE
ESCORTS SERVICE IN KOLKATA
BENGOLI ESCORTS GIRLS
BENGOLI CALL GIRLS KOLKATA
KOLKATA ESCORTS SERVICE
FREE ESCORTS SERVICE KOLKATA
FREE WHATSAAP NUMBER ESCORTS
FREE WHATSAAP NUMBER OF CALL GIRLS
PHONE NUMBER OF ESCORTS SERVICE
FREE PHONE NUMBER ESCORTS SERVICE
MOBILE NUMBER OF ESCORTS SERVICE
ReplyDeleteOur Russian Escorts in Aerocityhave smooth bodies that can prepare you for a night of sexual loving. You don’t want to feel self-conscious as you’re safe here at our Call Girls agency. Our Services are all-time service that is accessible for you 24*7 hours. Our Escorts Girls are always prepared to make you fill you’ll have a fantastic alluring night with our lovely girls and you only need for believing simultaneously. Check our other Services also...
Russian Escorts in Aerocity
Russian Escorts in Aerocity
Russian Escorts in Aerocity
Russian Escorts in Aerocity
Russian Escorts in Aerocity
Russian Escorts in Aerocity
I am often to blogging and i really appreciate your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information.
ReplyDeleteRabbitroom.com
Information
Click Here
Playing lottery games are very common and people are playing this game for several centuries. The real fact is this lottery site is exactly reputable site which gives importance to safety and security of its players constantly. mơ thấy đỉa cắn đánh con gì ăn lớn
ReplyDeleteAn impressive share, I just given this onto a colleague who was doing a little analysis on this. And he in fact bought me breakfast because I found it for him.. smile. So let me reword that: Thnx for the treat! But yeah Thnkx for spending the time to discuss this, I feel strongly about it and love reading more on this topic. If possible, as you become expertise, would you mind updating your blog with more details? It is highly helpful for me. Big thumb up for this blog post!
ReplyDeleteCareer.habr.com
Information
Click Here
Would you be interested in exchanging links?
ReplyDeleteClick Here
Visit Web
Tetongravity.com
I really wonder why there is no such like this post elsewhere. This article is so awesome no. 1 indeed.
ReplyDeletehttps://www.aarinkaur.com/
https://www.aarinkaur.com/Contact-Mumbai-Escorts-Aarin.html
https://www.aarinkaur.com/Escort services in Mumbai.html
https://www.aarinkaur.com/Independent escorts in Mumbai.html
https://www.aarinkaur.com/Call Girls in Mumbai.html
There are some interesting points in time in this article but I don’t know if I see all of them center to heart. There is some validity but I will take hold opinion until I look into it further. Good article, thanks and we want more! Added to FeedBurner as well
ReplyDeletePlancontingenciaunitec.com
Information
Click Here
Visit Web
Survey is the right way to explore customer or any other identity's requirement or need. Great article.
ReplyDeleteget free robux
Survey always helps both parties to get their advantages.
ReplyDeleteglance.intuit.com
There are certainly a lot of details like that to take into consideration. That is a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith. I don?t know if best practices have emerged around things like that, but I am sure that your job is clearly identified as a fair game. Both boys and girls feel the impact of just a moment’s pleasure, for the rest of their lives.
ReplyDeleteOurblogginglife.com
Information
Click Here
Visit Web
ReplyDeleteI really enjoy simply reading most of your weblogs.
Simply wanted to inform you that you've people like me who appreciate your work.
Definitely a fantastic publish. The information you have provided is actually very worthwhile
Thank you for very usefull information
Here's my website : -- 안마
(jk)
Your place is valueble for me. Thanks!…
ReplyDeleteCreator.wonderhowto.com
Information
Click Here
Visit Web
This really answered my problem, thank you!
ReplyDeleteMuabs.com
Information
Click Here
Visit Web
Aw, this was a really nice post. In idea I would like to put in writing like this additionally – taking time and actual effort to make a very good article… but what can I say… I procrastinate alot and by no means seem to get something done.
ReplyDeleteClick Here
Visit Web
Chillispot.org
Information