[quote]If you’re a psychologist, the news has to make you a little nervous—particularly if you’re a psychologist who published an article in 2008 in any of these three journals: Psychological Science, the Journal of Personality and Social Psychology, or the Journal of Experimental Psychology: Learning, Memory, and Cognition.
Because, if you did, someone is going to check your work. A group of researchers have already begun what they’ve dubbed the Reproducibility Project, which aims to [B]replicate every study from those three journals for that one year.[/B] The project is part of Open Science Framework, a group interested in scientific values, and its stated mission is to “estimate the reproducibility of a sample of studies from the scientific literature.” This is a more polite way of saying “We want to see how much of what gets published turns out to be bunk.”
For decades, literally, there has been talk about whether what makes it into the pages of psychology journals—or the journals of other disciplines, for that matter—is actually, you know, true. Researchers anxious for novel, significant, career-making findings have an incentive to publish their successes while neglecting to mention their failures. It’s what the psychologist Robert Rosenthal named “the file drawer effect.” So if an experiment is run ten times but pans out only once you trumpet the exception rather than the rule. Or perhaps a researcher is unconsciously biasing a study somehow. Or maybe he or she is flat-out faking results, which is not unheard of. Diederik Stapel, we’re looking at you.
So why not check? Well, for a lot of reasons. It’s time-consuming and doesn’t do much for your career to replicate other researchers’ findings. Journal editors aren’t exactly jazzed about publishing replications. And potentially undermining someone else’s research is not a good way to make friends.
Brian Nosek knows all that and he’s doing it anyway. Nosek, a professor of psychology at the University of Virginia, is one of the coordinators of the project. He’s careful not to make it sound as if he’s attacking his own field. “The project does not aim to single out anybody,” he says. He notes that being unable to replicate a finding is not the same as discovering that the finding is false. It’s not always possible to match research methods precisely, and researchers performing replications can make mistakes, too.
But still. If it turns out that a sizable percentage (a quarter? half?) of the results published in these three top psychology journals can’t be replicated, it’s not going to reflect well on the field or on the researchers whose papers didn’t pass the test. In the long run, coming to grips with the scope of the problem is almost certainly beneficial for everyone. In the short run, it might get ugly.
Nosek told Science that a senior colleague warned him not to take this on “because psychology is under threat and this could make us look bad.” In a Google discussion group, one of the researchers involved in the project wrote that it was important to stay “on message” and portray the effort to the news media as “protecting our science, not tearing it down.”
The researchers point out, fairly, that it’s not just social psychology that has to deal with this issue. Recently, a scientist named C. Glenn Begley attempted to replicate 53 cancer studies he deemed landmark publications. He could only replicate six. Six! Last December I interviewed Christopher Chabris about his paper titled “Most Reported Genetic Associations with General Intelligence Are Probably False Positives.” Most!
A related new endeavour called Psych File Drawer allows psychologists to upload their attempts to replicate studies. So far nine studies have been uploaded and only three of them were successes.
Both Psych File Drawer and the Reproducibility Project were started in part because it’s hard to get a replication published even when a study cries out for one. For instance, Daryl J. Bem’s 2011 study that seemed to prove that extra-sensory perception is real — that subjects could, in a limited sense, predict the future — got no shortage of attention and seemed to turn everything we know about the world upside-down.
Yet when Stuart Ritchie, a doctoral student in psychology at the University of Edinburgh, and two colleagues failed to replicate his findings, they had a heck of a time getting the results into print (they finally did, just recently, after months of trying). It may not be a coincidence that the journal that published Bem’s findings, the Journal of Personality and Social Psychology, is one of the three selected for scrutiny.
Nosek acknowledges that Bem’s study and Stapel’s fraud were among the motivators for the project. “Right now we have an opportunity to do something about it rather than writing another article about what we can do about it,” he says. He hopes that the replications for all three journals will be completed by the fall and the results published online next spring.
Like most researchers, Nosek is interested in advancing his own research agenda rather than simply running someone else’s experiments. That said, he thinks it’s better for researchers to know whether they’re discovering “true stuff” or just fooling themselves, their colleagues, and the general public. “Ultimately it’s a waste of everyone’s time if I can’t replicate the effects,” he says. “Otherwise, what are we working on?[/quote]
[url]http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/29045[/url]
so science might beet phycology
This should be done in all areas, not just psychology.
[QUOTE=Native Hunter;35623029]so science might beet phycology[/QUOTE]
I dunno, beets are pretty soft.
[QUOTE=Electrocuter;35623042]This should be done in all areas, not just psychology.[/QUOTE]
It's expensive in most fields. Psychology is pretty cheap for the most part, all you have to do is kidnap a bunch of undergrads, stick them in a room and make them fill out questionnaires
so as a tl;dr for you guys, psychologists (and basically everyone in any field of science ever) are inclined to lie about their findings for more publicity, so what these guys are gonna do is replicate the tests most modern psychology is based on and see if they were made from biased or even flat-out fabricated information
[QUOTE=Native Hunter;35623029]so science might beet phycology[/QUOTE]
No, they're just double checking some studies to make sure they aren't bullshit.
[QUOTE=Native Hunter;35623029]so science might beet phycology[/QUOTE]
psychology is a science and beet isn't a verb
I'm more inclined to say Psychology isn't an exact science.
"Hey guys we're just gonna double check your work to see if it's right, okay"
"OH GOD OUR ENTIRE FIELD IS GOING TO FALL APART"
[QUOTE=Sanius;35623302]psychology is a science and beet isn't a verb[/QUOTE]
[img]http://facepunch.com/avatar/32610.png?garryis=awesome[/img]
"I'm crying. we are gonna be poor. oh fuck oh fuck"
This is a big undertaking, but one of the most important parts of psychology and science in general is replication. I learned that in a psychology survey course. If you publish something, expect it to be fact checked, [i]especially[/i] if it's in psychology.
[QUOTE=Last or First;35623478]"Hey guys we're just gonna double check your work to see if it's right, okay"
"OH GOD OUR ENTIRE FIELD IS GOING TO FALL APART"[/QUOTE]
I don't know about the whole field falling apart, but the article points out how this research doesn't exactly have a very good track record. If the majority of published studies end up getting exposed as a load of crap it could completely shake up the field.
[QUOTE=Native Hunter;35623029]so science might beet phycology[/QUOTE]
Science might [IMG_thumb]http://jackdempsey.me/beet/beet.jpg[/IMG_thumb] [URL="http://en.wikipedia.org/wiki/Phycology"]The scientific study of algae?[/URL]
[QUOTE=Noble;35623966]I don't know about the whole field falling apart, but the article points out how this research doesn't exactly have a very good track record. If the majority of published studies end up getting exposed as a load of crap it could completely shake up the field.[/QUOTE]
Good, uncovering the truth is more important than stopping a "shake up", it's science not politics.
[QUOTE=Jonii;35623672]This is a big undertaking, but one of the most important parts of psychology and science in general is replication. I learned that in a psychology survey course. If you publish something, expect it to be fact checked, [i]especially[/i] if it's in psychology.[/QUOTE]Yeah, and if you don't get fact-checked right away, watch out. They'll wait until you get complacent and then when you least expect it, BAM, fact checking for everything that year.
I hope the review the effects of psychoactive substances in the long term, far too many issues there
Doubtful there'll be any significant changes, especially not this hyperbole about the entire field collapsing. Psychology, like any scientific field, sheds bad studies over time even if there's no concerted effort to purge them. Someone'll do a study about something tangentially related, get results that conflict with part of an older study, that study is re-examined and thusly reclassified.
That having been said, it's always healthy for science to double-check things.
[QUOTE=Biotoxsin;35625357]I hope the review the effects of psychoactive substances in the long term, far too many issues there[/QUOTE]
They're reviewieng Psychology not Psychiatry
[QUOTE=Cone;35623168]so as a tl;dr for you guys, psychologists (and basically everyone in any field of science ever) are inclined to lie about their findings for more publicity, so what these guys are gonna do is replicate the tests most modern psychology is based on and see if they were made from biased or even flat-out fabricated information[/QUOTE]
thank fuck
i'm so sick of people using a one time study with a small sample size as outright fact
Psychology is a hard field because everyone thinks and acts differently in different situations. Sometimes, experiments aren't easily verifiable like physics or chemistry experiments.
Coming undone? This is just science. We do this all the time in research psychology: You find an article, look at its hypothesis and what its conditions were, replicate the test using new variables, see if you find something new. If it did, the hypothesis/theory holds up to the new variables. If not, double check. If still not, you've found one of the many needles that'll crack the hypothesis/theory. Yes, the "file drawer" effect is very real and some researchers would rather publish the finding that is different but only found once. Shouldn't happen, but it does. What these guys are doing is just science. See if it holds true or is just silliness that only works under distinct conditions.
I'll say the title had me holding my seat for a moment.
[QUOTE=King Tiger;35626593]Psychology is a hard field because everyone thinks and acts differently in different situations. Sometimes, experiments aren't easily verifiable like physics or chemistry experiments.[/QUOTE]
use a big & varied test group.
The DSM has grown so much over the past few decades and contains so many contradicting conditions as a result of the majority of the writers and editors being under the payroll of big pharma.
When you take something relative like the study of mental processes/patterns from outside the affected brain as described by the alleged sufferer and observed to a small degree and call it a science, it gets taken advantage of because it can just be bent to fit lots of different holes because it is vague and pseudo-scientific.
The ambiguity of psychology is used to justify all sorts of things from alleged illicit drug induced conditions to locking someone up just because their moral hierarchy is radically different from the majority to the point that they become a danger or an obstacle.
It would be nice if they investigated the brain chemistry side some more and found a provable basis for some conditions which would hopefully reduce the ambiguity therefore reducing psychology's' vulnerability to profiteering and being used as a moralising vehicle.
[QUOTE=JustExtreme;35627370]The DSM has grown so much over the past few decades and contains so many contradicting conditions as a result of the majority of the writers and editors being under the payroll of big pharma.
When you take something relative like the study of mental processes/patterns from outside the affected brain as described by the alleged sufferer and observed to a small degree and call it a science, it gets taken advantage of because it can just be bent to fit lots of different holes because it is vague and pseudo-scientific.
The ambiguity of psychology is used to justify all sorts of things from alleged illicit drug induced conditions to locking someone up just because their moral hierarchy is radically different from the majority to the point that they become a danger or an obstacle.
It would be nice if they investigated the brain chemistry side some more and found a provable basis for some conditions which would hopefully reduce the ambiguity therefore reducing psychology's' vulnerability to profiteering and being used as a moralising vehicle.[/QUOTE]
I think you're more so leaning towards Psychiatry, especially in the last part of your statement. I agree with you though. It's quite silly how they come up with medications for certain conditions. "Well we took this anxiolytic and gave it to someone for their anxiety, oh well it also made that person fall asleep. The anxiolytic is now a sleep medication, huzzah!" or "Hey this high blood pressure medication we gave to these people really mellowed them out, the medication is now an anti-manic.".
phew all the reverse psychologists are safe
[QUOTE=InvaderNouga;35628003]I think you're more so leaning towards Psychiatry, especially in the last part of your statement. I agree with you though. It's quite silly how they come up with medications for certain conditions. "Well we took this anxiolytic and gave it to someone for their anxiety, oh well it also made that person fall asleep. The anxiolytic is now a sleep medication, huzzah!" or "Hey this high blood pressure medication we gave to these people really mellowed them out, the medication is now an anti-manic.".[/QUOTE]
Indeed and the overuse of SSRI's
Sure some people who have been depressed have been shown to have lower serotonin levels (I think? I don't remember) but for the majority it seems not to be that simple. The brain is a complex organ with many different chemicals with ratios that vary from person to person from day to day.
On another note I think a lot of the depression in the world today actually comes from people being dissatisfied with how things are (the ruling plutarchy [plutocracy + oligarchy], not really having a say in things that affect day to day life, etc. for example) and wanting them to change but feeling too insignificant to try to have an influence on their reality and perception thereof. I don't think people should be medicated just because they are upset about how the world works, that is a natural reaction.
[QUOTE=JustExtreme;35628169]Indeed and the overuse of SSRI's
Sure some people who have been depressed have been shown to have lower serotonin levels (I think? I don't remember) but for the majority it seems not to be that simple. The brain is a complex organ with many different chemicals with ratios that vary from person to person from day to day.
On another note I think a lot of the depression in the world today actually comes from people being dissatisfied with how things are (the ruling plutarchy [plutocracy + oligarchy], not really having a say in things that affect day to day life, etc. for example) and wanting them to change but feeling too insignificant to try to have an influence on their reality and perception thereof. I don't think people should be medicated just because they are upset about how the world works, that is a natural reaction.[/QUOTE]
Besides the overuse of SSRI'S, the lack of understanding on the mechanism of action regarding these drugs and other drugs used to correct and treat disorders concerns me. Lithium is a great example.
Somewhere out there, Stephen Glass is having a bad flashback.
Truth seekers these days.
Sorry, you need to Log In to post a reply to this thread.
