Moreover, while I'm not impressed with the evidence of this study, finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations.
Moreover, while I'm not impressed with the evidence of this study, finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations.
>Bad evidence may be perfectly reproducible.
That was part of my point, namely that a weakly supported interpretation of subjective reports is probably a lot easier to reproduce than more empiric studies, but it does nothing to further the art.
>finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations.
I'm actually not arguing that subjective reports aren't legitimate data though. I think this study's methodology was also flawed due to the way it was structured. Group A was asked to describe an act of self-promotion of their choosing, and in a separate survey, Group B was asked to describe an instance of their choosing when someone had self-promoted to them. It's disjointed and lacking context.
There is a distinct possibility that people in Group A chose to report a time when they felt that their bragging was socially acceptable and perhaps the people they bragged to were actually happy for them (e.g. health of their child). Likewise, I think the people in Group B were more likely to report a particularly obnoxious brag simply because that would be more memorable.
A better (albeit admittedly more difficult to execute) study would involve investigating real-world and online instances of humblebragging, where there is direct relationship between promoter and recipient, and the subjects don't get to decide which experience they describe.
In case not: reproducibility is a major concern with psychological studies in general. Drawing broad/subjective conclusions will make the results easier reproduce, but it also leaves more room for confirmation bias, so it's not a valid solution to the problem.
"Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.
Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group".
Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.
As I understand it, reproducibility means "you can run this same experiment and expect to get the same effect." Reproducibility is an issue because of study designs that are unusually to get accidental results. So you have your p value < .05, but the worry is that it's just an accident. If it's not an accident, then you should be able to run the exact same study again and get a similar effect. The problem of reproducibility is that even _that_ may not possible for some notable studies. Because of bad experimental design, we were just reporting meaningless results (the 5% chance that you get a nice p-value for an effect that isn't real). So in the case of this experiment, reproducibility is "if you survey people about bragging, will coders report their comments in the same way?" not "is bragging perceived negatively?"
I think what you're describing is sometimes called a "conceptual reproduction" where you run a different study that you can argue supports the same conclusion. For instance, you've run an experiment where people seem to show a cognitive bias. Then someone else runs a different experiment, with a different design, and says "my results show the same cognitive bias that dbbolton showed in his study". In this case, the experimental design is different, and the second study doesn't directly demonstrate that you results could be reproduced. But in theory, it does suggest that there's a real phenomenon, because the same explanation seems natural for both experiments.
There has been an overuse of conceptual reproductions in psychology, but what really has people worried is that major studies may literally not be reproducible.
P.S. all of this is based on my understanding of how the terms are used, and I'm not 100% confident I'm right about it. I think I am, but I'm not a psychologist, take my opinion with a grain of salt.