For accuracy I would prefer the title had been:
> Trusting science [without understanding the scientific method] leaves people vulnerable to believing pseudoscience
But it was already a wordy title.
> Trusting science [without understanding the scientific method] leaves people vulnerable to believing pseudoscience
But it was already a wordy title.
Understanding the design of a particular experiment and the theories and values it attempts to put to the test is quite another matter. Scientists and laymen in completely different fields all understand “the scientific method”, but they often cannot even begin to read or understand each other's work or experiments.
Luckily, the convenient thing is that the experiments that are used to support very policalized claims are often quite easy to understand to the lay man with some rudimentary knowledge, and it's the experiments in things such as particle physics or quantum chemistry that have almost no political influence that are incomprehensible without at least a 8 years of formal education in the specific field.
For a layperson, simply talking about experiments and repeatability doesn't really convey the reality of how science works.
The way I describe the scientific method to regular people is its purpose: It is simply the growing and improving collection of methods we have found for separating fact, from fiction, illusion, bias, wishful thinking, tradition, coincidence, and every other form of misconception or non-predictive source.
It is the only system of knowledge, that is based on relentlessly weeding out any false or unproven conjectures we have about reality.
This conforms to actual practice: actual scientific activity includes falsifiable predictions, experiments, reviews and repeatability, but also math (a biggie obviously), statistics, revealed conflicts of interest, regular meta-studies of active areas, tests for data bias, simulations for exploring implications, discussion groups vetting and encouraging each other's ideas, cataloging of fruitful concepts for guiding inquiry such as symmetry, ... and on and on and on.
And as new tests, methods or ideas are found useful, the scientific method accumulates them.
What all these activities have in common is they decrease the chance that something incorrect will be accepted.
And given all that testing and vetting and care, science holds itself to a higher standard than any other search for knowledge except pure math.
And given that science is an inherently decentralized activity, carried out with experts in numerous overlapping fields, organizations, and areas of research, and the fact that nature itself contains no contradictions, it is very difficult for errors to stick around unnoticed in any area attracting much attention.
Another practical part of science many people don't get, is most scientists are actively working at the intersection of the known and unknown, so are very aware of sciences limitations, and also the destructive danger to their own progress and credibility if they unwittingly or strategically choose to accept false premises, for political, economic or other reasons.
Not that bad science is never done, but it is very risky for the scientists own credibility and ability to make any meaningful discovery. Coalitions to intentionally push false narratives would know they were severely handicapping themselves from further discovery, and sabotaging their own relevance to downstream scientific discovery and technological applications.
Those of us familiar with science tend to take all this context for how and why science works for granted.
> I don't think the scientific method is actually that easy for most people to grasp.
Do you think most people have the actual opportunity to learn and grasp it though?
My worries/experience: https://news.ycombinator.com/item?id=28661179
I. How science works so well. I.e. the body of methods for vetting ideas, scientists brutal incentives to be honest and competent, natures inherent consistency as the final unassailable arbiter of truth, etc.
II. Then how to judge the products of science:
None of us can understand a tiny fraction of science directly, we must all learn to judge what science we can trust.
From General Relativity to the game theory behind self-organizing ants, I think people need to recognize that although with effort they could understand individual effects, everyone has to trust other experts for most of their scientific knowledge.
The best indicators of solid science, after personal understanding, are ideas that are widely accepted by many experts in the field, especially experts working across multiple overlapping fields.
(Climate science is a fantastic example of the latter: Experts including chemists, atmospheric physicists, geologists, biologists, ecologists, solar physicists, .... almost every kind of science.)
And another strong indicator of both stable and promising science is that it provides a workable foundation for further scientific and even technological advances.
And finally, when non-scientist experts, whose career success objectively depends on recognizing good science (i.e. the military, insurance companies, ...) treat particular areas of science as credible, we can assume it is safe for us to as well.
This is something I personally want to figure out. We are working on math/scientific educational software (as a spin off of a larger mission) and addressing basic issues of illiteracy is a big deal.
I will continue to think very hard about your question!
--
A random idea: a simulation game of doing science, including brainstorming, making conjectures, testing, getting reviewed, etc. while doing the same for others would be great.
The blockchain aspect happening - i.e. building on others ideas and having others build on one's own. The need to vet some ideas, while learning the efficiency need to trust the output of reliable groups whose results have already been widely accepted.
Getting to see both how the system spreads good ideas and removes bad ideas, the incentives for individual self-honesty, etc would be great.
Not an easy "game" to design, but something very active like that, maybe even between schools would be really neat.
And there would be a whole game design issue as to topic choice and progression for this to work.
That's a pie in the sky!!!! - and not a solution for the public at large. I will re-watch your video and be thinking more.
--
For now, we are only working with one school as our lab for educational redesign in the areas of music and math, with science and crafting areas to follow later.
An improvement common to all topics is an RPG-like skill graph, broken down into the dependencies of each micro-skill. This is both to give students greater awareness of their progress relative to material, and also let them manage their own learning.
Students are unable to move past a task until it is very well understood, and teachers can see when a student is moving slowly so needs individual help. That avoids the problem of slower students being undermined by pressure to skip ahead too soon, and their hurdles being overlooked.
Both required and many optional skills are included to give students a chance to just stick to what is required, or follow their curiosity if they are able to move more quickly. This results in greater education, but also solves the problem of keeping faster students bored by providing neat topics and a clear sense of special accomplishment.
I love that! I would have benefitted from that a lot in school.
> The need to vet some ideas, while learning the efficiency need to trust the output of reliable groups whose results have already been widely accepted.
Your project sounds great! I'm curious if you like Dr. Alex Freeman's approach to a new type of scientific publishing system, with her http://Science-Octopus.org (aims to make science more accountable and open access):
"Q: What problem are you aiming to tackle?
A: Fundamentally, I believe that the whole concept of the scientific paper drives people to tell stories rather than worry about the quality of the actual research. It's true that during the scientific process, you usually start with a problem and you finish with a real-world use of your solutions. But the whole chain, from one end to the other, takes a whole range of different skills and different people and different amounts of time. Forcing researchers to guard their work and get to the end of the chain before they can publish anything makes people work in silos, when we'd all be much better off if everybody shared their work as they went along and everybody collaborated to get to the end of the process.
There is a real problem in science—with widespread questionable research practices, lack of replication, publication bias, inequalities in access, slow progress, and wasted resources. And I think that the root of it all is the publication system, and the pressure to publish as the only way that scientists are being judged. The new publishing platform I envisage would stop forcing researchers to try to tell a story, allowing them to concentrate on doing their work to the best of their ability and on constructively collaborating.
Q: How would this new publication system work?
A: The overall idea behind the new platform, which I called Octopus, is to break the standard unit of publication up into eight smaller stages or pieces. These include formulated scientific problems, hypotheses, methods and protocols, data, analysis, interpretation, and real-world applications. The eighth piece is reviews where people share their comments about the other stages, but it's also treated as a publication itself because it is a constructive piece of collaboration. Researchers will be able to revisit their publications to acknowledge others' comments or suggestions and even include some of the useful reviewers as authors, with the old version remaining available so that readers can track the evolution of ideas.
The formulated scientific problem automatically becomes the beginning of a chain, and all the other pieces are vertically linked upward. If you have a great hypothesis, you can publish that and link it straight up to the problem that it is trying to address. And if you've collected some small amount of data, you can publish that, linking it directly up to the protocol that you followed. It will also be possible to create horizontal links between publications; maybe somebody has come up with an amazing algorithm whilst studying starling flocking behavior and somebody else working on oil pipelines has decided to use it.
Octopus will thus allow researchers to publish in much smaller author groups, and that can make the system much more accountable and meritocratic. If, say, you're an applied statistician, at the moment you're very rarely a first author on anything. But with Octopus, statisticians can do analyses of anybody else's data and get credit for it as the author of that publication. Octopus will also be made language-agnostic through automatic translation to lower language barriers and broaden access.
Another way to make sure that we are judging people on the actual quality of their work and not on the stories is through rating. Anybody who's logged in to the system will be able to rate entries as soon as they are published based on pre-set criteria reflecting what we as a scientific community value at each stage of the scientific process. So, for instance, there might be a rating for the size of the dataset, and another rating for how well it is annotated and how usable it is. Just like on eBay or Amazon, the most highly rated pieces of work will automatically become more easily visible.
Octopus will also allow the evaluation of individual authors through a professional page that shows their affiliations, their conflicts of interest, and all their publications and their ratings. Seeing these metrics, you will be able to get a feel for what type of researcher this is. Are they publishing lots of well-rated hypotheses? That's an idea-generating person. Are they writing a lot of constructive reviews? That's a very collaborative person."
Source: Meet Octopus, a new vision for scientific publishing: https://www.science.org/careers/2018/11/meet-octopus-new-vis...
Talk: Dr. Alexandra Freeman | Octopus - a radical new approach to scientific publishing | 29 October 2018: https://www.youtube.com/watch?v=Af6aITLEoD8
Or have I misunderstood your aims/interests and you are not interested in open access?
Experiments? Is that actually a required part of the scientific method? Apparently not: lots of fields presented as scientific to the public can't do experiments. Most obviously, neither climatology, epidemiology nor economics do experiments, despite being fields with enormous direct impact on governments and therefore people's lives. You can try to refine it to "making testable predictions" but then it would require that the predictions are actually tested - real scientific papers often make predictions without testing them against reality.
What about computer science? Is it really a science? Most people would say the scientific method is a way to discover knowledge about the natural world, but there's nothing natural about computers. I myself am not convinced computer science is a science, but, it's treated as one by the institutions.
The prevalence of papers that simply present unvalidated statistical/algebraic models raises huge questions about what science is in theory vs what it is in practice. In theory, science is about discovering knowledge about the natural world, by refining hypotheses which make testable predictions into validated theories. In practice science is whatever granting agencies, universities and journals find interesting. Nothing anywhere in the so-called scientific system actually tests that the work is following the scientific method, nor is there even a working definition of what that would mean. The result is that scientific journals routinely publish pseudo-scientific claims without realizing that it's what they're doing (see my other comment).
In fact many fields are almost entirely a-theoretical: social psychology papers like the one this thread is about aren't rooted in an attempt to refine a grand unified theory of the mind. They're just testing academic folk intuitions, and are often driven by political or ideological concerns more than an attempt to derive theory.
In such a world, deification of The Scientific Method™ is naive. There is no such method, just a hodgepodge of tactics and social conventions that seems to work better than nothing, but which may or may not be actually used by something labelled as scientific.
> Now let us begin our course. The course is in scientific method—it is called ‘Introduction to Scientific Method’—and the first thing that I want to say as an introduction is that this is a subject that does not exist."
-- Karl Popper
https://twitter.com/DavidDeutschOxf/status/14391227643261255...
The real scientific method is: collect every means you can to test and vet ideas against nature as directly, efficiently and reliably as possible. Then do those things. As you do them, keep growing and fine tuning those methods.
The only thing the "methods" of the "scientific method" have, are their purpose - eliminating bad ideas, in order to more quickly find promising ideas, and then narrow those to the most promising at any given time.
Experiments, repeatability, etc. are just some of many activities, and don't even apply to all situations where they are not practical but other methods help.