How Beliefs Change What We See in Starlight

[unlike most of my other posts, this one is written to be accessible for children who want to know about the pitfalls of the scientific method.]

If you would like to hear this read aloud, try this video!

When you believe in a theory that predicts the existence of gigantic, gravitational black holes that eat light, will you see evidence of these black holes all over the night sky? Will you construct pretty, artistic representations of the things in which you believe.

When you believe in a theory that predicts black hole collisions that release waves which will be detected on Earth as tiny changes in very sensitive measurement devices, will you see evidence of these collisions everywhere you look? Will you construct simulations of what you think these collisions would look like?

When the data from a single measurement device is so noisy that it shows you nothing, is it possible to combine data from multiple measurement devices to see something, or is there the risk that by picking and choosing which measurements to combine and look at, you might see something that isn’t really there? For example, if you took photographs every midnight of a dark, creepy hallway and none of them showed anything unusual, but when you superimposed all of the images together, all of the dust particles accumulated to form the shape of a ghost, can you believe that measurement?

There is always the risk that you might accidentally measure something other than what you intended, for example, instead of measuring a black hole shadow in the starry night sky, you might end up measuring light leaking around your earthly measurement device.

The purpose of the scientific method is to avoid fooling ourselves about what causes what we see. How can we avoid fooling ourselves about what we see in the stars? There are, after all, many ideas that are consistent with our theories of nature. Dragons and unicorns are perfectly consistent with the theory of evolution, yet they do not exist.

In a laboratory experiment, you have a closed system in which changing one variable should produce a predictable effect in another, but in astronomy, we can’t do these sorts of controlled experiments. At best, we can describe what we see. A cosmologist might see the results of a big bang explosion, a crystallographer might see a diffraction pattern, and a materials scientist might see a pattern of localized light in an inhomogeneous medium. When it comes to astronomy, there can never be one, definitively true description because we can never create a controlled experiment with the stars.

Some people build experiments to detect small changes in gravity and they attribute these changes to invisible things that are happening in the stars, but they can’t be sure that the stars are truly the cause of the changes. The changes could be caused by something much closer to Earth. They also can’t be sure that they haven’t made a mistake in how they have interpreted and filtered their data.

To avoid accidentally seeing something in noisy data that isn’t there, scientists will take multiple, independent measurements of something happening. If ten people all independently observe a dragon fighting a unicorn or a gravitational wave from colliding black holes in a distant galaxy, they were probably not all hallucinating. However, if ten people are all looking at a blurry, filtered, and enhanced image and they all agree that it is probably a picture of a dragon fighting a unicorn, they might be fooling themselves and looking at something much more mundane.

In the case of gravitational waves, back in the 1970s, hundreds of independent research groups constructed simple devices to measure them and they all compared their results. Each research group thought that it had measured gravitational waves, but when the results were combined, they all had to conclude that no one had been measuring gravitational waves. They had all been measuring different sources of noise.

Today, we have a new sort of gravitational wave detector that is very expensive and there are only three of them in existence. They all believe that they are measuring gravitational waves, but it is possible that they are all measuring different sources of noise because it is difficult to get enough results to compare with only three, independent measurement devices.

When evaluating a scientific result, it is important to remember that raw measurements should always be believed, but the interpretation we give to those measurements should always be treated skeptically because you might be measuring something you hadn’t intended to measure. It is also important to be wary of those who construct a result by combining biased, filtered, or calibrated measurements.

In conclusion, it is a good idea to be wary of scientists’ conclusions but to trust in the raw data. The conclusions might be biased by beliefs, even if the result was ‘peer-reviewed’. Groups of scientists can be just as unaware of their blind-spots and biases as individual scientists.

The scientific method itself is the source of the power of science. This power is not contained within the peer-review system or within the community, especially when that community is motivated to see things that are not there.