The science of consciousness has not lived up to expectations.

Over the summer, the neuroscientist Christof Koch conceded defeat on his 25-year bet with the philosopher David Chalmers, a lost wager that the science of consciousness would be all wrapped up by now. In September, over 100 consciousness researchers signed a public letter condemning one of the most popular theories of consciousness—the integrated information theory—as pseudoscience. This in turn prompted strong responses from other researchers in the field. Despite decades of research, there’s little sign of consensus on consciousness, with several rival theories still in contention.

Your consciousness is what it’s like to be you. It’s your experiences of color and sound and smell; your feelings of pain, joy, excitement or tiredness. It’s what makes you a thinking, sentient being rather than an unfeeling mechanism.

In my new book, entitled Why? The Purpose of the Universe, I take head-on the question of why it’s so hard to make progress on consciousness. The core difficulty is that consciousness defies observation. You can’t look inside someone’s brain and see their feelings and experiences. Science does deal with things that can’t be observed, such as fundamental particles, quantum wave functions, maybe even other universes. But consciousness poses an important difference: In all of these other cases, we theorize about things we can’t observe in order to explain what we can observe. Uniquely with consciousness, the thing we are trying to explain cannot be publicly observed.

How then can we investigate consciousness? Although consciousness can’t be directly observed, if you’re dealing with another human being, you can ask them what they’re feeling, or look for external indications of consciousness. And if you scan their brain at the same time, you can try to match up the brain activity, which you can observe, with the invisible consciousness, which you can’t. The trouble is there are inevitably multiple ways of interpreting such data. This leads to wildly different theories as to where consciousness resides in the brain. Believe it or not, the debates we are currently having in the science of consciousness closely resemble debates that were raging in the 19th century.

There may be a way forward. I argue that we can account for the evolution of consciousness only if we reject reductionism about consciousness. Most consciousness researchers employ a reductionist view of the universe, where physics is running the show. Thus insofar as there are some future possibilities left open by the arrangements of particles in our brains, they are settled by nothing more than the random chanciness implicit in quantum mechanics.

Some challenges have lately emerged to this reductionist paradigm. The neuroscientist Kevin Mitchell has argued that the free will of conscious organisms plays a role in determining what will happen in the brain, over and above what is settled by the laws of physics. And the assembly theory of chemist Lee Cronin and physicist Sara Walker decisively rejects reduction to microscopic-level equations, arguing for a kind of memory inherent in nature that guides the construction of complex molecules.

Evolution offers one of the strongest challenges to reductionist approaches to consciousness. Natural selection only cares about behavior, as it’s only behavior that matters for survival. Rapid progress in AI and robotics has made it clear, however, that extremely complex behaviour can exist in a system that entirely lacks conscious experience. Natural selection could have constructed survival mechanisms: complex biological robots able to track features of their environment and initiate survival-conducive behavioral responses, without having any kind of inner life. For any adaptive behaviour associated with consciousness, there could be a nonconscious mechanism that instigates the same behaviour. Given all this, it is a deep mystery why consciousness evolved at all.

Or rather, the evolution of consciousness is a deep mystery under the reductionist paradigm, according to which the behavior is determined at the micro level, making it irrelevant whether or not consciousness pops up at higher levels. But suppose instead that the emergence of biological consciousness brings into existence radically new forms of behavior, over and above what physics alone could produce. Perhaps organisms that have conscious awareness of the world around them, and thereby freely respond based on that awareness, behave very differently than mere mechanisms. Consequently, they survive much better. With these assumptions in place, we can make sense of natural selection’s preference for conscious organisms.

If consciousness does defy reduction, this could revolutionize the science of consciousness. What it would essentially provide is a new empirical marker of consciousness. If the neural processes that correspond to consciousness have a novel causal profile, one that could not be predicted—even in principle—from underlying chemistry and physics, then this would amount to a giant “HERE IT IS!” in the brain.

Would we not have noticed already if there were processes in the brain that didn’t reduce to underlying chemistry and physics? The truth is we know very little about how the brain works. We know a lot about the basic chemistry: how neurons fire, how chemical signals are transmitted. And we know a fair bit about the large functions of various brain regions. But we know almost nothing about how these large-scale functions are realized at the cellular level. To an extent, abstract theorizing has stood in for detailed neurophysiological investigation of what is actually going on in the brain.

As a philosopher, I’m not opposed to abstract theorizing. However, it’s crucial to distinguish the scientific questions of consciousness from the philosophical questions. The scientific task is to work out which kinds of brain activity correspond to consciousness, and it’s this task that detailed neurophysiological investigations—equipped to catch the HERE IT IS marker of consciousness—will help us make progress on. But what we ultimately want from a theory of consciousness is an explanation of why brain activity—of whatever form—is correlated with consciousness in the first place. Because consciousness is not an observable phenomenon, the “why” question is not one we can make progress on with experiments. In Why? I develop a radical form of panpsychism—the view that consciousness goes right down to the fundamental building blocks of reality—aimed at addressing the philosophical challenges of consciousness, as well as providing a framework for scientists to make progress on the scientific issues.

We’re still not at first base in dealing with consciousness. It requires working on many fronts, exploiting many different areas of expertise. We need to let the philosophers do the philosophy and the scientists study the brain. Each provides a different piece of the puzzle. It is a pincer movement of science and philosophy that will, ultimately, crack the mystery of consciousness.

This is an opinion and analysis article, and the views expressed by the author or authors are not necessarily those of Scientific American.