Modern physics deals with some truly mind-boggling extremes of scale. Cosmology reveals the Earth as a tiny dot amid an observable universe that is a staggering 93bn light years across. Meanwhile, today’s particle colliders are exploring a microcosmic world billions of times smaller than the smallest atom.

These two extremes, the biggest and smallest distances probed by science, are separated by 47 orders of magnitude. That’s one with 47 zeros after it, a number so ludicrously huge that it isn’t worth trying to get your head around. And yet, despite exploring such radically different distances and phenomena, cosmology and particle physics are deeply connected. Observing the motions of stars and galaxies can reveal the influence of as-yet-undiscovered particles, while studying fundamental particles in the lab can tell us about the birth and evolution of the cosmos.

Intriguingly, both disciplines are grappling with unexplained results that could be pointing to the existence of a new force of nature. If such a new force were to be confirmed, the implications for our understanding of the universe, its history and makeup would be profound.

There are four forces that we already know about. Gravity governs the grandest scales, marshalling the planets in their orbits and shaping the evolution of the universe as a whole. Electromagnetic force gives rise to a vast range of phenomena, from the magnetic field of the Earth to radio waves, visible light and X-rays, while also holding atoms, molecules and, by extension, the physical world together. Deep within the atomic nucleus, two further forces emerge: the vice-like “strong force”, which binds atomic nuclei, and the “weak force”, which among other things causes radioactive decay and enables the nuclear reactions that power the sun and the stars.

Studying these forces has transformed our understanding of nature and generated revolutionary new technologies. Work on electromagnetism in the 19th century gave us the electric dynamo and radio broadcasts, the discovery of the strong and weak forces in the 1930s led to nuclear energy and atomic bombs, while understanding gravity has made it possible to put astronauts on the moon and to develop GPS satellites that can tell us our location anywhere on Earth to within a few metres. Uncovering a fifth force would be one hell of a prize.

Hints that physicists may be on the brink of making such a breakthrough have been accumulating over the past decade. The first tranche of evidence comes from particle physics experiments here on Earth, the results of which appear to conflict with our current best theory of fundamental particles, the standard model.

Notwithstanding its rather uninspiring name, the standard model is one of humankind’s greatest intellectual achievements, the closest we have come to a theory of everything, and has passed almost every experimental test thrown at it with flying colours. So far at least.

However, the BaBar experiment in California, the Belle experiment in Japan and the LHCb experiment at Cern have all spied exotic fundamental particles known as “beauty quarks” behaving in ways that go against the predictions of the standard model. Meanwhile, just outside Chicago, Fermilab’s Muon g–2 experiment has been busily studying another type of fundamental particle called a muon, finding that it emits a slightly stronger magnetic field than expected.

The most exciting explanations for these anomalies involve hitherto unknown forces of nature that subtly alter the way beauty quarks transform into other particles or mess with the muon’s magnetism. Such a new force could help unlock a deeper structure at the base of reality, explaining why we have the fundamental particles in nature that we do. Another tantalising possibility is that it could act as a link to the unseen “dark universe”, made from invisible dark matter.

That said, for now the overall picture remains frustratingly murky. Just over a year ago, new results from LHCb poured cold water on the prospects of a big breakthrough, after missed biases were found in some of the earlier measurements. Meanwhile, theorists have been debating just how magnetic the muon really ought to be, leaving open the possibility that this anomaly is down to a calculational problem.

Perhaps the most compelling evidence for a new force at work in the universe comes from the other end of the cosmic scale. For the past few years, cosmology has been riven by what has become known as the “Hubble crisis” – a dramatic disagreement over how fast the universe is expanding.

According to the accepted cosmological story, the universe as we know it began with the big bang around 13.8bn years ago and has been expanding ever since, with galaxies carried ever farther apart as the space between them stretches. Cosmologists have two ways of figuring out how fast space is stretching. One involves studying a host of far-off galaxies through telescopes, then determining the relationship between their distance and how fast they appear to be rushing away from us. The other exploits exquisitely precise maps of faded light from the fireball of the big bang – known as the “cosmic microwave background” – to infer the properties of the infant universe. Then you apply current cosmological theory to run the clock forward and predict how fast the universe ought to be expanding today.

The fact that these two methods give different answers is the strongest evidence we have that there is more to the universe than we’ve imagined so far. Possibilities abound. A popular proposal involves a form of energy that drove the universe to expand even more rapidly than thought soon after the big bang. Others involve “dark forces” acting in the hidden world of dark matter. Some have even proposed that gravity itself behaves differently over the vast spaces between galaxies.

How the story of these anomalies will end is unclear. But the wealth of emerging evidence does suggest that physics may be on the brink of something big. The discovery of a new force would mark the start of a new age of exploration, perhaps offering a deeper understanding of the basic building blocks of nature, or opening the door to a vast, unknown dark realm, which, despite being invisible, contains 95% of everything that exists. Such breakthroughs are always hard won, but following nature’s breadcrumb trail may soon lead to a profound new view of the universe.

• Harry Cliff is the author of Space Oddities: The Mysterious Anomalies Challenging Our Understanding of the Universe (Picador).

Further Reading:

White Holes: Inside the Horizon by Carlo Rovelli (Allen Lane, £14.99)

The End of Everything: (Astrophysically Speaking) by Katie Mack (Penguin, £10.99)

The Elephant in the Universe: Our Hundred-Year Search for Dark Matter by Govert Schilling (Belnap, £24.95)