Alan Lightman has made a unique career finding imaginative ways to bridge the “two cultures” of science and humanities. A novelist and physicist, he was the first person to be awarded a joint professorship in literature and astrophysics at Massachusetts Institute of Technology (MIT). His work in the latter discipline has resulted in notable contributions to the theory of astrophysical processes under extreme temperatures; he has helped to map the behaviour of such out-of-this-world concepts as “accretion discs” and “relativistic plasmas”.

He made his name as a writer of fiction, meanwhile, with Einstein’s Dreams in 1993 – a lyrical series of short sketches, each one an attempt at a different understanding of time, and all rooted in the freewheeling mind of Einstein as he grappled with relativity in Berne, Switzerland in 1905. That book drew comparisons with the playful philosophical fiction of Italo Calvino and Jorge Luis Borges, and became an international bestseller. Since then, Lightman has published 17 books, each of which dwell in different ways on life, the universe and everything.

In a TED talk, Lightman drew the distinction between his two habits of mind in this way: “The scientist tries to name things; the artist tries to avoid naming things.” As he approaches his three score and 10, the gap between those two positions apparently becomes ever more urgent to him. This latest curious book of essays is another stab at resolving that universal either/or.

The starting point for its reflections is a kind of epiphany Lightman experienced 12 months or so ago. For many years, he has spent his summers and, occasionally, parts of his winters, on a tiny island off the coast of Maine. The island is a mile long and three quarters of a mile wide. Six families have houses on it, each with their own jetty. By necessity all the families have become at home in boats. One clear night, Lightman was chugging out to the island alone when he decided to turn off his engine and his lights and just drift. He lay on his back and gazed at the “sky vibrating with stars”, and not for the first time in his life experienced a kind of intense weightlessness, a powerful sense of eternity, a loss of self; he suggests when he returned from that Wordsworthian mental journey he had no idea how long he had been travelling.

This experience inevitably got him thinking, somewhat in the manner of the star-gazing ancients or visionary Romantics (he nicknames his singular offshore vantage point “Lute Island” and watches and waits for inspiration). The book, a series of fragments of philosophy of mind, and insights into the creative lives of the great scientists, and attention to the materiality of the universe in both its largest and smallest components, is the result of those wandering meditations. Thoreau on Walden Pond is one acknowledged antecedent; the book also puts you in mind from time to time of Calvino’s last (and to my mind, greatest) work, the collected observations of Mr Palomar.

Lightman does not possess Calvino’s structural rigour, his brilliant hold on irony as the defining principle of the human condition, but his discursive method is full of insight into some of the mysteries of the physical world, as well as the physics of mystery. He uses his own biography – the little science lab he created in his bedroom closet in Memphis, Tennessee, aged 12 – to demonstrate an intact sense of wonder at what we know and what we don’t. At the heart of his mediation is this neat formulation of the boundaries of scientific understanding: “The infinite is not merely a lot more of the finite.”

Lightman has a sympathetic gift for recreating the leaps of faith in scientific advance. Of Galileo grinding his lenses and bringing far-off galaxies a thousand times closer. Of the way, in the 20th century one set of “absolutes” was quickly supplanted by the next – of how JJ Thomson’s discovery of electrons (his “plum pudding” notion of the atom) gave way to Ernest Rutherford’s idea of the “peach” structure of matter (a whole lot of space with a hard nut of protons at the centre). And how that certainty was divided again by the quark.

At the same time, he feels himself in a wonderland of shifting scales. He maps out the heavens, concentrates on the veins of a leaf, tries to fathom the evolution of the humming bird, the veracity of Van Gogh’s Starry Night, and returns often to the thrilling chutzpah of Einstein rethinking time and space. Does he end up much the wiser after this latest record of attention to his pattern-making mind? Of course not. Does that make the effort of tracking his progress worthwhile? Of course.

• Searching for Stars on an Island in Maine by Alan Lightman is published by Corsair (£12.99). To order a copy for £11.04 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99