The novelist Gerald Murnane is something of a writer’s writer’s writer. He is celebrated but not much read in his native Australia, or in Britain (the last of his novels to be published here, so far as I can tell, was Inland, in 1988). But elsewhere his reputation as an author of strange and luminous books is well established: Ben Lerner, JM Coetzee and Teju Cole are all fans, and a profile in the New York Times last year tipped him for the Nobel prize.

Now Sheffield-based small press And Other Stories is publishing five of his novels in the UK, starting with his most recent, Border Districts (which Murnane has said will be his last, though he has given up writing fiction before: he turns 80 this year), and his first, Tamarisk Row. Reading these books alongside each other shows that, while his style has developed over the years, many of his interests have remained the same. His central themes – parochialism, the workings of memory, the Australian landscape, Catholicism, horse racing – have been there from the beginning. So too have many of what he calls the individual “mental images” (instances in which characters keep records, or perceive things through coloured glass, or half-closed eyes) from which he assembles his novels. Though his books are deftly crafted, he’s not very interested in plot. Instead he wants to show the world as it seems (“seems” is a very Murnane word); how our minds and memories mediate it.

Tamarisk Row is written in the present tense in close third-person narration, as a series of short chapters with descriptive, perfunctory titles such as “Mr Glasscock ill-treats his family” and “Augustine tells Clement how to avoid temptation”. The sentences are long, sweeping and rhythmically complex, taking convoluted digressions before returning, eventually, to their beginnings in order to end.

The novel focuses on a boy named Clement Killeaton growing up in the town of Bassett just after the second world war. Clement’s Catholic father, Augustine, is a gambler who spends his weekends at distant racecourses hoping for a big win. Early in his career he tasted success with a horse of his own, Clementia, which was then injured, and he has lived on the fumes of his dreams ever since.

Clement, an only child, is a loner who spends much of his time in his garden, which he has turned in his imagination into a vast landscape populated by farms and racecourses. When he is not using marbles to recreate imaginary horse races he tries to find out about sex and has run-ins with a local gang.

It is often difficult to know who the narrator of the novel is supposed to be, or to ascribe the frequent passages of free indirect discourse to individual characters within it. At one point Clement’s friend Desmond tells him that as soon as school has broken up he is going on holiday to Melbourne where “in a long street in a […] suburb named after a tree or a flower, in a house with lawn between the footpath and the gutter, his little sweetheart waits for him”. This isn’t quite Desmond’s language, and nor is it Clement’s, though the indeterminacy of “a tree or a flower” suggests the narrator is not omniscient, either. Rather, with that “little sweetheart” – which is both indulgent and condescending – it reads as though an adult storyteller is trying to imagine the mind of a boy at one remove.

Tamarisk Row is filled with such moments of fertile ambiguity or misreading. In another scene Augustine is talking to his horse racing friends outside church. Clement asks his father for money to buy a newspaper. “Augustine makes a face,” the narrator reports, “that is meant to show the men that he is exasperated with his son but not able to refuse him.” After he has handed over the money, Clement “asks in a voice that is meant to sound innocent and girlish – can I buy myself a chocolate malted milk too please Dad?” Here Augustine must perform his exasperation to his friends, just as his son performs for him, and the narrator’s knowingness allows us to see through both performances. It’s a method that draws attention to the way adults talk over and beyond children – referring to things they cannot understand in ways they cannot understand – but also to the fact that children are just as likely as adults to have secret lives of their own.

Border Districts is a far sparser book. It presents itself not as a novel but as a “report” written by an old man after he has moved from the capital of the district to a small town in the interior “so that I could spend most of my time alone and so that I could live according to several rules that I have long wanted to live by”. Mainly this involves trying to excavate and interpret the “mental scenes” and “thought images” that make up the narrator’s memories. He is a bit like Beckett’s Krapp, or Borges’s Funes, but actually it sounds as if the bones of the story are pretty autobiographical: Murnane rarely travels, and for the last decade he has lived in the town of Goroke, Victoria (pop 623) with his son, whom he describes as a recluse. There he lives quietly, indulging visiting academics who come to interrogate him about his writing. He also maintains an extensive filing system, organising his papers under headings such as “I decide that most books are crap” and “Peter Carey exposed at last”. You get the feeling he revels in his own eccentricity.

If Border Districts is a stripped down example of late-style Murnane, Tamarisk Row feels like an authentically modernist novel, as though mid-century experiments in maximalism and postmodernist play had never happened. Its themes, as well as its technique, place him in the tradition of Katherine Mansfield and James Joyce. Like both Mansfield and Joyce, Murnane writes from the edges of English literary culture, and his stories are similarly sensitive to the feelings of children and the little worlds they inhabit (in this there is something of Henry James about him, too).

But where Mansfield and Joyce were in flight from the provincial and parochial, Murnane has embraced it. Joyce’s Stephen Dedalus said that the artist, like God, should remain “within or behind or beyond or above his handiwork, invisible, refined out of existence, indifferent, paring his fingernails”. His big innovation, as critic Hugh Kenner put it, was to do away with the storyteller altogether. Murnane’s technique is almost the opposite of this. In a 2007 foreword he said that his aim with Tamarisk Row was to bring “to life the fictional personage responsible for it: the narrator through whose mind the text is reflected”. This storyteller he calls variously “the awesome personage” (Murnane trained as a priest at one point in his life) or “implied Author”. In the end his books aren’t really about the characters they describe, but about the mind behind those characters: the singular, fascinating consciousness that gives them life.

• To order Tamarisk Row (And Other Stories, RRP £10) for £8.80 or Border Districts (And Other Stories, RRP £8.99) for £7.91, go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min. p&p of £1.99.