Sometime in the late 18th century an Indian painter, clearly frustrated with his patron, scribbled a small prayer in the margins of a manuscript on which he was working: “Protect me O Lord, from oil, from water, from fire and from poor binding,” he wrote. “And save me from falling into the hands of a fool.”

Most historians of Indian art have tended to look at their subject from the point of view of the patron. The great master bronzes of southern India are known after their Pallava and Chola patrons; the most accomplished court miniatures, such as the Padshahnama of Shah Jahan, tend to be seen through the prism of the Mughals who commissioned them. The patronage of individual rulers – the emperor Jahangir, Ibrahim Adil Shahi II of Bijapur, Maharaja Man Singh of Jodhpur – are the subject of detailed academic studies and exhibitions. But until recently, few scholars have attempted to look at the production of Indian art from the point of view of the artists who actually held the brushes and burnished the paper.

There is a reason for this: very little evidence survives to illuminate the lives of Indian artists. In particular, there is no Indian Vasari providing the kind of detail that has illuminated lives of the artists of the Renaissance: the hot-blooded womanising of Fra Angelico, say, or Uccello’s passion for geometry. All we have to go on is a series of minute inscriptions, often hidden in the details of paintings, sometimes in a deliberately humble position: the Mughal master Abu’l Hasan, who won from Jahangir the title Nadir al-Zaman, “wonder of the times”, deliberately chose to sign his name on the spade used to clear up the dung of his patron’s elephant.

BN Goswamy, the highly respected historian of Indian painting, has been trying for nearly five decades to look down the other end of the art historical telescope. Like an Indian avatar of Bernard Berenson, who dug in the Tuscan Ducal archives to unearth the bills of exchange between the artists and patrons that would enable him toprovide attributions to a host of anonymous canvases, Goswamy has succeeded in reconstructing whole dynasties of previously obscure artists, given them names, and restored their identities and honour.

This is no easy task: many painters came from the humble carpenter caste, in ancient India ranked alongside lowly musicians and dancing girls. There survives in the Jahangir album, now in Berlin, a heartbreaking self-portrait of the Mughal master painter Keshav Das coming in old age to beg for assistance from his former patron. The old artist shows himself ragged, hollow-chested, bowed and emaciated. In his hands he holds a petition to the emperor who had once numbered him among the greatest talents of his court – but before the old man can present himself, a lathi-weilding attendant advances on him, stick raised, driving him back. In a similar mood, a moving letter found by Goswamy was written by the 18th-century painter Shiba asking his patron Raja Sansar Chand of Kangra for permission to return home, “for your humble servant here has fallen on bad days. Your servant has been living on debts, but now no one will give him a loan. He is helpless and goes without food.”

In 1968, Goswamy wrote a ground-breaking article, “Pahari Painting: The Family as the Basis of Style”. Employing a combination of detective work and intuition, he managed to marry the evidence from inscriptions on the back of miniatures with 18th-century pilgrim records kept in the Ganges holy town of Haridwar. In this way he reconstructed the entire family network of arguably the greatest of all Indian painter families: that of Pandit Seu and his sons, Nainsukh and Manaku, as well as their numerous artist grandchildren.

He then showed how many members of the family shared a common style, and that their mobility between different noblemen effectively made nonsense of the existing system of categorising miniatures by courts and patrons. What was important, Goswamy made clear, was not where a particular painting was produced, or who paid the bills, but which artist, or family of artists, was holding the brush. Court styles could vary hugely, depending on who was at work; but families had recognisable techniques and stylistic idiosyncrasies.

Since then Goswamy has been working at reconstructing the lives first of the painters of the Punjab hills, and then of those elsewhere in India. The culmination of his work was the Master Painters of India show three years ago, which travelled from Zurich to New York and which amounted to a dramatic re-evaluation of the human and biographical reality behind Indian painting.

Now, in The Spirit of Indian Painting – a book that is in many ways the summation of Goswamy’s whole career – he tries to get inside the heads of those artists, to understand what made them paint the way they did, how they came to choose their iconography and what were the daily circumstances of their lives.

The process was painstaking. As Goswamy writes, we are dealing with “a world of silence in which one has to strain very hard to pick up whispers from the past … a layered world that does not reveal all its treasures immediately … One has to fall back on one’s own resources … to piece things together, the willingness to construct a narrative, the imagination to flesh it out … One needs to make an effort to receive from these paintings all the riches that reside within.” But if we strain hard, he says, it is still possible to “feel the breath of those times – even if lightly – upon our skin”, and so gain access to the highest state of pure aesthetic pleasure – to experience what Indian aesthetic theory describes as romaharshana, meaning literally: “the hair on my body has become happy”.

In the Hindu scriptures, “time moves in a cyclical fashion, making bends and loops, turning back on itself”. As a result, the art of the Hindu courts, and even more that of the Mughals, often shows the same figure appearing more than once in the same frame; this indicates that the artists are “completely at home with the notion of time as manipulable and elusive”.

We follow Goswamy into the workshops of his painters as they collect their materials – brushes made from a single hair from a calf’s ear or a squirrel’s tail – or as they grind their pigments from Afghan lapis, saffron derived from the flower of the palash tree (which later gave its name to the battle of Plassey) or the gaogoli yellow, concocted from the urine of cows fed on mango leaves. Goswamy also carefully teaches us the difference between paintings produced in the family workshops of the Rajput or Pahari courts, where the artists worked at home and all generations lent a hand, and that of the Mughal ateliers, where the best talent from across the empire was deployed under the strict discipline of a master ustad.

As with any sweeping survey, there are insights one can quibble with, here perhaps the lack of space Goswamy gives to India’s rich tradition of mural painting, which is dismissed in a single paragraph: this is a book exclusively about works on paper. His distinction between the Rajput and Mughal ateliers is too absolute – later Mughal master artists such as Ghulam Ali Khan, while proudly calling themselves “palace born”, had the freedom to travel around Hindustan taking commissions from other nobles and East India Company officials. These were men who went where they pleased. Equally, the suggestion that even the grandest Mughal painters were sometimes treated as chattels is countered by their own self-portraits: Jahangir’s beloved master painter, Govardhan, for example, portrayed himself as an eager, sharp-eyed and intelligent young man with raffishly long sideburns and a carefully trimmed moustache, an immaculate white jama and dashing black cape. Govardhan knew he was the crown prince’s protege, and proudly depicted himself as such.

Yet these are small matters. Old age, Goswamy writes, referring to a specific Mughal portrait of an old man, is often a time when “the meaning of things begins to dimly unfold”. Certainly the historian, now in his 80s, has never been more prolific, lectured so brilliantly or written so well. The Spirit of Indian Painting is that rarity: an out-and-out masterpiece, and will undoubtedly come to be looked on as one of the greatest books ever written on Indian art.

• William Dalrymple’s most recent book, Return of a King: An Indian Army in Afghanistan, is out in paperback from Bloomsbury.

• The Spirit of Indian Painting is published by Allen Lane India