Newly declassified documents from the FBI reveal how the US federal agency under J Edgar Hoover monitored the activities of dozens of prominent African American writers for decades, devoting thousands of pages to detailing their activities and critiquing their work.

Academic William Maxwell first stumbled upon the extent of the surveillance when he submitted a freedom of information request for the FBI file of Claude McKay. The Jamaican-born writer was a key figure in the Harlem Renaissance, author of the sonnet If We Must Die, supposedly recited by Winston Churchill, and Maxwell was preparing an edition of his complete poems. When the file came through from the FBI, it stretched to 193 pages and, said Maxwell, revealed “that the bureau had closely read and aggressively chased McKay” – describing him as a “notorious negro revolutionary” – “all across the Atlantic world, and into Moscow”.

Maxwell, associate professor of English and African American studies at Washington University in St Louis, decided to investigate further, knowing that other scholars had already found files on well-known black writers such as Langston Hughes and James Baldwin. He made 106 freedom of information requests about what he describes as “noteworthy Afro-modernists” to the FBI; 51 of those writers had files, ranging from three to 1,884 pages each.

“I suspected there would be more than a few,” said Maxwell. “I knew Hoover was especially impressed and worried by the busy crossroads of black protest, leftwing politics, and literary potential. But I was surprised to learn that the FBI had read, monitored, and ‘filed’ nearly half of the nationally prominent African American authors working from 1919 (Hoover’s first year at the Bureau, and the first year of the Harlem Renaissance) to 1972 (the year of Hoover’s death and the peak of the nationalist Black Arts movement). In this, I realised, the FBI had outdone most every other major institution of US literary study, only fitfully concerned with black writing.”

Maxwell’s book about his discovery, FB Eyes: How J Edgar Hoover’s Ghostreaders Framed African American Literature, is out on 18 February from Princeton University Press. It argues that the FBI’s attention was fuelled by Hoover’s “personal fascination with black culture”, that “the FBI is perhaps the most dedicated and influential forgotten critic of African American literature”, and that “African American literature is characterised by a deep awareness of FBI ghostreading”.

Princeton said that while it is well known that Hoover was hostile to Martin Luther King and the civil rights movement, Maxwell’s forthcoming book is the first exposé of “the extent to which the FBI monitored and influenced African American writing” between 1919 and 1972.

Taking its title from Richard Wright’s 1949 poem The FB Eye Blues, in which the Native Son novelist writes that “every place I look, Lord / I find FB eyes / I’m getting sick and tired of gover’ment spies”, the work also posits that for some authors, suspicion of the surveillance prompted creative replies.

Digital copies of 49 of the FBI files have been made available to the public online. “The collected files of the entire set of authors comprise 13,892 pages, or the rough equivalent of 46 300-page PhD theses,” Maxwell writes in the book. “FBI ghostreaders genuinely rivalled the productivity of their academic counterparts.”

The academic told the Guardian that he believes the FBI monitoring stems from the fact that “from the beginning of his tenure at the FBI ... Hoover was exercised by what he saw as an emerging alliance between black literacy and black radicalism”.

“Then there’s the fact that many later African American writers were allied, at one time or another, with socialist and communist politics in the US,” he added, with Wright and WEB Du Bois both becoming Communist Party members, Hughes a “major party sympathiser”, and McKay “toasted by Trotsky and published in Russian as a significant Marxist theorist”.

The files show how the travel arrangements of black writers were closely scrutinised by the FBI, with the passport records of a long list of authors “combed for scraps of criminal behaviour and ‘derogatory information’”, writes Maxwell. Some writers were threatened by “‘stops’, instructions to advise and defer to the Bureau if a suspect tried to pass through a designated point of entry” to the US.

When McKay went to the Soviet Union, a “stop notice” instructed that the poet should be held for “appropriate attention” if he attempted to re-enter the US. In Baltimore, writes Maxwell, FBI agents “paraded their seriousness in a bulletin sent straight to Hoover, boasting of a clued-in ‘Local Police Department’ on the ‘lookout’ for one ‘Claude McKay (colored)’ (23 Mar. 1923)”.

They also reveal how, with the help of informers, the agency reviewed works such as Lorraine Hansberry’s A Raisin in the Sun and Ralph Ellison’s Invisible Man before publication.

“What did the FBI learn from these dossiers? Several things,” said Maxwell. “Where African American writers were travelling, especially during their expatriate adventures in Europe, Africa, and Latin America. What they were publishing, even while it was still in press.” In the 1950s, he said, the FBI aspired to “a foreknowledge of American publishing so deep that literary threats to the FBI’s reputation could be seen before their public appearance”.

The bureau also considered “whether certain African Americans should be allowed government jobs and White House visits, in the cases of the most fortunate”, and “what the leading minds of black America were thinking, and would be thinking”.

But, he added, “the files also show that some FBI spy-critics couldn’t help from learning that they liked reading the stuff, for simple aesthetic reasons”.