Professor Peter Murray-Rust was looking for new ways to make better drugs. Dr Heather Piwowar wanted to track how scientific papers were cited and shared by researchers around the world. Dr Casey Bergman wanted to create a way for busy doctors and scientists to quickly navigate the latest research in genetics, to help them treat patients and further their research.

All of them needed access to tens of thousands of research papers at once, so they could use computers to look for unseen patterns and associations across the millions of words in the articles. This technique, called text mining, is a vital 21st-century research method. It uses powerful computers to find links between drugs and side effects, or genes and diseases, that are hidden within the vast scientific literature. These are discoveries that a person scouring through papers one by one may never notice.

It is a technique with big potential. A report published by McKinsey Global Institute last year said that "big data" technologies such as text and data mining had the potential to create €250bn (£200bn) of annual value to Europe's economy, if researchers were allowed to make full use of it.

Unfortunately, in most cases, text mining is forbidden. Bergman, Murray-Rust, Piwowar and countless other academics are prevented from using the most modern research techniques because the big publishing companies such as Macmillan, Wiley and Elsevier, which control the distribution of most of the world's academic literature, by default do not allow text mining of the content that sits behind their expensive paywalls.

Any such project requires special dispensation from – and time-consuming individual negotiations with – the scores of publishers that may be involved.

"That's the key fact which is halting progress in this field," said Robert Kiley, head of digital services at the Wellcome Trust. "For a lot of people, though there is promise there, the activation effort is just too great."

The restrictions placed by publishers on text mining has led campaigners to view the issue as another front in the battle to make fruits of publicly funded research work available through "open access", free at the point of use. That would allow researchers to mine the content freely without needing to request any extra permissions.

The scale of new information in modern science is staggering: more than 1.5m scholarly articles are published every year and the volume of data doubles every three years. No individual can keep up with such a volume, and scientists need computers to help them digest and make sense of the information.

Bergman, an evolutionary biologist at the University of Manchester, used text mining to create a tool to help scientists make sense of the ever-growing research literature on genetics. Though genetic sequences of living organisms are publicly available, discussions of what the sequences do and how they interact with each other sits within the text of scientific papers that are mostly behind paywalls.

Working with Max Haeussler, of the University of California, Santa Cruz, Bergman came up with Text2genome, which identifies strings of text in thousands of papers that look like the letters of a DNA sequence – a gene, say – and links together all papers that mention or discuss that sequence. Text2genome could allow a clinician or researcher who may not be an expert on a particular gene to access the relevant literature quickly and easily. Haeussler's attempts to scale up Text2genome, however, have hit a wall, and his blog is a litany of the problems in trying to gain permissions from the scores of publishers to download and add papers to the project. "If we don't have access to the papers to do this text mining, we can't make those connections," says Bergman.

Murray-Rust, a chemist at the University of Cambridge, has used text mining to look for ways to make chemical compounds, such as pharmaceuticals, more efficiently.

"If you have a compound you don't know how to make and it's similar to one you do know how to make, then the machine would be able to suggest a number of methods which would allow you to do it."

But, although his university subscribes to the journals he needs to do this work, he is forbidden from using the content in what he calls "a modern manner using machines".

A member of his research group accidentally tripped the alarms of a publisher's website when he downloaded several dozen papers at once from journals to which the university had already paid subscription fees. The publisher saw it as an attempt to illegally download content and immediately blocked access to its content for the entire university.

Asking for permission from publishers is an option, though time-consuming. The University of British Columbia (UBC) researcher, Heather Piwowar, was trying to map the ways scientists use and share papers.

She was eventually contacted by Alicia Wise, Elsevier's director of universal access, who convened a conference call with Piwowar, a UBC librarian and five Elsevier colleagues. That conversation led to permission for UBC researchers to text mine the Elsevier journals to which they already had access.

Piwowar said: "It takes a lot of time and a lot of energy and doesn't scale at all. To me it's a good result because now I have access to things I didn't have access to before and also it will also hopefully drive change by people saying, 'This is not an OK way to build on our scholarly literature.'"

Wise said that, in principle, her company was happy to enable text mining for its content. "We want to help researchers deepen their insight and understanding, we want to help them to advance science and healthcare and we want to be able to do that in ways that help realise the maximum benefit from the content we publish. Text mining is clearly a part of this landscape and it will continue to be and we're keen to support it."

The UK government supports open access to publicly funded research and the text mining that it would allow. In a report for the Intellectual Property Office last year on intellectual property and growth, Professor Ian Hargreaves proposed that researchers should be allowed to text mine articles to which they had already subscribed – a position supported by science funding organisations such as the Wellcome Trust.

"Imagine a world where you weren't allowed to use powerful computers to use weather patterns and astronomical data – it's just nonsensical," said Kiley. "Even in commerce, the reason Amazon knows what records I should buy or what books is because it knows what I've bought before, it knows what other people have bought similar to what I've bought and it can suggest things.

"To not be able to exploit that technology in healthcare and life sciences, that doesn't make much sense nowadays."

Warning for publishers

The brewing controversy between scientists and publishers over access to scientific information has also caught the attention of investors. In a briefing note on the publishing company Elsevier, Claudio Aspesi of Bernstein Research warned investors that publishers might be on the verge of falling out with scientists. "We continue to be baffled by Elsevier's perception that controlling everything (for example by severely restricting text and data mining applications) is essential to protect its economics," he wrote.

He said some of the commercial restrictions from publishers seemed not only to be restricting access to the scientific community, but also hindering the work of researchers. "Elsevier needs to take a much harder look at what it is doing to work well with the academic community at large, since it believes that its future lies in tapping the funding for science," he wrote.

Elsevier bosses have long told investors that the publisher's relationship with academics is excellent. But Aspesi doubted that things were so rosy. "If the academic community were to conclude that the commercial terms imposed by Elsevier are also hindering the progress of science or their ability to efficiently perform research, the risk of a further escalation in what is already an acrimonious debate would rise substantially," he wrote.

None of which would be beneficial for Elsevier's bottom line. "Adding confrontational relationships with the research community to the difficult ones it already has with academic librarians looks self-defeating," wrote Aspesi"Elsevier needs to rethink altogether how it thinks of researchers as customers, or it could end up, in a few years, facing the same hostility it encounters with much of the academic librarian community. Governments and other funding bodies may then look a lot less kindly on subscription publishers if they antagonise scientists as well." The note was written before Heather Piwowar's discussions with Elsevier had concluded but Aspesi said those results did not change his conclusions. "If anything I would say that ... my impression is that more issues were raised by the meeting with Elsevier rather than fewer," he said.

A Reed Elsevier spokesman pointed to reports from other analysts which, he claimed, demonstrated that Elsevier still had good relations with librarians. "We continue to look at ways in which we can benefit the research community, and our position to enable text mining is just one recent example," he said.