I was living in Norwich when the Brexit referendum happened. The city voted remain, an island of European feeling separated from Cambridge – the next dot in the remainer archipelago – by 60 miles of rural and small town leaverdom. One morning not long afterwards, in the small hours, a local deli selling Romanian produce was firebombed. No one was hurt, although people were sleeping in a flat over the shop when it happened. A crowdfunding appeal quickly raised almost £30,000 for the damaged business.

A few days later I went with my infant son to a rally for tolerance held in the market square in response to the attack. The numbers were pitifully few. I remember looking round at my fellow demonstrators, the elderly liberals and the hippies, and thinking none of us would be much good in a street fight. Tolerance was all very well, but what if we had to defend ourselves? My psychic response to the leaver neurosis of remainers as “the arrogant elite” was an equally neurotic identification with the persecuted of history, as if the 52% were about to systematically Other and repress the 48% en masse.

It was a jittery and unsettling time. I doubt I was alone among remain voters in feeling, for the first week or two after the referendum, that sleep-reducing ball of anxiety in the pit of my stomach which signifies the perception not just that a deeply wrong event has occurred, but that the very framework within which events of any kind occur has been broken.

I should have recognised the feeling sooner. It was a feeling I perceived intellectually, but didn’t experience personally, when I talked to people in the former Soviet Union in the 1990s about their regret for the collapse of the USSR – its economic collapse bound together with the belittlement and dissolution of its cultural landscape.

It’s a feeling, I imagine, very similar to learning that the mine or factory where you and your family have worked for a generation or two is closing, or that new people who look and talk and dress differently to your people are eligible for benefits you’ve been queueing for, or that the social values that defined your youth are now officially wrong – which means, by extension, that your youth was wrong, too. And now we remainers were feeling it: people like me, who had remotely observed others, were experiencing it for ourselves. For leavers, the merit of voting to leave the EU wasn’t only in winning. It was in getting their opponents to feel loss – to feel what they had felt, that deep unease at the shattering of their representation of the world. My bad feeling was somebody else’s catharsis.

I’ve been angry about Brexit. I’ve been ashamed, sometimes fearful, often bored by the futility and pettiness of it all. As I write, although the fight goes on for a second-thoughts referendum, it seems likely Britain won’t be in the European Union by year’s end – perhaps as soon as the end of next month – and I’ll be angry, ashamed and fearful again. These feelings are forgivable. What isn’t forgivable, however, is bitterness.

If and when Brexit happens, I don’t want to be a remainer equivalent of Rush Limbaugh, uttering his odious response to Barack Obama’s election as US president: “I hope he fails.” While it will be hard to forgive Boris Johnson his lies, or David Cameron his haughty recklessness, while I’ll always struggle to see Nigel Farage as anything other than the outcome of a liaison between Toad of Toad Hall and the Chief Weasel, I don’t want my dislike of the leavers’ champions hardening into a crude ideology of hatred. To fume for ever at the tricks and strength of the winners, without questioning my own, British liberal way of thinking, would be to waste one of the few opportunities Brexit offers.

It’s not that the country should put aside its differences and come together: a horrible, stifling thought. It often seems, in the heat of Brexit rage, that ardent leavers and remainers have forgotten how normal it is for nations to be divided. “United” nations made up of people who agree about everything are fictions invented by authoritarian regimes. The point of democracies is not to unite a nation so that everyone is happy, but to manage the unhappiness of whoever feels they’ve lost out without resorting to civil war.

A renunciation of bitterness doesn’t mean remainers giving up their struggle, either, reining in their attacks on folly and social media jiggery-pokery. It means remainers like me examining and beginning to resolve the contradictions in our own, not always rational, way of representing the world. We need to raise our eyes from Brexit and consider not just the Britain, not just the Europe, but the kind of world we are aiming for. The alternative is to end up as the remainer version of a French Bourbon aristocrat, learning nothing, forgetting nothing.

Who are we, the remainers? We know what we don’t want, but we don’t seem sure what we do want in the long term. Matthew Goodwin, an academic and writer prominent in analysis of the deep roots of English Euroscepticism, has characterised Britain’s current political landscape as the embodiment of Michael Oakeshott’s distinction between “politics as faith” and “politics as scepticism”, moderated by the political scientist Margaret Canovan’s more optimistic version: she saw populism as pragmatic democracy’s necessary redemptive shadow.

The trouble with this is that it puts all the faith and redemptive hope, all the inspiration, all the archetypes of surrogate religion, all the dreaming, on the leave side, stranding the remainers in a dreamless, materialist Britain of cold calculation and joyless efficiency.

Besides being offensive to remainers, it’s inaccurate. Sure, the enlightened self-interest is there. But when the reality of EU administration is often so bumbling, awkward and lame, when the great majority of remain voters will never actually exercise the freedom to live and work on mainland Europe, isn’t there something irrational and romantic about the yearning to continue to be “part of Europe” – an act of faith, a response to impulses in one’s political unconscious?

When I try to interrogate my own inclinations honestly, I find not only rational, pragmatic fears of Brexit, not only the personal legacy of past trans-European love affairs and a brother with a German wife living in Berlin, but a more intangible and perhaps not altogether wholesome dream of power and abandonment. This is the dream of being at once an element in a vast, variegated, rich continent, and of losing myself in wanderings in the furthest and least fashionable corners of that continent. It does not seem either rational or pragmatic, but nor is it “wrong”.

Remainers, on the whole, aspire to a morality that is universal. We are liberals. We endorse the principle of the right to cultural self-determination not just for the native English, but for all peoples. At the same time we support other abstract, universal principles – women’s rights, sexual minority rights, ethnic minority rights, animal rights, children’s rights, the primacy of environmental protection, migrant rights, equal opportunities, education for all, democracy. Some liberals believe in free trade, free movement of people or free movement of capital.

I have to face the truth that these strands are deeply contradictory. A belief in the imperative to conserve the traditional, authentic and distinctive in local cultures clashes with an equally fervent promotion of universal rights and freedoms. This is the liberal bourgeois dilemma: an irreconcilable desire for both universality and particularity.

British liberals both supported large-scale coal mining as the enabler of traditional, authentic, working-class communities, and abominated it as global warming’s lead culprit. Only an accident of history allowed these sentiments to be consecutive, rather than concurrent. Liberals prize the authentically local, the person or product or way of being that is truly, traditionally, of place; we are sad and angry when that authenticity is cheapened, commercialised, tainted by globalism, even though our own restless, flocking quest for local authenticity in multiple locations each year contributes to the very globalism that torments us.

And there are consequences. At the libertarian, neoliberal end of liberalism, there are occasional personal shocks for those whose wealth usually enables them to keep the contradictions at a distance. A friend who prospered as a fund manager in London stepped back from finance and became involved in a worthy British charity helping to empower young people. On her way to a meeting of the organisation in a northern English town, the taxi driver began telling her of the area’s economic woes and she realised the multinational responsible for hurting the people she was trying to help was one of the companies she’d invested her clients’ money in.

At the communitarian end – the part of the liberal spectrum where I like to think I reside – there’s a tendency to assume “good” localism (the ideal of the “thriving local community”, locally sourced food, preservation of vernacular local architecture and the traditional local landscape) can be neatly separated from “bad” localism (hostility to immigrants and new ways of doing things). But can it?

The trickiest contradiction in the remainer-liberal view is the European Union itself. British liberals like to see support for remaining in the EU as a marker, of and by itself, of good universal values: openness, receptiveness to other cultures and ideas, freedom for people to move and work in different places, widely pooled security. Indeed, Britain in the EU does approach these values more nearly than Britain on its own. But they aren’t applied universally. Britain, on its own, is an exclusionary, overwhelmingly white, post-Christian society; Britain in the EU is part of a larger, overwhelmingly white, post-Christian society that still excludes 14 out of 15 of the world’s population from freely moving, living or working there.

In the wake of the Brexit vote a number of EU citizens living in Britain have had horrible experiences with the immigration authorities. But the experiences of the Windrush generation, people who have lived in Britain for decades and find themselves officially persecuted, in some cases forcibly expelled to countries they were brought from as children, happened while Britain was still in the EU. Two hundred people a month are dying trying to illegally cross the EU’s closed external borders.

Most remainers I know feel it’s a strong enough stance on openness to be firm in their insistence on free movement within Europe, firm on the rights of migrants and refugees, and vague on the ideal degree of permeability of the external borders of the jurisdiction in which they live. There aren’t many takers for absolute global freedom of movement. But to spell out the exact nature and justification of the consequent limits to absolute freedom provokes in liberals a fear of infection with the disease of racial exclusivity. The shudders that greeted Gordon Brown’s “British jobs for British workers” comment, or Jeremy Corbyn’s complaint of “wholesale importation of underpaid workers from central Europe”, are proof enough.

¶

The idea that remain voters were winners who only cared about themselves and leave voters were losers screaming for help is simplistic. Most remain voters don’t deny the damage the country has suffered since the crash of 2008, if not earlier.

The damage is economic and cultural. Economically, high-status non-graduate jobs with good pay and conditions have been replaced by low-status jobs with bad pay and conditions, housing has become scarcer and more expensive, taxpayer-funded public services such as health, education, police and the criminal justice system have got worse, and privatised user-funded public services such as water and energy have increased in cost to the benefit of their mainly overseas owners. Culturally, distinctiveness of place is being erased by a creeping uniformity whose paradoxical hallmark is a shallow diversity. In a narrow sense the blame for these ills is distributed differently; one side blames immigration and the EU, the other, austerity and globalisation.

Both halves of the electorate, then, are dismayed by the same phenomena – economic decay and corrosion of identity. To see a relationship between hostility to immigrants from far away (which shades into overt racism) and hostility to capitalists from far away is not to equate them, but to acknowledge the coincidence of mood. In the same way, the blaming of immigrants and the blaming of government spending cuts are two quite different responses to the strain on public services, but they meet at the heart of the problem, which is that British government spending is not increasing at the same pace as the population of Britain. Taking inflation and population into account, the British state now spends 6% less on public services than it did in 2010 – an absolute cut that most affects the poorest and the oldest.

Sometimes since the Brexit referendum I’ve sensed the divergent dreams of leavers and remainers reducing to the same primal personal desire, the desire to live in a community that is naturally, effortlessly kind to them and their families – one where a structural benignity of laws, customs and officialdom is matched by a social benignity of shared outlook and common values. Among some leavers floats a dream of Britain as an island purged of incomers – out with the Muslim, the Pole, the Russian oligarch and his dirty money. Remainers are haunted by a dream of sanctuary, of some other country marvellously preserving the spirit of a more generous and tolerant society, ready to make available for a select few Britons that freedom and openness of outlook Britain has lost for ever. “I’m trying to get an Irish passport.” “I’m eligible for German citizenship.” “I’m going to be spending more time at my house in Spain. This country’s finished.”

When “dream” refers to your representation of the world, it’s not a state you’re going to wake from. The disruption to your dream comes otherwise; not only in the form of a momentous event like military defeat, imperial collapse, workplace disappearance or Brexit. It comes when you can’t avoid awareness that your dream clashes with another’s: when the remainer beds deeply in her new home abroad only to find that there, too, resentment of the outsider and the cosmopolitan runs deep, and she encounters local versions of the dark island dream of purification she fled. Where, now, in the world is a sanctuary for tolerance more reliable than the one you might get if you spoke up for the values you claim to hold in the place you already are? In a US where almost a hundred million adults think Donald Trump is doing a good job? In a France where a third of the electorate wanted the virulently anti-immigrant Eurosceptic Marine Le Pen as president? In a Denmark which is establishing immigrant ghettos where different laws apply?

For the leavers and their island dream, it’s harsher. They know their exclusionary dream clashes with the sanctuary-seeking dream of immigrants and refugees from other countries – people who yearn not just for better-paid work than at home but for a refuge from corruption and extreme pro-rich policies that render basic public services in their countries weak, patchy or non-existent.

¶

Whatever course the Brexit process now takes, the future for those who believe themselves tolerant and outward-looking, but who resist Britain’s economic and cultural corrosion, must be to continually challenge acceptance of a world where the division of the spoils of labour varies so wildly from one country to the next. The failed liberal slogan for the world that sounded so loudly in the afterclap of communism’s fall – “Democracy and the free market” – lacks the Robin Hood kicker: democracy, trade and fairly shared wealth.

The scale to measure the success of globalisation shouldn’t be how easily the wealthiest can suck rents from the majority and keep the proceeds through a combination of capital mobility, tax havens and wage suppression, by shuffling production, populations and profits from national jurisdiction to national jurisdiction. It should be by how far the world as a whole approaches a high baseline of shared security and prosperity, within which cultural distinctiveness can flourish – how far Indonesia or China or Egypt are reaching up to establish universal networks such as education, healthcare, housing, water and energy supply; how far Britain, the United States or Italy are levelling down; and where the levels meet.

As the world is, if a country reneges on the trappings of democracy, doesn’t hold free elections, arrests and tortures dissenters, crushes minorities, starts invading its neighbours and smothers free speech, it can expect, at the minimum, odium from the global human rights establishment, at worst sanctions, even (if it’s small and weak) military action. Similarly, if a poor country reneges on its commitment to open markets and trade, it will be hung out to dry by the global financial system.

No similar set of responses, not even a conceptual framework, exists by which intensively networked societies such as Britain or Japan – countries, that is, that have universal networks of health, education, welfare and so on, built and maintained on the basis of wealth shared through taxation – might pressurise the wealthy elite of other nations to share more of their wealth with the mass of their own people. If sanctions are the weapons of global democracy, and capital flight the weapon of global finance, what might the weapon of global social justice be? Perhaps a global wealth tax; perhaps something like tariffs, tied to a global minimum wage.

This doesn’t sound terribly exciting, until you think about the consequences, and reflect on where we in Britain might be now if, before the former communist countries of eastern Europe joined the EU, Brussels had brought in a Europe-wide minimum wage, pegged to west European living standards. As we prepare to leave the EU, the discourse of the most prominent Brexiters, who are almost all Conservatives, is thick with neo-imperialist gobbledygook about global trade. Their opponents, scattered through the parties, must not think on a smaller scale.

The best way for remainers to avoid bitterness over being deprived of membership of the EU is not to settle for less, but to ask for more. Not to grimly resent the construction of Britain as a fortress against immigration by mourning our departure from a bigger fortress – Europe – but to ask why life needs be so grim outside the fortresses that those outside try so hard to get in.

• Dreams of Leaving and Remaining by James Meek will be published by Verso on 5 March (£16.99). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99.