Myth, monsters and making sense of a disenchanted world: why everyone is reading fantasy

I have made the leap from literary fiction to fantasy – for those who think it’s mere wish-fulfilment, here’s why we need that thing with the dragons

Fantasy doesn’t need defending. It is one of the great cultural forms at the moment, all-pervading, ubiquitous. Maybe even the dominant form of writing just now, in line with the bookseller’s joke that contemporary publishing divides into A: romantasy and B: everything else.

But it might need explaining a little bit, for those who don’t get its pleasures; who still see it as wish-fulfilment, or as a low form that literary fiction gets to look down upon or direct a puzzled tolerance towards. As a writer of literary fiction who has borrowed and rejoiced in fantasy tropes for years, and has now himself written an out-and-out fantasy, I’m beyond embarrassment. I’ve been reading and loving fantasy all my life, and for me its best creators stand comfortably alongside the greats of any genre. And yet, I’m still encountering a faint sense that there is something to be accounted for in writing fantasy. That I ought to have reasons for wanting to do that thing with the dragons, no matter how culturally pervasive it is.

None of what I’m about to say is going to strike my fellow lovers of the genre as even slightly necessary. We can just assume its joys, assume that like any form of writing it features solid good stuff, works of brilliance, and also extruded polystyrene product – and then proceed to specifics. Portal fantasy or epic? Urban fantasy or fantasy of manners? Romantasy or grimdark? Cosy or horror-inflected? And then, in which line of descent do your tastes lie? Are you part of the endlessly ramifying Clan of Tolkien, or is the feminist fantasy that descends from Ursula K Le Guin the genealogy that matters to you? Are you in it for the decolonising inventiveness of NK Jemisin, the LGBTQ+ friendliness of Katherine Addison, the remixed history of Guy Gavriel Kay, the surrealism of Jeff VanderMeer, the political wit of China Miéville, the queered gothic of Tamsyn Muir? For any of those, there is a conversation waiting to be had, a corner we can head into together, excitedly jabbering to each other.

But for anyone else, here’s a case for fantasy rehearsed from scratch. Fantasy, in the first place, is true to the experience of the human psyche. Specifically, it’s true to what’s wild in it that the reasonable, consensual, self-restrained daylight world doesn’t easily express, but which everyone feels. Children and teenagers acutely so, for a set of reasons to do with the conflict between dependence in their lives and how big they feel themselves to be, what shadowy powers they half-see in themselves. Also, how gigantic and new-seen the world’s evils seem to them, making dragons and monsters natural. But it’s true too, on different grounds, for everyone of every age at times. To use the philosopher Charles Taylor’s term, we all inhabit the constraints and reassurances of “the buffered self”. We take the world to be reliably disenchanted; we take there to be a secure line between our interior self and everything else, which can’t be transgressed by ghouls, demons, fairies, visions, spirits, malign or benevolent powers.

This keeps us safe, but it also cuts off or cuts down to size the unruly and imaginative in us. It makes us long untidily for the enchantment it excludes; it makes us want the magical to be allowed to billow out sometimes.

Or maybe it’s a question of needing rather than wanting. The strictly disenchanted world, where nothing exists but physical processes describable without metaphor, and even consciousness is just a material problem waiting to be solved, can be a desiccated place. It keeps heart and mind on inadequate rations. This is the point Philip Pullman is making in The Rose Field, the recent last volume of The Book of Dust, where he has Lyra think about the human need for the kind of things we can’t prove, but would suffocate without. The imagination, above all. “Maybe the imagination is a sort of wind that blows through all the worlds … It shows us true things.” For Pullman, of course, the enemy of imagination is religious dogma even more than narrow scientism – but there are a lot of different ways of figuring what deadens in the modern world, just as there are other ways of naming the unpredictable wind that blows through all the worlds, showing us true things.

Yet a lot of what we eliminated from the world by disenchanting it, we really don’t want back. At least, not seriously. There’s a compelling origin story for fantasy as a genre – you can find it brilliantly and subtly analysed in Adam Roberts’s recent Fantasy: A Short History – in which it works as a kind of regulated return of the repressed. A partial haunting. With its kings and quests and chosen ones and battles and powers of earth and air, it lets back all the things we miss in the world of science, contract, employment, regularity, but don’t want to return all the way. Roberts identifies the turning point as the first world war, giving a generation of young men such as Tolkien and CS Lewis an experience of modernity as utter mechanical savagery, and brewing in them the desire for a literature in which the old stories of myth – with the space in them for individual human agency – came back, remixed, in modern form. We like to dream of having enormous muscles like Conan, when office life makes seven-stone weaklings of us all; we like to think of ourselves as the singular and remarkable Chosen One, when really we form a pixel in a crowd. But having picked these dreams up, we want to put them safely down again, rather than deal with a world in which unaccountable kings and barbarians with zero impulse control really shape our fate. Hence (runs this argument) our packaging of the fantastic in trilogies that end, and books that close.

But there’s another story of fantasy that needs telling. In this one, it isn’t just the wonder-book of our impulses, or an organised nostalgia for a more romantic world. Here it exists because it is (paradoxically) a kind of necessary realism, arising in response to qualities of the contemporary world that we couldn’t properly attend to, couldn’t narrate, any other way. I’d argue that, as well as expressing our frustrations with the disenchanted world, it’s also our best means for capturing the ways in which the world remains enchanted, for all our strenuous buffering. I read and write fantasy because it’s the literature that sees the recurrent unearthliness in human experience. That knows we’re hopelessly metaphorical creatures, who find meaning by tying together patterns of resemblance that might as well be spells. That knows there are some struggles where the stakes really are overwhelming, and good and evil in something like their pure forms really do pivot on human choices. Fantasy understands that to undertake the risks of love is to venture beyond safety, into landscapes strange to you, on perilous and wonderful journeys.

• Nonesuch by Francis Spufford is published by Faber on 26 February. To support the Guardian order your copy at guardianbookshop.com. Delivery charges may apply.