Creating the Dystopian Universe
Out of necessity, you'll have to set your dystopian novel in a world not our own. This doesn't mean it can't be on Earth, but it usually has to be an alternate Earth. Before you can put pen to paper, you'll have to construct your world. You'll need and entire backstory that explains how the world got to be the way it was. Though you'll only need to dole out this information in small and necessary bits throughout your story, but you'll still need to have the backstory in your mind (and probably written down) in order to craft a believable story.
All of this means that you'll have to make some careful decisions involving oppression, uprising, and rebellion. This might involve crafting a story based on war, natural disaster, revolution, overpopulation, or any other serious problem the society you perceive as a threat.
Creating Your Dystopian Hero
Though sometimes compared to utopian fiction, your dystopian novel will almost certainly not feature a utopian hero. In utopian fiction, the hero is often an outsider intent on disrupting the status quo. But a dystopian hero is usually from within the society (though there are exceptions to this rule). This hero, called the protagonist, is often a doubter, someone who questions the way the society has become and is determined to do something about it.
The protagonist will usually take one of three paths: escape the system, overthrow the system, or change the system. He takes action, almost certainly risking his life, even if he's not entirely sure of what he's doing. His viewpoint is typically unpopular (or at least silent) at the beginning of the novel. This quickly changes as the story progresses as the protagonist works toward his goal.
The Conflict of a Dystopian Novel
As with any good novel, there needs to be some obvious conflict. This usually means an antagonist, and it's not unusual to have an antagonist who is a faceless entity. In most cases, the antagonist will take the form of a representative of the dystopian society. Someone in charge, someone responsible for the way things run.
Though the protagonist and antagonist may not meet until later in the story, they do interact on some level, usually through two distinct groups of people. One of these is under the direct control of the antagonist. They control and patrol the society, keeping everyone in their place. The second group is usually not under the complete control of the state or society. They are rebels, perhaps, but without clear guidance. At least until your protagonist gets involved. Where things go from there is anyone's guess.
The Resolution of a Dystopian Story
The interesting thing about dystopian literature is that at the end things are often unresolved. Yes, the protagonist and his friends may escape the clutches of the state (or not), but the state itself continues on much as it has been. But the rebellion may accomplish very little. There are exceptions, of course, but they are few. In fact, in some cases, the protagonist not only fails to change anything, but is changed himself, conforming to the expectations of society, such as in With Folded Hands. This may sound frustrating, but it rarely turns out that way. In fact, this little trait of dystopian fiction leads many authors to create additional works based on the same universe.
Dystopian fiction is popular because of its interesting narrative arc. A successful dystopian novel gives a sense of hopelessness which is carried throughout the story and is embodied in the characters that exist in the dystopian world. If you can convey this and create characters your audience can connect with, you can write a dystopian novel.
