I accidentally A-Z challenge twice, is it bad? So this is individual post :D
Monsters in fantasy games are every day encounters what trouble villages, block dungeon paths and give challenge and experience points for players. In my opinion monsters are a bit too common in fantasy games to be really terrifying anymore. Usually monster becomes terrifying in fantasy games if:
- It's really huge, final boss type.
- It's something player's haven't met before and in meta-game players don't know how much damage it does, how hard it is to hit, or what abilities it has.
Other monsters are just there to kill. One eyed octopus with visible brain? Kill it. Orc raiders with human skulls hanging on their armor decoration? Kill them. Woman with snake lower body? Kill it.
But how monsters could be more horrible?
Low Fantasy Setting
Fantasy games, what are more fantasy medieval and human are the main race, monsters could be more terrifying. If in game characters don't posses all these special abilities, but are more "normal" but more skilled people who encounter mostly "normal" challenges and antagonists, monsters might be more terrifying. If seven game sessions have been fighting against hungry wolf pack at wilderness, defending against raiding tribesmen, hunting down rogues down the city streets... a strange mutant in cave could be more terrifying. If game master is not shuffling his monster manual every few minutes during the campaign, but once reaches on it, it could mean some serious business.
Other method is, don't shuffle your monster book at all. Just describe something horrible. Players don't know if it is something you already have prepared for in advance taking from monster book, or is it something unique you have created for this encounter. Monsters you create are more terrifying than monsters you can find from some book.
One example is Praedor. Praedor is low-magic fantasy setting, where every single monster is unique. Every monster player characters encounter are created individually. Even smaller ones can be dangerous, but if you meet a pack of human sized monsters what are intelligent enough to use weapons, or meet a monster size of house... it is scary. Also, monsters don't roam the countryside. They live behind the borders of known world in the infinite corrupted labyrinth of ancient now forgotten city where wild and strange phenomenoms of wild magic, ancient temples with residential demons and corrupted and twisted monsters live. Sometimes one or two of these could wander by accident to known world...
Another good example of terrifying monsters could be LotFP. As far as I know, there is no monster list but Raggi encourages game masters to create their own monsters. Cannot tell you more about LotFP, because haven't read it but it sounds really good OSR game.
What is great about Call of Cthulhu is that there are monsters. Smaller ones you might fight against, and huge enough to make you almost instantly insane. You might pack yourself with shotguns and dynamites, but they are no help if you fall into madness before you have your own turn. Most of the bigger Cthulhu monsters are the ones you cannot defeat at all. They are ancient, immortal, but still they might be the plot of the game.
Monsters Of World Of Darkness
In several WoD game lines you actually play a monster yourself. Werewolf or vampire for example. But still, even though humanity considers you a monster, you have your own monsters. Vampires are vulnerable to werewolven, werewolven fight against corruption of spirit world. Even your own kind can be monster for you.
Also, if you play mortal in WoD, you might find out that world and reality are not what you are used to. Humans are not top of the food chain. There are more powerful beings, what can be evil in you view. If you play a mortal, you don't care what clan or bloodline this one vampire antagonist is. It's a freaking vampire!
Also for human monsters (or what human is left in them) I suggest you to look at Hunter: the Vigil sourcebook Slasher. These are the monster serial killers you see in the movies from Jason, Freddy, Norman and Hannibal. And as WoD games are personal horrors, what if you are the monster yourself for yourself (V:tM humanity).
Reality Is A Lie
What if this world was just an illusion and behind the curtains of what we see is the real world? Ruins of times of gods and angels what is now corrupted and really messed up. What if the curtain of reality rips off and you get to see the real truth? Kult has one of the best settings ever. The reality behind this illusion is truly horrible where you might get lost. And Kult monster gallery? It is twisted and horrifying.
Zombies, All Time Favourite
Who doesn't like zombies? Lots of people must like, as zombies are present in various games. Fantasy games have zombies, horror games have zombies, even science fiction game Fading Suns has its version of zombies. And there's a game about zombie scenarios (All Flesh Must Be Eaten). Zombies are one of basic monsters. They are dead people who walk again with their rotten corpses wanting for flesh. They might be cursed, magically raised, toxic catastrophe aftermath, anything. But are zombies terrifying? It depends. If zombies are just another monster gets on the way of dungeon crawling group, they might not be that scary. But read CoC example play with a zombie, make whole city full of zombies in AFMBE, put zombies where they don't setting wise belong. Make them tough and hard to kill. Describe them how they sniff the air when player characters are hiding. Make them block important route with a mass of rotting flesh. Make huge horrific mutant zombies size of a building constructed from several dead bodies. Make them faster than running men.