Because games are games and books are books.
The idea of books is to read whatever the writer wanted to say. The story is fixed, the ending is fixed, you experience it without any control over it. Others are the heroes, you are just a side observer. In games and especially in RPGs the player has control, the player is the hero, the player decides what to happen in many situations.
In this aspect, for me books written in game universes are an insult to the player, treating him like a fanboy, milking him for cash. Not to mention that I cannot imagine any good, self-respecting author to agree to write books in an already established game universe instead of creating his own; meaning that such writers are probably mediocre, the ones who couldn't "hack it with the big boys".
But that's just how I feel.
Even though you get some agency in a game the developers still have a story to tell, all the player gets to do is decide how the story is told.
Also are you say guys like Dan Abnett, Greg Rucka and Warren Ellis are mediocre, really?
So do you think all in-game lore books are a waste of time?
I've never heard of any of those guys, so I'm guessing they are not top class.
I don't read in-game lore books much if that's what you ask. I don't understand what exactly do you mean by "waste of time". I know people who read them, so it's generally better that they exist.