Kanye isn't a rapper, he is an artist. I agree that Kanye is an average rapper. The College Dropout, Late Registration, and, to a lesser extent, Graduation are just good albums, not much more. But each of his last 4 albums all are completely genre-redefining, innovative, and continue to be the best thing ever put out in their sub-genre.
I obviously can see liking one band/artist better than another. But that isn't (or at least shouldn't) be the standard that is being discussed if looking for the "greatest." It is just objectively true that Kanye is the most important artist of the 21st century.
It's kind of like arguing that Seurat was a greater artist than, say, Cezanne. They did different things, sure, and I actually really love the whole pointillism thing, probably more than most Cezanne stuff. But Seurat just doesn't matter to the history of art really at all, whereas Cezanne truly saw where art was going and pushed it forward.