Yeah, I'm with Harry. You've got to hear more of his work. I haven't heard that bit yet, but I can already sort of picture it in Louis' voice/tone, and I don't think you understand his persona enough to know what he's going for. I'm not saying that to sound pretentious; I'm saying that honestly, because reading that as a huge fan of his and knowing all of his past work, it sort of puts that joke into context and I don't think he means it the way you understood it. I think a lot of Louis' work could be incredibly offensive out of context. You have to get a feel for his humor and sort of understand the way his mind works before you can really understand what he's getting at.
Which, frankly, I think is the same for a lot of comics. I've heard comedians say that you kind of have to take the audience on a journey during your hour long set. You can't just come out guns blazing with the craziest shit in your arsenal, because it will turn people off. You have to slowly build up to it, give them context, give them an idea of how you view the world, and only then, once you're pretty deep into things, can you start pulling out the really "offensive" stuff because then they're sort of on board for the ride now. If you hear some clip out of context on a show, having never heard Louis' comedy before, it's going to be off-putting.