As a reader, I find it enormously irritating when a writer refers to a book, movie, fictional character, or tv show I am either unfamiliar with or I can’t remember well enough to ‘get’ the reference. I feel stupid, because I don’t know what the pun is about, and I feel that the writer is making an implied statement about my intelligence.
The reality is, there are many more books than I have time to read. Even if I’ve read a book, there’s a fair chance I won’t remember it that well, especially if it was long time ago.
I think as writer you should play this on the safe side, and assume very little about a reader’s familiarity with aspects of culture.
Some considerations about readership: English-speaking countries have a large immigrant population, who often haven’t had the same exposure to English classics. Even different English-speaking countries have radically different tv shows. If you’re talking classics, I think many people would have read them under sufferance in high school, and not paid all that much attention. Younger or newer readers have a different starting point. For example, I’m too young to have been around in Heinlein’s heydays, and while I’ve read some of his work, I prefer to read more current publications.
When a cultural reference is necessary, I’d try to do it in such a way that those unfamiliar with the reference can understand and maybe learn something. You don’t really want the reader to feel stupid.