You could. People do it. But, imho, storing pointers in an in-memory cache can be messy business.

Any user of the cache who receives a pointer for a cache hit can innocently mutate cache contents, perhaps without realizing they are doing so.