Memories serve to establish some permanence to our inner lives despite the fleeting nature of subjective experience. Most neurobiological theories of memory assume that this mental permanence reflects an underlying cellular permanence. Namely, it is assumed that the cellular changes which first occur to store a memory are perpetuated for as long as the memory is stored. But is that really the case? In an opinion piece in this issue of Hippocampus, Aryeh Routtenberg raises the provocative idea that the subjective sense of memory persistence is not in fact a result of persistence at the cellular level, rather, that “supple synapses” and multiple “evanescent networks” that are forever changing are responsible for our memories. On one level, his proposal could be construed as a radical challenge to some of our most fundamental theories of the neurobiology of memory, including Donald Hebb’s proposal that memories are stored by networks that strengthen their connections to increase the likelihood of the same activity patterns being recreated at a later date. However, it could also be seen as a moderating call, a call for a greater acknowledgement of the dynamic, stochastic, and distributed nature of neural networks. In this response to Routtenberg’s article, we attempt to provide a clarification of the dividing line between these two interpretations of his argument, and in doing so, we provide some overview of the empirical evidence that bears on this subject. We argue that the data that exists to date favors the more moderate interpretation: that memory storage involves a process in which activity patterns are made more likely to reoccur, but that an under-appreciated reality is that mnemonic traces may continue to change and evolve over time.