A generic burst of classical gravitational radiation will cause an array of freely falling test masses far from the source to experience a permanent displacement, called the "Gravitational Memory Effect". This effect is responsible for the, previously unexpected, enlargement of the asymptotic symmetry group from the usual Poincar Group to the BMS group. Furthermore, Memory (and its field theory analogs) are responsible for all infrared divergences in any formulation of Quantum Gravity as well as in Quantum Field Theory. Hilbert spaces containing states with differing Memories are unitarily inequivalent to each other and, consequently, "out" scattering states live in an uncountably infinite set of unitarily inequivalent Hilbert spaces (one for each Memory). The longstanding "infrared problem" is the construction of an "IR-finite" S-matrix by constructing "in" and "out" Hilbert spaces which (1) includes states with Memory, (2) is separable and (3) unitarily implements the BMS group. In this talk, I will describe these classical and quantum connections from a geometric and intuitive perspective. I will then clarify that while there is an essentially unique construction of such a Hilbert space in QFT using known as "Faddeev Kulish dressing", this construction cannot be applied to Quantum Gravity. In Quantum Gravity, we prove that conditions (1) and (2) imply that the Hilbert space is a Direct Integral (a generalization of the Direct sum) with some choice of infinite-dimensional measure on the space of Memories. We prove new constraints on the space of allowed Memories and that there are no BMS invariant Gaussian measures which satisfy these constraints. In totality, our results suggest that the "in" and "out" states are more appropriately described in the Algebraic framework. I conclude by commenting on the implications of our results for recent holographic proposals.