I lived in the same house until I was 22. As an adult I've moved about 13 times in the last 25 years. Quite frankly I love living in different houses and would have relished the chance to try different homes as a child. Rarely do I reminsce about my childhood home. My memories relate to people, not the house. You gave you son a loving home and stability, and I'm sure he doesn't see this in the same way you do.