You can miss it and still not go back.
Missing something doesn't mean it was right for you.
You can miss a place and know you'd never move back. You can miss a person and know you were right to leave. You can miss who you were back then, and still be glad you're not that person anymore.
We tend to think feelings are instructions. Miss it → go back. But missing is just love remembering. It's allowed to visit without being obeyed.
So let the train move. Look out the window if you need to. The house getting smaller behind you doesn't mean you made a mistake. It means you're going somewhere.