You can remember the meaning of 回去 with this easy mnemonic:
When food has decomposed or changed, it is not possible to return (回) it into the previous state. Have to go (去) with the flow. That's how new food get discovered too. Someone must have discovered blue cheese after forgetting about it.
Using mnemonics to remember the meaning of Chinese characters like can have an enormous impact on how long to learn Mandarin it takes.