That carpeted wall you are thinking of leaving out, at least on the starboard side, is protection for your gas filler and vent lines from anything that gets stuffed into the rear compartment - not just cosmetic. May protect your bilge line as well if it is routed through there.

I'd strongly suggest putting it back in place and deal with a crumpled bag.