You do not need to remove it, but you do need to stow it (as per Crane Man's post above) if you're going to be driving over about 40mph or expect any strong winds.

And yes, it's a pain in the arse to put up and take down. Let's just say it's my least favorite "feature" of my boat. It really takes some practice and best when done w/2 people.