Christopher at graphpaper visited Governor's Island and has a great account of it. It has special resonance for him, as his father's family had lived there during the 1950s. He also has some thoughts about its future redevelopment:

The college campus proposal is a no-brainer: a more idyllic place of learning cannot be imagined, with quad-like open spaces, august halls ready for hundreds of classrooms, military barracks that are a renovation away from being dormitories, stately mansions for college administrators and professors, and a short ferry ride to Manhattan and Brooklyn for adventurous students, yet seclusion from the hustle and bustle for students concerned with intensive focused learning. That plan has my vote.

There are great events at the Governor's Island every weekend - check here to see what's happening. And the ferries from lower Manhattan are free.