Oslo is great for swimming. Sørenga by the Munch museum is really nice, and a swim next to the Opera can also make for a really nice experience. Bygdøy has great swimming spots a bit outside of the city, and this is also where you will find the viking ship museum. For a swim in the river, there's a great spot around Nydalen. The lakes (like Sognsvann) are a bit more outside the city and can be colder, but going up there and sleeping in a hammock by the water can be a great way to combine a nice experience with saving money on hotels. If it's cold, check out Oslo Badstueforening for really affordable floating saunas.
I guess in general what to do depends a lot on your interests. If you just want to walk around in the city a bit, Grünerløkka is known to be one of the nicer neighbourhoods.
I left a while ago, and since then both the Munch museum and the National Gallery have moved locations. People say both are pretty great, so if you're into art they're probably worth checking out. Astrup Fernley is more geared towards contemporary art, which is good if you enjoy the whole "is this art" debate and all that jazz.