I think the key aspect here is that the magnetic field doesn't do work on *freely moving* charges. That is, at any given point in time, the force exerted on the charge is perfectly perpendicular to the motion (courtesy of the Lorentz force).
In a current loop, or in a magnet, the charges are constrained in their movement. That means, the Lorentz force and the charge's motion is no longer perfectly perpendicular, with the result of work being done.
Mind you, I'm only 95% sure on this. But it seems in line with this site's explanation:
http://van.physics.illinois.edu/QA/listing.php?id=17176