The Pilgrims had looked set for at least a point when Gavin Allott headed home in the 80th minute, before striking the post a minute from time.
Jack Sampson had handed Port a 16th minute lead and they nicked all three points when a combination of Ryan Astles and United's James Jones saw the loose ball bundled home from a long Reagan Ogle throw-in five minutes into added time.