I suspect that the Navy may have picked left as easy to remember (Port being left in matelot - why the hell is that? Now there's a question).
Long before Port and Starboard the correct naval terms were Larboard (from loading board) and Starboard (from steering board), which on a windy storm battered deck on a high sea could quite easily be confused for one another. Potential for disaster.
The term Port for the left-hand side of a ship dates to the 16th century, but it was not until the 1840s that both the Royal and US Navies officially abandoned the term Larboard in favor of Port.
Most sources believe it is because the left-hand side of a ship was the side typically put next to the wharf or port, but it is quite possible that Port was chosen to replace the term for left because the Port was passed to the left.