Port Jefferson was first settled by colonists in the 17th century and remained a rural community until its development as an active shipbuilding center in the mid-19th century. The village has since transitioned to a tourist-based economy. The port remains active as the terminus of the Bridgeport & Port Jefferson Ferry, one of two commercial ferry lines between Long Island and Connecticut, and is also supplemented by the terminus of the Long Island Railroad's Port Jefferson Branch. It is also the center of the Greater Port Jefferson region of northwestern Brookhaven, serving as the cultural, commercial, and transportation hub of the neighboring Port Jefferson Station, Belle Terre, Mount Sinai, Miller Place, and the Setauket communities.

The original settlers of the Town of Brookhaven, based in the neighboring hamlet of Setauket, bought a tract of land from the Setalcott Indians in 1655. The deed included the area of contemporary Port Jefferson along with all other lands along the North Shore from the Nissequogue River eastward to Mount Misery Point. Port Jefferson's original name was Suwassett, a Native American term for either "place of small pines" or "where water opens."

The first known home within the present village boundaries was erected in the early 1660s by Captain John Scott, an interesting character in Long Island's early history. This house, named Egerton, was a grand one on the western end of Mount Sinai Harbor at Mount Misery Neck. The first settler in Port Jefferson's current downtown was an Irish shoemaker from Queens named John Roe, who built his still-standing home in 1682. It remained a small community of five homes through the 18th century and was renamed to "Drowned Meadow" in 1682. The villagers voted to change its name to Port Jefferson in 1836. The village was incorporated in 1963.

Port Jefferson was originally called Drowned Meadow because the downtown area was tidal and "drowned" by the tide twice a day. Today’s Main Street was created when Capt. William L. Jones built a causeway across the 22-acre salt marsh in 1836. The marsh was then gradually filled in. Today’s Main Street was first called Jones Street. Its businesses supplemented the village’s shipbuilding industry, which was the largest in Suffolk County. Four out of ten ships built in the county were constructed here. The village was the home of coastal schooner captains, blue water sailors, and seamen. Most families had at least one or two members building ships or at sea.