Port Jefferson was first settled in the 17th century and remained a rural community until its development as an active shipbuilding center in the mid-19th century. The village has since transitioned to a tourist-based economy. The port remains active as terminus of the Bridgeport & Port Jefferson Ferry, one of two commercial ferry lines between Long Island and Connecticut, and is supplemented by the terminus of the Long Island Railroad's Port Jefferson Branch. It is also the center of the Greater Port Jefferson region of northwestern Brookhaven, serving as the cultural, commercial and transportation hub of the neighboring Port Jefferson Station, Belle Terre, Mount Sinai,Miler Place, and the Setaukets.

Incorporated in 1963, Port Jefferson was originally called Drowned Meadow because the downtown area was tidal and "drowned" by the tide twice a day. Today’s Main Street was created when Capt. William L. Jones built a causeway across the 22 acre salt marsh in 1836. The marsh was then gradually filled in. Today’s Main Street was first called Jones Street. Its businesses supplemented the village’s shipbuilding industry which was the largest in Suffolk County. Four out of ten ships built in the county were constructed here. The village was the home of coastal schooner captains, blue water sailors and seamen. Most families had at least one or two members building ships or at sea.

Port Jefferson was originally called Drowned Meadow because the area was tidal and was “drowned” by the tide twice a day. East Main Street followed the edge of the salt marsh, and was the original “Main Street”. The Villagers voted to change its name to Port Jefferson in 1836. The Village was the largest shipbuilding center in Suffolk County in the 1800’s with four of every ten ships built here. The Village incorporated in 1963.

The original settlers of the Town of Brookhaven, based in the neighboring hamlet of Setauket, bought a tract of land from the Setalcott Indians in 1655. The deed included the area of contemporary Port Jefferson along with all other lands along the North Shore from the Nissequogue River eastward to Mount Misery Point.

Port Jefferson's original name was Sowasset, a Native American term for either "place of small pines" or "where water opens.

The first known home within the present village boundaries was erected in the early 1660s by Captain John Scott, an important leader in Long Island's early history. This house, named Egerton, was a grand abode on the western end of Mount Sinai Harbor at Mount Misery Neck.

The first settler in Port Jefferson's current downtown was an Irishshoemaker from Queens named John Roe, who built his still-standing home in 1682. It remained a small community of five homes through the 18th century, and was renamed to "Drowned Meadow" in 1682.