Some 50 people were reported killed in the town of Roche-a-Bateau alone.
The nearby city of Jeremie saw 80% of its buildings levelled. In Sud province 30,000 homes were destroyed.
The hurricane, now a Category Three storm with sustained winds of 120mph (193km/h), is heading towards the US state of Florida.
At 02:00 local time (06:00 GMT) Matthew was still off the Florida coast, centred about 37 miles (60km) east of Vero Beach and was moving north-west at about 14mph (22km/h), the National Hurricane Center said.
Senator Herve Fourcand from southern Haiti told AFP news agency that more than 300 people had died. An unnamed official quoted by Reuters news agency put the death toll at 339.
Hurricane Matthew has pounded the Bahamas after slicing through Haiti and Cuba.
Trees and power lines were reportedly down in the Bahamas but no fatalities were reported.
Most of the deaths in Haiti were in towns and fishing villages around the southern coast, with many killed by falling trees, flying debris and swollen rivers.
The storm passed directly through the Tiburon peninsula, driving the sea inland and flattening homes with winds of up to 230km/h (145mph) and torrential rain on Monday and Tuesday.
The collapse of an important bridge on Tuesday had left the south-west largely cut off.
Non-governmental organisations said phone coverage and electricity were down and people were running out of food and water.
The BBC's Tony Brown in south-western Haiti said he had seen people trying to cope with the mass destruction on their own, trying to rebuild from the rubble but without the help of the army or police.
Les Cayes resident Jean Joseph described the scene in his town - one of the worst-hit - as "complete devastation".
"What's going on right now is a lot of people are walking around," he told the BBC.
"They have no home. A lot of them - they're just walking around. I don't know what they're going to do."
Across the country, there were some 350,000 in need of assistance, according to the UN Office for the Co-ordination of Humanitarian Affairs.
A spokesperson for the American Red Cross, Suzy DeFrancis, said the first priority was to get phone networks across the country back up and running.
"We will bring in technology to help do that," she said.
"We also have warehouses with relief supplies that we will be distributing. Some of the needs that families may have are kitchen kits so they can cook meals, any kind of hygiene kits and then we are most worried about cholera, so we will be helping to distribute aqua tabs to purify the water."
The US is also sending nine military helicopters to help deliver food and water to the hardest-hit areas.
The country is one of the world's poorest, with many residents living in flimsy housing in flood-prone areas. Four people also died in the storm in the neighbouring Dominican Republic on Tuesday.
Meanwhile in the US, evacuation orders have been issued for areas covering some three million inhabitants.
In Florida, heavy rains and high winds lashed the Miami area overnight. Some 200,000 homes and businesses have been left without power in the state already.
Sustained winds could reach 130mph, with gusts of up to 160mph, with rainfall of up to 15in (38cm).
Governor Rick Scott said: "Think about this: 11ft (3.3m) of possible storm surge. And on top of that, waves. So if you are close, you could have the storm surge and waves over your roof."
Orlando's theme parks, Walt Disney World, Universal Studios and SeaWorld, are shut.
Why Haiti is vulnerable to disasters
More than half of Haiti's city-dwellers live in overcrowded shantytowns that take the full force of any earthquake, hurricane, or disease outbreak. An ongoing cholera epidemic, triggered by the arrival of UN troops after the 2010 earthquake, has killed thousands of people.
Massive deforestation has also led to soil erosion, leaving hillside huts and poorly-built houses in the capital, Port-au-Prince, dangerously exposed. In rural areas, topsoil used for agriculture is often washed away.
Political instability and corruption have been a factor. Without effective government for decades, Haiti currently ranks 163rd out of the 188 countries on the UN Human Development Index. It spends little on storm defences.