Coast guard Mohamed al-Alay told Reuters the refugees, carrying official UNHCR documents, were on their way from Yemen to Sudan when they were attacked by an Apache helicopter near the Bab al-Mandeb strait.
Mohammed Abdiker, emergencies director at the International Organization for Migration (IOM), said 42 bodies had been recovered.
He said the attack at around 3:00am on Friday (local time) was "totally unacceptable" and that responsible combatants should have checked who was aboard the boat "before firing on it".
Shabia Mantoo, the Yemen spokeswoman for refugee agency UNHCR, said refugees and asylum seekers were moving out of Yemen and heading north because of deteriorating conditions.
IOM spokesman Joel Millman said 80 people had been rescued and taken to hospitals in Hodeidah.
Yemen's Shiite rebels, known as Houthis, accused the Saudi-led air coalition of carrying out the attack.
The spokesperson for the Saudi-led coalition, General Ahmed al-Asseri, said they did not conduct any operations or have any engagement in the Hodeidah area on Thursday.
He said Hodeidah remained under the control of the Houthis and the port continued to be used for "trafficking people, smuggling weapons and attacks against the line of communications in the Red Sea".
Hodeidah on the Red Sea is controlled by Iran-allied Houthi fighters who in 2014 overran Yemen's capital Sanaa and forced the Saudi-backed government of President Abd-Rabbu Mansour Hadi to flee into exile.
A Saudi-led coalition was formed in 2015 to fight the Houthis and troops loyal to former president Ali Abdullah Saleh who have fired missiles into neighbouring Saudi Arabia.