The American world No 5 advanced to her fifth semi-final at the season finale after a determined 7-5 6-4 victory in one hour and 41 minutes.
Wiliams' win, which acts as some form of revenge for her Wimbledon final defeat to Muguruza, sees her finish second in the White Group behind Karolina Pliskova and the seven-time Grand Slam champion will next meet either Caroline Wozniacki or Caroline Garcia.
"I was happy to win the last point," Williams said. "It is not easy out here, you have to play every point well otherwise you lose the game. "I had some great coaching coming into the game, I hope we can win some more."
The first set was a tight affair with both players fighting off break point opportunities before the Spanish world No 2 claimed the opening break of serve in the seventh game of the match only to hand the advantage back straight away.
Both players then held serve until the 12th game of the match when the 37-year-old Williams, who has qualified for the WTA Finals for the first time since 2009, broke Muguruza to claim the opener on her third set point.
Muguruza began the second set with a positive reaction breaking Williams' opening service game only to once again hand the advantage straight back to the American.
Both players struggled to gain a stranglehold on their serve and despite Muguruza securing her third break in the second set to move 4-3 clear, the Wimbledon champion then lost three unanswered games in a row to fall to her second successive round-robin exit at the tournament.
Meanwhile, Ostapenko finished her maiden appearance at the WTA Finals with an impressive straight-sets victory over the in-form Karolina Pliskova, who had already booked her place in the semi-finals.
The French Open champion won the White Group dead rubber 6-3 6-1 in 66 minutes over her Czech opponent thanks in part to her greater initiative from the back of the court.
Pliskova, who had already clinched first place in the group, put in a subdued performance and suffered her first loss to the powerful 20-year-old.
"It was really tough to push myself and play my best tennis when you know you've qualified," the 25-year-old said. "I haven't ever been in this situation. I didn't know how I was going to feel and how she was going to play. I think she played amazing."
Ostapenko, who fired 25 winners during the contest, broke her opponent three times in the opening set while the second set was one-way traffic.
The Latvian secured a further two breaks of serve and grew in confidence behind her delivery to secure the comprehensive victory with a love service hold.
"I think it is great to finish the season this way," Ostapenko said. "Today I had nothing to lose as it was my last match here. I just tried to enjoy it as I really love to play here."