The match, lasting 3 hours, 13 minutes, featured 20 service breaks, with neither player finding much confidence on their service games.
The 37-year-old Williams dropped her serve nine times, while the 20-year-old French Open champion conceded serve 11 times.
"I don't know if there was any luck for me tonight, I just had to work for every point,'' the fifth-seeded Williams said. "It's not easy to lose the first match, and both of us were in the same situation. What else can you do but fight to live another day?''
By contrast, Karolina Pliskova capitalised on a fourth match point to easily beat Garbine Muguruza 6-2, 6-2. Pliskova, at 2-0 in the round-robin, is guaranteed a place in the semifinals.
"I'm just happy that I don't have to play the next match (against Ostapenko) for qualifying, because last year it was pretty stressful to do that,'' Pliskova said. "If I win a few matches in a row it gives me confidence.''
The second semifinalist from the White Group will be decided by Thursday's match between Williams and Muguruza, who both are at 1-1 in their group.
Williams and Muguruza met in this year's Wimbledon final with the Spaniard winning the title.
Williams's first match point came on Ostapenko's serve in the 10th game of the second set, but she squandered the opportunity by netting a forehand.
Ostapenko kept the pressure on to force a tiebreaker and levelled the match. Prior to this match, Williams had only lost two tiebreakers in 12 played this season.
But Williams, the runner-up at the Australian Open and Wimbledon this year, then started taking Ostapenko's second serve from well within the baseline. She broke her Latvian opponent in the 11th game by forcing her into a backhand error.
Williams, the only player in the field to have won the WTA Finals - back in 2008 - sealed victory with a backhand cross-court winner.
Seventh-seeded Ostapenko - who is making her WTA Finals debut - fell to 0-2 in round-robin play.
Photo by: GETTY IMAGES (Caption: Venus Williams plays a backhand in her singles match against Jelena Ostapenko of Latvia)