Seeded ninth, Nadal looked in full control when up two sets to love before Monfils launched a comeback which fell short 6-3, 6-3, 4-6, 6-4.
Nadal next faces third-seeded Canadian Milos Raonic.
It will be the 30-year-old's ninth appearance in the final eight at Melbourne Park, having won the title in 2009.
It is also the former world number one's first grand slam quarter-final showing since 2015.
Nadal took advantage early of Monfils' inconsistent play, but Monfils showed there was more to his game than flamboyance as he made the veteran earn the win.