The 50-metre mural on the side of the historic Dimmeys building in the inner suburb of Richmond features dozens of portraits of famous Australians.
Dimmeys commissioned artist Hayden Dewar to paint the mural in 2003 in celebration of the department store's 150th anniversary.
Dewar said the disgraced entertainer's depiction in the mural had been on his mind since Harris was convicted of indecent assault in 2014.
He approached the building's owners about replacing Harris with someone "more deserving" after learning that the section of the mural depicting the entertainer had been defaced.
"Once it starts looking in a state of disrepair it's potentially going to spread to other areas of the mural," Dewar said.
Mural features Kylie Minogue, Hills Hoist
Dewar said the mural was a chronology of memorable Australian people, places and events between 1853 and 2003.
Other people depicted include children's author May Gibbs, boxer Lionel Rose, singer Kylie Minogue and musician Nick Cave.
Also depicted are Australian inventions such as the Hills Hoist and the Holden Kingswood, as well as Melbourne landmarks such as Luna Park and the Arts Centre.
Dewar said delays and interruptions meant he completed the mural over a period of four years.
He said he didn't want to think about how many hours he put into the work.
"I think you do your best work when you don't think about the time it's taking you."
Harris replaced with author
Dimmeys sold the heritage-listed building to a developer in 2008 and it reopened as a supermarket and apartment complex in 2015.
Dewar provided the new owners with a list of candidates to replace Harris.
They settled on Joan Lindsay, author of Picnic At Hanging Rock, who Dewar said was a "well-loved" figure in the arts.
"It was also important to make it a woman, because the men on the wall probably do outnumber the women."
He said there had been some heritage-related issues to work through before he could be given the go-ahead to update the mural.
On Wednesday morning Dewar was finally able to get out his paint roller and erase Harris from the wall once and for all.
"Good riddance to bad rubbish," Dewar said as he painted