The world number one held her nerve to win 6-2 1-6 6-3 and reach the semi-finals in New York.
Serena, 33, will play unseeded Italian Roberta Vinci in the last four on Thursday.
The American could become the first player to win all four majors in the same year since Steffi Graf in 1988.
Venus, 35, was playing in the fourth round at Flushing Meadows for the first time since 2010, but threatened to derail her sister’s grand ambitions.
The 23rd seed battled back to force a final set, only to see younger sister Serena capitalise on an early service break and claim her 33rd straight win at the Slams.
The Williams sisters had played each other 26 times on the professional circuit, but their 27th contest was as highly anticipated as any.
Serena might already hold all four Slam titles, but her bid to win them all in the same year has elevated tennis in the US sporting headlines.
A lengthy preceding match meant Tuesday’s night session was delayed, and expectancy rose outside Arthur Ashe Stadium as 23,000 spectators waited to take their seats for the night session.
The early excitement was tempered by Serena’s excellent start, but Venus gave a reminder of why she is a seven-time major champion by taking it to a final set.
“She’s the toughest player I’ve ever played in my life and the best person,” said Serena.
“It’s going against your best friend and the greatest competitor, for me, in women’s tennis, so it was really difficult for me today.”
Venus said: “Losing isn’t fun, but probably the most gratifying part is I’m still very excited to see Serena have an opportunity to win the four majors.”
The six-time champion began and ended the match in clinical fashion, dropping to her knees and screaming “Come on!” as she earned a match point and moments later firing down an ace.
After the first 33 minutes, Serena had succeeded in bringing a hush over the huge stadium as she began to dismantle her sister’s game.
Venus came out swinging but her younger sister made the breakthrough at 3-2, and would hit 15 winners and just two errors in a high-class opening set.
Serena has looked understandably on edge throughout her time in New York, however, as she closes in on a landmark achievement.
A double fault saw her slip 3-1 down in the second, and she looked upset as the crowd gave their backing to her underdog sister.
Venus broke again for 5-1 and came though a testing service game to force a final set that had seemed highly unlikely half an hour earlier.
Serena had prevailed in 10 final-set deciders at the Slams already this year though, and survived a tense opening game to then break with three blistering backhand winners.
There was plenty more huge hitting as Venus strained to recover the break, but Serena would not offer up another break point and clinched victory with her 12th ace after one hour and 38 minutes.
“Holding serve in the third set was huge,” said Serena. “She came out blasting and I was defending a lot. She has so much power so it was tough.”
