Smart injured his right ankle in the final minute of the first half and had to be helped to the locker room.
Williams stayed in the game after teammate Jaylen Brown collided with his left knee in the opening quarter, but Williams did not return for the second half because the knee was hyperextended.
Toronto led 57-50 at the break.
Williams had left knee surgery last March but returned in the playoffs.
The Celtics were already without leading scorer Jayson Tatum, who sat because of a sore left wrist. Tatum is averaging 31.2 points, the third-highest total in the league, and a career-high 8.5 rebounds.
The Associated Press contributed to this report.