Police said Adam Lanza, 20, killed his mother before driving to Sandy Hook Elementary School, where he killed the children and six other women before committing suicide Friday.
"Newtown, you are not alone," the president said. "As these difficult days have unfolded, you have also inspired us with stories of strength and resolve and sacrifice."
The auditorium filled to its capacity of about 900 well before the vigil got under way. Hundreds more streamed into the gymnasium, which was used for overflow.
Obama spoke from a simple podium adorned with the presidential seal, which was set up before a plain black backdrop on a black stage. The only other color was provided by the U.S. and Connecticut flags standing in the background and a small table bearing 26 small white votive candles, one for each of the victims at the school.
Saying society would be judged by how well it fosters its children, the president said: "We're all parents. And they're all our children. This is our first task: caring for our children.
Obama asked whether America was doing enough and bluntly concluded: "No. And this must change."
"We can't tolerate this any longer," he declared. "These tragedies must end."
The president challenged the audience asking: "Are we prepared to say that we're powerless in the face of such carnage? That the politics are too hard? Are we prepared to say that such violence visited on our children year after year after year is the price of our freedom?"
He vowed to "use whatever power this office holds to engage our citizens ... to save another child or another parent or another town" the anguish of Newtown.
The president concluded by reciting the names of all 20 children who had been killed and asked God to bless them "with His holy comfort."
A White House official said Obama, who made no comment as he left the White House, personally wrote most of his planned remarks. He was accompanied aboard Air Force One by Reps. John Larson and Rosa DeLauro, both D-Conn.
The president was like thousands of other parents who have been drawn to Newtown from all over the country after the shootings.
"We can always pray for the goodness of people to just come out and try to bury this evil that's happened and hug our children a little better," Terry Polvay of Newtown said Sunday.
Tom Winter and Jay Gray orosebudf NBC News contributed to this report.