I don't like Affleck, but I don't think he's the main problem with the Sum of All Fears. The whole production just seems cheaper than the previous movies. All those battle scenes shown from inside the war room, for instance, make it look like an HBO original movie. The on-location sets of Clear and Present Danger and Patriot Games are so much more effective. You've also got to love how the Sum of All Fears pretends the other Jack Ryan movies never happened. Ironically, it's the one Jack Ryan movie that most fans would like to pretend never happened.
In any case, the problems go far beyond the lead actor. If I could choose between having Phillip Noyce direct the next movie with Affleck starring, or having Phil Alden Robinson direct with Harrison Ford playing Ryan again, I would choose the former. Ideally, we will get both Noyce and Ford, but that seems pretty unlikely.