I disagree. While the potential is there for exactly that, there's really been nothing to suggest that that will actually ever happen. Yes, there are rumors of plans to have Downey back as Tony Stark, but (a) they're still just rumors, and (b) if they're true, we won't know the context until...
Was it a matter of reading/watching it, or did you also join a club, go to conventions, read fanzines, and otherwise get involved in fandom? That might be the difference. I watched SF (and adjacent) movies and read SF books as soon as I was able to watch/read. My first convention was Boskone 6...
When the term "sci-fi" came into wide use, it was mainly used by people who weren't science fiction fans. Science fiction fans continued to use "SF". As science fiction became more mainstream, more and more people became fans and "sci-fi" became more common.
Those of us who are old fart fans of...