Yes, except until that point he (and the rest of the multiverse) didn’t know that the infinity stones existed elsewhere and were being used as paperweights.

So the use of the word “infinity” to describe them couldn’t have been a reference to how many of them there were.