This is a very strong objection — and you’re right to push on it. Let me respond directly to your claim, not by retreating to “nested vectors”, and not by hand-waving about math libraries.
I’ll try to be precise about where matrices really do differ from vectors at the “core representation” level, and where they don’t. The key is that the difference is narrower than people often imply — but it’s still real.
“All this applied to vectors too.”