Alright, let me address it this way
Cannot ignore random folders
When you say "referenced" in composer.json, I assume you mean require and require-dev.
I don't think it is correct to ignore random folders, if I require package A and it requires package B, C and D, B, C and D are not going to appear in composer.json, but language server cannot just ignore B, C, D, correct?
Dev packages not placed in vendor manually
The reason I place my packages in /vendor folder is so that composer can help take care of putting them in the right place, it is completely automated. What I would say is to just treat my project like a normal one.
Anyway, I just checked again and I no longer have the red underline/strange error, will get back again if I do. Thank you.