As always, thanks for the great feedback!
First I will have to investigate the relationship between laggyness and big phppub.user file. There shouldn't be any, but in your case it seems there is.
Second, there is Publishable property on folders, but vendor is an exception. It always published, I will take a look what we could do about it.
Third, publish should upload just the change files, but it checks all the files. We will optimize this and really take just the changed subset of files. This should speed things up considerably.