Brandi Glanville and Caroline Manzo departed the fourth season of The Real Housewives Ultimate Girls Trip ahead of schedule, three sources close to the show tell PEOPLE, after an incident deemed "inappropriate" by higher ups