Apparently it is a known issue with Microsoft?!? There is a way to add or remove a person without all attendees being informed.

If you wish to remove someone from the list, open the meeting and manually remove their name from the To list. To add an attendee, click the To button and get their name from the address book. Click on Send update and take the option to send to those added or removed and it will not send to all.

Do not click on the Add or Remove Attendees button, or click on Scheduling Assistant to add or remove an attendee.

Mike.