Sort of the final update which I got from MS:

Lot of discussion going between Outlook team and the VSTO team (after
we discovered the issue and pointed it to them) and they are planing
to roll the fix in a hotfix which might be relased in the near
future.

Hopefully that will resolve this issue.