Experts claim one royal was left 'so angry' by Harry's exit plans that he refused to attend an infamous meeting