After 16 times with West Ham United, Baroness Karren Brady has left her place as vice-chair.