A Congress leader said they have nothing against the Speaker but avoided joining the meeting as nothing substantial was discussed