The vote marks a major breakthrough for the party and a fresh challenge to Germany's mainstream political establishment.