Iceland has voted against a government proposal to resume talks to join the European Union, according to a public broadcaster.