Iceland has voted to reject the resumption of European Union accession talks, in a referendum that swung back and forth through the night.