Icelanders have voted in a pivotal referendum on whether to reopen European Union membership talks, with polls suggesting the result will go down to the wire.