The 85-year-old retired miner reminded the Labour Party that Britons were never offered with a referendum to extend their membership of the bloc after they voted to become a member in the early 1970s.
Mr Moss told Sky News: “We have had the vote and it were in and out. Why do they need a next one?
“They did not give us a vote. We had a vote in 74 when they took us in.
“They didn’t give us another vote. We had to abide by that and what percentage did we go in with and then they extended it and we didn’t get another vote on extension?”
When asked by correspondent Gerard Tubb why the retired miner does not think having EU membership is good for Britain, he said: “Why? Because of the promises we went in on.”
Shadow Brexit secretary Sir Keir Starmer delivered his keynote speech on Brexit at the Labour Party conference and pleaded for Britain to have another Brexit referendum.
Sir Keir said that if Labour could not secure a general election “we must have other options”.
He said: “That must include campaigning for a public vote.
“It is right that Parliament has the first say but if we need to break the impasse, our options must include campaigning for a public vote and nobody is ruling out Remain as an option.”
Shadow Chancellor John McDonnell previously dismissed holding a second referendum as he claims it will “divide the country again”.
The party will debate and vote on a motion to remain in the bloc on the table if a General Election cannot be triggered by Labour.
The final draft for the vote, written by Sir Keir Starmer, says: “If we cannot get a general election, Labour must support all options remaining on the table, including campaigning for a public vote.”