A preliminary projection published by national broadcaster SRF indicated about 45 per cent of voters were in favour of the proposal and 55 per cent against.
The referendum, which was likened to Britain's 2016 Brexit vote, had put businesses on edge due to concerns it could lead to the end of free movement of labour between Switzerland and the EU, its main trading partner.
The proposal championed by the right-wing Swiss People's Party stipulated that the population must not exceed 10 million before 2050, and that if it did so for two years, Switzerland should scrap its freedom of movement accord with the EU.
Urs Bieri from polling firm GFS Bern said the vote failed because although many people were worried about the rising population, they were not convinced by the plan and worried about the possible side-effects.
"From the very beginning it has been presented as the chaos initiative. Voters were worried about negative consequences for Switzerland's relationship with the EU and for the labour market," he said.
"People are also worried about things like having enough care and health workers. Also there's a feeling that in the current international environment it's not sensible for a small country to do this."
The Swiss population already stands at 9.1 million and has grown far more quickly than in the surrounding EU. Foreigners make up nearly 28 per cent.
Polls had forecast a close outcome, and the result tallied with a final survey by pollsters GFS Bern, which had predicted the proposal would be narrowly rejected.
Still, the substantial backing for the measure sits alongside growing support for policies aimed at curbing immigration across Europe.
Opponents dubbed the plan a recipe for chaos because of the upheaval it would cause for Swiss companies, workers and Bern's ties with the EU.
They also questioned whether it was wise to clash with Brussels after a bruising 2025, when President Donald Trump slapped the highest US tariffs in Europe on Swiss goods.
Patrick Leisibach, a migration expert at think tank Avenir Suisse, said economic arguments had played a role, with people wary of how a "yes" vote would affect their daily lives.
"They wonder 'who is going to serve me at the restaurant?' and 'who is going to care for me when I get old?' It's more about personal welfare which made people reject this initiative," Leisibach said.
The unprecedented scheme to dictate population levels by law sits alongside burgeoning efforts by the political right in Europe to set tougher curbs on immigration, fed by discontent about the cost of living, weak economic growth and crime.
"If it goes above 10 million, it will become tight, and immigration should be restricted," said Helen Gulea, a 58-year-old seamstress and part-time kiosk worker in Zurich, originally from Kenya, who voted by post for the cap.
Under its system of direct democracy, the Swiss electorate generally votes four times a year on national referendums, which also require the support of the majority of its cantons to pass.
The government and parliament have urged voters to reject the SVP's so-called "sustainability initiative" as folly at a delicate time for Switzerland's export-oriented economy.
But SVP lawmaker Thomas Matter argued Swiss gains in prosperity had not kept pace with overall immigration and the country needed to step on the brakes.
