At least 23 Cambodians were injured in the incident, according to Cambodian authorities, while Thailand's military said an unspecified number of Thai officials had also sustained injuries.
The clash took place at a disputed frontier settlement which Thailand says is part of its Ban Nong Ya Kaew village in Sa Kaeo province but Cambodia says is part of Prey Chan village in Bantheay Meanchey province.
Thai authorities erected barbed wire fences in the area last month and for weeks there have been protests by civilians from both sides of the border.
Thailand and Cambodia have for more than a century contested sovereignty at various un-demarcated points along their 817km land border, which was first mapped by France in 1907 when Cambodia was its colony.
Tensions over disputed areas spiralled into a border conflict in July, when the fiercest fighting between the neighbours in decades killed at least 48 people and temporarily displaced hundreds of thousands.
The fighting ended after both countries agreed a ceasefire brokered in Malaysia on July 28, and the border has largely remained calm since.
On Wednesday, Cambodia's information minister Neth Pheaktra accused Thai officials of encroaching across the border and said they used "tear gas, rubber bullets and noise-making devices against Cambodian civilians".
Thailand's use of force was in response to provocation from about 200 Cambodian protesters, some of whom threw sticks and stones and fired slingshots at Thai officials, causing injuries, the Thai army said in a statement.
