Research by the Institute of Economic Affairs found the cumulative effect of traffic regulation measures "imposes an enormous burden on the UK economy".
The think tank found that just a two-minute delay to every car journey equates to a loss of approximately £16bn a year.
The report entitled Seeing Red: Traffic Controls And The Economy said: "Not only is a high proportion of traffic regulation detrimental to road safety, the economy and the environment, it also imposes huge costs on road-users, taxpayers and communities."
It states: "Traffic signals could be taken out where they cause unnecessary delays, perhaps following Portishead-style trials where lights are switched off for several weeks to observe the impact.
"Successful schemes in Drachten in the Netherlands (in 2002) and Bohmte in Germany (in 2007) scrapped over 80% of their traffic lights. Together with the Portishead experiment, this suggests a broadly similar proportion of signals could be removed in the UK."
Report author and head of transport at the Institute of Economic Affairs Dr Richard Wellings said: "For too long policy makers have failed to make a cost-benefit analysis of a range of regulations - including traffic lights, speed cameras and bus lanes - making life a misery from drivers nationwide.
"It's quite clear that traffic management has spread far beyond the locations where it might be justified, to the detriment of the economy, environment and road safety.
"The evidence of shared space schemes shows the transformational benefits of less regulated approach, whilst the removal of a high proportion of traffic lights would deliver substantial economic and social benefits."
The authors suggest there is a "strong economic case for replacing standard traffic regulation with strategies that harness voluntary cooperation among road-users".
They said: "A high proportion of traffic lights should be replaced by filter-in-turn or all-way give-ways. Many bus lanes, cycle lanes, speed cameras and parking restrictions should also go. Culling such traffic management infrastructure would deliver substantial economic and social benefits."
From 2000 to 2014 the number of traffic lights on Britain's roads increased by 25%, the report said.
Britain's first speed camera was installed in 1992, but by 2012 there were over 3,000 at 2,300 fixed sites, it added, nothing that the "rapid expansion" of bus lanes began in the late 1990s, growing from 59 miles in 1997 to 172 miles in 2007.
Responding to the report, the Department for Transport said the "safety of Britain's roads is absolutely paramount".
It added: "Road accidents come with a human cost which unfortunately, as families across the country know, is far too high.
"Local councils are responsible for managing their networks in such a way as to balance the needs of all users. We provide guidance on designing and implementing measures but it is up to the authorities to decide how best to implement them."