More than 5,700 staff will be moved from Whitehall to offices currently occupied by Barclays, as part of a wider push to slash the government's estate and cut costs by around £2.2bn over 10 years.
The plans will see the government occupy the whole of 10 South Colonnade by the end of 2018 because it offers "significantly lower costs" compared to other central London locations.
The move comes as banking giant Barclays looks to save £35m a year by moving its staff out of the building to its existing Canary Wharf offices at 1 Churchill Place and 5 North Colonnade.
Ben Gummer, minister for the Cabinet Office, said: "This new hub will provide a better working environment for many London-based civil servants at considerably less cost to the taxpayer.
"We will be replicating this approach across the United Kingdom, putting right the historic mistake of forcing public servants to work in ugly and expensive buildings."
The government will take the 50,354 sq m site - owned by the Canary Wharf Group - on a 15-year lease, helping it save £20m per year in running costs.
BNP Paribas Real Estate, which advised the government on the relocation, said total property costs - including rent, business rates and service charge - can be as low as £60 per sq ft in Canary Wharf, compared to £100 per square foot in Westminster.
Barclays aims to transfer its staff out of the building by the end of 2017, with the government completing its move the year after.
The bank said no jobs will be impacted as part of its plans.