The company said it had to face the "stark reality" that the service was making "significant losses."
The Larne to Troon route had been operating between Northern Ireland and Scotland since 2003.
P&O said it would continue to operate its Larne to Cairnryan service.
The company said the decision to end the Troon service, which did not operate in the winter months, came after a "comprehensive review."
"P&O Ferries has proudly operated this additional seasonal route since 2003, but the stark reality is that the company is continuing to make significant losses, and sadly the income from ticket sales is not sufficient to cover the annual vessel and port operating costs," said a company statement.
"In a final attempt to make the route economic the company reduced the service in the off-peak months last year, whilst maintaining the full service in the peak months of July and August. Whilst this measure did significantly reduce the operating costs, the route remained loss-making.
"Since the last sailing at the end of September, P&O Ferries has continued to negotiate with a number of suppliers and other organisations, including the search for a lower cost chartered fast-craft to be able to continue, but we have been unable to develop a solution for the company.
"There will be no redundancies, with all permanent staff being found alternative roles within the company."
Our BQ Bulletin emails will land in your inbox at 7.30am, Monday to Friday, with a mix of the latest local business news, national news, and features to inspire you. Sign up here!
Click here to read our privacy statement