A Toronto-based investment research firm has pinpointed what it sees as the key risk to Canada’s housing market: Money-losing property investors.

In an informal poll of its clientele earlier this month, Veritas found that only half of those who own real estate as an investment are cash-flow positive, earning more money than the property costs them. About 18% are breaking even, while 33% said they are losing money on their investments. Presumably, they are counting on rising house prices to turn a profit; the Veritas survey showed 84% of investors don’t plan on selling for now. But if they do, it could be enough to turn the market downwards.

“Our belief is that house prices in the future will be driven by residential real estate investors, whether they are selling or buying,” Veritas said in a note to clients.

According to data from Statistics Canada released this year, about 38% of condos in Toronto and 46% in Vancouver are not occupied by their owners, meaning they are being rented out or are sitting empty. And that trend seems to be intensifying. A report last year from CIBC estimated that about half of all condos delivered to market recently were bought by investors.