The investors being counted on to revive the mortgage market are getting off to a difficult start.

In recent months, a number of hedge funds have been dabbling in down-and-out securities linked to the fate of U.S. homeowners, including bonds backed by risky subprime mortgages. These distressed investors, as they are known, buy assets that others are afraid to touch in the hopes that prices will rise eventually. Economists and policy makers are watching their moves closely, because by putting a floor on prices, they can play a crucial role in ending the turmoil that has gripped the financial markets.

So far, though, the going has been tough. Some funds have had trouble attracting investors, limiting their impact on the $1.3 trillion market. With house prices still falling and mortgage defaults rising, putting a value on mortgage securities remains a tricky endeavor. Beleaguered banks keep dumping more securities, potentially making cheap assets even cheaper and precipitating losses for the brave few who chose to get in early.

"If somebody can pick the right spot there and can hold out long enough, there is definitely money to be made," says Karen Weaver, global head of securitization research at Deutsche Bank AG. But "there is so much concern about further supply out there...that people are just wary of buying anything."

Still, the case to buy can be compelling. On average, the prices of highly rated subprime-backed securities have declined roughly 40% since the credit crunch took hold last summer. At current prices, some securities offer double-digit yields even if 80% of borrowers default on mortgages -- worse than many analysts expect -- and only 30 cents on the dollar is recovered for every dollar borrowed, says Dan Ivascyn, a portfolio manager for asset-backed and mortgage-backed securities at Pacific Investment Management Co.

The landscape is littered with funds that took the bait and failed. London-based Peloton Partners LLP imploded in February after its bets on highly rated mortgage-backed securities soured. Shortly after, Carlyle Capital Corp., run by Washington, D.C., private-equity firm Carlyle Group, saw the collapse of its $22 billion mortgage-backed securities fund.

Jitters about the sector were on display earlier this year, when Lehman Brothers Holdings Inc. managed to raise only $100 million for its Mortgage Opportunity Fund Ltd., falling far short of the $500 million it planned. In one rough indicator of sentiment toward such funds, investors withdrew about $2.1 billion in the first quarter of 2008 from "relative-value" funds that focus on mortgage and other asset-backed securities, according to data tracker Hedge Fund Research. That compares with the $2.3 billion investors put into such funds in all of 2007, in many instances hoping to profit from a potential rebound in the sector.

Those figures, however, include funds that made negative bets on the housing market, and therefore also reflect withdrawals by investors who had turned a profit on declines in mortgage securities.

Those funds that have raised money are placing conditions on investors and taking other precautions to ensure they can hold on to securities long enough to reap the potential payoffs. One big early mover, New York money manager Marathon Asset Management LLC, requires that investors leave their money in the fund for longer than a year. The fund also doesn't use borrowed money to boost returns, a strategy that went awry for Peloton and Carlyle. With about half of Marathon's roughly $1 billion war chest in cash, its managers say they see future price dips as buying opportunities.

"You're never sure when you are at the bottom," says Marathon's chief operating officer, Andrew Rabinowitz.

It has been a bumpy ride. Marathon's Distressed Subprime Fund has declined about 2.4% since its launch last fall. At the beginning of May, the portion of the ABX index that tracks highly rated subprime securities had rebounded about 19% from the lows in mid-March, according to data provider Markit Group Ltd. It has since retreated to mid-March levels.

To figure out the value of mortgage securities, portfolio managers must pore over the sometimes thousands of loans packaged into each security, and the technical details of how each security channels payments on those loans to different classes of investors. In addition to publicly available data on house values and credit scores, managers typically use proprietary sources of information. Marathon, for example, gleans information from broker networks and data on actual home sales from a loan-servicing company that it owns, Marix Servicing LLC.

An added complication: Congress is considering new legislation aimed at making it easier for homeowners to refinance by offering government insurance on certain qualified loans. While good for homeowners, the rescue effort could establish an incentive for borrowers to miss payments to qualify for the program.

With the U.S. economy in what could be a prolonged downturn and a fresh round of losses at such big banks as Lehman Brothers and UBS AG, some money managers think the prices of many mortgage-backed securities have further to fall.

Alistair Lumsden, a senior portfolio manager at London hedge fund CQS LLP, which has a roughly $375 million asset-backed securities fund, expects mortgage defaults and the resulting losses to get worse, particularly in states such as California, Florida and Nevada, where large numbers of homeowners already have fallen behind on mortgage payments. Recent data showed a further sharp rise in delinquency rates in May.

"We think that there are limited opportunities currently," Mr. Lumsden says.

Write to Cassell Bryan-Low at cassell.bryan-low@wsj.com and Carrick Mollenkamp at carrick.mollenkamp@wsj.com

Real Estate MFs to Draw in More Foreign Money

Realty Mutual Funds: New Avenue for Retail Players

Subprime Losses May Be Overstated

Bonds Tied to Mortgages Have Hope

Mortgage Firms Cool to Principal-Cut Plan