Just days after UBS AG said that it would gut its fixed-income trading business last October, three of the Swiss bank's senior executives left to launch a firm to go where big banks won't anymore.

The fund, called Melody Capital Partners, is raising about $750 million to make loans or buy debt of companies and other entities with bad credit or no credit rating. Often the borrowers are in financial distress and the financing arrangements tend to be unusual in nature.

For instance, Melody recently lent tens of millions of dollars to a private-equity fund that wanted to buy more shares of a company it already had a stake in, but had all its capital tied up in other investments. Melody also hunts for cash-starved companies that have collateral to seize if loans go bad in industries ranging from telecommunications to real estate.

Once, such risky but lucrative deals fattened the profits of Wall Street banks, including UBS, Goldman Sachs Group and Deutsche Bank AG.

But five years after the collapse of Lehman Brothers turned a mortgage crisis into a full-blown financial panic, banks are pulling back their balance sheets from the fringes of the credit markets, with more and more risk being driven to unregulated lenders that comprise the $60 trillion "shadow-banking" sector.

Fragmented and opaque, shadow banking comes in many forms - from payday loans and “crowdfunding” websites on Main Street, to securitised products, money-market funds and repurchase agreements on Wall Street.

Banking regulators traditionally paid little attention to it, but when Lehman went under, shadow banking intensified the tumult: money markets and repurchase markets seized up, and investment banks and major corporations suddenly had trouble funding themselves. The financial sector, which had spent years trying to avoid regulation, began begging for government rescues.

Rulemakers acknowledge they need to fix the shadow banking sector as they reform the financial system. But their progress on that front has been halting, while regulation for the conventional banking sector has ramped up quickly and severely. In interviews, regulators, bank executives and academics said the increasing regulation of the big banks may be spurring growth in shadow banking, which in turn could be sowing the seeds for the next financial crisis.

"We haven't done as much as we should to fix this," said former US Treasury secretary Henry Paulson, who recently reissued his 2011 book "On the Brink" with a new prologue that among other things speaks to the risks in shadow banking.

“I’m not making light of ‘too big to fail’ and challenges posed by big banks,” Mr Paulson said, “but I think we’ve come a long way there, whereas we have not when it comes to dealing with some of the structural issues of the financial markets.”

While no one can predict what precisely the seeds of the next crisis will be, a big concern is that one part of the shadow banking sector will go belly up and infect larger parts of the system. Not only would it come as a shock, but it also would be more difficult to contain if regulators had no grasp of the systemic risks that were building.

According to Mr Paulson’s definition, shadow banking is non-bank short-term funding, which is used to finance longer-term obligations by financial institutions and industrial companies. Therefore, funds that use long-term investor money to make long-term loans are less of a concern than money-market funds or repurchase agreements, which provide short-term cash to corporations and financial firms.

To be sure, the riskiest parts of the shadow banking system may still be too small and too diffuse to create systemic risk. And regulators globally plan to collect data on now-opaque segments of the financial markets, and enforce tougher rules on a host of areas including credit funds, exchange-traded funds, hedge funds, private equity funds, securities broker-dealers, credit insurance providers, securitisation and finance companies.

Some shadow banking segments have already started shrinking. For example, the size of the US market for repurchase agreements is down 3 per cent to $1.65 trillion since 2010.

Fannie Mae and Freddie Mac, the government-controlled mortgage finance giants that now back nearly 90 per cent of new home loans, have also been shrinking their balance sheets, as mandated by terms of their rescue in the financial crisis. There is no blueprint yet, however, for what will replace them.

Joseph Abate, a money market strategist at Barclays, said he expects most major shadow-banking markets to shrink further as new regulations are devised.

“If you talk to the regulators, they would generally acknowledge that the system is safer but it could be even safer,” Mr Abate said. “If you talk to the industry, you would get the view that enough has been done.”

But even as some shadow banking areas shrink, others are growing. For example, the market for collateralised loan obligations, or CLOs, is on track to double in 2013 from last year, with $51.2 billion having been sold so far this year. These securities package pools of loans into bonds that are sliced up and sold to investors.

The precise size and number of middle-market credit funds like Melody Capital Partners is difficult to estimate, but many such funds have launched lately, or are raising money.

Since the financial crisis, 232 hedge funds that perform direct lending have launched - 3.5 times as many as had ever launched before then, according to the research firm Preqin. Those funds have performed better and shown less volatility than the overall hedge fund industry, Preqin data shows.

Firms including Blackstone, Cerberus and Avenue Capital, have all raised or are in the process of raising sizeable middle-market loan funds. Business development companies, which make loans to small companies and have to pay little corporate tax, have also grown at a rapid clip since the financial crisis. Prospect Capital Corp, which makes $5 million to $100 million loans to middle-market companies, for example, had grown its assets to nearly $4.5 billion in June 2013 from about $500 million in June 2008, representing an annualised growth rate of over 50 per cent.

“Prospect and other non-bank lenders are growing at the expense of banks for regulatory reasons that are not expected to abate in the years to come,” said Grier Eliasek, Prospect’s president.

Bill Winters, a former JPMorgan Chase & Co executive, said shifting risk from regulated banks to unregulated entities is not necessarily a bad thing. Winters launched Renshaw Bay in 2011, a firm that makes investments in credit in areas where banks are withdrawing.

“This is all very healthy,” he said. “Banks are supposed to be central to the transmission of credit, but not be the only game in town.”

But others are alarmed. Steven Schwarcz, a professor of law and business at Duke University, said the Dodd-Frank financial reform law - the main regulatory reform package introduced in the US as a result of the financial crisis - fell short by focusing primarily on banks.

“Dodd-Frank hasn’t addressed financial market risk,” said Schwarcz, who has written papers on shadow banking and systemic risk.

The growth comes as banks become more cautious. Several banks, including Goldman Sachs and Morgan Stanley have put in place committees that must approve new products and certain transactions to avoid introducing inappropriate risks to the firms or to their clients. Big universal banks like JPMorgan and Citigroup have pulled out of lending businesses such as student loans.

Regulators, bank executives and others say that the idea of unregulated entities stepping further into banks’ traditional businesses is troubling.

"We should have no confidence that the system is safe," said Richard Carnell, associate professor of law at Fordham University Law School and a former assistant Treasury secretary for financial institutions under president Bill Clinton. (Reuters)