By this time of year, wealthy bankers in the City of London have usually jetted off to their summer holiday destinations. Tuscany, Provence and the Balearic Islands beckon. For all its charm, Dublin seems unlikely to top many of their lists.

But senior European executives at some big Wall Street banks do have the Irish capital on their minds in at least one respect. For many of them, it is the natural destination should they need to move activities out of the UK as it drifts away from the European Union.

British prime minister David Cameron has promised to hold a referendum on a renegotiated EU membership if his Conservative party wins next May's election.

London hosts more than 250 foreign banks, many of which have based their main European subsidiaries in the UK capital and gained an automatic passport to operate across the other 27 countries in the EU single market for wholesale financial services.

If the UK were to leave the EU – the so-called “Brexit” scenario – senior bankers worry that Britain would be unable to negotiate the same passporting rights for its financial services industry. If these were lost, it would force many corporate and investment banking operations to leave the UK.

It is hard to argue with the importance of the financial services industry to the UK. According to the Treasury, the industry provided 1.4m jobs and paid £27.5bn in income tax and national insurance in 2011-12, or 12 per cent of the total. The International Monetary Fund says the UK is the largest net exporter of financial services, insurance and pensions in the world, with a trade surplus of $67bn in those industries. A third of that surplus came from trading with the EU.

In many cases, the US banks are as worried about the eurozone's impending banking union as they are about Brexit. As the European Central Bank starts to exert its authority over the main banks in the single currency zone, it threatens to leave Britain isolated. The UK has already launched four legal challenges to EU financial services regulation including the power to ban short selling; the requirement that clearing houses handling euro transactions be based in the eurozone; the financial transaction tax; and the bonus cap.

The Americans are not the only banks that need to worry about Brexit. The Swiss would also be hit, because they do not have full EU passport rights from their home market, so have chosen to set up large subsidiaries in the UK. Even British banks would face upheaval – particularly those with big investment banking operations such as Barclays.

Bank of America Merrill Lynch, Citigroup and Morgan Stanley are among those hoping for the best, but preparing for the worst by considering Ireland as a "Brexit" option. Dublin's attractions for foreign banks include its low corporate tax rate, English speaking population, English-style legal system and eurozone membership. The International Financial Services Centre was established in Dublin in 1987 and it now hosts more than 500 companies employing 32,700 people and contributing €2.1bn to the Irish exchequer.

Yet, if anything, banks have been moving operations out of Dublin recently. Goldman Sachs last year sold its Irish-based hedge fund administration business and gave up its banking licence in the country. BofA moved about $600bn worth of fixed income and derivatives assets from Ireland to the UK, partly for tax reasons, but also because of pressure from regulators in both countries.

These moves raise the question of whether Ireland would be able to cope with an influx of foreign banking activities from the UK. It has nothing like the regulatory, accounting and legal infrastructure that has built up to support the banks in London, or even Paris and Frankfurt. US banks that have looked seriously at moving some operations to Ireland reckon that if they moved their European corporate deposit books to the country, they would be big enough to be regulated by the ECB rather than the Central Bank of Ireland.

Earlier this month, London mayor Boris Johnson painted an optimistic picture of Brexit, arguing that the UK could "negotiate a generous exit securing access to the common market" that would create "a lower regulatory burden and undiminished trade access".

However, most senior executives at US banks in London disagree with Mr Johnson. They say other European countries are already trying to prise business away from the UK in what one calls “a competitive arms race”. Given the importance of financial services to the UK economy, this is a race that the City of London can ill-afford to lose.

Martin Arnold is the Financial Times’s Banking Editor