High oil prices will dominate the energy landscape of this decade as much as low oil prices punctuated the 1990s. Not as high as the $50 plus change currently needed to buy a barrel of Brent, but senior oil analysts don't expect oil to drop below $30 a barrel for the rest of this decade.

The US brokerage firm Sanford Bernstein has revised upwards its oil price forecasts to $40 a barrel in 2004, $35 in 2005 and $32 in 2006.

"Oil prices are likely to inflate by 2.5 per cent each year from $32 in 2006 out as far as 2010," says Sanford Bernstein associate analyst Mr Ben Dell.

But it isn't just concerns about supply disruptions from key producers Saudi Arabia and Iraq, among others, that has prompted this bullish assessment of the market, but deep structural flaws in the industry.

Oil demand is surging, mostly on the back of spiralling consumption in developing countries, particularly China and India. But there are still enough global supplies to meet demand.

"Saudi Arabia is sitting on spare capacity, but the problem is it's the wrong type of oil," says Mr Aiden Bradley, London-based global oil sector specialist at investment bank Citigroup.

"That is heavy oil and there are just not enough complex refineries needed to break this type of oil down into gasoline, diesel and jet fuel. The bottleneck in the industry comes from huge under-investment in the midstream."

The midstream sector comprises the refineries and pipeline infrastructure needed to transport and convert oil from the upstream sector (production from oil fields and rigs) to the downstream sector (petrol pumps and other end users).

The last wave of investment in the midstream was in the early 1980s. Supply soon exceeded demand, causing a collapse in refining margins. Consequently, barely a new refinery has been built in the intervening period.

Now there is a global shortage of diesel, gasoline and jet fuel, which is putting a solid floor under oil prices. But there is also a global shortage of the oil refineries needed to make these products. Most of the extra oil Opec has recently put on the market is heavy oil and the current global refining infrastructure is incapable of converting this blend of oil into the products needed.

Because of these tight fundamentals, any exogenous shocks - bombings in Saudi Arabia, escalating violence in Iraq or labour unrest in Nigeria - will trigger disproportionate bouts of volatility in the market, says Mr Bradley.

How to rectify the structural imbalance has prompted a round of finger-pointing in the industry. The heads of western oil companies, Royal Dutch/Shell, ExxonMobil, BP and Total, have called on Opec to open the taps and ramp up supplies. But this just shifts the blame, Bradley says.

"Why should Opec countries invest another $30-$40 billion to create extra capacity and get the right type of oil on the market, when they have such big financial constraints themselves."

Most Opec countries are single commodity economies. De facto Opec leader Saudi Arabia needed $25 a barrel to balance its domestic budget prior to the recent slide in the value of the dollar, which was the mid-point of the official Opec trading band of $22-$28 a barrel.

Since the tumble in the dollar, the kingdom now needs $30 a barrel to balance its books, which is why the official trading band has been abandoned in favour of an unofficial band of $26-$34 a barrel, say industry observers.

Western oil companies have made record profits over the past few years and could easily build more complex refineries with extra midstream capacity, which would smooth out many of the current bottlenecks in the industry and cause a considerable easing in oil prices. But western oil companies won't fork out the billions needed in midstream investment, because returns are far more attractive in the upstream division, says Mr Bradley.

Oil companies tend to shift the bulk of their investment to increase oil production in periods of high oil prices, because it has a much bigger impact on profit margins than adding extra refining capacity or building new petrol stations.

But an unwillingness to invest in the midstream sector isn't the only reason western oil companies want Opec to increase production.

"Opec countries cannot meet quotas. Now they have to invest to make sure that they have extra capacity and have a margin to control the market," according to Mr Christophe de Margerie, upstream head of French firm Total at his company's strategy review presentation. "But first they have to convince themselves that Opec countries are part of the world system. It is not just about balancing budgets."

Mr De Margerie was calling on all Opec countries to remove barriers to foreign investment in order to increase investment and lift capacity.

But calls by Mr de Margerie and other western oil company executives for Opec to increase capacity isn't motivated entirely by concerns over surging oil prices.

Oil is a finite resource, but reports that it will run out soon are wide of the mark. However, non-Opec reserves, with the exception of Russia and deepwater Africa, are dwindling rapidly. That is why western oil companies, faced with a dearth of growth opportunities in OECD regions, need Opec countries to open up their reserves to foreign oil companies.

So far this access has largely been denied. And as long as Opec countries reap the benefits of high oil prices, they won't want to share the spoils with anybody else.

"But to conclude that Opec is happy with prices at these [ $50 a barrel] levels is wrong. They know if prices continue at these levels, then it will cause an economic slowdown and people will look to alternative fuels. If prices drop to $30 a barrel that incentive to look elsewhere disappears and these countries are still making enough money," says Mr Bradley.

Without access to Opec acreage, western oil firms have to rely on ultra-deepwater exploration and non-conventional oils such as Canadian tar sands. But these types of projects are highly capital intensive and have much higher cost bases than traditional shallow water and onshore drilling plays.

This is the double whammy for oil consumers. Opec countries are happy if prices settle in the low $30s, but western oil firms need oil prices at these levels to maintain healthy profit margins. "That, as much as the refining imbalance, is why oil prices will remain high until Opec opens up reserves or new frontier basins are discovered," says Mr Dell.

John Walsh is an energy journalist based in London