Glassfish Woe (Part 2)

I did some tests on our development server:
1) When connecting to development database, I ran a load test to have 10 users login/logout at the same time. Everything looked fine.
2) Then I switched to the production database, same load test, now I could reproduce the transaction error.

So it seems that something got changed on our production database to cause the transaction error. But it is hard to pinpoint what it was. So for now, I implemented a workaround and it seems to be working. Instead of using resource type javax.sql.XADataSource, I am now using javax.sql.DataSource. The difference is that XA type can handle global transactions while non-XA can only handle local transactions. Since we are not running in distributed environment, this seems to be working for us. And the server has been quiet over the weekend.

Content is not allowed in prolog

While working on the glassfish crash issue, I stumbled upon an error in the log:

[Fatal Error] servicetag-registry.xml:1:1: Content is not allowed in prolog.

It happens to be an extra character in the lib/registration/servicetag-registry.xml file. And it was introduced 6 months ago!

Glassfish Woe

Our website running on Glassfish 2.1.1 crashed over the weekend. We found a lot of transaction errors in the log:

JTS5041: The resource manager is doing work outside a global transaction
oracle.jdbc.xa.OracleXAException
...

When this happens, the connection associated with the transaction will become unusable, then the connection pool has to create more and more connections. At some point, it will reach the maximum, and when all the connections in the pool become stale, the site will crash.

After some research on google, it seems to be caused by a parallel transaction bug described in http://java.net/jira/browse/GLASSFISH-11920.

But the weird thing is that we didn't change anything recently and all this started to happen out of nowhere.

This really drives me nuts!