Making sure we are using Narayana

Notice how “Narayana (JTA)” has been added to the “Selected Dependencies”.

Making sure we are using the right version of Narayana

Now, we need to make sure we’re running the latest version of Narayana. There are a number of options to do that as explained in a post on a Spring blog. But in this case, overriding version property is the easiest option:

<narayana.version>5.5.1.Final</narayana.version> 

<jboss-transaction-spi.version>7.5.0.Final</jboss-transaction-spi.version>

You can see this in the applications pom over here.

Observing the transaction demarcation

In this application, both buying and selling operations are transactional. Buy method looks like this:

And sell method looks like this:

Note on Artemis

Artemis broker is not available on Spring Initializr and to make our application self contained we would like to use an embedded broker. To do this we add the following dependency into the applications pom:

<dependency>

    <groupId>org.apache.activemq</groupId>

    <artifactId>artemis-jms-server</artifactId>

</dependency>

Everything else is quite self explanatory. Go ahead and try it out over here.
If you have comments or feedback please do contact us over on our forum.

ACID

• Atomicity signifies all or nothing. If one operation from the set fails then all others have to fail too. Looking from a different angle we can call the property abortability[1].
• Consistency states that system is consistent before transaction starts and after it finishes. The definition of consistency varies from resource to resource. In database systems, we can consider it as not breaking any constraints defined by the primary key, foreign key, uniqueness etc. But being consistent in broader term is understood having the application in the consistent state. That is not maintained by the transactional system itself but by the application programmer. Of course, the transaction helps to defend it.
• Isolation refers to the behaviour of concurrent operations. The ACID definition defines it as transactions being serializable - the system behaves like all transactions being processed in a single thread one by one. Such implementation is not required and it widely differs[2]. Besides such behaviour (even pretended) brings performance implications. It’s usual that isolation property is relaxed to comply with one of the isolation levels with t lower guarantee than serializability[3].
• Durability means that if the transaction commits all data are persisted and even the system crashes it will be accessible after the restart.

Two-phase commit (2PC)

Saga

• Backward recovery - when a running operation fails then compensating action are executed for any previously finished operation. The compensation actions are executed in reverse order than the operations were run in.
• Forward recovery - when the running operation is aborted but in the following step, it’s replayed to finish with success. The benefit is that forward progress is ensured and work is not wasted even when the failure occurs - the particular operation is aborted, replayed and Saga continues from that operation forward.
  You don’t need to specify any compensation logic but you can be trapped in an infinite retry loop. This recovery is defined as the pure forward recovery in the paper[4].
• Forward/backward recovery - for sake of completeness the paper[4] defines the way of combining the forward and backward recovery. The application logic or the saga manager defines save-points. A savepoint declares a place in the Saga processing where operations can be safely retried from. When the running operation fails then compensation action is executed for previously finished operations up to the defined save-point. From there, operations are replied with a try for finishing the Saga successfully.
  We say that the forward recovery defines save-points after each of the successfully finished operation.

1. First saga manager starts Saga
2. The business logic sends a message to the JMS queue. This business operation is attached to Saga which brings obligation from to define a compensation handler (a chunk of code which is executed in case of Saga needs to be aborted).
   Sending of the message is wrapped into a short local transaction which ends when JMS message is acknowledged by the receiver.
3. The business logic inserts a row into the database table - again, the operation is attached to Saga, the SQL insert command is covered by short local transaction and a compensation handler has to be defined to undo the insertion in case of abortion of the Saga
4. If everything goes fine this is the end of the Saga and business logic can continue with another saga.

• As database insertion fails the ACID resource-located transaction reached an error state and has to be rolled-back by the resource. The error is returned to the Saga.
• The saga manager is responsible for data integrity being preserved which is ensured by aborting the whole Saga which means executing compensating actions for previously completed business operations. The compensation action for database insertion is not needed as the local transaction has been already rolled-back. But compensation action of sent JMS message is waiting to be executed.
• The compensating action is inevitably bound to the business context, thus we have to define at least a simple one.
  Let’s assume that sending of the JMS message meant placing an order in a warehouse system. Now the compensating action is expected to undo it. Probably the easiest way to model such cancellation is by sending a different message with cancel command to the JMS broker queue. We expect there is a system listening to messages which process the real order cancellation later on.

Sagas and ACID

Blending Sagas and ACID

Where does Narayana fit in

1. Ordering of invocations on participants in the transaction

 // LRA completion pseudo-code:  
 for (Participant participant : lra.getParticipants()) {  
  if (!participant.complete()) {      
   for (Participant participant : Collections.reverse(lra.getParticipants())  
    if (!participant.compensate()) {  
     // hand this to recovery manager but continue  
    }  
   }  
   break;  

2. Idempotency

3. Structure

4. Developer-friendliness

5. Locking

6. Orchestration

"Vert.x is the leading JVM-based stack for developing asynchronous, event-driven applications. Traditional ACID transactions, especially distributed transactions, are typically difficult to use in such an environment due to their blocking nature. However, the transactional actor model, which pre-dates Java, has been successful in a number of areas over the years. In this talk you will learn how they have been integrating this model using Narayana transactions, Software Transactional Memory and Vert.x. Michael will go through an example and show how Vert.x developers can now utilise volatile, persistent and nested transactions in their applications, as well as what this might mean for enterprise deployments."

1. Firstly I showed how to write a simple Vert.x flight booking application that maintains an integer count of the number of bookings.
2. Run the application using multiple Vert.x verticle instances.
3. Demonstrate concurrency issues using parallel workloads.
4. Fix the concurrency issue by showing how to add volatile STM support to the application.

Saga - what we are talking about

NOTE: From ACID point of view the Saga transaction relaxes the I (isolation) and because of that gets availability and is representation of BASE principle, see http://queue.acm.org/detail.cfm?id=1394128.

LRA - Long Running Actions

1. A client makes a call to the first service (Microservice #1)
2. The Microservice #1 is responsible for starting LRA (creating the saga). It calls LRA coordinator to endpoint starting LRA. The coordinator announces lra identifier in response.
3. The Microservice #1 enlists itself with the created LRA by calling coordinator along with the particular LRA identifier and handing over addresses of REST endpoints for compensation (optionally completion) callbacks. Those are endpoint the coordinator can call back to Microservice #1.
4. The Microservice #1 takes the LRA identifier and adds it as an HTTP header (long-running-action) to the REST call to the next service - Microservice #2. If the Microservice #2 distinguishes the LRA header it can enlist itself (by announcing REST endpoints for compensation/completion callbacks) to coordinator.
5. On way back the first service is responsible for finishing saga by calling close (when finishing with success) on the LRA coordinator with the saga identifier.
6. Some of the other services could fail saga by calling cancel on the LRA coordinator, in which case the close won't succeeds and reports back an error.

Code examples

• https://github.com/ochaloup/ola/blob/lra/src/main/java/com/redhat/developers/msa/ola/OlaController.java#L94
• https://github.com/ochaloup/aloha/blob/lra/src/main/java/com/redhat/developers/msa/aloha/AlohaVerticle.java#L141
• https://github.com/ochaloup/bonjour/blob/lra/lib/api.js#L72

Transactional aware JDBC connections

The word multiple is important here as if you want to run only single database connection you are fine to use local JDBC transaction (Connection.setAutoCommit(false), https://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#setAutoCommit(boolean)). But if you want to manage transactionally multiple JDBC connections (different databases, running at different database accounts...) then you need transaction manager.

Managing the transaction enlistment on your own

// here we get instance of Narayana transaction manager
TransactionManager tm = com.arjuna.ats.jta.TransactionManager.transactionManager();
// and beginning the global transaction for XAResources could be enlisted into
tm.begin();

// getting DB2 datasource which then provides XAResource
XADataSource dsXA = neworg.h2.jdbcx.JdbcDataSource();
// the xa datasource has to be filled with information to connection happens, using setters
dsXA.set...();
// from XADataSource getting XAConnection and then the XAResource
XAConnection xaConn = dsXA.getXAConnection();
// transaction manager to be provided with the XAResource
tm.getTransaction().enlistResource(xaConn.getXAResource());

// the business logic in database in the transaction happening here
PreparedStatement ps = xaConn.getConnection().prepareStatement("INSERT INTO TEST values (?, ?)");
ps.setInt(1, 1);
ps.setString(2, "Narayana");

// statement executed and transaction is committed or rolled-back depending of the result
try {
  ps.executeUpdate();
  tm.commit();
} catch (Exception e) {
  tm.rollback();
} finally {
  xaConn.close(); // omitting try-catch block
}

Managing the transaction enlistment with use of the JDBC transactional driver

<dependency>
<groupId>org.jboss.narayana.jta</groupId>
<artifactId>narayana-jta</artifactId>
<version>5.7.2.Final</version>
</dependency>

<dependency>
<groupId>org.jboss.narayana.jta</groupId>
<artifactId>jdbc</artifactId>
<version>5.7.2.Final</version>
</dependency>

XADataSource provided within Narayana JDBC driver properties

// XADataSource initialized to be passed to transactional driver
XADataSource dsXA = new org.h2.jdbcx.JdbcDataSource();

dsXA.set...();

// the datasource is put as property with the special name
Properties connProperties = new Properties();
connProperties.put(TransactionalDriver.XADataSource, dsXA);

// getting connection when the 'url' is 'jdbc:arjuna' prefix which determines
// the Naryana drive to be used
Connection con = DriverManager.getConnection(TransactionalDriver.arjunaDriver, connProperties);

// starting transaction
TransactionManager tm = com.arjuna.ats.jta.TransactionManager.transactionManager();
tm.begin();

// db business logic (sql insert query) preparation
PreparedStatement ps = conn.prepareStatement("INSERT INTO TEST values (?, ?)");
ps.setInt(1, 41);
ps.setString(2, "Narayana");

// execution, committing/rolling-back
try {
  ps.executeUpdate();
  tm.commit();
} catch (Exception e) {
  tm.rollback();
} finally {
  conn.close();
}

XADataSource bound to JNDI

// the JNDI name has to start with the Narayana transactional driver prefix,
// for would be determined that we want connection of transactional driver
// the suffix (here 'ds') is used as JNDI name that XADataSource will be bound to
XADataSource dsXA = JdbcDataSource ds = new JdbcDataSource();
dsXA.set...();

// binding xa datasource to JNDI name 'ds'
InitialContext ctx = new IntitialContext();
ctx.bind("ds", dsXA);

// passing the JNDI name 'ds' as part of the connection url that we demand
// the first part is narayana transactional driver prefix
String dsJndi = TransactionalDriver.arjunaDriver + "ds";
Connection conn = DriverManager.getConnection(dsJndi, new Properties());

// get transaction driver and start transaction
TransactionManager tm = com.arjuna.ats.jta.TransactionManager.transactionManager();
tm.begin();

// data insertion preparation
PreparedStatement ps = conn.prepareStatement("INSERT INTO TEST values (?, ?)");
ps.setInt(1, 42);
ps.setString(2, "Narayana");

// execute, commit or rollback
try {
  ps.executeUpdate();
  tm.commit();
} catch (Exception e) {
  tm.rollback();
} finally {
  conn.close();
} 

XADataSource connection data provided in properties file

// jdbc url is defined with path to the properties file
// see the ./ds.properties as 'getConnection' url parameter
Properties props = new Properties();
props.put(TransactionalDriver.dynamicClass, PropertyFileDynamicClass.class.getName());
Connection conn = DriverManager.getConnection(TransactionalDriver.arjunaDriver
  + "./ds.properties", props);

// starting transaction
TransactionManager tm = com.arjuna.ats.jta.TransactionManager.transactionManager();
tm.begin();

// data insertion preparation
PreparedStatement ps = conn.prepareStatement("INSERT INTO TEST values (?, ?)");
ps.setInt(1, 43);
ps.setString(2, "Narayana");

// execute, commit or rollback
try {
  ps.executeUpdate();
  tm.commit();
} catch (Exception e) {
  tm.rollback();
} finally {
  conn.close();
}

# implementation of XADataSource
xaDataSourceClassName=org.h2.jdbcx.JdbcDataSource
# properties which will be invoked on dynamically created XADataSource as setters.
# For example there will be call
# JdbcDataSource.setURL("jdbc:h2:mem:test1;DB_CLOSE_DELAY=-1")
URL=jdbc:h2:mem:test1;DB_CLOSE_DELAY=-1
User=sa
Password=sa

In summary

• TransactionalDriver.arjunaDriver is a prefix of jdbc url which defines the Narayana transactional driver is in demand. Data after this prefix is used as parameter for later use.

• TransactionalDriver.XADataSource when used it defines that implementation of XADataSource was provided as parameter and will be used for connection withdrawal 
• TransactionalDriver.dynamicClass defines name of class implementing interface com.arjuna.ats.internal.jdbc.DynamicClass which is then used for dynamically creating of the XADataSource. 
• TransactionalDriver.userName and TransactionalDriver.password you can use if you the particular connection needs to be specified with the values. They will be used in call of XADataSource.getXAConnection(username, password).
• TransactionalDriver.poolConnections is default as true. The current behavior is simple. There is created a pool for each jdbc url, internal check verifies if the particular connection is in use, if not then it's passed for the usage otherwise it's created a new connection. The pool capacity is defined by property TransactionalDriver.maxConnections. When the connection is closed then it is returned to the pool and marked as 'not under use'. For some more sophisticated pool management some 3rd party pool management project is recommended.
  If this property is set to false then a new connection is returned each time is asked for. That way you need to pass this connection over the application.

After reading the post about transaction recovery you will find out that for the recovery manager would consider any resource that we enlist into the global transaction we have to:

• either ensure that resource could be serialized into Narayana transaction log store (resource has to be serializable), in which case the recovery manager deserialize the XAResource and use it directly to get data from it
• or to register recovery counterpart of the transaction enlistment which is capable to provide the instance of XAResource to the RecoveryModule
  in other words we need implemntation of XAResourceRecoveryHelper or XAResourceRecovery (and that's where transactional driver can help us).

For configuration of the recovery properties, we use the jbossts-properties.xml descriptor in our quickstart. We leave the most of the properties with their default values but we still need to concentrate to set up the recovery.
You can observe that it serves to define recovery modules, orphan filters or xa resource recovery helpers. Those are important entries for the recovery works for our transaction JDBC driver.
For better understanding what was set I recommend to check the comments there.

JDBC transaction driver and the recovery

XADataSource provided within Narayana JDBC driver properties

This is a variant where XADatasource is created directly in the code and then the created(!) instance is passed to the jdbc transactional driver.
As transactional driver receives the resource directly it does not have clue how to create such resource on its own during recovery. We have to help it with our own implementation of the XAResourceRecovery class. That has to be of course registered into environment bean (it's intentionally commented out as for testing purposes we need to change different variants).

• link to section of previous blogpost
• quickstart class enlisting the resource
• the implementation class providing the XAResource (in our case pretty dummy one)
• testcase with recovery

XADataSource bound to JNDI

This variant bound the XADatasource to JNDI name. The recovery can lookup the provided jndi name and receive it to create an XAConnection to find the indoubt transactions at database side.
The point here is to pass information about jndi name to the recovery process for it knowing what to search for.
The jdbc driver uses a xml descriptor for that purpose in two variants

• BasicXARecovery
• JDBCXARecovery

• link to section of previous blogpost
• quickstart class enlisting the resource
• testcase with recovery #1
• testcase with recovery #2
• configuration in the jbossts-properties.xml file

XADataSource connection data provided in properties file

The last variant uses properties file where the same connection information is used already during resource enlistment. And the same property file is then automatically used for recovery. You don't need to set any property manually. The recovery is automatically setup because the placement of the properties file is serialized into object store and then loaded during recovery.

• link to section of previous blogpost
• quickstart class enlisting the resource
• testcase with recovery

In this case you configured the PropertyFileDynamicClass providing datasource credentials for the transaction manager and the recovery too. If you would like to extend the behaviour you can implement your own DynamicClass (please consult the codebase). For the recovery would work automatically you need to work with the RecoverableXAConnection.

Summary

There is currently available three approaches for the setting up recovery of jdbc transactional driver
creation of your own XAResourceRecovery/XAResourceRecoveryHelper which is feasible if you want to control the creation of the datasource and jdbc connection on your own. Or using one of the prepared XAResourceRecovery classes - either JDBCXARecovery or BasicXARecovery where you provide xml file where you specify the JNDI name of the datasource. The last option is to use properties file which defines credentials for connection and for recovery too.

Let's talk about the transaction recovery with details specific to Narayana.
This blog post is related to JTA transactions. If you configure recovery for JTS, still you can find a relevant information here but then you will need to consult the Narayana documentation.

What is the transaction recovery

The transaction recovery is process needed when an active transaction fails for some reason. It could be a crash of process of the transaction manager (JVM) or connection to the resource (database)could fail or any other reason for failure.
The failure of the transaction could happen at various points of the transaction lifetime and the point define the state which the in-progress transaction was left at. The state could be just an in-memory state which is left behind and transaction manager relies on the resource transaction timeout to release it. But it could the transaction state after prepare was called (by successful prepare call the 2PC transaction confirms that is capable to finish transaction and more of what it promises to finish the transaction with commit). There has to be a process which finishes such transaction remainders. And that process is the transaction recovery.
Let's review three variants of failures which serves three different transaction states. Their results and needs of termination will guide us through the work of the transaction recovery process.

Transaction manager runs a global transaction which includes several transaction branches (when using term from XA specification). In our article about 2PC we used (not precisely) term resource-located transactioninstead of the transaction branch.
Let's say we have a global transaction containing data insertion to a database plus sending a message to a message broker queue.
We will examine the crash of transaction manager (the JVM process) where each point represents one of the three example cases. The examples show the timeline of actions and differ in time when the crash happens.

1. The global transaction was started and insertion to database happened, now JVM crashes. (no message was sent to queue). In this situation, all the transaction metadata is saved only in the memory. After JVM is restarted the transaction manager has no notion of existence of the global transaction in time before.
   But the insertion to the database already happened and the database has some work in progress already. But there was no promise by prepare call to end with commit and everything was stored only as an in-memory state thus transaction manager relies on the database to abort the work itself. Normally it happens when transaction timeout expires.
2. The global transaction was started, data was inserted into the database and message was sent to the queue. The global transaction is asking to commit. The two-phase commit begins – the prepare is called on the database (resource-located transaction). Now the transaction manager (the JVM) crashes. If an interleaving data manipulation would be permitted then the 2PC commitwould fail. But calling of prepare means the promise of the successful end. Thus the call of prepare causes locks to be taken to prevent other transactions to interleave.
   When transaction manager is restarted but again no notion of the transaction could be found as all the in-memory state was cleared. And there was nothing to be saved in the Narayana transaction log so far.
   But the database transaction is in the prepared state and with locks. On top of it, the transaction in the prepared state can't be rolled-back by transaction timeout and needs to wait for some other party to finish it.
3. The global transaction was started, data inserted into the database and message was sent to the queue. The transaction was asked to commit. The two-phase commit begins – the prepare is called on the database and on the message queue too. A record success of the prepare phase is saved to the Narayana transaction log too. Now the transaction manager (JVM) crashes.
   After the transaction manager is restarted there is no in-memory state but we can observe the record in the Narayana transaction log and that database and the JMS queue resource-located transactions are in the prepared state with locks.

In the later cases the transaction state survived the JVM crash - once only at the side of locked records of a database, in other cases, a record is present in transaction log too. In the first case only in memory transaction representation was used where transaction manager is not responsible to finish it.
The work of finishing the unfinished transactions belongs to the recovery manager. The purpose of the recovery manager is to periodically check the state of the Narayana transaction log and resource transaction logs (unfinished resource-located transactions– it runs the JTA API call of XAResource.recover().
If an in-doubt transaction is found the recovery manager either roll it back - for example in the second case, or commit it as the whole prepare phase was originally finished with success, see the third case.

Narayana periodic recovery in details

Narayana periodic recovery configuration

• CoreEnvironmentBean
• CoordinatorEnvironmentBean
• JTAEnvironmentBean
• RecoveryEnvironmentBean
• JDBCEnvironmentBean

• via system property – see example in the quickstart pom.xml
  . We can see that the property is passed at the JVM argument line.
• via use of the descriptor file jbossts-properties.xml.
  This is usually the main source of configuration in the standalone applications using Narayana. You can see the example jbossts-properties.xml and observe that as the standalone application is not the exception.
  The descriptor has to be at the classpath for the Narayana will be able to access it.
• via call of bean setter methods.
  This is the programatic approach and is normally used mainly in managed environment as they are application servers as WildFly is.

• The usage of the system property has precedence over use of the descriptor jbossts-properties.xml.
• The usage of the programatic call of the setter method has precedence over use of system properties.

• arjPropertyManager for CoreEnvironmentBean etc.
• recoveryPropertyManager for RecoveryEnvironmentBean
• jdbcPropertyManager for JDBCEnvironementBean

We will examine the programmatic approach at the example of the jdbc transactional driver quickstart inside of the recovery utility class where property controlling values of XAResourceRecovery is reset in the code.
If you search to understand what should be the exact name of the system property or entry in jbossts-properties.xml the rule of thumb is to take the short class name of the bean, add the dot and the name of the property at the end.
For example let's say you want to redefine time period for the periodic recovery cycle. Then you need to visit the RecoveryEnvironmentBean, find the name of the variable – which is periodicRecoveryPeriod. By using the rule of thumb will use the name RecoveryEnvironmentBean.periodicRecoveryPeriod for redefinition of the default 2 minutes value.
Some bean uses annotations @PropertyPrefix which offers other way of naming for the property for settings it up. In case of the periodicRecoveryPeriod we can use system property with name com.arjuna.ats.arjuna.recovery.periodicRecoveryPeriod to reset it in the same way.

Thinking about XA recovery

I hope you have a better picture of recovery setup and how that works now.
The XARecoveryModule has the responsibility for handling recovery of 2PC XA transactions. The module is responsible for committing unfinished transactions and for handling orphans by running registered XAResourceOrphanFilter.

As you could see we configured two RecoveryModules – XARecoveryModule and AtomicActionRecoveryModulein the jbossts-properties.xml descriptor.
The AtomicActionRecoveryModule is responsible for loading resource from object store and if it is serializable and as the whole saved in the Narayana transaction log then it could be deserialized and used immediately during recovery.
This is not the case often. When the XAResource is not serializable (which is hard to achieve for example for database where we need to have a connection to do any work) the Narayana offers resource initiated recovery. That requires a class (a code and a settings) that could provide XAResources for the recovery purposes. For getting the resource we need a connection (to database, to jms broker...). The XARecoveryModule uses objects of two interfaces to get such information (to get the XAResources for recovery).
Those interfaces are

• XAResourceRecoveryHelper
• XAResourceRecovery

Both interfaces then contain method to retrieve the resources (XAResourceRecoveryHelper.getXAResources(),XAResourceRecovery.getXAResource()). The XARecoveryModule then ask all the received XAResources to find in-doubt transactions (by calling XAResource.recovery()) (aka. resource located transactions).
The found in-doubt transactions are then paired with transactions in Narayana transaction log store. If the match is found the XAResource.commit() could be called.
Maybe you wonder what both interfaces are mostly the same – which kind of true – but the use differs. The XAResourceRecoveryHelper is designed (and only available) to be used in programatic way. For adding the helper amogst other ones you need to call XARecoveryModule.addXAResourceRecoveryHelper(). You can even deregister the helper by method call XARecoveryModule.removeXAResourceRecoveryHelper.
The XAResourceRecovery is configured not directly in the code but via property com.arjuna.ats.jta.recovery.XAResourceRecovery. This is not viable for dynamic changes as in normal circumstances it's not possible to reset it – even when you try to change the values by call of JTAEnvironmentBean.setXaResourceRecoveryClassNames().

Running the recovery manager

We have explained how to configure the recovery properties but we haven't pointed down one important fact – in the standalone application there is no automatic launch of the recovery manager. You need manually to start it.
A good point is that's quite easy (if you don't use ORB) and it's fine to call just

      RecoveryManager manager = RecoveryManager.manager();
      manager.initialize()

      RecoveryManager manager = RecoveryManager.manager(RecoveryManager.DIRECT_MANAGEMENT);
      manager.initialize();
      manager.scan();

      manager.terminate();

Summary

This blog post tried to introduce process of transaction recovery in Narayana.
The goal was to present settings necessary to be set for the recovery would work in an expected way for XA transactions and shows how to start the recovery manager in your application.

Narayana implements JTA specification in Java. It's flexible and easy to be integrated to any system which desires transaction capabilities. As proof of the Narayana extensibility check our quickstarts like Spring Boot one or Camel one.
But this blogpost is different integration effort. It talks in details about Narayana integration with Apache Tomcat server.

If you do not care about details then just jump directly to the Narayana quickstarts in this area and use the code there for yourself.

• dbcp2-and-tomcat
• transactionaldriver-and-tomcat
• jca-and-tomcat

Narayana, database resources and JDBC interface

All the proclaimed Narayana capabilities to integrate with other systems come from requirements for the system to conform with the JTA specification. JTA expects manageable resources which follows XA specification in particular. For case of the database resources the underlaying API is defined by JDBC specification. JDBC assembled resources manageable by transaction manager under package javax.sqlIt defines interfaces used for managing XA capabilities. The probably most noticable is XADataSource which serves as factory for XAConnection. From there we can otain XAResource. The XAResource is interface that the transaction manager works with. The instance of it participates in the two phase commit.

The workflow is to get or create the XADataSource, obtains XAConnection and as next the XAResource which is enlisted to the global transaction (managed by a transaction manager). Now we can call queries or statements through the XAConnection. When all the business work is finished the global transaction is commanded to commit which is propagated to call the commit on each enlisted XAResources.

It's important to mention that developer is not expected to do all this (getting xa resources, enlisting them to transaction manager…) All this handling is responsibility of the "container" which could be WildFly, Spring or Apache Tomcat in our case.
Normally the integration which ensures the database XAResource is enlisted to the transaction is provided by some pooling library. By the term pooling library we means code that manages a connection pool with capability enlisting database resource to the transaction.

We can say at the high level that integration parts are

• the Apache Tomcat container
• Narayana library
• jdbc pooling library

Narayana configuration with Tomcat

After the brief overview of integration requirements, let's elaborate on common settings needed for any integration approach you choose.
Be aware that each library needs a little bit different configuration and especially IronJacamar is specific.

JDBC pooling libraries integration

Narayana provides integration code in maven module tomcat-jta. That contains the glue code which integrates Narayana to the world of the Tomcat. If you write an application you will need the following:

• providing Narayana itself to the application classpath
• providing Narayana tomcat-jta module to the application classpath
• configure WEB-INF/web.xml with NarayanaJtaServletContextListener which ensures the intialization of Narayana transaction manager
• add META-INF/context.xml which setup Tomcat to start using implementation of JTA interfaces provided by Narayana
• configure database resources to be XA aware and cooperate with Narayana by setting them up in the META-INF/context.xml

If we take a look at the structure of the jar to be deployed we would get the picture possibly similar to this one:

├── META-INF
│   └── context.xml
└── WEB-INF
├── classes
│   ├── application…
│   └── jbossts-properties.xml
├── lib
│   ├── arjuna-5.8.1.Final.jar
│   ├── jboss-logging-3.2.1.Final.jar
│   ├── jboss-transaction-spi-7.6.0.Final.jar
│   ├── jta-5.8.1.Final.jar
│   ├── postgresql-9.0-801.jdbc4.jar
│   └── tomcat-jta-5.8.1.Final.jar
└── web.xml

From this summary let's overview the configuration files one by one to see what's needed to be defined there.

Configuration files to be setup for the integration

WEB-INF/web.xml

<web-app>
<listener>
<listener-class>org.jboss.narayana.tomcat.jta.NarayanaJtaServletContextListener</listener-class>
</listener>
</web-app>

The web.xml needs to define the NarayanaJtaServletContextListener to be loaded during context initialization to initialize the Narayana itself. Narayana needs to get running, for example, reaper thread that ensures transaction timeouts checking or thread of recovery manager.

WEB-INF/clases/jbossts-properties.xml

This file is not compulsory. The purpose is to configure the Narayana itself.
If you don't use your own configuration file then the default is in charge. See more at blogpost Narayana periodic recovery of XA transactions or consider settings done by the default descriptor jbossts-properties.xml at narayana-jts-idlj.

META-INF/context.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<Context antiJarLocking="true" antiResourceLocking="true">
<!-- Narayana resources -->
<Transaction factory="org.jboss.narayana.tomcat.jta.UserTransactionFactory"/>
<Resource factory="org.jboss.narayana.tomcat.jta.TransactionManagerFactory"
        name="TransactionManager" type="javax.transaction.TransactionManager"/>
<Resource factory="org.jboss.narayana.tomcat.jta.TransactionSynchronizationRegistryFactory"
        name="TransactionSynchronizationRegistry" type="javax.transaction.TransactionSynchronizationRegistry"/>

<Resource auth="Container" databaseName="test" description="Data Source"
        factory="org.postgresql.xa.PGXADataSourceFactory" loginTimeout="0"
        name="myDataSource" password="test" portNumber="5432" serverName="localhost"
        type="org.postgresql.xa.PGXADataSource" user="test" username="test"
        uniqueName="myDataSource" url="jdbc:postgresql://localhost:5432/test"/>
<Resource auth="Container" description="Transactional Data Source"
        factory="org.jboss.narayana.tomcat.jta.TransactionalDataSourceFactory"
        initialSize="10" jmxEnabled="true" logAbandoned="true" maxAge="30000"
        maxIdle="16" maxTotal="4" maxWaitMillis="10000" minIdle="8"
        name="transactionalDataSource" password="test" removeAbandoned="true"
        removeAbandonedTimeout="60" testOnBorrow="true" transactionManager="TransactionManager"
        type="javax.sql.XADataSource" uniqueName="transactionalDataSource"
        username="test" validationQuery="select 1" xaDataSource="myDataSource"/>
</Context>

I divide explanation this file into two parts. First are the generic settings - those needed for transaction manager integration (top part of the context.xml). The second part is on resource declaration that defines linking to the JDBC pooling library.

Transaction manager integration settings

We define implementation classes for the JTA api here. The implementation is provided by Narayana transaction manager. Those are lines of UserTransactionFactory and resources of TransactionManager and TransactionSynchronizationRegistry in the context.xml file.

JDBC pooling library settings

We aim to define database resources that can be used in the application. That's how you get the connection typically with code DataSource ds = InitialContext.doLookup("java:comp/env/transactionalDataSource"), and eventually execute a sql statement.
We define a PostgreSQL datasource with information how to create a new XA connection (we provide the host and port, credentials etc.) in the example.
The second resource is definition of jdbc pooling library to utilize the PostgreSQL one and to provide the XA capabilities. It roughtly means putting the PostgreSQL connection to the managed pool and enlisting the work under an active transaction.
Thus we have got two resources defined here. One is non-managed (the PosgreSQL one) and the second manages the first one to provide the ease work with the resources. For the developer is the most important to know he needs to use the managed one in his application, namely the transactionalDataSource from our example.

A bit about datasource configuration of Apache Tomcat context.xml

Let's take a side step at this place. Before we will talk in details about supported pooling libraries let's check a little bit more about the configuration of the Resource from perspective of XA connection in the context.xml.

Looking at the Resource definition there are highlighted parts which are interesting for us

<Resource auth="Container"databaseName="test" description="Data Source"
factory="org.postgresql.xa.PGXADataSourceFactory"
    loginTimeout="0"name="myDataSource"password="test"portNumber="5432"serverName="localhost"
type="org.postgresql.xa.PGXADataSource" uniqueName="myDataSource"
    url="jdbc:postgresql://localhost:5432/test"user="test" username="test"/>

name

defines the name the resource is bound at the container and we can use the jndi lookup to find it by that name in application

factory

defines what type we will get as the final created Object. The factory which we declares here is class which implements interface ObjectFactory and from the provided properties it construct an object.
If we would not define any factory element in the definition then the Tomcat class ResourceFactory is used (see default factory constants). The ResourceFactory will pass the call to the BasicDataSourceFactory of the dbcp2 library. Here we can see the importantce of the type xml parameter which defines what is the object type we want to obtain and the factory normally checks if it's able to provide such (by string equals check usually).
The next step is generation of the object itself where the factory takes each of the properties and tries to applied them.
In our case we use the PGXADataSourceFactory which utilizes some of the properties to create the XADataSource.

serverName, portNumber, databaseName, user, password

are properties used by the object factory class to get connection from the database
Knowing the name of the properties for the particular ObjectFactory is possibly the most important when you need to configure your datasource. Here you need to check setters of the factory implementation.
In case of the PGXADataSourceFactory we need to go through the inheritance hierarchy to find the properties are saved at BaseDataSource. For our case for the relevant properties are user name and password. From the BaseDataSource we can see the setter for the user name is setUser thus the property name we look for is user.

After this side step let's take a look at the setup of the Resources in respect of the used pooling library.

Apache Commons DBCP2 library

Quickstart:https://github.com/jbosstm/quickstart/tree/master/dbcp2-and-tomcat

The best integration comes probably with Apache Common DBCP2 as the library itself is part of the Tomcat distribution (the Tomcat code base uses fork of the project). The XA integration is provided in Apache Tomcat version 9.0.7 and later. There is added dbcp2 package managed which knows how to enlist a resource to XA transaction.

The integration is similar to what we discussed in case of the JDBC transactional driver. You need to have configured two resources in context.xml. One is the database datasource (see above) and other is wrapper providing XA capabilities.

<Resource name="transactionalDataSource" uniqueName="transactionalDataSource"
    auth="Container" type="javax.sql.XADataSource"
    transactionManager="TransactionManager" xaDataSource="h2DataSource"
    factory="org.jboss.narayana.tomcat.jta.TransactionalDataSourceFactory"/>

The integration is here done over the use of the specific factory which directly depends on classes from Apache Tomcat org.apache.tomcat.dbcp.dbcp2 package. The factory ensures the resource being enlisted to the recovery manager as well.
The nice feature is that you can use all the DBCP2 configuration parameters for pooling as you would used when BasicDataSource is configured. See the configuration options and the their meaning at the Apache Commons documentation.

Summary:

• Already packed in the Apache Tomcat distribution from version 9.0.7
• Configure two resources in context.xml. One is the database datasource, the second is wrapper providing XA capabilities with use of the dbcp2 pooling capabilities integrated with TransactionalDataSourceFactory.

<Resource name="transactionalDataSource" uniqueName="transactionalDataSource"
    auth="Container" type="javax.sql.XADataSource"
    transactionManager="TransactionManager" xaDataSource="h2DataSource"
    jmxEnabled="true" jmxName="org.apache.commons.dbcp2:name=transactionalDataSource"
    factory="org.jboss.narayana.tomcat.jta.TransactionalDataSourceFactory"/>

Narayana jdbc transactional driver

Quickstart:https://github.com/jbosstm/quickstart/tree/master/transactionaldriver/transactionaldriver-and-tomcat

With this we will get back to other two recent articles about jdbc transactional driver and recovery of the transactional driver.
The big advantage of jdbc transactional driver is its tight integration with Narayana. It's the dependecy of the Narayana tomcat-jtamodule which contains all the integration code needed for Narayana working in Tomcat. So if you take the tomcat-jta-5.8.1.Finalyou have packed the Narayna integration code and jdbc driver in out-of-the-box working bundle.

Configuration actions

Here we will define two resources in the context.xml file. The first one is the database one.

<Resource name="h2DataSource" uniqueName="h2Datasource" auth="Container"
    type="org.h2.jdbcx.JdbcDataSource" username="sa" user="sa" password="sa"
    url="jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1" description="H2 Data Source"
    loginTimeout="0" factory="org.h2.jdbcx.JdbcDataSourceFactory"/>

The database one defines data needed for preparation of datasource and creation of the connection. The datasource is not XA aware. We need to add one more layer on top which is transactional JDBC driver here. It wraps the datasource connection within XA capabilities.

<Resource name="transactionalDataSource" uniqueName="transactionalDataSource"
    auth="Container" type="javax.sql.DataSource" username="sa" password="sa"
    driverClassName="com.arjuna.ats.jdbc.TransactionalDriver"
    url="jdbc:arjuna:java:comp/env/h2DataSource" description="Transactional Driver Datasource"
    connectionProperties="POOL_CONNECTIONS=false"/>

As we do not define the element factory then the default one is used which is org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory. Unfortunately, this is fine up to the time you need to process some more sophisticated pooling strategies. In this aspect the transactional driver does not play well with the default factory and some further integration work would be needed.

This configuration is nice for having transactionalDataSource available for the transactional work. Unfortunately, it's not all that you need to do. You miss here configuration of recovery. You need to tell the recovery manager what is the resource to care of. You can setup this in jbossts-properties.xml or maybe easier way to add it to environment variables of the starting Tomcat, for example by adding the setup under script $CATALINA_HOME/bin/setenv.sh
You define it with property com.arjuna.ats.jta.recovery.XAResourceRecovery.

-Dcom.arjuna.ats.jta.recovery.XAResourceRecovery1=com.arjuna.ats.internal.jdbc.recovery.BasicXARecovery;abs://$(pwd)/src/main/resources/h2recoveryproperties.xml

You can define whatever number of the resources you need the recovery is aware of. It's done by adding more numbers at the end of the property name (we use 1 in the example above). The value of the property is the class implementing com.arjuna.ats.jta.recovery.XAResourceRecovery. All the properties provided to the particular implementation is concatenated after the ; character. In our example it's path to the xml descriptor h2recoveryproperties.xml.
When transactional driver is used then you need to declareBasicXARecovery as recovery implementation class and this class needs connection properties to be declared in the xml descriptor.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<entry key="DB_1_DatabaseUser">sa</entry>
<entry key="DB_1_DatabasePassword">sa</entry>
<entry key="DB_1_DatabaseDynamicClass"></entry>
<entry key="DB_1_DatabaseURL">java:comp/env/h2DataSource</entry>
</properties>

Note: there is option not defining the two resources under context.xml and use the env property for recovery enlistment. All the configuration properties are then involved in one properties file and transactional driver dynamic class is used. If interested the working example is at ochaloup/quickstart-jbosstm/tree/transactional-driver-and-tomcat-dynamic-class.

Summary:

• Already packed in the tomcat-jta artifact
• Configure two resources in context.xml. One is database datasource, the second is transactional datasource wrapped by transactional driver.
• Need to configure recovery with env variable setup com.arjuna.ats.jta.recovery.XAResourceRecovery while providing xml descriptor with connection parameters

IronJacamar

Quickstart:https://github.com/jbosstm/quickstart/tree/master/jca-and-tomcat

The settings of IronJacamar integration differs pretty much from what we've seen so far. The IronJacamar implements whole JCA specification and it's pretty different beast (not just a jdbc pooling library).

The whole handling and integration is passed to IronJacamar itself.
You don't use tomcat-jta module at all.
You need to configure all aspects in the IronJacamar xml descriptors. Aspects like datasource definition, transaction configuration, pooling definition, up to the jndi binding.

The standalone IronJacamar is needed to be started with command org.jboss.jca.embedded.EmbeddedFactory.create().startup() where you defines the descriptors to be used. You can configure it in web.xml as ServletContextListener.

What are descriptors to be defined:

• jdbc-xa.rar which is resource adapter provided by IronJacamar itself. It needs to be part of the deployment. It's capable to process ds files.
• ds.xml which defines connecion properties and jndi name binding
• transaction.xml which configures transaction manager instead of use of the jbossts-properties.xml.

Summary: IronJacamar is started as embedded system and process all the handling on its own. Developer needs to provide xml descriptor to set up.

Summary

This article provides the details about configuration of the Narayana when used in Apache Tomcat container. We've seen the three possible libraries to get the integration working - the Narayana JDBC transactional driver, Apache DBCP2 library and IronJacamar JCA implementation.
On top of it, the article contains many details about Narayana and Tomcat resource configuration.

If you hesitate what alternative is the best fit for your project then this table can help you

CMR is neat Narayana feature enabling full XA transaction capability for one non-XA JDBC resource. This gives you a way to engage a database resource to XA transaction even the JDBC driver is not fully XA capable (or you just have a design restriction on it) while transaction data consistency is kept.

Last resource commit optimization (aka. LRCO)

Maybe you will say "adding one non-XA resource to a transaction is well-known LRCO optimization". And you are right. But just partially. The last resource commit optimization (abbreviated as LRCO) provides a way to enlist and process one non-XA datasource to the global transaction managed by the transaction manager. But LRCO contains a pitfall. When the crash of the system (or the connection) happens in particular point of the time, during two-phase commit processing, it causes data inconsistency. Namely, the LRCO could be committed while the rest of the resources will be rolled-back.

Let's elaborate a bit on the LRCO failure. Let's say we have a JMS resource where we send a message to a message broker and non-XA JDBC datasource where we save information to the database.

NOTE: The example refers to the Narayana two-phase commit implemenation.

1. updating the database with INSERT INTO SQL command, enlisting LRCO resource under the transaction
2. sending a message to the JMS broker, enlisting the JMS resource to the transaction
3. Narayana starts the two phase commit processing
4. prepare is called to JMS XA resource, the transaction log is stored at the JMS broker side
5. prepare phase for the LRCO means to call commit at the non-XA datasource. That call makes the data changes visible to the outer world.
6. crash of the Narayana JVM occurs before the Narayana can preserve information of commit to its transaction log store
7. after the Narayana restarts there is no notion about the existence of any transaction thus the prepared JMS resource is rolled-back during transaction recovery

Note: roll-backing of the JMS resource is caused by presumed abort strategy applied in the Narayana. If transaction manager does do not apply the presumed abort then you end ideally not better than in the transaction heuristic state.

The LRCO processing is about ordering the LRCO resource as the last during the transaction manager 2PCprepare phase. At place where transaction normally calls prepare at XAResources there is called commit at the LRCO's underlaying non-XA resource.
Then during the transaction manager commit phase there is called nothing for the LRCO.

Commit markable resource (aka. CMR)

The Commit Markable Resource, abbreviated as CMR, is an enhancement of the last resource commit optimization applicable on the JDBC resources. The CMR approach achieves capabilities similar to XA by demanding special database table (normally named xids) that is accessible for transaction manager to write and to read via the configured CMR datasource.

Let's demonstrate the CMR behavior at the example (reusing setup from the previous one).

1. updating the database with INSERT INTO SQL command, enlisting the CMR resource under the transaction
2. sending a message to the JMS broker, enlisting the JMS resource to the transaction
3. Narayana starts the two phase commit processing
4. prepare on CMR saves information about prepare to the xids table
5. prepare is called to JMS XA resource, the transaction log is stored at the JMS broker side
6. commit on CMR means calling commit on underlaying non-XA datasource
7. commit on JMS XA resource means commit on the XA JMS resource and thus the message being visible at the queue, the proper transaction log is removed at the JMS broker side
8. Narayana two phase commit processing ends

From what you can see here the difference from the LRCO example is that the CMR resource is not ordered as last in the resource processing but it's ordered as the first one. The CMR prepare does not mean committing the work as in case of the LRCO but it means saving information about that CMR is considered to be prepared into the database xids table.
As the CMR is ordered as the first resource for processing it's taken as first during the commit phase too. The commit call then means to call commit at the underlying database connection. The xids table is not cleaned at that phase and it's normally responsibility of CommitMarkableResourceRecordRecoveryModule to process the garbage collection of records in the xids table (see more below).

The main fact to understand is that CMR resource is considered as fully prepared only after the commit is processed (meaning commit on the underlaying non-XA JDBC datasource). Till that time the transaction is considered as not prepared and will be processed with rollback by the transaction recovery.

NOTE: the term fully prepared considers the standard XA two-phase commit processing. If the transaction manager finishes with the prepare phase, aka. prepare is called on all transaction participants, the transaction is counted as prepared and commit is expected to be called on each participant.

It's important to note that the correct processing of failures in transactions which contain CMR resources is responsibility of the special periodic recovery moduleCommitMarkableResourceRecordRecoveryModule. It has to be configured as the first in the recovery module list as it needs to check and eventually process all the XA resources belonging to the transaction which contains the CMR resource (the recovery modules are processed in the order they were configured). You can check here how this is set up in WildFly.
The CMR recovery module knows about the existence of the CMR resource from the record saved in the xids table. From that it's capable to pair all the resources belonging to the same transaction where CMR was involved.

xids: database table to save CMR processing data

As said Narayana needs a special database table (usually named xids) to save information that CMR was prepared. You may wonder what is content of that table.
The table consists of three columns.

• xid : id of the transaction branch belonging to the CMR resource
• transactionManagerID : id of transaction manager, this serves to distinguish more transaction managers (WildFly servers) working with the same database. There is a strict rule that each transaction manager must be defined with unique transaction id (see description of the node-identifer).
• actionuid : global transaction id which unites all the resources belonging to the one particular transaction

LRCO failure case with CMR

In the example, we presented as problematic for LRCO, the container crashed just before prepare phase finished. In such case, the CMR is not committed yet. The other transaction participants are then rolled-back as the transaction was not fully prepared. The CMR brings the consistent rollback outcome for all the resources.

Commit markable resource configured in WildFly

We have sketched the principle of the CMR and now it's time to check how to configure it for your application running at the WildFly application server.
The configuration consists of three steps.

1. The JDBC datasource needs to be marked as connectable
2. The database, the connectable datasource points to, has to be enriched with the xids table where Narayana can saves the data about CMR processing
3. Transaction subsystem needs to be configured to be aware of the CMR capable resource

In our example, I use the H2 database as it's good for the showcase. You can find it in quickstart I prepared too. Check out the https://github.com/jbosstm/quickstart/tree/master/wildfly/commit-markable-resource.

Mark JDBC datasource as connectable

You will mark the resource as connectable when you use attribute connectable="true" in your datasource declaration in standalone*.xml configuration file. When you use jboss cli for the app server configuration you will use commands

/subsystem=datasources/data-source=jdbc-cmr:write-attribute(name=connectable, value=true)
:reload

The whole datasource configuration then looks like

<datasource jndi-name="java:jboss/datasources/jdbc-cmr" pool-name="jdbc-cmr-datasource"
          enabled="true" use-java-context="true" connectable="true">
<connection-url>jdbc:h2:mem:cmrdatasource</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>

When datasource is marked as connectable then the IronJacamar (JCA layer of WildFly) creates the datasource instance as implementing org.jboss.tm.ConnectableResource (defined in the jboss-transaction-spi project). This resource defines that the class provides method getConnection() throws Throwable. That's how the transaction manager is capable to obtain the connection to the database and works with the xids table inside it.

Xids database table creation

The database configured to be connectable has to ensure existence of the xids before transaction manager starts. As described above the xids allows to save the cruical information about the non-XA datasource during prepare. The shape of the SQL command depends on the SQL syntax of the database you use. The example of the table cleation commands is (see more commands under this link)

-- Oracle
CREATE TABLE xids (
  xid RAW(144), transactionManagerID VARCHAR(64), actionuid RAW(28)
);
CREATE UNIQUE INDEX index_xid ON xids (xid);

-- PostgreSQL
CREATE TABLE xids (
  xid bytea, transactionManagerID varchar(64), actionuid bytea
);
CREATE UNIQUE INDEX index_xid ON xids (xid);

-- H2
CREATE TABLE xids (
  xid VARBINARY(144), transactionManagerID VARCHAR(64), actionuid VARBINARY(28)
);
CREATE UNIQUE INDEX index_xid ON xids (xid);

I addressed the need of the table definition in the CMR quickstart by adding the JPA schema generation create script which contains the SQL to initialize the database.

Transaction manager CMR configuration

The last part is to configure the CMR for the transaction subsystem. The declaration puts the datasource under the list JTAEnvironmentBean#commitMarkableResourceJNDINames which is then used in code of TransactionImple#createResource.
The xml element used in the transaction subsystem and the jboss cli commands look like

<commit-markable-resources>
<commit-markable-resource jndi-name="java:jboss/datasources/jdbc-cmr"/>
</commit-markable-resources>

/subsystem=transactions/commit-markable-resource="java:jboss/datasources/jdbc-cmr":add()
:reload

CMR configuration options

In addition to such simple CMR declaration, the CMR can be configured with following parameters

• jndi-name : as it could be seen above the jndi-name is way to point to the datasource which we mark as CMR ready
• name : defines the name of the table which is used for storing the CMR state during prepare while used during recovery.
  The default value (and we've reffered to it in this way above) is xids
• immediate-cleanup : If configured to true then there is registered a synchronization which removes proper value from the xids table immediatelly after the transaction is committed.
  When synchronization is not set up then the clean-up of the xids table is responsibility of the recovery by the code at CommitMarkableResourceRecordRecoveryModule. It checks about finished xids and it removes those which are free for garbage collection.
  The default value is false (using only recovery garbage collection).
• batch-size : This parameter influences the process of the garbage collection (as described above). The garbage collection takes finished xids and runs DELETE SQL command. The DELETE contains the WHERE xid in (...) clause with maximum of batch-size entries provided. When there is still some finished xids left after deletion, another SQL command is assembled with maximum number of batch-size entries again.
  The default value is 100.

<subsystem xmlns="urn:jboss:domain:transactions:4.0">
<core-environment>
<process-id>
<uuid/>
</process-id>
</core-environment>
<recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
<object-store path="tx-object-store" relative-to="jboss.server.data.dir"/>
<commit-markable-resources>
<commit-markable-resource jndi-name="java:jboss/datasources/jdbc-cmr">
<xid-location name="myxidstable" batch-size="10" immediate-cleanup="true"/>
</commit-markable-resource>
</commit-markable-resources>
</subsystem>

And the jboss cli commands for the same are

/subsystem=transactions/commit-markable-resource="java:jboss/datasources/jdbc-cmr"\
  :write-attribute(name=name, value=myxidstable)
  /subsystem=transactions/commit-markable-resource="java:jboss/datasources/jdbc-cmr"\
  :write-attribute(name=immediate-cleanup, value=true)
/subsystem=transactions/commit-markable-resource="java:jboss/datasources/jdbc-cmr"\
  :write-attribute(name=batch-size, value=10)
:reload

NOTE: the JBoss EAP documentation about the CMR resource configuration can be found at section About the LRCO Optimization for Single-phase Commit (1PC)

Conclusion

The article explained what is the Narayana Commit Markable resource (CMR), it compared it with LRCO and presented its advantages. In the latter part of the article you found how to configure the CMR resource in your application deployed at the WildFly application server.
If you like to run an application using the commit markable resource feature, check our Narayana quickstart at https://github.com/jbosstm/quickstart/tree/master/wildfly/commit-markable-resource.

• The STM approach of mutating data inside of a transaction has some nice features:
• It is less error prone since the demarcation of an atomic block of code is primitive but other synchronisation approaches are many and varied. Techniques such as locks, semaphores, signals etc are tricky to get right, for example the programmer must ensure that accesses are protected with the correct locks and in the correct order. With conventional concurrency control, imagine trying reverse all the changes made during a computation if a problem such as deadlock or data race is detected, whereas code changes that are protected by STM can be aborted in a single statement.
• Transactional updates make it easier for the programmer to reason about his code (it is clear how different threads affect each other) and data (because it simplifies the sharing of state between threads).

• The declarative approach (where the programmer simply marks which code blocks are transactional) means concurrent programming is more intuitive with no explicit locks or synchronisation to worry about.
• Can perform much better than fine grained locking (which can lead to deadlock) and coarse grained locking (which inhibits concurrency):
• If a thread takes a lock and is context switched or incurs cache misses or page faults etc then other threads that need the lock are stalled until the thread is rescheduled or until the needed data is retrieved.
• With STM, updates can be batched up and speculatively committed together.
• The runtime manages lock acquisition and release and resolves conflicts (using approaches such as timeouts and retries).
• It is easier to compose operations using a technique called nesting (traditionally composing two operations can produce concurrency problems unless one analyses in detail the locking approach used by those operations).

    @Transactional
    public interface AtomicInt {
        int get() throws Exception;
        void set(int value) throws Exception;
    }

    public class AtomicIntImpl implements AtomicInt {
        private int state;

        @ReadLock
        public int get() throws Exception {
            return state;
        }

        @WriteLock
        public void set(int value) throws Exception {
            state = value;
        }
    }

Property 1: Interaction with non transactional code

    public void testWeakIsolation() throws Exception {
        AtomicIntImpl aiImple = new AtomicIntImpl();
        // STM is managed by Containers. Enlisting the above implementation
        // with the container returns a proxy which will enforce STM semantics
        AtomicInt ai = new RecoverableContainer().enlist(aiImple);
        AtomicAction tx = new AtomicAction();

        // set up the code that will access the memory outside of a transaction
        Thread ot = new Thread(() -> {
            try {
                synchronized (tx) {
                    tx.wait(); // for the other thread to start a transaction

                    // weak isolation implies that this thread (which is running
                    // outside of a transaction) can observe transactional updates
                    assertEquals(2, aiImple.get()); // the other thread set it to 2
                    aiImple.set(10); // this update is visible to transactional code
            } catch (Exception e) {
                e.printStackTrace();
            }
        });

        ot.start();

        ai.set(1); // initialise the shared memory
        tx.begin(); // start a transaction
        {
            ai.set(2); // conditionally set the value to 2

            synchronized (tx) {
                tx.notify(); // trigger non-transactional code to update the memory
            }

            // weak isolation means that this transactional code may see the
            // changes made by the non transactional code
            assertEquals(10, ai.get()); // the other thread set it to 10
            tx.commit(); // commit the changes made to the shared memory
        }

        // changes made by non transactional code are still visible after commit
        assertEquals(10, ai.get());
        assertEquals(aiImple.get(), ai.get());
    }

    public void testIsClosedNestedCommit() throws Exception {
        AtomicInt ai = new RecoverableContainer().enlist(new AtomicIntImpl());
        AtomicAction parent = new AtomicAction();
        AtomicAction child = new AtomicAction();

        ai.set(1); // initialise the shared memory
        parent.begin(); // start a top level transaction
        {
            ai.set(2); // update the memory in the context of the parent transaction
            child.begin(); // start a child transaction
            {
                ai.set(3); // update the memory in a child transaction
                // NB the parent would still see the value as 2
                // (not shown in this test)
                child.commit();
            }
            // since the child committed the parent should see the value as 3
            assertEquals(3, ai.get());
            // NB other transactions would not see the value 3 however until
            // the parent commits (not demonstrated in this test)
        }
        parent.commit();

        assertEquals(3, ai.get());
    }

Property 3: Exception Handling

    public void testExceptionDoesNotAbort() throws Exception {
        AtomicInt ai = new RecoverableContainer().enlist(new AtomicIntImpl());
        AtomicAction tx = new AtomicAction();

        ai.set(1);
        tx.begin();
        {
            try {
                ai.set(2);
                throw new Exception();
            } catch (Exception e) {
                assertEquals(2, ai.get());
                // the transaction should still be active
                ai.set(3);
                tx.commit();
            }
        }

        assertEquals(3, ai.get());
    }

Project Agroal defines itself as “The natural database connection pool”. And that’s what is it.

It was developed by Luis Barreiro. He works for WildFly as a performance engineer. This prefigures what you can expect – a well performing database connection pool. As Agroal comes from the porfolio of the WildFly projects it offers smooth integration with WildFly and with Narayana too.

In the previous posts we checked other connection pools that you can use with Narayana - either the transactional driver provided by Narayana or DBCP2 which is nicely integrated to be used with Narayana in Apache Tomcat. Another option is the use of the IronJacamar which lives in the long-termed brotherhood with Narayana. All those options are nicely documented in our quickstarts.

Agroal is a party member and you should consider to check it. Either when running standalone application with Narayana or when you run on WildFly. Let’s take a look how you can use it in the standalone application first.

Agroal with Narayana standalone

In case you want to use the Agroal JDBC pooling capabilities with Narayana in your application you need to configure the Agroal datasource to know

• how to grab the instance of the Narayana transaction manager
• where to find the synchronization registry
• how to register resources to Narayana recovery manager

Narayana setup

First we need to gain all the mentioned Narayana objects which are then passed to Agroal which ensures the integration by calling the Narayana API at appropriate moments.

// gaining the transction manager and synchronization registry
TransactionManager transactionManager
    = com.arjuna.ats.jta.TransactionManager.transactionManager();
TransactionSynchronizationRegistry transactionSynchronizationRegistry
    = new com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionSynchronizationRegistryImple();

// intitialization of recovery manager
RecoveryManager recoveryManager
    = com.arjuna.ats.arjuna.recovery.RecoveryManager.manager();
recoveryManager.initialize();
// recovery service provides binding for hooking the XAResource to recovery process
RecoveryManagerService recoveryManagerService
    = new com.arjuna.ats.jbossatx.jta.RecoveryManagerService();
recoveryManagerService.create();

Agroal integration

Now we need to pass the Narayana's object instances to Agroal. With that being done we can obtain a JDBC Connectionwhich is backed by the transaction manager.

AgroalDataSourceConfigurationSupplier configurationSupplier
    = new AgroalDataSourceConfigurationSupplier()
  .connectionPoolConfiguration(cp -> cp
    .transactionIntegration(new NarayanaTransactionIntegration(
        transactionManager, transactionSynchronizationRegistry,
"java:/agroalds1", false, recoveryManagerService))
      cf.connectionFactoryConfiguration(cf ->
        .jdbcUrl("jdbc:h2:mem:test")
        .principal(new NamePrincipal("testuser"))
        .credential(new SimplePassword("testpass"))
        .recoveryPrincipal(new NamePrincipal("testuser"))
        .recoveryCredential(new SimplePassword("testpass"))
        .connectionProviderClassName("org.h2.jdbcx.JdbcDataSource"))
      .maxSize(10)
    );
AgroalDataSource ds1 = AgroalDataSource.from(configurationSupplier);

transactionManager.begin();

conn1 = ds1.getConnection();
...

Those are steps needed for the standalone application to use Narayana and Agroal. The working code example could be seen in the Narayana quickstart at github.com/jbosstm/quickstart#agroal - AgroalDatasource.java

Agroal XA datasource in WildFly

If you want to use the power of Narayana in the WildFly application you need XA participants that Narayana can drive. From Agroal perspective you need to define a xa datasource which you use (linked via JNDI name) in your application.

DISCLAIMER: for you can use the Agroal capabilities integrated with Narayana you will need to run WildFly 15 or later. Currently only WildFly 14 is available so for testing this you need to build the WildFly from sources by yourself. The good message is that’s an easy task – see at https://github.com/wildfly/wildfly/#building.

Agroal datasource subsystem is not available by default in the standalone.xml file so you need to enable that extension. When you run the jboss clicommands then you do it like this

cd $JBOSS_HOME
./bin/jboss-cli.sh -c

#  jboss-cli is started, run following command there
/extension=org.wildfly.extension.datasources-agroal:add
/subsystem=datasources-agroal:add()
:reload

From now you can work with the datasources-agroal subsystem. For you can create the xa-datasource definition you need to have a driver which the datasource will use. The driver has to define it’s XA connection provider.

NOTE: if you want to check what are options for the Agroal configuration in the jboss cli then read the resource description with command /subsystem=datasources-agroal:read-resource-description(recursive=true)

Agroal driver definition works only with drivers deployed as modules. You can’t just copy the driver jar to $JBOSS_HOME/standalone/deployments directory but you need to create a module under $JBOSS_HOME/modules directory. See details either by creating module.xmlby yourself or the recommended way is using the jboss cli with command

module add --name=org.postgresql
    --resources=/path/to/jdbc/driver.jar --dependencies=javax.api,javax.transaction.api

NOTE: The command uses the name of the module org.postgresql as I will demonstrate adding the xa datasource for the PostgreSQL database.

When the module is added we can declare the Agroal’s driver.

/subsystem=datasources-agroal/driver=postgres:add(
    module=org.postgresql, class=org.postgresql.xa.PGXADataSource)

We’ve used the class org.postgresql.xa.PGXADataSource as we want to use it as XA datasource. When class is not defined then standard jdbc driver for PostgresSQL is used (org.postgresql.Driver) as declared in the META-INF/services/java.sql.Driver file.

NOTE: If you would declare the driver without the XA datasource being defined and then you try to add it to XA datasource definition you will get an error

/subsystem=datasources-agroal/driver=non-xa-postgres:add(module=org.postgresql)
/subsystem=datasources-agroal/xa-datasource=AgroalPostgresql:add(
    connection-factory={driver=non-xa-postgres},...)
{
"outcome" =>"failed",
"failure-description" => {"WFLYCTL0080: Failed services" => {"org.wildfly.data-source.AgroalPostgresql"
        =>"WFLYAG0108: An xa-datasource requires a javax.sql.XADataSource as connection provider. Fix the connection-provider for the driver"}
},
"rolled-back" => true
}

When the JDBC driver module is defined we can create the Agroal XA datasource. The bare minimum of attributes you have to define is shown in the following command

/subsystem=datasources-agroal/xa-datasource=AgroalPostgresql:add(
    jndi-name=java:/AgroalPostgresql, connection-pool={max-size=10}, connection-factory={
    driver=postgres, username=test, password=test,url=jdbc:postgresql://localhost:5432/test})

NOTE: this is the most simple way of define the credentials for the connection to database. If you consider more sophisticated method, than just username/password as clear strings saved in the standalone.xml, take a look at the Elytron capabilities.

To check if the WildFly Agroal datasource is able to connect to the database you can use test-connection command

/subsystem=datasources-agroal/xa-datasource=AgroalPostgresql:test-connection()

If you are insterested how the configuration looks as a xml element in standalone.xml configuration file then the Agroal subsystem with PostgreSQL XA datasource definition would look like

<subsystem xmlns="urn:jboss:domain:datasources-agroal:1.0">
<xa-datasource name="AgroalPostgresql" jndi-name="java:/AgroalPostgresql">
<connection-factory driver="postgres" url="jdbc:postgresql://localhost:5432/test"
            username="test" password="test"/>
<connection-pool max-size="10"/>
</xa-datasource>
<drivers>
<driver name="postgres" module="org.postgresql" class="org.postgresql.xa.PGXADataSource"/>
</drivers>
</subsystem>

If you want use the Agroal non-xa datasource as commit markable resource (CMR) it’s possible too. You need to create a standard datasource and define it as connectable. For more information what the commit markable resource means and how it works check our previous blogpost about CMR.

<subsystem xmlns="urn:jboss:domain:datasources-agroal:1.0">
<datasource name="AgroalPostgresql" connectable="true" jndi-name="java:/AgroalPostgresql"
            statistics-enabled="true">
<connection-factory driver="postgres" url="jdbc:postgresql://localhost:5432/test"
            username="test" password="test"/>
<connection-pool max-size="10"/>
</datasource>
<drivers>
<driver name="postgres" module="org.postgresql" class="org.postgresql.Driver"/>
</drivers>
</subsystem>

NOTE: In addition to this configuration of Agroal datasource you need to enable the CMR in the transaction subsystem too – check the blogpost for detailed info.

Summary

This blogpost showed way how to configure Agroal JDBC pooling library and how to integrate it with Narayana.
The code example is part of the Narayana quickstart and you can check it at https://github.com/jbosstm/quickstart/tree/master/agroal

The Narayana release 5.9.5.Finalcomes with few nice CDI functionality enhancements. This blogpost introduces these changes while placing them to the context of the JTA and CDI integration, particularly with focus to Weld.

TL;DR

The fastest way to find out the way of using the JTA with the CDI is walking through the Narayana CDI quickstart.

JTA and CDI specifications

JTA version 1.2 was published in 2013. The version introduced the integration of JTA with CDI. The specification came with the definition of annotations javax.transaction.Transactionaland javax.transaction.TransactionScoped. Those two provide a way for transaction boundary definition and for handling application data bounded to the transaction.

Narayana, as the implementation of the JTA specification, provides those capabilities in the CDI maven module.
Here we come with the maven coordinates:
<groupId>org.jboss.narayana.jta</groupId>
<artifactId>cdi</artifactId>

The module brings Narayana CDI extensionto the user's project. The extension installs interceptors which manage transactional boundaries for method invocation annotated with @Transactional. Then the extension defines a transaction scopedeclared with the @TransactionScoped annotation.

On top of the functionality defined in the JTA specification, it's the CDI specification which defines some more transaction-related features. They are the transactional observer methods and the definition of the javax.transaction.UserTransactionbuilt-in bean.

Let's summarize what that all means in practice.

@Transactional

With the use of the @Transactional annotation, transaction boundary could be controlled declaratively. The use of the annotation is really similar to the container-managed transactions in EJB.

When the annotation is used for a bean or a method the Narayana CDI extension (CDI interceptor is used) verifies the existence of the transaction context when the method is called. Based on the value of the value parameter an appropriate action is taken. The value is defined from enumeration Transactional.TxType
For example when @Transactional(Transactional.TxType.REQUIRES_NEW) is used on the method then on the start of its execution a new transaction is started. If the incoming method call contains an existing transaction it's suspended during the method execution and then resumed after it finishes. For details about the other Transactional.TxType values consider the javadoc documentation.

NOTE: be aware of the fact that for the CDI container can intercept the method call the CDI managed instance has to be used. For example, when you want to use the capability for calling an inner bean you must use the injection of the bean itself.

@RequestScope
public class MyCDIBean {
  @Inject
  MyCDIBean myBean;

  @Transactional(TxType.REQUIRED)
  public void mainMethod() {
    // CDI container does not wrap the invocation
    // no new transaction is started
    innerFunctionality();

    // CDI container starts a new transaction
    // the method uses TxType.REQUIRES_NEW and is called from the CDI bean
    myBean.innerFunctionality();
  }

  @Transactional(TxType.REQUIRES_NEW)
  private void innerFunctionality() {
    // some business logic
  }
}

@TransactionScoped

@TransactionScoped brings an additional scope type in addition to the standard built-in ones. A bean annotated with the @TransactionScoped, when injected, lives in the scope of the currently active transaction. The bean remains bound to the transaction even when it is suspended. On resuming the transaction the scoped data are available again. If a user tries to access the bean out of the scope of the active transaction the javax.enterprise.context.ContextNotActiveException is thrown.

Built-in UserTransaction bean

The CDI specification declares that the Java EE container has to provide a bean for the UserTransaction can be @Injected. Notice that the standalone CDI container has no obligation to provide such bean. The availability is expected for the Java EE container. In Weld, the integration for the Java EE container is provided through the SPI interface TransactionServices.

If somebody wants to use the Weld integrated with Narayana JTA implementation in a standalone application he needs to implement this SPI interface (see more below).

Transaction observer methods

The feature of the transaction observer methods allows defining an observer with the definition of the during parameter at @Observes annotation. During takes a value from the TransactionPhase enumeration. The during value defines when the event will be delivered to the observer. The event is fired during transaction processing in the business logic but then the delivery is deferred until transaction got status defined by the during parameter.
The during parameter can obtain values BEFORE_COMPLETION, AFTER_COMPLETION, AFTER_FAILURE, AFTER_SUCCESS. Using value IN_PROGRESS means the event is delivered to observer immediately when it's fired. It behaves like there is no during parameter used.

The implementation is based on the registration of the transaction synchronization. When the event is fired there is registered a special new synchronization which is invoked by the transaction manager afterwards. The registered CDI synchronization code then manages to launch the observer method to deliver the event.

For the during parameter working and for the events being deferred Weld requires integration through the TransactionServices SPI. The interface defines a method which provides makes for Weld possible to register the transaction synchronization. If the integration with the TransactionServices is not provided then the user can still use the during parameter in his code. But(!) no matter what TransactionPhase value is used the event is not deferred but it's immediately delivered to the observer. The behaviour is the same as when the IN_PROGRESS value is used.

Maybe it could be fine to clarify who fires the event. The event is fired by the user code. For example, take a look at the example in the Weld documentation. The user code injects an event and fires it when considers it necessary.

@Inject @Any Event productEvent;
...
public void persist(Product product) {
  em.persist(product);
  productEvent.select(new AnnotationLiteral(){}).fire(product);
}

void addProduct(@Observes(during = AFTER_SUCCESS) @Created Product product) {
...
}

A bit more about TransactionServices: Weld and JTA integration

As said for the integration of the Weld CDI to JTA it's needed to implement the TransactionServices SPI interface. The interface gives the Weld the chance to gain the UserTransaction thus the built-in bean can provide it when it's @Injected. It provides the way to register transaction synchronization for an event could be deferred until particular transaction status occurs. Up to that, it demands the implementation of the method isTransactionActive. The TransactionScoped is active only when there is some active transaction. This way the Weld is able to obtain the transaction activity state.

Regarding the implementation, you can look at how the interface TransactionServicesis implemented in WildFly or in the more standalone way for SmallRye Context Propagation.

A new Narayana CDI features

Narayana brings two new CDI JTA integration capabilities, up to those described above.

The first enhancement is the addition of the transactional scope events. Up to now, Narayana did not fire the scope events for the @TransactionScoped. From now there is fired the scope events automatically by Narayana. The user can observe the initialization and the destruction of the transaction scope. The code for the observer could be like

void transactionScopeActivated(
  @Observes @Initialized(TransactionScoped.class) final Transaction event,
  final BeanManager beanManager) {
...
}

The second enhancement is the addition of two built-in beans which can be @Injected in the user code. Those are beans TransctionManager and TransactionSynchronizationRegistry.

The implementation gives priority to the JNDI binding. If there is bound TransactionManager/TransactionSynchronizationRegistry in the JNDI then such instance is returned at the injection point.
If the user defines his own CDI bean or a CDI producer which provides an instance of those two classes then such instance is grabbed for the injection.
As the last resort, the default Narayana implementation of both classes is used. You can consider the TransactionManagerImple and the TransactionSynchronizationRegistryImple to be used.

Using the transactional CDI extension

The easiest way to check the integration in the action is to run our JTA standalone quickstart. You can observe the implementation of the Weld SPI interface TransactionServices. You can check the use of the observers, both the transaction observer methods and the transactional scoped observers. Up to that, you can see the use of the transaction scope and use of the injection for the TransactionManager.

Acknowledgement

Big thanks to Laird Nelsonwho contributed the new CDI functionality enhancements to Narayana.
And secondly thanks to Matěj Novotný. for his help in understanding the CDI topic.

What are the expiry scanners?

Narayana object store and transaction records

Object store records for JTA and JTS

data/tx-object-store/
ShadowNoFileLockStore
└── defaultStore
├── EISNAME
│   ├── 0_ffff0a000007_6d753eda_5d0f2fd1_34
│   └── 0_ffff0a000007_6d753eda_5d0f2fd1_3a
└── StateManager
└── BasicAction
└── TwoPhaseCoordinator
└── AtomicAction
└── 0_ffff0a000007_6d753eda_5d0f2fd1_29

data/tx-object-store/
ShadowNoFileLockStore
└── defaultStore
├── CosTransactions
│   └── XAResourceRecord
│       ├── 0_ffff0a000007_-55aeb984_5d0f33c3_4b
│       └── 0_ffff0a000007_-55aeb984_5d0f33c3_50
├── Recovery
│   └── FactoryContact
│       └── 0_ffff0a000007_-55aeb984_5d0f33c3_15
├── RecoveryCoordinator
│   └── 0_ffff52e38d0c_c91_4140398c_0
└── StateManager
└── BasicAction
└── TwoPhaseCoordinator
└── ArjunaTransactionImple
└── 0_ffff0a000007_-55aeb984_5d0f33c3_41

Now, what about the failures?

ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource

• https://planet.jboss.org/post/norecoveryxa
• https://developer.jboss.org/wiki/TxNonSerializableXAResource

-DRecoveryEnvironmentBean.expiryScannerClassNames="com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner com.arjuna.ats.internal.arjuna.recovery.AtomicActionExpiryScanner"

A transaction is finished either with commit or rollback. But have you considered that the third transaction outcome is <<unspecified>>? This third type of outcome may occur when the participant does not follow the coordinator's orders and makes a decision on the contrary. Then the outcome of the whole transaction could be inconsistent. Some participants could follow the coordinator guidelines to commit while the disobedient participant rolls-back. In such a case, the coordinator cannot report back to the application that "the work was successfully finished". From the perspective of the outside observer, the consistency is damaged. The coordinator itself cannot do much more. It followed the rules of the protocol but the participants disobeyed. Such transaction can only be marked with <<unspecified>> result &mdash which is known as heuristic outcome.
The resolution of that outcome requires a third-party intervention. Saying differently somebody has to go and verify the state of data and make corrections.

XA specification

In scope of this article we talk about the two phase commit protocol and how the XA specification uses it.
XA specification takes the two-phase commit, the abstractly defined consensus protocol, and carries it down to the ground of implementation. It defines rules for communication amongst parties, prescribes state model and assesses outcomes, exceptions etc.
Where the 2PC talks about coordinator and participants the XA specification define a model where the coordinator is represented by a transaction manager (TM), the participant is modelled as the resource manager (RM) and the employer of them is an application program which talks to both of them.
The transaction manager drives a global transaction which enlists resource managers as participants responsible for part of work. For the work managed by RM is used term transaction branch. The resource manager is represented normally by a Java code that implements the JTA API while it uses some internal mechanism to call the underlying data resource. The resource manager could be represented by a JDBC driver and then underlying resource could be the PostgreSQL database running in some distinct datacenter.

JTA Specification

The Java Transaction API is a projection of XA specification to the language of Java. In general, it's a high-level API that strives to provide a comprehensible tool for handling transactional code based on the XA specification.

Several purposes of JTA

The JTA is meant to be used first by an application developer. In terms of XA specification, it would be the application program. Here we have the UserTransaction interface. It gives the chance to begin and commit the transaction. In terms of XA specification, it's a representation of the global transaction. In a newer version (from version 1.2) JTA defines the annotations like Transactional That gives the user chance to control the application scope declaratively. As you can notice neither with the UserTransaction nor with he @Transactional annotation you can't do much more than to define where the transaction begins and where it ends. How to put a participant into the transaction? That limited capability is because developer is anticipated running the application in a managed environment. It could be for example Java EE application server or a different container.

The container is a second consumer of the JTA API. JTA gives the entry point to the world of transaction management. The container uses the interface TransactionManager&mdash which provides ability of managing transaction scope but also gives access to the Transaction object itself. Transaction is used to enlist the resource (the participant) to the global transaction. As the resource for enlistment is used the XAResource interface in JTA. The XAResource is managed by the transaction manager and is operated by the resource manager. Then container may arrange Synchronizations are callbacks called at the start and the end of the 2PC processing (used e.g. by JPA).

The third perspective where JTA participates in is communication with resource managers. The API defines the class XAResource (represents a participant resource to be enlisted to the global transaction), XAException (represents an error state defined in the XA specification) and Xid. The Xid represents an identifer which is unique for each transaction branch and consists of two parts — first an unique identifier of the global transaction and second an unique identifier of the resource manager. If you want to see how the transaction manager uses these classes to communicate with the resource manager take a look at the example from documentation of SQL Server.

Note:If you study the JTA API by looking into the javadoc and strolling the package summary then you can wonder about some other classes which were not mentioned here. Part of the javax.transaction package are interfaces used exclusively by JTS (transaction services running with ORB). That's mitigated by the fact the Java SE 11 removed support of ORB and those classes were removed from JDK as well.
Plus the JTA classses are now (from Java SE 11) split over Java SE and Java EE bundles as the package javax.transaction.xa is solely part of the Java SE while javax.transaction belongs to the Jakarta EE API.

Type of failures

Now when we have talked the model let's see the failure states. First, it's needed to realize that a failure in the application does not mean failure from the protocol perspective. If there is trouble in the application code or a network is transiently not available such occurrences can lead to rollback. It's a failure from the perspective of the application but for the transaction manager the rollback is just another valid state to shift to.
Even if the whole transaction manager crashes (e.g. the underlying JVM is killed) the system still should maintain data consistency and the transaction is recovered when the transaction manager (or rather recovery manager) comes back to life.

What's the trouble for the protocol is an unexpected behaviour of the resource manager (or the backed resource). We can track basically two incident types. A heuristic outcome where the resource deliberately decides to process some action (which is different from transaction manager decision). Or a heuristic outcome caused by a bug in the code (either of the transaction manager or the resource manager).

Let's discuss some examples for these types.
The deliberate decision could be a situation where the transaction manager calls the prepare on the database. The database confirms the prepare— it promises to finish the transaction branch with commit. But unfortunately the transaction manager crashes and nobody comes to restart it for a long time. The database decides that it's pretty long time to hold resources and delaying other transactions to proceed. Thus it decides to commit the work. The processing in the database may continue from that time. But later the transaction manager is restarted and it tries to commit all other branches belonging to the global transaction (let's say e.g. a JMS broker). That resource responds with an error and the transaction manager decides to rollback the whole global transaction. Now it accesses the database with the request for the rollback. But the database already committed its branch — the heuristic outcome just occurred.

An example for the bug in the code could be the PostgreSQL database driver. The driver was returning a wrong error code in case of intermittent connection failure. The XA specification defines that in such case the XAException should be thrown and it has to carry one of the following error codes — either the XAException.XAER_RMFAIL or the XAException.XA_RETRY. But the JDBC driver was returning XAException.XAER_RMERR. Such error code means that an irrecoverable error occurred. It makes the transaction manager think there is no way of automatic recovery and it switches the state of such transaction to heuristic immediately.

Heuristic exceptions

As the last part of this article we take a look on the heuristic outcomes of the transaction. The heuristics is represented with an exception being thrown. The exception reports the reason of the failure. It does so with error code or with type of class.

There are two main types of exception classes. First type is the XAException. This one is part of the communication contract between the transaction manager and the resource manager. It should not happen for the application code to obtain this type of exception. But for sure you can observe the XAException in the container log. It shows that there happened an error during transaction processing.

The second type is represented with multiple classes named Heuristic*Exception. These are exceptions that application code works with. They are thrown from the UserTransaction methods and they are checked.

Heuristic outcome with XAResource

The XAException reports reason of failure with the use of error codes. XA specification defines the meaning. It depends on the context in which it's used. For example the code XAException.XA_RETRY could be used for reporting error from commit with meaning to retry the commit action. But on the other hand it's not permitted to be used as an error code for the one-phase commit.

Then where are those heuristic states? Let's check what could happen when the transaction manager calls the XAResource calls of prepare, commit and rollback.
If the prepare is called then there is not many chances that heuristic occurs. At that time no promise from the resource is placed and the work can be easily roll-back or the worse timed-out. The only occurrence that can bring the system to the heuristic state is if the resource manager returns undefined code for this phase. But that is cause the most probably only by a bug in the implementation. Consult the XA specification which are those.
The more interesting are the commit and rollback calls. The commit and rollback are (or could be) called after the prepare. Heuristic exception means that the resource promised to commit (he acknowledge the prepare call) but it does not wait for transaction manager to command it for the next action and it finished the transaction branch on its own. The error codes are those with prefix XA_HEUR*. The decision on its own does not mean an error for the protocol in all cases.

Let's talk about rollback now. The global transaction was successfully prepared but the transaction manager decided at the end to roll-back it. It calls the rollback to the XAResource. The error XAException.XA_HEURRB announces that the resource manager decided to roll-back the transaction branch prior it was asked for it by the transaction manager. But as the transaction manager decided to go for the roll-back too the heuristic outcome followed the decision.
The XAException.XA_HEURCOM means that all work represented by the transaction branch was committed (at time the rollback is executed on the XAResource). That's bad from the data consistency as some other transaction branches could be already rolled-back.
To explain the meaning of the XAException.XA_HEURMIX it's needed to mention that the transaction branch could consist of several "local transactions". For example, PostgreSQL JDBC driver starts a database transaction to insert data to the database. Later (still in the scope of the same global transaction) it decides to update the data. It starts another database transaction. The transaction manager is clever enough to join those two database transactions which belong to the same database resource (controlled by the same resource manager) under the one transaction branch. It's good as it could reduce the communication overhead. So the XA_HEURMIX says that part of workload involved in the transaction branch was committed and the other part was rolled-back.
The XAException.XA_HEURHAZ says that the resource manager made a decision on its own but it's not capable to say what was the result of such an independent decision.

The most interesting part is the commit call. First it uses the XA_HEUR* exceptions in the same meaning as the rollback call and all what is said in the previous paragraph pays for the commit too. But up to that there are three new error codes. They do not contain word HEUR but in result they mean it. Those are XAER_RMERR which announces that an unspecified error happened during the currently executing commit call. But instead of committing the resource manager had just rolled-back the transaction branch. That means we are in the same state as with the XA_HEURRB The XAER_NOTA says that resource manager does not know anything about this transaction branch. That means the resource manager lost the notion about it and it either commits it or rolled-back it or it may do an arbitrary one in the future. That means we are in the same state as with the XA_HEURHAZ. The last one is the XAER_PROTO which says that the commit was called in a wrong context — for example it was called without the prepare being invoked before. This seems being similar to XAER_NOTA and thus have the same impact as the XA_HEURRB.

Heuristic outcome with the "application exceptions"

For the "application exceptions" it could be considered easier. The heuristic exceptions can be thrown only from the commit call (see UserTransaction javadoc). The UserTransaction gives chance to finish the transaction with commit or roll-back. The roll-back means that transaction branches should be aborted and all work discarded. When UserTransaction.rollback() is called the resource manager had not promised succesful outcome yet. The time the rollback is called is time when all transaction processing data is available only in memory. Thus resource manager has nothing onto decide differently. If there is some trouble the the other types of exceptions are thrown — like IllegalStateException or SystemException (see the UserTransaction javadoc).
But with the UserTransaction.commit it is different. This call means that two-phase commit protocol is to be started and XAResource.prepare/commit/rollback calls are involved. Here the JTA uses the checked exceptions to inform the application to handle the trouble. The application checked exceptions are RollbackException,HeuristicMixedException,HeuristicRollbackException. The RollbackException is not a heuristic exception (at least by name) but still. That exception informs that even the code asked for commit all work (in all transaction branches) was undone by the rollback. The HeuristicMixedException means that some transaction branches were committed and others were rolled-back. This is exception thrown for example if during the commit phase of 2PC. One of the XAResource.commit calls returns XAException.XA_HEURRB (aka it was rolled-back) while the others were succesfully committed.
The HeuristicRollbackException has the same final outcome from the global transaction perspective as the RollbackException. It only emphasizes that the fact that the roll-back was deliberately chosen by all the resources prior to the commit was executed by the transaction manager. In comparison, the RollbackException means that the transaction manager was just trying to commit all resources but during the process of committing trouble occurred and all the work was rolled-back (all resources rolled-back). To be perfectly honest I'm not sure I can't see a real difference between these two.

We talked about all exceptions defined at the UserTransaction.commit definition so we are done. Oh wait, we are not!
There is one more exception defined in the javax.transaction package. It's the HeuristicCommitException. The HeuristicCommitException is not defined at the UserTransaction.commit as even all resources would idependently decide to commit the global transaction result is still just committed. Which is intended as UserTransaction.commit is called. Then what is the purpose of it then?
We need to look into the implementation. It's used at calls of commit and rollback at a subordinate transaction. The subordinate transaction is a transaction which is driven by a parent transaction. The parent transaction (named as top-level as well) manages the subordinate and decides the overall outcome.
When the subordinate transaction is commanded it reports the outcome back to the top-level one. It's a similar relation as the XAResource has to the global transaction. Because the subordinate transaction needs to report heuristic decisions back from the commit and rollback calls the HeuristicCommitException serves for cases when subordinate transaction decided to commit prior the top-level transaction commanded for a final action.

NOTE: Don't interchange the subordinate transaction for the nested transaction. If the nested transaction is aborted the upper transaction can continue in processing and it can finish with commit at the end (but if the top-level transaction rolls-back the nested transaction has to roll-back as well).
The subordinate transaction is a composite part of the top-level transaction. If the subordinate transaction aborts the top-level one aborts as well.

Summary

That's all. Hopefully, you understand a bit more on the meaning of the heuristic outcomes for the XA and JTA specifications. And for sure you won't be writing code like

try {
  UserTransaction.begin();
  ...
  UserTransaction.commit();
} catch (Throwable t) {
  // some strange error happened so we print it to the log
  t.printStackTrace();
}