/ 
Rinci-1.1.23
River stage one • 1 direct dependent • 1 total dependent
/ Rinci::function::Transaction

NAME

Rinci::function::Transaction - Transactional system based on functions

VERSION

version 1.1.23

SPECIFICATION VERSION

1.1

SPECIFICATION

This document describes a transactional system based on functions, where several function calls participate in a single transaction. This transactional system has the following properties:

  • Client/server architecture

    Transaction can be performed over Riap. Client can start more than one active transaction on the server. Each transaction-management request and the function calls are requested separately (each one is a separate Riap request).

    For more details on this, see Riap::Transaction.

  • Undo/redo

    Committed transactions are still recorded in the database along with its undo information. Client can request to undo/redo the transactions. Thus the system is also an undo/redo system.

  • Relies on the functions for reliability/ACID properties

    Server or framework provides the transaction manager (TM), but each function acts as the resource manager (RM). It is the responsibility of the functions to maintain ACID properties while modifying resources. For best results, each function should be written carefully and tested extensively, and utilize a real, robust RM (like an RDBMS to store data or a transactional filesystem layer to read/modify files). In the absence of a real RM, some ACID properties like isolation and consistency might be compromised. For example: one transaction TX1 modifies a file in an ordinary (i.e. non-transactional) filesystem. Another transaction TX2 can see TX1's modification in the middle of uncommitted transaction (violates isolation principle).

How transaction works

Transaction works by relying on undo/redo capability of its participating functions.

Start

User starts a transaction by instantiating TM. TM sets up its data directory and performs cleanup and crash recovery. User then calls $tm->begin(tx_id => $tx_id), providing a unique $tx_id as identifier for the transaction. TM will create an entry for the transaction in its journal. Initial transaction status is i (in-progress).

Calls

User calls one or more functions using $tm->call($name, $args). TM will first check whether function named $foo is indeed eligible to be used in a transaction (see "Function requirements" for more details on this). If function is eligible, TM will first call function using -dry_run=>1 to get undo data. TM then records the undo data first in its journal, to be able to undo the call later (either for rollback in the event of failure, or for undoing the transaction). After that, TM will call the function again, using -undo_action => 'do'. If function fails (returns a non-success status code), rollback will happen.

If function is not eligible, rollback is also triggered.

Commit

If everything goes smoothly, user can call $tm->commit now. TM will mark the transaction as C (committed). TM still stores the transaction and its undo data in its database for some time, because TM allows udoing (and redoing) a transaction.

Transaction status progress:

i -> C

Rollback a transaction in in-progress (i) status

If a call fails, or some other error happens, rollback happens. Rollback can also be started by user using $tm->rollback.

TM will first mark the transaction status to a (aborting an in-progress transaction, rolling back). This will prevent other clients trying to add new calls to this transaction, since aborted transaction can longer accept new calls, it can only be rolled back.

TM will then perform undo for each function, in reverse order, using the undo data previously recorded. If an error happens, rollback fails and transaction status is changed into X (inconsistent, ignored). If rollback succeeds, transaction status is changed into R (rolled back) and the in next cleanup TM can remove the data for this transaction (i.e., forget the transaction).

Transaction status progress:

i -> a -> R  # successful rollback
i -> a -> X  # failed rollback

Undo

TM allows undoing committed transaction, so the transaction system also serves as an undo/redo system.

To perform undo, user invokes $tm->undo(tx_id => $tx_id) where $tx_id is the ID of a committed transaction.

TM will first check that transaction status is indeed C. Then, TM will set status transaction to u (undoing). This will prevents other clients trying to undo the same transaction.

TM then performs a call for each function in reverse order, just like in rollback. However, for each function, the same procedure is applied as in "Calls": first the function is called using -dry_run => 1 as well as passing -undo_action => 'undo_data' and -undo_data. Function returns undo data (a.k.a. redo data, since this data will be used to undo the undo operation). TM records the redo data first in its journal before calling the function again.

If everything went smoothly until the last call, TM will finally set transaction status to U (committed and undone transaction).

Transaction status progress:

C -> u -> U

Rollback a transaction in undoing status

If undo fails in the middle, rollback will happen. TM will first mark the transaction status to v (mnemonic: v comes after u). This will give information to crash recovery process (since handling status a is different, the final goal is R, while for v the final goal is C). TM then will perform the undo calls (in this case, redo) up to that point and finally mark the TM back to C (committed) status.

If rollback fails, TM will set transaction status to X.

Transaction status progress:

u -> v -> C # rollback succeeds
u -> v -> X # rollback fails

Redo

To redo an undone transaction, the same process is applied like in redo. User invokes $tm->redo(tx_id => $tx_id). TM will first check that only transaction with status U is possible to redo. TM then sets transaction status to d (redoing) to prevent other clients to try to redo the same transaction. It then performs the redo calls, recording the undo data before each step. Finally, transaction status is set to C again.

Transaction status progress:

U -> d -> C

Rollback a transaction in redoing status

If redo fails in the middle, rollback will happen. Rolling back a transaction in redoing (d) status is similar to rolling back a transaction in undoing (u) status. TM will first change status of transaction to e (mnemonic: e comes after d). This will give information to crash recovery process that the final goal of rollback is U (not R or C like in a or v recovery). TM then perform the undo calls up to that point and finally mark the TM back to U (committed) status.

If rollback fails, TM will set transaction status to X.

Transaction status progress:

d -> e -> U # rollback succeeds
d -> e -> X # rollback fails

Cleanup

Cleanup is done at TM startup and at regular intervals. TM should delete (forget) all C and U transactions that are too old, or keep the number of those transactions under a certain limit, according to its settings. As soon as those transactions are deleted, they can no longer be undone/redone, since the undo data has been deleted too.

The cleanup process also deletes all X transactions, since they cannot be resolved anyway (TODO: perhaps some retry mechanism can be applied, if desired?)

Cleanup process also deletes all R transactions.

Cleanup process should also rolls back any transactions with status i that have been going for too long.

Crash recovery

Crash can happen at any point during a transaction. TM must perform crash recovery during its startup to resolve things back to a consistent state, usually by doing rollback (but can also by continuing previous undo or redo process).

  • Crash during calls

    Illustration:

     1. client invokes $tm->begin
 2. $tm creates tx entry, status=i
 3. client invokes $tm->call($f1, $args1)
 4. $tm checks $f1, calls $f1 with -dry_run=>1, gets $undo_data1
 5. $tm records $undo_data1 in journal
 6. $tm calls $f1
 7. client invokes $tm->call($f2, $args2)
 8. (like step 4) $tm checks $f2, calls $f2 with -dry_run=>1, gets $undo_data2
 9. (like step 5) $tm records $undo_data2 in journal
10. (like step 6) $tm calls $f2
11. client calls $tm->commit
12. $tm marks tx status=C

    If crash happens right after step 1, 2, 3, or 4, TM does not need to recover anything for these transactions, since there is nothing to recover. After recovery, transaction status is still at i.

    If crash happens right after step 5, TM will perform rollback for this transaction. $f1 will be called with -undo_action => 'undo' and -undo_data => $undo_data1. Function needs to be idempotent (see "Function requirements") and realize that step 6 has not been performed, so undoing a not-done action will not result in an inconsistent state.

    If crash happens right after step 6-11, TM also rolls back the transaction.

    If crash happens right after step 12, no recovery is neeeded since transaction is already committed.

    Crash right after this moment no longer rolls back tx since tx is already committed.

  • Crash during rollback of in-progress transaction

    Illustration:

    1. rollback is started, tx status changed from i to a
2. $tm calls $f2 with -undo_action => 'undo' and -undo_data => $undo_data2
3. $tm marks that f2 has been processed
4. $tm calls $f1 with -undo_action => 'undo' and -undo_data => $undo_data1
5. $tm marks that f1 has been processed
6. $tm marks tx status=R

    If crash happens after step 1, recovery should just continue the rollback process. In other words, during recovery, all transactions with status a should be (continued to be) rolled back.

    However, rollback can continue at the first unprocessed call. So if crash happens after step 2, in recovery rollback continues from step 2. If crash happens after step 3, in recovery rollback can continue from step 4, since $f2 has been marked as processed. If crash happens after step 5, in recovery rollback can continue from step 6.

  • Crash during undoing

    Illustration:

     1. client invokes $tm->undo
 2. $tm changes tx status from C to u
 3. $tm calls $f2 with -undo_action => 'undo', -undo_data => $undo_data2, and
    -dry_run => 1, gets $redo_data2
 4. $tm records $redo_data2 in journal
 5. $tm calls $f2 with -undo_action => 'undo' and -undo_data => $undo_data2
 6. $tm marks $f2 as processed
 7. (like step 3) $tm calls $f1 with -undo_action => 'undo',
    -undo_data => $undo_data2, and -dry_run => 1, gets $redo_data1
 8. (like step 4) $tm records $redo_data1 in journal
 9. (like step 5) $tm calls $f1 with -undo_action => 'undo' and
    -undo_data => $undo_data1.
10. $tm marks $f1 as processed
11. $tm marks tx status=U

    If crash happens after steps 2-10, TM should just continue the undo process.

  • Crash during rollback of undoing transaction

    Illustration:

    1. rollback is started, tx status changed from u to v
2. $tm calls $f1 with -undo_action => 'undo' and -undo_data => $redo_data1
3. $tm marks that f1 has been processed
4. $tm calls $f2 with -undo_action => 'undo' and -undo_data => $redo_data2
5. $tm marks that f2 has been processed
6. $tm marks tx status=C

    If crash happens after steps 2-5, TM should just continue the rollback process.

  • Crash during redoing

    Illustration:

     1. client invokes $tm->redo
 2. $tm changes tx status from U to d
 3. $tm calls $f1 with -undo_action => 'undo', -undo_data => $redo_data1, and
    -dry_run => 1, gets $undo_data1'
 4. $tm records $undo_data1' in journal
 5. $tm calls $f1 with -undo_action => 'undo' and -undo_data => $redo_data1
 6. $tm marks $f1 as processed
 7. (like step 3) $tm calls $f2 with -undo_action => 'undo',
    -undo_data => $redo_data2, and -dry_run => 1, gets $undo_data2'
 8. (like step 4) $tm records $undo_data2' in journal
 9. (like step 5) $tm calls $f2 with -undo_action => 'undo' and
    -undo_data => $redo_data2.
10. $tm marks $f2 as processed
11. $tm marks tx status=C

    If crash happens after steps 2-10, TM should just continue the redo process.

  • Crash during rollback of redoing transaction

    Illustration:

    1. rollback is started, tx status changed from d to e
2. $tm calls $f2 with -undo_action => 'undo' and -undo_data => $undo_data2'
3. $tm marks that f2 has been processed
4. $tm calls $f1 with -undo_action => 'undo' and -undo_data => $redo_data1'
5. $tm marks that f1 has been processed
6. $tm marks tx status=U

    If crash happens after steps 2-5, TM should just continue the rollback process.

Function requirements

Function participating in a transaction is required to declare tx feature to have at least use => 1 or req => 1. What this means is:

  • undo

    Function must support undo operation (undo feature set to true). In addition, there is a stricter requirement for the undo data. Undo data must be a series of calls:

    [[$f1, $args1], [$f2, $args2], ...]

    where $fN are fully-qualified function names (like Foo::Bar::func) and $argsN are hashrefs containing arguments for the functions.

    This allows flexibility and modularity. Functionality (and undo functionality) can be organized into one or more functions.

  • dry_run

    Function must support dry run operation (dry_run feature set to true). In addition, even when called under dry_run => 1, function must still return undo data. This allows TM to record undo data before performing function, to allow for rollback.

Calling function inside another

During transaction, $tm will call functions passing itself in special argument -tx_manager => $tm. Function which wants to call another function inside the scope of transaction must do this using $tm->call.

FAQ

Why is this useful?

The protocol is a pretty generic and simple way to build transactional system, even on heterogenous, multiuser environment. If the functions are written carefully, the system can be reliable. And even if some of the ACID properties are compromised due to lack of real RM, the system is still useful for its undo/redo capability.

What are the drawbacks?

The reliability of the system rests on the reliability of each involved function. One buggy function can break the transaction.

SEE ALSO

Related specifications: Rinci::function::Undo, Rinci::function, Riap::Transaction

Implementations: Perinci::Tx::Manager

AUTHOR

Steven Haryanto <stevenharyanto@gmail.com>

COPYRIGHT AND LICENSE

This software is copyright (c) 2012 by Steven Haryanto.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.