nti.transactions

Extensions to the transaction package.

Transaction Management

nti.transactions.loop.TransactionsLoop is a retryable transaction manager. It is conceptually similar to the attempts context manager provided by the transaction package itself, but much more powerful and extensible via subclasses. Features include:

• Configurable commit vetos.

• Extensible tests for which exceptions should be retried.

• The ability to abort the transaction and bypass a potentially expensive commit when there are expected to be no side-effects.

• Sleeping between retries.

• Extensive logging and timing.

The TransactionLoop can be used as-is, or it can be subclassed for customization. For use in a Pyramid tween, for example, a minimal subclass might look like this (see nti.transactions.pyramid_tween for a full-featured tween):

>>> class PyramidTransactionLoop(TransactionLoop):
...    def prep_for_retry(self, number, request):
...        request.make_body_seekable()
...    def describe_transaction(self, request):
...        return request.url

Data Managers

A few data managers are provided for convenience.

The first data manager is used to put an object in a queue (something with the full and put_nowait methods) when a transaction succeeds. If the queue is full, then the transaction will not be allowed to commit:

>>> from nti.transactions.queue import put_nowait
>>> put_nowait(queue, object)

This is a special case of the ObjectDataManager, which will call one method with any arguments when a transaction commits. It can be configured to vote on whether the transaction should be allowed to commit. or not. This is useful for, say, putting an item in a Redis queue when the transaction is successful. It can be constructed directly, but the do function is a shorthand way of joining one to the current transaction:

>>> from nti.transactions.manager import do
>>> do(print, args=("Committed"))

Caution

See the documentation of this object for numerous warnings about side-effects and its interaction with the transaction machinery. Use it with care!

• Changes

nti.transactions.interfaces

Interfaces for nti.transactions.

exception AbortFailedError

Bases: TransactionError

Aborting the active transaction failed for an unknown and unexpected reason.

This is raised instead of raising very generic system exceptions such as ValueError and AttributeError.

exception CommitFailedError

Bases: TransactionError

Committing the active transaction failed for an unknown and unexpected reason.

This is raised instead of raising very generic system exceptions such as TypeError.

exception ForeignTransactionError

Bases: TransactionLifecycleError

Raised when a transaction manager has its transaction changed while a controlling transaction loop believes it is in control.

The handler first aborted or committed the transaction, and then began a new one. Possibly many times.

A kind of TransactionLifecycleError. Applications must not raise this exception.

This is a programming error.

interface IAfterTransactionBegan

Extends: nti.transactions.interfaces.ILoopEvent

A new transaction has begun.

tx

The transaction.

interface IExtendedTransaction

Extends: transaction.interfaces.ITransaction

Extensions to the transaction api.

nti_commit()

Like commit, but produces a perfmetrics.Metric transaction.commit metric.

nti_abort()

Like abort, but produces a perfmetrics.Metric transaction.abort metric.

interface ILoopEvent

Base class for event loop events.

The handler_args and handler_kwargs should not be mutated in place or assigned to. These are a convenience for accessing the attributes of the invocation attribute.

Changed in version 4.2.0: Add the handler_args and handler_kwargs.

invocation

An ILoopInvocation.

handler_args

The positional arguments for the handler, or ()

handler_kwargs

The keyword arguments for the handler, or {}

interface ILoopInvocation

Description of why a loop was invoked.

handler

The handler to run.

loop

The loop doing the running.

args

The arguments passed to the handler

kwargs

The keyword arguments passed to the handler.

interface IWillAttemptTransaction

Extends: nti.transactions.interfaces.ILoopEvent

Base class for attempt events.

tx

The transaction.

attempt_number

The number of the attempt. Starts at 0.

interface IWillFirstAttempt

Extends: nti.transactions.interfaces.IWillAttemptTransaction

The first attempt.

interface IWillFirstAttemptWithRequest

Extends: nti.transactions.interfaces.IWillFirstAttempt

The first attempt, but with a request object.

Added in version 4.2.0.

request

The request object that will be used.

interface IWillLastAttempt

Extends: nti.transactions.interfaces.IWillRetryAttempt

The final retry.

interface IWillLastAttemptWithRequest

Extends: nti.transactions.interfaces.IWillLastAttempt

The final retry with a request object.

Added in version 4.2.0.

request

The request object that will be used.

interface IWillRetryAttempt

Extends: nti.transactions.interfaces.IWillAttemptTransaction

A retry attempt.

interface IWillRetryAttemptWithRequest

Extends: nti.transactions.interfaces.IWillRetryAttempt

A retry attempt with a request object.

Added in version 4.2.0.

request

The request object that will be used.

interface IWillSleepBetweenAttempts

Extends: nti.transactions.interfaces.ILoopEvent

Will sleep between attempts.

If the sleep_time attribute is modified, that will be the time slept.

sleep_time

The time to sleep.

exception TransactionLifecycleError

Bases: TransactionError

Raised when an application commits or aborts a transaction while the transaction controller believes it is in control.

Applications must not raise this exception.

This may have happened many times; we cannot detect that.

This is a programming error.

nti.transactions.manager

Support for data managers.

class ObjectDataManager(target=None, method_name=None, call=None, vote=None, args=(), kwargs=None)

Bases: object

A generic (and therefore relatively expensive) transaction.interfaces.IDataManager that invokes a callable (usually a method of an object) when the transaction finishes successfully. The method should not raise exceptions when invoked, as they will be caught and ignored (to preserve consistency with the rest of the data managers). If there’s a chance the method could fail, then whatever actions it does take should not have side-effects.

These data managers have no guaranteed relationship to other data managers in terms of the order of which they commit, except as documented with sortKey().

Because these data managers execute exactly one operation on a complete transaction commit, implementing a savepoint is trivial: do nothing when it is rolled back. A savepoint is created to checkpoint a transaction and rolled back to reverse actions taken after the checkpoint. Only data managers that were active (joined) at the time the transaction savepoint is created are asked to create their own savepoint, and then potentially to roll it back. We do no work until the transaction is committed, so we have nothing to rollback. Moroever, if a transaction savepoint is activated before a manager joins, then that manager is not asked for its own savepoint: it is simply aborted and unjoined from the transaction if the previous savepoint is rolledback.

Construct a data manager. You must pass either the target and method_name arguments or the call argument. (You may always pass the target argument, which will be made available on this object for the use of tpc_vote(). )

Parameters:

• target – An object. Optional if call is given. If provided, will be used to compute the sortKey().

• method_name (string) – The name of the attribute to get from target. Optional if callable is given.

• call (callable) – A callable object. Ignored if target and method_name are given.

• vote (callable) – If given, then a callable that will be called during the voting phase. It should raise an exception if the transaction will fail.

• args – A sequence of arguments to pass to the callable object. Optional.

• kwargs – A dictionary of arguments to pass to the callable object. Optional.

abort_sub(tx)

Does nothing

afterCompletion(tx)

Does nothing

beforeCompletion(tx)

Does nothing

commit_sub(tx)

Does nothing

sortKey()

Return the string value that, when sorted, determines the order in which data managers will get to vote and commit at the end of a transaction. (See transaction.interfaces.IDataManager.sortKey()).

The default implementation of this method uses the ID of either the target object we were initialized with or the ID of the actual callable we will call. This has the property of ensuring that all calls to methods of a particular object instance (when target is given), or calls to a particular callable (when target is not given) will execute in the order in which they were added to the transaction.

Note

This relies on an implementation detail of the transaction package, which sorts using list.sort(), which is guaranteed to be stable: thus objects using the same key remain in the same relative order. (See transaction._transaction.Transaction._commitResources().)

To execute only calls to a particular method of a particular instance in the order they are added to the transaction, but allow other methods to execute before or after them, do not provide the target.

It is not advisable to use the ID of this object (self) in the implementation of this method, because the ID values are not guaranteed to be monotonically increasing and thus instances of a particular class that did this would execute in “random” order.

class OrderedNearEndObjectDataManager(target=None, method_name=None, call=None, vote=None, args=(), kwargs=None)

Bases: ObjectDataManager

A special extension of ObjectDataManager that attempts to execute after all other data managers have executed. This is useful when an operation relies on the execution of other data managers.

Added in version 1.1.

Construct a data manager. You must pass either the target and method_name arguments or the call argument. (You may always pass the target argument, which will be made available on this object for the use of tpc_vote(). )

Parameters:

• target – An object. Optional if call is given. If provided, will be used to compute the sortKey().

• method_name (string) – The name of the attribute to get from target. Optional if callable is given.

• call (callable) – A callable object. Ignored if target and method_name are given.

• vote (callable) – If given, then a callable that will be called during the voting phase. It should raise an exception if the transaction will fail.

• args – A sequence of arguments to pass to the callable object. Optional.

• kwargs – A dictionary of arguments to pass to the callable object. Optional.

sortKey()

Sort prepended with z’s in an attempt to execute after other data managers.

do(*args, **kwargs)

Establishes a IDataManager in the current transaction. See ObjectDataManager for the possible arguments.

do_near_end(*args, **kwargs)

Establishes a IDataManager in the current transaction that will attempt to execute after all other DataManagers have had their say. See ObjectDataManager for the possible arguments.

Added in version 1.1.

nti.transactions.queue

Support for transactionally working with queues.

put_nowait(queue, obj)

Transactionally puts obj in queue. The obj will only be visible in the queue after the current transaction successfully commits. If the queue cannot accept the object because it is full, the transaction will be aborted.

See gevent.queue.Queue and Queue.Full and gevent.queue.

nti.transactions.loop

Support for working with the transaction module.

This module imports the dm.transaction.aborthook module and directly provides the add_abort_hooks() function; you should call this if you need such functionality.

class TransactionLoop(handler, retries: int = None, sleep: float = None, long_commit_duration: float = None, transaction_manager=None)

Bases: object

Similar to the transaction attempts mechanism, but less error prone and with added logging and hooks.

This object is callable (passing any arguments along to its handler) and runs its handler in the transaction loop.

The handler code may doom the transaction, but they must not attempt to commit or abort it. A doomed transaction, or one whose commit is vetoed by should_abort_due_to_no_side_effects() or should_veto_commit() is never retried.

Aborting or committing the transaction will set these perfmetrics timers:

transaction.commit

Time taken to commit the transaction. Sampled.

transaction.abort

Time taken to abort the transaction. Sampled.

Running the loop will increment these perfmetrics counters (new in 3.1):

transaction.retry

The number of retries required in any given loop. Note that if the handler raises non-retryable exceptions, this number will be inflated.

transaction.side_effect_free

How many side-effect free transactions there have been.

transaction.side_effect_free_violation

How many side-effect free transactions actually had resource managers joined to the transaction, and so potentially aborted work that wanted to be committed. (3.1.1)

transaction.vetoed

How many transactions were vetoed.

transaction.doomed

The number of doomed transactions.

transaction.successful

The number of transactions that successfully returned.

transaction.failed

The number of transactions that did not successfully return.

Important

Instances of this class must be thread safe, and running the loop should not mutate the state of this object.

Changed in version 3.0: When this object is called, the thread-local default or global transaction.TransactionManager is placed into explicit mode (if it wasn’t already). The handler callable is forbidden from beginning, aborting or committing the transaction. Explicit transactions can be faster in ZODB, and are easier to reason about.

If the application begins, commits or aborts the transaction, it can expect this object to raise transaction.interfaces.NoTransaction, transaction.interfaces.AlreadyInTransaction or nti.transactions.interfaces.TransactionLifecycleError.

Changed in version 3.1: zope.event is used to publish events after each transaction begins, before a transaction is retried or the first attempt is made, and before we sleep between retries. See nti.transaction.interfaces.IAfterTransactionBegan, nti.transaction.interfaces.IWillFirstAttempt, nti.transaction.interfaces.IWillRetryAttempt, nti.transaction.interfaces.IWillSleepBetweenAttempt.

Parameters:

• sleep (float) – Sets the sleep.

• retries (int) – If given, the number of times a transaction will be retried. Note that this is one less than the total number of attempts

• transaction_manager – If not None, this is what will be returned from get_transaction_manager_for_call() instead of the default transaction manager.

Changed in version 5.0.0: Add the transaction_manager argument.

exception AbortAndReturn(response, reason)

Bases: Exception

This private exception is raised here to cause us to break out of our loop, abort the transaction, and return the result.

Changed in version 3.0: Previously this was called AbortException. Until 4.0, that name remains available as a deprecated alias.

AbortException

Deprecated alias for AbortAndReturn

alias of AbortAndReturn

EVT_WILL_FIRST_ATTEMPT

The event to send before running the handler the first time. Subclasses may override.

..versionadded:: 4.2.0

alias of WillFirstAttempt

EVT_WILL_LAST_ATTEMPT

The event to send before running the handler the last time. Subclasses may override.

..versionadded:: 4.2.0

alias of WillLastAttempt

EVT_WILL_RETRY_ATTEMPT

The event to send before running the handler the second time, up until the last time. Subclasses may override.

..versionadded:: 4.2.0

alias of WillRetryAttempt

describe_transaction(*args, **kwargs)

Return a note for the transaction.

This should return a string or None. If it returns a string, that value will be used via transaction.note()

get_transaction_manager_for_call(*args, **kwargs)

Return the transaction manager to use for a particular call.

By default, this is the global/default thread-local transaction manager.

The arguments are the arguments passed to __call__.

Subclasses may override. The expected use case is for transactions that are isolated and scoped to a particular well-defined location. Most likely, you’ll keep a reference to the transaction manager outside the loop object (or, if the arguments don’t matter, you can just call this method without supplying them) for later use. For example, when getting a connection from ZODB:

txm = TransactionManager()
loop = TransactionLoop(..., transaction_manager=txm)
db = ... # get ZODB DB
with db.open(loop.get_transaction_manager_for_call()) as conn:
    ...

Added in version 5.0.0.

on_begin_failed(exc_info, txm, args, kwargs)

Called when begin() raised an error other than AlreadyInTransaction, probably due to a bad synchronizer. Subclasses may override this method.

After this is called, the tranasction manager txm will be aborted.

This method must not raise an exception.

Added in version 4.0.1.

prep_for_retry(attempts_remaining, tx, *args, **kwargs)

Called just after a transaction begins if there will be more than one attempt possible. Do any preparation needed to cleanup or prepare reattempts, or raise AbortAndReturn if that’s not possible.

Parameters:

• attempts_remaining (int) – How many attempts remain. Will always be at least 1.

• tx – The transaction that’s just begun.

Changed in version 3.1: Add the tx parameter.

run_handler(*args, **kwargs)

Actually execute the callable handler.

This is called from our __call__ method. Subclasses may override to customize how the handler is called.

static setUp()

Called by __call__ before making any attempts or beginning any transaction.

When this method is called, it is guaranteed that transaction.manager.explicit is true.

Subclasses may override this method. If they are not a direct subclass of this class, they should be sure to call the super implementation; it is not necessary to call this implementation.

Added in version 3.0.

should_abort_due_to_no_side_effects(*args, **kwargs)

Called after the handler has run. If the handler should have produced no side effects and the transaction can be aborted as an optimization, return True.

This defaults to the value of side_effect_free.

should_veto_commit(result, *args, **kwargs)

Called after the handler has run. If the result of the handler should abort the transaction, return True.

static tearDown()

Called by __call__() just before returning, in all cases, once setUp has been called.

When this method is called, it is guaranteed that transaction.manager.explicit is at its original value.

Subclasses may override this method. If they are not a direct subclass of this class, they should be sure to call the super implementation; it is not necessary to call this implementation.

If this method raises an exception, the original return value of the handler, or its exception, will be lost.

Added in version 3.0.

attempts = 3

How many total attempts will be made, including the initial. This can be set in a subclass, if not passed to the constructor.

long_commit_duration = 6

Commits longer than this (seconds) will trigger a warning log message. This can be set in a subclass if not passed to the constructor.

side_effect_free = False

The default return value from should_abort_due_to_no_side_effects(). If you are not subclassing, or you do not need access to the arguments of the called function to make this determination, you may set this as an instance variable.

side_effect_free_log_level = 10

The log level at which to report violations of side-effect-free transactions (those that are reported as should_abort_due_to_no_side_effects(), but which nonetheless have resource managers joined). This usually signifies a programming error, and results in throwing away work. If this is set to logging.ERROR or higher, an exception will be raised when this happens; this is useful for testing. The default value is logging.DEBUG.

Added in version 4.1.0.

side_effect_free_resource_report_limit = 5

If the number of resources joined to a transaction exceeds this, only a summary will be logged.

Added in version 4.1.0.

sleep = None

If not None, this is a number that specifies the base amount of time (in seconds) to wait after a failed transaction attempt before retrying. Each retry attempt will pick a delay following the binary exponential backoff algorithm: sleep * (random number between 0 and 2^retry-1). (A simple increasing multiplier might work well if there is only one other transaction that we conflict with, but in cases of multiple conflicts or even new conflicts, the random backoff should provide higher overall progress.) This can be set in a subclass if not passed to the constructor.

nti.transactions.pyramid_tween

A Pyramid tween that begins and ends transactions around its handler using the nti.transactions.loop.TransactionLoop.

This very similar to earlier versions of pyramid_tm, but with the following substantial differences:

• The transaction is rolled back if the request is deemed to be side-effect free (this has intimate knowledge of the paths that do not follow the HTTP rules for a GET being side-effect free; however, if you are a GET request and you violate the rules by having side-effects, you can set the environment key nti.request_had_transaction_side_effects to True)

• Logging is added to account for the time spent in aborts and commits.

• Later versions of pyramid_tm were split into two parts, with pyramid_retry being used to handle retry using an “execution policy”, which was new in Pyramid 1.9. This library is compatible with all versions of Pyramid from 1.2 onward.

Install this tween using the add_tween method:

pyramid_config.add_tween(
    'nti.transactions.pyramid_tween.transaction_tween_factory',
    under=pyramid.tweens.EXCVIEW)

You may install it under or over the exception view, depending on whether you need the transaction to be active in exception views.

If you have a tween that manages a ZODB connection, it should be installed above this tween. That’s because ZODB connections cannot be closed while joined to a transaction; the transaction must be committed or aborted first.

class TransactionTween(handler, retries: int = None, sleep: float = None, long_commit_duration: float = None, transaction_manager=None)

Bases: TransactionLoop

A Pyramid tween for retrying execution of a request.

When the request body is executing, the functions is_last_attempt() and is_error_retryable() can be used to influence handler behaviour.

Changed in version 4.2.0: For convenience, now emits nti.transactions.interfaces.IWillFirstAttemptWithRequest, and the other WithRequest events, instead of events without the request object.

Parameters:

• sleep (float) – Sets the sleep.

• retries (int) – If given, the number of times a transaction will be retried. Note that this is one less than the total number of attempts

• transaction_manager – If not None, this is what will be returned from get_transaction_manager_for_call() instead of the default transaction manager.

Changed in version 5.0.0: Add the transaction_manager argument.

EVT_WILL_FIRST_ATTEMPT

alias of WillFirstAttemptWithRequest

EVT_WILL_LAST_ATTEMPT

alias of WillLastAttemptWithRequest

EVT_WILL_RETRY_ATTEMPT

alias of WillRetryAttemptWithRequest

describe_transaction(request)

Return a note for the transaction.

This should return a string or None. If it returns a string, that value will be used via transaction.note()

prep_for_retry(attempts_remaining, tx, request)

Prepares the request for possible retries.

Buffers the body if needed using pyramid.request.Request.make_body_seekable().

The first time this is called for a given request, if the method is expected to have a body and the body appears to be JSON, but the content type specifies a browser form submission, the content type is changed to be application/json. This is a simple fix for broken clients that forget to set the HTTP content type. This may be removed in the future.

Caution

This doesn’t do anything with the WSGI environment; changes to that persist between retries.

Changed in version 4.2.0: Now, transient attributes of the request object (those whose name begins with _v_) are deleted by this function before it returns for the second and susquent attempts. Previously, there were left with their existing values. This is intended to make request-specific caching easier, following a similar model as persistent and allowing for common patterns such as zope.cachedescriptors.property.Lazy.

run_handler(request)

Actually execute the callable handler.

This is called from our __call__ method. Subclasses may override to customize how the handler is called.

should_abort_due_to_no_side_effects(request)

Tests with is_side_effect_free().

If the request’s environ has a true value for the key 'nti.request_had_transaction_side_effects', this method will return false.

should_veto_commit(response, request)

Tests with commit_veto().

commit_veto(request, response)

When used as a commit veto, the logic in this function will cause the transaction to be aborted if:

• An X-Tm response header with the value 'abort' (or any value other than 'commit') exists. A value of 'commit' overrides any later match in this list.

• The response status code starts with '4' or '5'.

• The request environment has a true value for 'nti.commit_veto'

Otherwise the transaction will be allowed to commit.

is_error_retryable(request, exc)

Return True if the exception is one that can be retried.

This will return False if the request is on its last attempt.

is_last_attempt(request)

Return True if the request is on its last attempt, meaning that the TransactionTween will not invoke the handler again, regardless of what happens during this attempt.

is_side_effect_free(request)

Is the request side-effect free? If the answer is yes, we should be able to quietly abort the transaction and avoid taking out any locks in the DBs.

A request is considered to be free of side effects if:

• It is a GET or HEAD request; AND

• The URL DOES NOT match a configured exception list.

In this version, the configured exception list is not actually configurable. It is hardcoded to handle the case of socket.io polling (which does have side effects on GET requests), while still supporting serving the static resources of socket.io.

transaction_tween_factory(handler, registry)

The factory to create the tween.

The registry argument from Pyramid is used to access the Pyramid “deployment settings” to configure the behaviour of the transaction loop. The following settings are used.

retry.attempts

Int. The number of retry attempts. See TransactionLoop.attempts for the default.

retry.long_commit_duration

Float. The number of seconds after which a commit will be considered “too long” and a warning logging message issued. See TransactionLoop.long_commit_duration for the default.

retry.sleep_ms

Int. The base number of milliseconds to sleep between retry attempts. See TransactionLoop.sleep for more documentation.

See TransactionTween

nti.transactions.transactions (deprecated)

This module only contains backwards compatibility imports.

Indices and tables

• Index

• Module Index

• Search Page