PyBook 07: Transactions

What this notebook proves: "the rich transaction model + EVM decode: parts[], acts[], fees as siblings, the type derivation, reorg handling. Done when: a reorg fixture produces removed + re-added."

Two models, on purpose:

the rich decode.models.Transaction, parts[], fees[], acts[], data_quality, everything an app needs to explain a transaction;
the ledger LedgerTransaction, entries[] only, the durable, syncable row. ledger/bridge.py projects rich into ledger, and the projection is where fees get dropped, deliberately.

Rule #4 is the load-bearing one: every movement is a part[], and a fee is a sibling of the parts, never a movement. Fold gas into the ETH outflow and cost basis is wrong forever.

Everything here is in-memory: explorer rows in, decisions out. No network.

from auradefi.decode.models import BorneBy, Direction, TxStatus, TxSubtype, TxType
from auradefi.decode.pipeline import decode_account
from auradefi.sources.evm.txlist import NormalTxRecord, TokenTxRecord

CHAIN, ACCOUNT_ID = "eip155:1", "acct_1"
ME = "0x" + "11" * 20
COUNTERPARTY = "0x" + "44" * 20
USDC_CONTRACT = "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"
GAS_PRICE = 10**10  # 10 gwei

# One swap, seen twice by the explorer: as a normal tx (ETH out) and as a
# token transfer (USDC in). The decoder must fuse them into ONE transaction.
swap_normal = NormalTxRecord(
    tx_hash="0x" + "cc" * 32, block_number=102, time_stamp=1_700_000_200,
    from_address=ME, to_address=COUNTERPARTY, value_wei=10**18,
    gas_used=120_000, gas_price_wei=GAS_PRICE, is_error=False,
)
swap_token = TokenTxRecord(
    tx_hash="0x" + "cc" * 32, block_number=102, time_stamp=1_700_000_200,
    from_address=COUNTERPARTY, to_address=ME, contract_address=USDC_CONTRACT,
    value_raw=3_000_000_000, token_decimal=6, token_symbol="USDC",
    gas_used=120_000, gas_price_wei=GAS_PRICE,
)

(swap,) = decode_account(CHAIN, ACCOUNT_ID, ME, [swap_normal], [swap_token])

assert swap.id == "txn_557113c18fb02870"      # sha256("chain|hash|account")[:16]
assert swap.status is TxStatus.CONFIRMED
assert swap.type is TxType.TRADE              # DERIVED from part directions
assert swap.subtype is TxSubtype.SWAP
assert swap.initiated_at == 1_700_000_200_000  # seconds in, ms epoch out
print(swap.id, swap.type, swap.subtype, "block", swap.block_number)

txn_557113c18fb02870 trade swap block 102

`parts[]`: one row per movement, both legs visible

An in and an out in the same transaction is what makes it a trade; type is derived from the part directions rather than guessed from the to address (decode.models.derive_tx_type). Each part carries its own from/to, so a UI can render the counterparty without re-reading chain state.

assert [(part.direction, part.asset_id, str(part.quantity)) for part in swap.parts] == [
    (Direction.OUT, "eip155:1/slip44:60", "1"),
    (Direction.IN, f"eip155:1/erc20:{USDC_CONTRACT}", "3000"),
]
assert all(part.act_id == "act_0" for part in swap.parts)
assert swap.parts[0].from_address == ME and swap.parts[0].to_address == COUNTERPARTY

for part in swap.parts:
    print(f"{part.direction.value:>3}  {str(part.quantity):>6}  {part.asset_id}")

out       1  eip155:1/slip44:60
 in    3000  eip155:1/erc20:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48

`acts[]`: the grouping key inside one transaction

An act is a sub-operation; every part and every fee back-references its act_id. The EVM decoder emits exactly one act (act_0) per transaction, and multi-act decomposition (multicalls, ERC-4337 bundles) arrives with protocol decoders, so the shape is right today even though the depth is not. data_quality says so out loud instead of pretending.

assert [act.act_id for act in swap.acts] == ["act_0"]
assert swap.acts[0].subtype is TxSubtype.SWAP
assert swap.acts[0].protocol is None            # no protocol decoder yet: stated, not faked
assert swap.data_quality.incomplete == ("fiat_value",)
assert swap.data_quality.sources == ("etherscan",)
assert swap.data_quality.decoder_version == 1
print(swap.acts[0])
print(swap.data_quality)

Act(act_id='act_0', subtype=<TxSubtype.SWAP: 'swap'>, protocol=None)
DataQuality(incomplete=('fiat_value',), confidence=1.0, decoder_version=1, sources=('etherscan',))

Fees are siblings: and they say who paid

Fee lives beside parts[], never inside it, and carries borne_by:

self, this account sent the transaction and paid the gas;
counterparty, somebody else paid; the fee is reported so the row is explainable, but it is not this account's expense.

Getting this wrong is invisible: the balance still reconciles, the cost basis is just quietly wrong.

from auradefi.money.quantity import Quantity

# 120,000 gas x 10 gwei = 0.0012 ETH, paid by us.
(fee,) = swap.fees
assert fee.quantity == Quantity(1_200_000_000_000_000, 18)
assert str(fee.quantity) == "0.0012"
assert fee.borne_by is BorneBy.SELF
assert fee.asset_id == "eip155:1/slip44:60"
assert fee.act_id == "act_0"
# ...and it is NOT among the movements:
assert all(part.quantity != fee.quantity for part in swap.parts)

# An INCOMING transfer: the sender paid, so the fee is theirs, not ours.
incoming = NormalTxRecord(
    tx_hash="0x" + "aa" * 32, block_number=100, time_stamp=1_700_000_000,
    from_address="0x" + "99" * 20, to_address=ME, value_wei=10**18,
    gas_used=21_000, gas_price_wei=GAS_PRICE, is_error=False,
)
(received,) = decode_account(CHAIN, ACCOUNT_ID, ME, [incoming], [])
assert received.type is TxType.RECEIVE
assert received.fees[0].borne_by is BorneBy.COUNTERPARTY

print(f"swap fee     {fee.quantity} ETH  borne_by={fee.borne_by.value}")
print(f"received fee {received.fees[0].quantity} ETH  borne_by={received.fees[0].borne_by.value}")

swap fee     0.0012 ETH  borne_by=self
received fee 0.00021 ETH  borne_by=counterparty

A failed transaction still cost money

is_error=1 means nothing moved, but the gas was burned. So: status failed, zero parts, one fee. This is the case that makes "a fee is not a movement" pay for itself.

failed_row = NormalTxRecord(
    tx_hash="0x" + "dd" * 32, block_number=103, time_stamp=1_700_000_300,
    from_address=ME, to_address="0x" + "55" * 20, value_wei=5 * 10**17,
    gas_used=21_000, gas_price_wei=GAS_PRICE, is_error=True,
)
(failed,) = decode_account(CHAIN, ACCOUNT_ID, ME, [failed_row], [])

assert failed.status is TxStatus.FAILED
assert failed.parts == ()                       # 0.5 ETH did NOT move
assert failed.fees[0].quantity == Quantity(210_000_000_000_000, 18)
assert failed.type is TxType.INTERACTION
print(f"{failed.id}  status={failed.status.value}  parts={len(failed.parts)}  fee={failed.fees[0].quantity} ETH")

txn_a30f49051566e03d  status=failed  parts=0  fee=0.00021 ETH

The bridge: rich in, ledger out

ledger.bridge.to_ledger_transaction keeps the movements as entries[] and drops the fees: the ledger answers "what did this account hold", and gas is answered by the rich model. A failed transaction therefore bridges to a row with zero entries: present in the ledger (it happened, it has a hash) but moving nothing.

from auradefi.ledger.bridge import to_ledger_transaction
from auradefi.ledger.models import Direction as LedgerDirection

bridged_swap = to_ledger_transaction(swap)
assert [(entry.direction, entry.asset_id) for entry in bridged_swap.entries] == [
    (LedgerDirection.OUT, "eip155:1/slip44:60"),
    (LedgerDirection.IN, f"eip155:1/erc20:{USDC_CONTRACT}"),
]
assert bridged_swap.id == swap.id and bridged_swap.tx_hash == swap.tx_hash
assert to_ledger_transaction(failed).entries == ()   # fee survives ONLY at the rich level
assert not hasattr(bridged_swap, "fees")
print(bridged_swap.id, [str(entry.quantity) for entry in bridged_swap.entries])
print("failed tx bridges to", len(to_ledger_transaction(failed).entries), "entries")

txn_557113c18fb02870 ['1', '3000']
failed tx bridges to 0 entries

Reorg: `removed` + re-`added`

Blocks get orphaned. The contract is Plaid's: a client that only ever replays the cursor stream must converge on the truth, so a reorg emits a removed event for what is gone and an added event for what came back at its new height. Both bump the last-modified sequence, so the cursor stays strictly monotonic and a client that was mid-page loses nothing.

Below: four transactions land, we sync them, then a reorg from block 101 orphans C and re-mines B at block 105.

from auradefi.ledger.backends.memory import MemoryLedger
from auradefi.ledger.models import SyncEventKind
from auradefi.ledger.reorg import plan_reorg

TENANT = "tenant-a"


def normal(tx_hash, block, ts, sender, to, value, gas, is_error=False):
    return NormalTxRecord(
        tx_hash=tx_hash, block_number=block, time_stamp=ts, from_address=sender,
        to_address=to, value_wei=value, gas_used=gas, gas_price_wei=GAS_PRICE,
        is_error=is_error,
    )


def token(tx_hash, block, ts, sender, to, raw, gas):
    return TokenTxRecord(
        tx_hash=tx_hash, block_number=block, time_stamp=ts, from_address=sender,
        to_address=to, contract_address=USDC_CONTRACT, value_raw=raw,
        token_decimal=6, token_symbol="USDC", gas_used=gas, gas_price_wei=GAS_PRICE,
    )


def bridge(normals, tokens):
    return [to_ledger_transaction(rich)
            for rich in decode_account(CHAIN, ACCOUNT_ID, ME, normals, tokens)]


b_normal = normal("0x" + "bb" * 32, 101, 1_700_000_100, ME, USDC_CONTRACT, 0, 50_000)
b_token = token("0x" + "bb" * 32, 101, 1_700_000_100, ME, "0x" + "33" * 20, 25_000_000, 50_000)
fixture = bridge([incoming, b_normal, swap_normal, failed_row], [b_token, swap_token])

ledger = MemoryLedger()
events = ledger.upsert(TENANT, fixture)
assert [event.kind for event in events] == [SyncEventKind.ADDED] * 4
assert [event.transaction.last_modified_seq for event in events] == [1, 2, 3, 4]

first_page = ledger.sync(TENANT, None)
assert first_page.next_cursor == "00000000000000000004"   # f"{seq:020d}", pinned
assert first_page.has_more is False
print("initial sync:", [event.transaction.id for event in first_page.events])
print("cursor:", first_page.next_cursor)

initial sync: ['txn_f7e3f7aba9d6775a', 'txn_e5e727672fb4ada6', 'txn_557113c18fb02870', 'txn_a30f49051566e03d']
cursor: 00000000000000000004

TXN_B, TXN_C = "txn_e5e727672fb4ada6", "txn_557113c18fb02870"

# The canonical chain from block 101 onwards, re-read after the reorg:
# B is back (at block 105), C is gone, D is unchanged.
b_prime = bridge(
    [normal("0x" + "bb" * 32, 105, 1_700_000_500, ME, USDC_CONTRACT, 0, 50_000)],
    [token("0x" + "bb" * 32, 105, 1_700_000_500, ME, "0x" + "33" * 20, 25_000_000, 50_000)],
)[0]
stored = [ledger.get(TENANT, txn.id) for txn in fixture]
plan = plan_reorg(stored, [b_prime, fixture[3]], from_block=101)

assert plan.remove_ids == (TXN_C,)      # orphaned: absent from the canonical set
assert plan.add == (b_prime,)           # re-mined: payload changed, so re-added

reorg_events = ledger.apply_reorg(TENANT, plan)
assert [(event.kind, event.transaction.id) for event in reorg_events] == [
    (SyncEventKind.REMOVED, TXN_C),
    (SyncEventKind.ADDED, TXN_B),
]
assert reorg_events[0].transaction.removed is True
assert reorg_events[1].transaction.block_number == 105

# The client that only replays the cursor sees exactly the delta:
page = ledger.sync(TENANT, "00000000000000000004")
assert [(event.kind.value, event.transaction.id) for event in page.events] == [
    ("removed", TXN_C), ("added", TXN_B),
]
assert page.next_cursor == "00000000000000000006"
assert "00000000000000000004" < page.next_cursor  # lexicographic == numeric, by design
for event in page.events:
    print(f"{event.kind.value:>7}  {event.transaction.id}  block={event.transaction.block_number}")

removed  txn_557113c18fb02870  block=102
  added  txn_e5e727672fb4ada6  block=105

Resurrection

An orphaned transaction that resurfaces in a later block is not a new row: same derived id, removed flips back to False, and the sequence bumps again so the cursor stream carries it. Ids are derived from (chain, hash, account), which is exactly what makes this expressible.

c_prime = bridge(
    [normal("0x" + "cc" * 32, 106, 1_700_000_600, ME, COUNTERPARTY, 10**18, 120_000)],
    [token("0x" + "cc" * 32, 106, 1_700_000_600, COUNTERPARTY, ME, 3_000_000_000, 120_000)],
)[0]
(resurrected,) = ledger.upsert(TENANT, [c_prime])

assert resurrected.kind is SyncEventKind.ADDED
assert resurrected.transaction.id == TXN_C          # the SAME id, not a new row
assert resurrected.transaction.removed is False
assert resurrected.transaction.block_number == 106

final = ledger.sync(TENANT, "00000000000000000006")
assert [event.transaction.id for event in final.events] == [TXN_C]
assert final.next_cursor == "00000000000000000007"
print(f"{TXN_C} is back at block {resurrected.transaction.block_number}; cursor {final.next_cursor}")

txn_557113c18fb02870 is back at block 106; cursor 00000000000000000007

Honest edges

The EVM decoder consumes Etherscan-shaped txlist + tokentx rows. There are no protocol decoders yet, so acts[] is always one act and protocol is always None: data_quality.incomplete names what is missing instead of inventing a label.
value/price on parts are None: historical pricing is not shipped, which is why data_quality.incomplete carries "fiat_value".
Internal transfers are not auto-detected here; the accounting layer takes an explicit internal_transfer_ids set (PyBook 11).

Next: 08_positions. What the balances mean.