Skip to content

Commit 58edb26

Browse files
saddlepaddleKyleAMathewsautofix-ci[bot]
authored
feat(sqlite-persistence): prune applied_tx by default with safe replay recovery (#1572)
* feat(node-db-sqlite-persistence): prune applied_tx by default createNodeSQLitePersistence left appliedTxPruneMaxRows/appliedTxPruneMaxAgeSeconds unset, so the applied_tx log grew without bound for every consumer. Default them to 1,000 rows and a 24h age backstop (both overridable; pass 0 to disable), exported as DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS / DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS. * fix: make sqlite replay pruning recovery-aware * ci: apply automated fixes * docs: clarify sqlite pruning scope * feat: default sqlite applied_tx pruning across wrappers --------- Co-authored-by: Kyle Mathews <mathews.kyle@gmail.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
1 parent 3827b62 commit 58edb26

18 files changed

Lines changed: 407 additions & 45 deletions

File tree

‎.changeset/node-default-applied-tx-pruning.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
'@tanstack/browser-db-sqlite-persistence': minor
3+
'@tanstack/capacitor-db-sqlite-persistence': minor
4+
'@tanstack/cloudflare-durable-objects-db-sqlite-persistence': minor
5+
'@tanstack/db-sqlite-persistence-core': minor
6+
'@tanstack/expo-db-sqlite-persistence': minor
7+
'@tanstack/node-db-sqlite-persistence': minor
8+
'@tanstack/react-native-db-sqlite-persistence': minor
9+
'@tanstack/tauri-db-sqlite-persistence': minor
10+
---
11+
12+
SQLite persistence wrappers now prune the `applied_tx` replay log by default so SQLite files no longer grow without bound. When prune options are omitted, wrappers that construct the shared SQLite core adapter apply `appliedTxPruneMaxRows: 1_000` and `appliedTxPruneMaxAgeSeconds: 86_400` (24h). Both remain overridable, and passing `0` disables that limit. The defaults are exported as `DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS` and `DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS` from the shared SQLite core package and re-exported by wrapper packages.
13+
14+
The shared SQLite core adapter now treats `applied_tx` as a bounded replay cache during `pullSince` recovery. If a recovery request starts before the retained replay window, `pullSince` returns `requiresFullReload: true` instead of returning partial deltas.

‎packages/browser-db-sqlite-persistence/src/browser-persistence.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
import {
2+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
3+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
24
SingleProcessCoordinator,
35
createSQLiteCorePersistenceAdapter,
46
} from '@tanstack/db-sqlite-persistence-core'
@@ -78,8 +80,11 @@ function resolveAdapterBaseOptions(
7880
`driver` | `schemaVersion` | `schemaMismatchPolicy`
7981
> {
8082
return {
81-
appliedTxPruneMaxRows: options.appliedTxPruneMaxRows,
82-
appliedTxPruneMaxAgeSeconds: options.appliedTxPruneMaxAgeSeconds,
83+
appliedTxPruneMaxRows:
84+
options.appliedTxPruneMaxRows ?? DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
85+
appliedTxPruneMaxAgeSeconds:
86+
options.appliedTxPruneMaxAgeSeconds ??
87+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
8388
pullSinceReloadThreshold: options.pullSinceReloadThreshold,
8489
}
8590
}

‎packages/browser-db-sqlite-persistence/src/index.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,11 @@ export type {
88
} from './browser-persistence'
99
export type { OpenBrowserWASQLiteOPFSDatabaseOptions } from './opfs-database'
1010
export type { BrowserCollectionCoordinatorOptions } from './browser-coordinator'
11-
export { persistedCollectionOptions } from '@tanstack/db-sqlite-persistence-core'
11+
export {
12+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
13+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
14+
persistedCollectionOptions,
15+
} from '@tanstack/db-sqlite-persistence-core'
1216
export type {
1317
PersistedCollectionCoordinator,
1418
PersistedCollectionPersistence,

‎packages/capacitor-db-sqlite-persistence/src/capacitor-persistence.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
import {
2+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
3+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
24
SingleProcessCoordinator,
35
createSQLiteCorePersistenceAdapter,
46
} from '@tanstack/db-sqlite-persistence-core'
@@ -81,8 +83,11 @@ function resolveAdapterBaseOptions(
8183
`driver` | `schemaVersion` | `schemaMismatchPolicy`
8284
> {
8385
return {
84-
appliedTxPruneMaxRows: options.appliedTxPruneMaxRows,
85-
appliedTxPruneMaxAgeSeconds: options.appliedTxPruneMaxAgeSeconds,
86+
appliedTxPruneMaxRows:
87+
options.appliedTxPruneMaxRows ?? DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
88+
appliedTxPruneMaxAgeSeconds:
89+
options.appliedTxPruneMaxAgeSeconds ??
90+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
8691
pullSinceReloadThreshold: options.pullSinceReloadThreshold,
8792
}
8893
}

‎packages/capacitor-db-sqlite-persistence/src/index.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,11 @@ export type {
55
CapacitorSQLiteSchemaMismatchPolicy,
66
SQLiteDBConnection,
77
} from './capacitor'
8-
export { persistedCollectionOptions } from '@tanstack/db-sqlite-persistence-core'
8+
export {
9+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
10+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
11+
persistedCollectionOptions,
12+
} from '@tanstack/db-sqlite-persistence-core'
913
export type {
1014
PersistedCollectionCoordinator,
1115
PersistedCollectionPersistence,

‎packages/cloudflare-durable-objects-db-sqlite-persistence/src/do-persistence.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
import {
2+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
3+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
24
SingleProcessCoordinator,
35
createSQLiteCorePersistenceAdapter,
46
} from '@tanstack/db-sqlite-persistence-core'
@@ -79,8 +81,11 @@ function resolveAdapterBaseOptions(
7981
`driver` | `schemaVersion` | `schemaMismatchPolicy`
8082
> {
8183
return {
82-
appliedTxPruneMaxRows: options.appliedTxPruneMaxRows,
83-
appliedTxPruneMaxAgeSeconds: options.appliedTxPruneMaxAgeSeconds,
84+
appliedTxPruneMaxRows:
85+
options.appliedTxPruneMaxRows ?? DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
86+
appliedTxPruneMaxAgeSeconds:
87+
options.appliedTxPruneMaxAgeSeconds ??
88+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
8489
pullSinceReloadThreshold: options.pullSinceReloadThreshold,
8590
}
8691
}

‎packages/cloudflare-durable-objects-db-sqlite-persistence/src/index.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,11 @@ export type {
44
CloudflareDOSQLitePersistenceOptions,
55
DurableObjectStorageLike,
66
} from './do-persistence'
7-
export { persistedCollectionOptions } from '@tanstack/db-sqlite-persistence-core'
7+
export {
8+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
9+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
10+
persistedCollectionOptions,
11+
} from '@tanstack/db-sqlite-persistence-core'
812
export type {
913
PersistedCollectionCoordinator,
1014
PersistedCollectionPersistence,

‎packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts‎

Lines changed: 61 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,21 @@ export type SQLitePullSinceResult<TKey extends string | number> =
7373

7474
const DEFAULT_SCHEMA_VERSION = 1
7575
const DEFAULT_PULL_SINCE_RELOAD_THRESHOLD = 128
76+
77+
/**
78+
* Default cap on retained `applied_tx` rows per collection. The log is a
79+
* replayable cache, so a bounded row count keeps SQLite files from growing
80+
* without limit. Pass `appliedTxPruneMaxRows: 0` to disable the row cap.
81+
*/
82+
export const DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS = 1_000
83+
84+
/**
85+
* Default age backstop for retained `applied_tx` rows, in seconds (24h). Rows
86+
* older than this are pruned on the next write. Pass
87+
* `appliedTxPruneMaxAgeSeconds: 0` to disable the age backstop.
88+
*/
89+
export const DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS = 24 * 60 * 60
90+
7691
const SQLITE_MAX_IN_BATCH_SIZE = 900
7792
const SAFE_IDENTIFIER_PATTERN = /^[a-zA-Z_][a-zA-Z0-9_]*$/
7893
const FORBIDDEN_SQL_FRAGMENT_PATTERN = /(;|--|\/\*)/
@@ -1540,42 +1555,64 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter {
15401555
const collectionTableSql = quoteIdentifier(tableMapping.tableName)
15411556
const tombstoneTableSql = quoteIdentifier(tableMapping.tombstoneTableName)
15421557

1543-
const [changedRows, deletedRows, latestVersionRows, replayRows] =
1544-
await Promise.all([
1545-
this.driver.query<{ key: string }>(
1546-
`SELECT key
1558+
const [
1559+
changedRows,
1560+
deletedRows,
1561+
latestVersionRows,
1562+
replayRows,
1563+
replayAvailabilityRows,
1564+
] = await Promise.all([
1565+
this.driver.query<{ key: string }>(
1566+
`SELECT key
15471567
FROM ${collectionTableSql}
15481568
WHERE row_version > ?`,
1549-
[fromRowVersion],
1550-
),
1551-
this.driver.query<{ key: string }>(
1552-
`SELECT key
1569+
[fromRowVersion],
1570+
),
1571+
this.driver.query<{ key: string }>(
1572+
`SELECT key
15531573
FROM ${tombstoneTableSql}
15541574
WHERE row_version > ?`,
1555-
[fromRowVersion],
1556-
),
1557-
this.driver.query<{ latest_row_version: number }>(
1558-
`SELECT latest_row_version
1575+
[fromRowVersion],
1576+
),
1577+
this.driver.query<{ latest_row_version: number }>(
1578+
`SELECT latest_row_version
15591579
FROM collection_version
15601580
WHERE collection_id = ?
15611581
LIMIT 1`,
1562-
[collectionId],
1563-
),
1564-
this.driver.query<{
1565-
tx_id: string
1566-
row_version: number
1567-
replay_json: string | null
1568-
replay_requires_full_reload: number
1569-
}>(
1570-
`SELECT tx_id, row_version, replay_json, replay_requires_full_reload
1582+
[collectionId],
1583+
),
1584+
this.driver.query<{
1585+
tx_id: string
1586+
row_version: number
1587+
replay_json: string | null
1588+
replay_requires_full_reload: number
1589+
}>(
1590+
`SELECT tx_id, row_version, replay_json, replay_requires_full_reload
15711591
FROM applied_tx
15721592
WHERE collection_id = ? AND row_version > ?
15731593
ORDER BY term ASC, seq ASC`,
1574-
[collectionId, fromRowVersion],
1575-
),
1576-
])
1594+
[collectionId, fromRowVersion],
1595+
),
1596+
this.driver.query<{ min_row_version: number | null }>(
1597+
`SELECT MIN(row_version) AS min_row_version
1598+
FROM applied_tx
1599+
WHERE collection_id = ?`,
1600+
[collectionId],
1601+
),
1602+
])
15771603

15781604
const latestRowVersion = latestVersionRows[0]?.latest_row_version ?? 0
1605+
const replayFloor = replayAvailabilityRows[0]?.min_row_version
1606+
if (
1607+
latestRowVersion > fromRowVersion &&
1608+
(replayFloor == null || replayFloor > fromRowVersion + 1)
1609+
) {
1610+
return {
1611+
latestRowVersion,
1612+
requiresFullReload: true,
1613+
}
1614+
}
1615+
15791616
const changedKeyCount = changedRows.length + deletedRows.length
15801617

15811618
if (changedKeyCount > this.pullSinceReloadThreshold) {

‎packages/expo-db-sqlite-persistence/src/expo.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
import {
2+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
3+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
24
SingleProcessCoordinator,
35
createSQLiteCorePersistenceAdapter,
46
} from '@tanstack/db-sqlite-persistence-core'
@@ -77,8 +79,11 @@ function resolveAdapterBaseOptions(
7779
`driver` | `schemaVersion` | `schemaMismatchPolicy`
7880
> {
7981
return {
80-
appliedTxPruneMaxRows: options.appliedTxPruneMaxRows,
81-
appliedTxPruneMaxAgeSeconds: options.appliedTxPruneMaxAgeSeconds,
82+
appliedTxPruneMaxRows:
83+
options.appliedTxPruneMaxRows ?? DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
84+
appliedTxPruneMaxAgeSeconds:
85+
options.appliedTxPruneMaxAgeSeconds ??
86+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
8287
pullSinceReloadThreshold: options.pullSinceReloadThreshold,
8388
}
8489
}

‎packages/expo-db-sqlite-persistence/src/index.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,11 @@ export type {
44
ExpoSQLitePersistenceOptions,
55
ExpoSQLiteSchemaMismatchPolicy,
66
} from './expo'
7-
export { persistedCollectionOptions } from '@tanstack/db-sqlite-persistence-core'
7+
export {
8+
DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS,
9+
DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS,
10+
persistedCollectionOptions,
11+
} from '@tanstack/db-sqlite-persistence-core'
812
export type {
913
PersistedCollectionCoordinator,
1014
PersistedCollectionPersistence,

0 commit comments

Comments
 (0)