| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Each successful lightning send was emitting 5 INFO lines:
"<user> sent lightning <amount>" (debit, KEEP)
"paying lightning invoice <pr> <amt> <fee>" (drop — redundant)
"payment completed <id> <preimage>" (drop — implicit from absence of warns)
"refunding fee <maxfee> <fee> <delta> <ref>" (drop — the warn for non-integer
delta already covers anomalies)
"sendLightning outcome <id> = finalized" (drop for "finalized" only —
keep for the recovery-via-fallback
variants since those are diagnostic)
Net: only the "<user> sent lightning <amount>" line fires per healthy send,
matching the other asset types' style. Anything that needed a fallback path
or threw still surfaces via the existing warn/err lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Each xpay with failed_parts > 0 that ultimately settled (multi-path
retry — common, not concerning) was firing 3 WARN lines:
"xpay returned (abnormal)"
"xpay returned failed_parts without throwing"
"xpay failed_parts but listpays complete — finalizing"
These are all the SAME successful payment. The presence of failed_parts
just means CLN tried multiple paths before one succeeded. Adam was
seeing noise.
Now:
- Only warn on actually-missing preimage (xpay silently failed).
- Drop the unconditional "without throwing" warn — the failed_parts
branch handles the recovery silently in the success case.
- Drop the "listpays complete — finalizing" warn — that's the success
path, not a concern.
- KEEP warns for the genuinely interesting cases: listpays says failed
(we're about to reverse), listpays says pending (debit left for
check() to reconcile), or finalize itself throws.
Net effect: a healthy multi-path payment now emits only the standard
"sendLightning outcome <id> = finalized-via-listpays-after-failed_parts"
info line instead of 3 WARN lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Triggered by a recurring "finalize threw despite no failed_parts: ERR
value is not an integer or out of range" warning. Root cause: when xpay's
response shape is unexpected (or invoice is zero-amount and decoded
amount_msat is undefined), the computed p.fee becomes NaN. The
subsequent db.incrBy(balance:<uid>, maxfee - NaN) throws because Redis
can't increment by NaN.
Symptom: leak guard fires "ref set in db despite throw" — payment is
actually fine (p.ref persisted before the throw) but finalize never
completes the fee refund. User loses up to maxfee sats (was already
debited maxfee, gets nothing back).
Fix: validate the computed fee is a finite integer; if not, keep the
originally-reserved maxfee (zero refund, no balance corruption) and warn
loudly. Same guard on the refund delta before db.incrBy.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two cleanups based on a day of production data:
1. Only log the xpay response shape when something is abnormal
(failed_parts > 0 or missing preimage). The success case (200+/day) was
pure noise — answers Adam's "do we still need these xpay warnings".
2. Re-read the payment record from db before the leak guard fires. Several
"LEAKED DEBIT" lines today were finalize() throwing on a late line
(likely db.incrBy for the fee refund) AFTER db.set had already
persisted p.ref. The payment is functionally settled in those cases —
only flag as a real leak if ref is genuinely missing from the db record.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The first deployment surfaced two LEAKED DEBIT lines that were actually
clean finalizes via the new failed_parts listpays-fallback path — exactly
the path that recovers payments that would have been stuck under the old
code (joho33's case). The leak guard's `outcome !== "finalized"` check
was too narrow: any path that finalizes the payment, regardless of how
it got there, is a success.
Bucket outcomes explicitly:
- finalizedOutcomes — clean successes (finalize ran, ref set)
- safeNonFinalized — reverse ran, OR debit left in pending for check()
to reconcile (these aren't leaks)
- anything else — actually stranded (finalize-threw-*, unknown) → err
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Diagnosing the stuck "optimistic" state seen on payment 30976bc9 (joho33,
2026-05-19): confirmed=true with no preimage, not in the pending set, so
check() couldn't reconcile. Most likely path: ln.xpay returned without
throwing but with failed_parts > 0; the previous `if (!r.failed_parts)`
guard skipped finalize, no catch ran, and the debit was left stranded.
Adds:
- Log the full xpay/keysend response shape (status, failed_parts, parts,
amount_sent_msat, has_preimage) so we can see what CLN actually returned.
- Handle the failed_parts branch explicitly — verify with listpays and
finalize, reverse, or leave pending based on actual outcome.
- Track an `outcome` string through every branch and log it before
returning or throwing.
- "LEAKED DEBIT" err line at the function-end if we return without a clean
finalize — that's the exact stuck state we want zero of. grep `LEAKED
DEBIT` in app logs to catch any future occurrences immediately.
Same instrumentation pattern applied to sendKeysend.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Out-of-band emergency stop. The companion nobal.sh creates a sentinel
file on the host (bind-mounted at /locks in the container) when a
wallet balance drops below its failsafe threshold; debit() checks for
the file on every external send and refuses with "External withdrawals
temporarily disabled". /locks/ALL.locked is a global override.
Lives outside the db so a db-write vector (the May 18 /gateway pattern)
can't disable it. Internal sends are exempt — receives and user-to-user
transfers keep working with the site fully online during a lock.
For bitcoin and liquid this is a secondary defense on top of daemon-
level walletpassphrase rotation. For lightning, it's the only available
mechanism since CLN has no walletlock equivalent.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previous calc summed our_amount_msat across every channel regardless of
state, including ONCHAIN (channels that have been closed and are waiting
for the on-chain sweep). Those funds are not actually usable for
lightning sends, but were inflating lnbalance — and therefore inflating
${type}:limit — by the locked-up amount. With several closed channels
pending sweep, the effective threshold floor was tens of megasats lower
than the configured lightning:threshold suggested.
Filter to CHANNELD_NORMAL (and CHANNELD_AWAITING_SPLICE so an in-flight
splice doesn't briefly pin the limit to zero).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
When ln.listpays throws inside the post-failure verification path in
sendLightning / sendKeysend, the outer catch was swallowing it silently.
That meant a CLN outage during a failing payment was indistinguishable
from clean failure-and-reverse in the logs.
We still want to not-reverse in this case (the debit stays, pending stays
in the reconciler set, check() retries when CLN is back), but the
operator deserves to know the verification step failed. Logged via
warnThrottled so a multi-hour outage doesn't flood the log.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The catch blocks for ln.xpay and ln.keysend reversed the user's debit
whenever the RPC call threw, regardless of the actual payment state in
CLN. If the RPC errored transiently while the payment was still pending,
the user's balance got refunded — and then the lightning payment could
still complete, leaving the user with both the refund and the outgoing
sats.
Both call sites now mirror the safe check() logic:
const failed = !pays.length || pays.every((p) => p.status === "failed");
if (completed) finalize ...
else if (failed) reverse ...
// else (pending) leave the debit; check() reconciles when CLN settles.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two related resilience fixes for when one chain backend (typically lq via
the cs tunnel) is down or unreachable:
- getAddressType: when lq.getAddressInfo throws on a connection error
(ECONNREFUSED / ETIMEDOUT / "Unable to connect"), fail with
"liquid temporarily unavailable" instead of the misleading
"unrecognized address". Users get an actionable message and we can
tell user error from infrastructure problems in the logs.
- catchUp: wrap the per-chain listTransactions call in its own try/catch
so a downed lq doesn't stop the bitcoin deposit-recovery loop (and
vice versa). Logged via warnThrottled so a multi-hour outage doesn't
spam the same error every tick.
Also drops a redundant /confirm double-call comment from catchUp — with
walletnotify reliable again, a single call is sufficient and avoids
piling on the /confirm race the new atomic lock is preventing.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
build() is called from both /payments (the actual send path) and /fee
(the fee-estimation endpoint). Locking UTXOs in build() meant that every
fee quote was reserving inputs that never got broadcast, slowly leaking
UTXO locks until a bitcoind restart cleared them — eventually starving
the wallet of selectable inputs even when balance was healthy.
The lock belongs at the point where we're actually about to broadcast,
which is sendOnchain. Released in the catch path on broadcast failure;
on success, the inputs become spent and the lock is moot.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Follows up the Apr 29 double-credit incident. Two parallel callers of
/confirm (walletnotify, catchUp, bulk sweeps) could each read
p.confirmed=false and each run the multi() that increments balance —
because main-db pointers were sometimes missing while arc still had them,
gf()'s arc fallback returned stale state. Three pieces here:
- routes/payments.ts: SET NX EX 60 lock around the /confirm credit block
keyed by `confirmlock:${txid}:${vout}`. Concurrent callers short-circuit
cleanly. The lock is released on success and on every early-return.
- lib/payments.ts credit() + routes/payments.ts /confirm: mirror the
payment/invoice records and the `payment:<txid>:<vout>` / `payment:<hash>`
pointer keys into arc as soon as they're written. gf() will then find
the prior credit via arc fallback even if the main-db copy is wiped by
a future data-loss event.
- routes/payments.ts hide `revertedDuplicate` payments from the user's
payment list — matches the same filter added to lib/nwc.ts list_transactions
so reverted records never surface in either rendering path.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Three small NWC robustness fixes that surfaced from the Apr 29
double-credit cleanup:
- fail loudly when an NWC request references a pubkey whose user record
no longer exists (migrated/deleted account); previously crashed inside
handle() on `user.id`
- swallow `pubkey not found` and `user not found` in the per-event catch
so deleted-app/migrated-user spam doesn't fill the error log — these
are client-side staleness, not server faults
- use gf() instead of g() when fetching payment records (consistent with
the rest of the codebase: arc fallback for wiped main-db pointers)
- filter `revertedDuplicate` payments out of list_transactions so
clients never see the doubled records that were later reverted
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous check at debit() read ${type}:limit, compared amount, and
deferred any decrement to the next freezeCheck tick (every 10s). That made
the check non-atomic: a burst of concurrent withdrawals could each pass
the same stale limit value before any reconciliation happened. Combined
with the /gateway abuse, this allowed the hot wallet to be drained.
withLimitLock() serializes the check + db.decrBy inside a per-type async
mutex, and freezeCheck also takes the lock when overwriting limits so its
10s reconciliation can't race in-flight checks. Single-process only;
distributed lock or TigerBeetle reservation needed if app ever scales out.
Trade-off: decremented limits are not refunded on payment failure — they
get reset by the next freezeCheck within 10s. Acceptable false-deny
window in exchange for correctness against bursts.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Skip non-lightning entries in pending payments check and handle
malformed password hashes gracefully during login.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Catch failed keysend payments
|
| | | |
|
| |\ \
| | |
| | | |
Fix incorrect extratlvs format in pay_keysend
|
| | |/ |
|
| |/ |
|
| | |
|
| | |
|
