1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
|
# Handoff — Ark / bridge / NWC work
Working note, not project docs. Delete when it's stale.
Persistent memory for all of this lives in
`~/.claude/projects/-home-adam-coinosv3/memory/` — start by reading
`MEMORY.md`, then `hal-nwc`, `coinos-ark-mainnet`, `ark-swap-bridge`,
`coinos-ark-asp`, `ark-third-party-htlc`.
---
## 1. ~~THE ONE UNRESOLVED BUG~~ — **RESOLVED 2026-08-01**
**Root cause:** nostr-tools ≥2.23 changed `SimplePool.subscribeMany` to take a
SINGLE filter object instead of an array. Our `[filter]` wrapping serialized
as `["REQ",id,[{...}]]` — invalid NIP-01 — and every relay answered
`["CLOSED","sub:N","ERROR: bad req: provided filter is not an object"]`, then
silently dropped the sub. Neither guard nor decrypt was ever reached; there
was simply no live subscription. The CDP probe's own harness had the same bug,
which is why it "received frames" but could never observe a 23195.
Fixed in `src/nostr.js` (`subscribeOn` + `NostrSync.subscribeEvents` — device
sync's live subscription was broken too) and `nwcpush/server.js` (deployed to
cs, restarted, resubscribed OK). Verified end-to-end: a real nip04
`get_balance` published from the stored connection's client key got back
`{"result_type":"get_balance","result":{"balance":20000000}}` over public
relays. Instrumentation (guard logging + sub lifecycle gens) is still in
`nwc.js` — strip once Amethyst is confirmed.
**Remaining human step:** paste the existing connection string into Amethyst
again — it should now show a balance.
**LAYER 3 (2026-08-01 evening — the zap):** NWC now round-trips; the zap
`pay_invoice` failed for stacked reasons, all found: (a) the swap bridge
defaults OFF in hal (`BRIDGE_DEFAULT.mainnet='off'`) so pays went to the ASP,
whose CLN routing budget = 90% of the ASP's 1-sat min fee = 0 sats → unroutable;
(b) once enabled, the bridge needed its bearer token (cs
`~/arkswap/config-main.json` `tokens[0]`, now set in the laptop wallet) and
(c) its state dir `cs:~/arkswap/data-main` had been deleted → dangling mount,
500s (recreated + container restarted); (d) with all that fixed the 21-sat zap
STILL fails: the recipient is a private node behind Megalithic.me and is
offline — even manual `xpay` with 10-sat maxfee dies on
temporary_channel_failure everywhere. Not our infra. Refund machinery worked.
ALSO fixed: pool ping+reconnect (silent socket death used to kill sync + NWC
subs until reload — that's why a phone-created connection stayed invisible to
the open laptop tab). RESOLVED SAME EVENING (Adam approved): ASP
min_fee_sat 1→3 (captaind-main restarted, healthy); bridge default ON for
mainnet (commit 0344733) and bridge opened — tokens cleared in
`cs:~/arkswap/config-main.json`. Still open: phone PWA needs two loads to
get new builds.
**LAYER 2 (found after the filter fix — Amethyst still spun):**
relay.coinos.io's `write-policy.pl` (strfry `sf` on desk,
`~/coinos-server/data/strfry/`, gitignored) restricted kinds 13194/23195 to
coinos server pubkeys, so every reply hal published there was rejected — and
Amethyst listens for replies ONLY on the first URI relay, which is
relay.coinos.io. The rejections also strike-counted the laptop IP into a
temp-ban. Fixed 2026-08-01: NWC kinds opened to all pubkeys on the relay
(rate limits intact), `publishOn` now logs refused publishes, 13194 goes out
once per session per connection. Amethyst-shaped probe (coinos-only, #e+#p
filter) round-trips get_balance green on the deployed build.
<details><summary>Original investigation notes (superseded)</summary>
**Symptom:** pasting a hal NWC connection string into Amethyst shows a
spinner forever. No balance, no error.
**This is narrowed, not mysterious. Do not re-derive the following — all of
it was verified against the live laptop tab over Chrome DevTools Protocol:**
- hal DOES subscribe correctly, on all three NWC relays. Captured from its own
socket:
`["REQ","sub:1",[{"kinds":[23194],"#p":["<servicePk>"],"since":<now-120>}]]`
- The stored connection's keypairs are self-consistent:
`getPublicKey(secret) === clientPk` and `getPublicKey(serviceSk) === servicePk`.
So the string Amethyst holds is valid.
- hal's sockets **did receive 23194 frames** (6 — one per relay) in one run.
- No 23195 reply is ever produced.
- The decrypt path is now instrumented (logs + Settings card + replies with an
INTERNAL error in both schemes). A fresh probe produced **no log, no UI
error, no reply** — so execution isn't reaching decrypt.
**Therefore the drop is in one of the silent guards before decrypt**, in
`onRequest` in `src/features/nwc.js`:
```js
if (handled.has(ev.id)) return; // replay set
if (nowSec() - ev.created_at > MAX_AGE_SEC) return; // 120s age check
if (ev.pubkey !== c.clientPk) return; // wrong client
```
…or `onRequest` isn't being invoked at all.
**Second suspect, untested:** `subscribeOn` reuses the shared `pool` in
`src/nostr.js`, and `listen()` is called from three places — feature `init()`,
`createConn()`, and the sync-snapshot `load()`. Each call does `stop()` then
re-subscribes. If a later `listen()` closes the subscription whose callback
would have fired, frames keep arriving on the socket with nothing handling
them. That fits every observation.
**Next step (one cycle should name it):** log at the TOP of `onRequest` —
every event id seen and which guard returns — plus subscription lifecycle
(each `listen()`/`stop()` and how many subs are live). Then re-run the probe
below.
### The probe (copy-paste, needs Chrome with `--remote-debugging-port=9222` and a hal tab open)
```js
// bun run this from ~/coinosv3
import puppeteer from 'puppeteer-core';
import { SimplePool } from 'nostr-tools/pool';
import { nip04, finalizeEvent } from './src/nostr.js';
import { hex } from '@scure/base';
const b = await puppeteer.connect({ browserURL: 'http://127.0.0.1:9222', defaultViewport: null });
const p = (await b.pages()).find(x => x.url().includes('halwallet.app'));
p.on('console', (m) => { const t = m.text(); if (/nwc/i.test(t)) console.log('PAGE:', t.slice(0, 240)); });
// watch hal's own relay traffic
const cdp = await p.createCDPSession();
await cdp.send('Network.enable');
let got23194 = 0;
cdp.on('Network.webSocketFrameSent', (e) => {
if (e.response.payloadData.startsWith('["REQ"')) console.log('SUB:', e.response.payloadData.slice(0, 200));
});
cdp.on('Network.webSocketFrameReceived', (e) => { if (e.response.payloadData.includes('23194')) got23194++; });
await p.reload({ waitUntil: 'networkidle2' });
await new Promise(r => setTimeout(r, 8000));
const c = await p.evaluate(() => {
const k = Object.keys(localStorage).find(x => x.endsWith(':nwc'));
return (JSON.parse(localStorage.getItem(k)).conns || []).filter(y => !y.revoked)[0];
});
const RELAYS = ['wss://relay.coinos.io', 'wss://relay.damus.io', 'wss://nos.lol'];
const pool = new SimplePool();
const cSk = hex.decode(c.secret);
const now = Math.floor(Date.now() / 1000);
let reply = null;
const sub = pool.subscribeMany(RELAYS, [{ kinds: [23195], '#p': [c.clientPk], since: now - 10 }], {
onevent: async (ev) => { if (!reply) reply = await nip04.decrypt(cSk, c.servicePk, ev.content).catch(e => '(undecryptable: ' + e.message + ')'); },
});
await new Promise(r => setTimeout(r, 2500));
const req = finalizeEvent({ kind: 23194, created_at: now, tags: [['p', c.servicePk]],
content: await nip04.encrypt(cSk, c.servicePk, JSON.stringify({ method: 'get_balance', params: {} })) }, cSk);
await Promise.allSettled(pool.publish(RELAYS, req));
for (let i = 0; i < 40 && !reply; i++) await new Promise(r => setTimeout(r, 500));
console.log('hal received 23194 frames:', got23194);
console.log('reply:', reply || 'NONE');
sub.close(); pool.close(RELAYS); await b.disconnect();
```
Also worth trying, as a cheap independent signal: is it Amethyst-specific?
Test the same connection string with Alby Go or a NIP-47 CLI. If those work,
the bug is in how Amethyst reads the URI, not in hal's handler.
</details>
---
## 2. WHAT IS LIVE
| Thing | Where | Notes |
|---|---|---|
| Mainnet ASP | `https://ark.coinos.io` | patched captaind, 5M sat funded, 500k vtxo cap, 1h rounds, CLN attached |
| Mutinynet ASP | `https://ark-staging.coinos.io` | 30s rounds, free boards |
| Mainnet bridge | `https://arkswap.coinos.io` | max(2 sat, 2000ppm), shares boltz's CLN |
| Mutinynet bridge | `https://arkswap-staging.coinos.io` | max(1 sat, 1000ppm) |
| NWC push notifier | `https://nwcpush.coinos.io` | `/health`, `/vapid`, `POST|DELETE /register` |
Containers on **cs**: `captaind-main`, `ark-pg-main`, `captaind-mut`,
`ark-pg-mut`, `arkswap-main`, `arkswap-mut`, `nwcpush`.
Configs in `~/ark/` and `~/arkswap/`. Backups: `~/backup-ark.sh`, hourly at
:17, mirrored to `desk:/home/adam/backups/ark-cs/`, restore-tested.
**Machines:** `lap` = this laptop (its `cl`/`clb`/`clc`/`bc` containers are a
**regtest** dev stack, despite the "coinos" alias). `desk` = prod box
(mainnet bitcoind :8332, mainnet CLN `cl`, `~/scripts/solvency-monitor.ts`).
`cs` = coinos server (everything above + cloudflared tunnels).
---
## 3. LANDMINES THAT COST TIME
1. **build.js has its own feature registry.** It swaps
`src/features/index.js` for a generated module from `ALL_FEATURES`. Adding
a feature to index.js alone ships its i18n strings but NOT its code, so it
looks deployed and silently does nothing. Guard:
`bun tools/feature-registry-check.js`. **Grep the built bundle for feature
code, not just a green build.**
2. **Cloudflare challenges CORS preflights.** Any browser POST with
`content-type: application/json` preflights, and the managed WAF 403s the
OPTIONS → the browser reports a bare "Failed to fetch". Bit us three times.
One skip rule covers it: **`d3ee4d654c1b4e3eb646e14bfe8d76d1`** in ruleset
`48c017dca49b40ea897cd2ed5185fa11`, zone `cc9b01de29250639d6b0b733e19cefda`.
**Any new coinos hostname a browser POSTs to must be added.** curl will not
reproduce it — curl doesn't preflight. Creds: `~/.cf_global_key`
(line 1 email, line 2 global key; use `X-Auth-Email` + `X-Auth-Key`).
3. **Android Chrome freezes backgrounded PWAs** and closes their sockets. A
phone with hal "open" in the background answers nothing. Desktop keeps
background sockets alive; mobile does not.
4. **captaind + CLN TLS:** address it as `https://cln:9736` / `https://hold:9737`,
never `bcln` — the certs' SANs are `cln`/`hold`/`localhost`, and a mismatch
shows only as an endless "Trying to connect to offline node".
5. **Board outputs must be vout 0.** `bitcoin-cli sendtoaddress` randomizes
output order; use `~/ark/fund-board.sh` (pins change to position 1).
6. **Service-worker staleness:** the PWA often needs two loads to pick up a
deploy. When verifying in a live tab, check `caches.keys()` matches the
deployed `cold-*` hash before concluding anything.
---
## 4. OTHER OPEN WORK
- **NWC step 3, remaining half:** the SW currently only *wakes* the user
(notification → tap → hal answers). Auto-answering needs the Ark spend stack
(musig2, arkoor build, gRPC-web) bundled into the service worker — a second
build entry point — plus the pouch funding UI. `src/nwc-pouch.js` (chain-6
keys in IndexedDB, seed stays in localStorage) is built and tested; nothing
funds it yet.
- **Upstream:** issue filed at
`https://gitlab.com/ark-bitcoin/bark/-/work_items/1388`, no reply yet. Draft
MR sits in the fork at `asoltys/bark!1`, branch `third-party-htlc`, rebased
and green. Once the live demo works, adding it to the issue would strengthen
the case. Personal GitLab token: `~/.gitlab_pat`.
- **Monitoring:** nothing watches the mainnet ASP's rounds wallet or bcln's
outbound. Was going to the agent on desk (`~/scripts/solvency-monitor.ts`).
Suggested alerts: rounds wallet under ~1M of 5M; bcln outbound low.
- **bcln is shared** by boltz, the ark bridge, and now the ASP. A dedicated
CLN is the clean fix if volume grows.
- **Liquidity model** (verified in bark's source): ASP working capital ≈ total
user balance under management, locked ~30 days per cycle. Boards and arkoor
sends cost the ASP nothing; rounds and offboards front the full value.
5M sat comfortably supports ~3–4M of user balances.
---
## 5. TEST COMMANDS
```
bun tools/feature-registry-check.js # features ship (membership + order)
bun tools/nwc-test.js # NWC protocol, budgets, sync merge
bun tools/nwc-pouch-test.js # pouch key isolation
bun tools/ark/manager-test.js # ark regression (needs regtest stack)
bun tools/ark/ln-e2e-test.js # ark<->LN over the ASP
bun tools/ark/bridge-e2e-test.js # trustless bridge, regtest
bun tools/ark/htlc-bridge-test.js # third-party HTLC + attack rejections
bun tools/ark/live-bridge-demo.js # live mutinynet demo (public infra)
```
Regtest stack: `docker start ark-postgres ark-electrs captaind cln-ark`
(plain `docker start`, not compose — those services aren't in the compose file).
|
