Negentropy (NIP-77): DM and notification catch-up fetch only what this device lacks.

A relay that speaks it compares the ids it holds for a filter with the ids we hold and answers with the difference — a few hundred bytes, one round — instead of re-serving its newest events every time we ask. Measured on relay.coinos.io: the DM subscription re-served 200 gift wraps (468 KB) on every boot and every resubscribe, the Notifications refresh 120 events (105 KB) a visit; a reconcile holding the newest 100 of 120 found the 20 missing in 1.8 KB and 41 ms.

src/negentropy.js: the protocol (version 1, as strfry speaks it), a port of Doug Hoyte's reference client (MIT): NegSet of [created_at, id] items, Negentropy.initiate / reconcile → { next, have, need }, WebCrypto SHA-256. Byte-identical opening message to the reference's; seven pairings of ours and theirs as client and relay agreed on every have/need.

src/nostr.js: reconcileOn(url, filter, items) on its own socket (the pool's parser has no ear for NEG-* frames) → { need, have } or null; a relay that doesn't speak it (NOTICE "negentropy disabled", CLOSED, a socket shut on the opening message, silence) is remembered for a day in coinos-neg-off, so the plain query goes straight out there next time; NEG-ERR blocked/closed is not remembered. syncOn(relays, filter, items): reconciliation where it works, the plain query where it doesn't, the ids we lack fetched by id in slices, each event once, verified.

messages.js — DMs: relay by relay, one that speaks it gets the live subscription with limit 0 (opened first, so nothing slips between) and a reconcile of its newest 400 against wrapItems (the wraps held, full id and time, 600 kept beside the 16-char prefix log; a wrap the prefix log knows but the list doesn't is added as the relay re-serves it, once); nos.lol, with negentropy disabled, keeps the plain subscription with its backlog, and so does every relay while the one-time heal pass runs. Notifications: the refresh reconciles the relay's newest 120 against notifHeld (every notification event seen, 300 kept — not just the rows, or a folded zap twin would come back every visit), persisted beside the rows; the watch is unchanged. Not applied to feeds: the outbox relays mostly lack it (nos.lol, damus, primal, nostr.wine, nostr.mom, nostrplebs say no; relay.coinos.io, purplepag.es, ditto, oxtr, nostr21, offchain yes) and a first older page has nothing to reconcile against.

Tests: tools/negentropy-test.js (the reference's bytes, one round for a small difference, eight for a 4 KB frame, nothing to say when equal); tools/negentropy-sync-test.js (an in-page relay speaking NEG with the library on its side: the wrap subscription opens limit 0 and the three wraps lacked are fetched by id, nos.lol's "disabled" gets the plain subscription and is remembered, the boot refresh fetches the three notifications by id and the page's refresh and a refresh after reload fetch nothing). notif-report-fold, notif-faces, dm-filter, dm-row-resort, dm-nip42 pass.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01J1Kon8bxJoHRpJdSzshCkp

1 files changed, 64 insertions, 0 deletions

diff --git a/tools/negentropy-test.js b/tools/negentropy-test.js
new file mode 100644
index 0000000..28a1f30
--- /dev/null
+++ b/tools/negentropy-test.js
@@ -0,0 +1,64 @@
+// The negentropy client (NIP-77, src/negentropy.js) on the wire as strfry
+// and the reference implementation speak it: the opening message for a
+// fixed set is byte-identical to the reference's (checked against
+// hoytech/negentropy's JS on 2026-10-16: seven pairings of ours and theirs
+// as client and relay agreed on every have/need), and a reconciliation
+// against our own relay side finds exactly what each side lacks — in one
+// round for a small difference, in several when the relay's frame is
+// small — and ends with nothing to say once the sets match.
+// Run: bun tools/negentropy-test.js
+import assert from 'node:assert/strict';
+import { createHash } from 'node:crypto';
+import { NegSet, Negentropy } from '../src/negentropy.js';
+const id = (n) => createHash('sha256').update('item-' + n).digest('hex');
+const sha = (s) => createHash('sha256').update(s).digest('hex');
+
+// the fixed sets the reference was run on
+const A = Array.from({ length: 120 }, (_, i) => [1000 + Math.floor(i / 3), id(i)]);
+const B = [...A.filter((_, i) => i % 17 !== 0), ...Array.from({ length: 9 }, (_, i) => [1000 + i * 4, id(1000 + i)])];
+const first = await new Negentropy(new NegSet(A)).initiate();
+assert.equal(first.length, 634, 'the opening message for the fixed set is 317 bytes');
+assert.equal(first.slice(0, 2), '61', 'protocol version 1');
+assert.equal(sha(first), 'c926c013652f6e2f727abff15c0838aebd7f613fb4749ee691952dfc8be8ff81', 'the opening message is the reference implementation\'s, byte for byte');
+
+async function talk(client, relay) {
+  let msg = await client.initiate(), rounds = 0;
+  const have = [], need = [];
+  while (msg) {
+    rounds++;
+    const r = await relay.reconcile(msg);
+    assert(r.next, 'the relay side always answers');
+    const c = await client.reconcile(r.next);
+    have.push(...c.have); need.push(...c.need); msg = c.next;
+  }
+  return { rounds, have: have.sort(), need: need.sort() };
+}
+{
+  const r = await talk(new Negentropy(new NegSet(A)), new Negentropy(new NegSet(B)));
+  assert.equal(r.rounds, 1, 'a small difference reconciles in one round');
+  assert.deepEqual(r.have, A.filter((_, i) => i % 17 === 0).map((x) => x[1]).sort(), 'have: the ids the relay lacks');
+  assert.deepEqual(r.need, Array.from({ length: 9 }, (_, i) => id(1000 + i)).sort(), 'need: the ids we lack');
+}
+{
+  const r = await talk(new Negentropy(new NegSet(A)), new Negentropy(new NegSet(A)));
+  assert.equal(r.rounds, 1); assert.deepEqual(r.have, []); assert.deepEqual(r.need, []);
+  const r2 = await talk(new Negentropy(new NegSet([])), new Negentropy(new NegSet(B)));
+  assert.equal(r2.need.length, B.length, 'holding nothing, we need everything');
+  const r3 = await talk(new Negentropy(new NegSet(B)), new Negentropy(new NegSet([])));
+  assert.equal(r3.have.length, B.length, 'the relay holding nothing lacks everything');
+}
+{
+  // a big set and a relay with a small frame: many rounds, the same answer
+  const big = Array.from({ length: 3000 }, (_, i) => [1000 + i, id(i)]);
+  const theirs = [...big.filter((_, i) => i % 101 !== 0), ...Array.from({ length: 25 }, (_, i) => [1000 + i * 7, id(5000 + i)])];
+  const r = await talk(new Negentropy(new NegSet(big)), new Negentropy(new NegSet(theirs), { frameSizeLimit: 4096 }));
+  assert(r.rounds > 1, 'a small frame takes several rounds: ' + r.rounds);
+  assert.equal(r.have.length, 30); assert.equal(r.need.length, 25);
+}
+{
+  // the set drops duplicates and bad ids, and sorts by time then id
+  const s = new NegSet([[5, id(1)], [5, id(1)], [3, 'nope'], [4, id(2)], [5, id(0)]]);
+  assert.equal(s.size, 3);
+  assert.deepEqual(s.items.map((x) => x.ts), [4, 5, 5]);
+}
+console.log('✓ negentropy: the reference\'s bytes, one round for a small difference, several for a small frame, nothing to say when equal');