https://gitlab.synchro.net/main/sbbs/-/commit/36cf709445ed412d4de99c06
Modified Files:
exec/chat_llm.js exec/llm_index/dokuwiki.js
Log Message:
chat_llm: strip emoji as UTF-8 bytes, and bridge query vocabulary

sanitize_reply() has never removed a supplementary-plane emoji. It
matched UTF-16 surrogate pairs, and strings in this engine hold BYTES:
U+1F310 is the four characters F0 9F 8C 90, so there is no code unit in
the D83C-D83E range for the pattern to find. A literal emoji in a .js
source file arrives the same way, which is why the pattern also could
not be checked against one. Every emoji the model emitted went to the
channel despite both the system prompt forbidding them and this
backstop appearing to exist.

Matching is done on the UTF-8 byte sequences instead. The E2 ranges
are kept narrow deliberately: box drawing (U+2500, E2 94) and arrows
(U+2190, E2 86) are ordinary text and must survive. The surrogate
patterns are retained after them for an engine that decodes to UTF-16;
they are inert here.

Retrieval still could not reach a page whose vocabulary differs from
the question's. BM25 matches literal terms, so "the name of the mail
server recycle file" never reaches the semaphore-files page -- the documentation calls it a semaphore file, the question does not, and a
section about a text string named "Recycle Servers" wins the literal
match instead. A query_synonyms map adds bridging terms at retrieval
time. Terms are ADDED, never substituted, so an unhelpful synonym
costs ranking rather than correctness, and expansion is applied only to retrieval -- is_casual_input() also tokenizes queries, and judges small
talk by how many content words it finds, so expanding there would make
a greeting look substantive. Keys are single words: a multi-word key
is split, and each word carries the expansion independently.

Wiki subsections now carry a lead-in from their parent section.
Splitting a page can separate an answer from the qualifier that makes
it usable -- the recycle section says to touch ctrl/recycle while the ".<service>" suffix yielding recycle.mail is stated one level up, so
chunked alone that section holds no "mail" token and a question naming
the mail server cannot retrieve it.

The appended "See <url> for the official docs" citation is now scored
by rarity among the retrieved candidates rather than plain overlap: a
word carried by most of the retrieved chunks says nothing about which
one the answer came from. It is also suppressed when the supporting
chunk is not near the top of the ranking, because when two chunks
support a reply about equally the pick is a guess, and a confidently
wrong "official docs" link stapled to a correct answer reads as the bot
not knowing its own documentation. index_url_append_rank tunes how
strict that is, and 0 disables the append entirely.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net