Semantic Search

Semantic Search & RAG

Two features need to understand tokens by meaning, not just by symbol: “tokens similar to X” and the retrieval-augmented chat. Both run on the same foundation — token profiles embedded into vectors with Cloudflare Workers AI and stored in a Vectorize index.

The embedding model

Text is embedded with @cf/baai/bge-small-en-v1.5 via the Workers AI binding, producing 384-dimensional vectors. The Vectorize index (seshat-vectors) is configured for exactly 384 dimensions and cosine similarity, so the small model is required — the 768-dim base model would fail upserts on a dimension mismatch.

// providers/embeddings.ts
export const EMBEDDING_MODEL = '@cf/baai/bge-small-en-v1.5';
export const EMBEDDING_DIMS  = 384;

async function embedText(env, text): Promise<number[] | null> {
  if (!env.AI) return null;                 // fail-soft: no binding → null
  const res = await env.AI.run(EMBEDDING_MODEL, { text: text.trim() });
  const vector = res.data?.[0];
  if (!vector || vector.length !== EMBEDDING_DIMS) return null;  // shape guard
  return vector;
}

Fail-soft by design

Every embedding function returns null / no-ops if the AI or Vectorize binding is missing or throws. Semantic features are an enhancement, never a hard dependency — “similar tokens” returns an empty list with a 200, and chat simply skips its semantic block. Nothing in the core pipeline breaks if Vectorize is down.

What gets embedded: the token profile

Symbols are noise for similarity — two AI-agent tokens with unrelated tickers should cluster. So Seshat embeds a composed profile string that blends identity, AI research findings, and developer context (capped at ~1500 chars, since bge truncates long inputs anyway).

// buildTokenProfileText() composes, e.g.:
"Token SYMBOL (Name). Twitter: handle.
 Summary: <research summary>.
 Key findings: <finding 1>; <finding 2>; ...
 Developer: <developer_category>. Developer Twitter: <handle>."

// key_findings may arrive as a JS array OR a JSON string from D1 — both handled.

Storing vectors

A token's vector is upserted into Vectorize keyed by its address, with a little metadata (symbol, name, composite score, developer score) for display. A backfill job embeds tokens that were never embedded or whose vector is older than 3 days, stampingembedded_at so it stays incremental.

// upsertTokenVector() — keyed by address, null metadata coerced to safe defaults
await env.VECTORIZE.upsert([{
  id:     address,
  values: vector,                       // 384 floats
  metadata: { symbol, name, composite_score, developer_score },
}]);

// Backfill: POST /api/admin/embed-backfill
//   WHERE embedded_at IS NULL OR embedded_at < datetime('now','-3 days')
//   → embed profile → upsert → UPDATE tokens SET embedded_at = now

Feature 1 — similar tokens

GET /api/tokens/:address/similar re-embeds the token's current profile (so it reflects the latest research), queries Vectorize for nearest neighbours, drops the token itself, and joins the results back to live token + score data — preserving the cosine-similarity ordering.

// similarTokens() — query topK = limit + 1 (token is its own nearest neighbour)
const vector = await embedText(env, profile);
if (!vector || !env.VECTORIZE) return { data: [], total: 0 };   // fail-soft

const result = await env.VECTORIZE.query(vector, { topK: limit + 1, returnMetadata: 'all' });
const neighbours = result.matches
  .filter(m => m.id.toLowerCase() !== addr)   // drop self
  .slice(0, limit);

// → join ids back to tokens + latest composite_score, ordered by similarity,
//   each result carrying a rounded similarity score.

Feature 2 — retrieval-augmented chat

The chat assistant grounds Claude in live data. Alongside platform stats and any token explicitly mentioned, it embeds the user's message, queries Vectorize for the top 5 semantically relevant tokens, and injects a compact context block — so a question about “yield optimisation projects” surfaces the right tokens even if the user never named one.

// buildSemanticContext() inside the chat context builder
const vector = await embedText(env, message);
if (!vector) return null;                                   // fail-soft → skip block

const result = await env.VECTORIZE.query(vector, { topK: 5, returnMetadata: 'all' });
// → join ids → tokens + latest score + research summary, preserve vector ordering
// → returns a "## Semantically Relevant Tokens (vector search)" block

// This block is appended to the system prompt; chat works fine without it.

Chat retrieval is one of several context sources stitched into the prompt — platform stats are always injected, a full token profile is pulled when an address or $SYMBOL is detected, and fee / top-score blocks are added based on the question. The semantic block rounds it out with meaning-based matches.

Cost note

Each embedText() call is a Workers AI inference, so upserts and backfills incur inference cost — backfills are paced to stay gentle on the account. Queries embed a single short string, so they're cheap.