Initialize service with database path

Use EITHER:

• service: Name of pre-configured service (from ServiceProvider.create)
• format: API format ("openai", "mock") to configure inline

Required:

• db_path: Full path to database file (e.g., "/var/data/memo.db")
• api_key: API key (not needed for mock format)

Optional:

• service: Name of pre-configured service to use
• format: API format for inline configuration (default "openai")
• base_url: Custom API endpoint (for OpenAI-compatible APIs)
• model: Embedding model (default depends on format)
• dimensions: Vector dimensions (auto-detected from model)
• max_tokens: Token limit (auto-detected from model)
• store_text: Enable text storage in texts table (default true)
• build_vocab: Enable incremental vocabulary building (default true, requires store_text)
• chunking_max_tokens: Max tokens per chunk (default 2000)

Example:

memo = Memo::Service.new(
  db_path: "/var/data/memo.db",
  format: "openai",
  api_key: ENV["OPENAI_API_KEY"]
)

Initialize service with existing database connection

Use this when caller manages the connection lifecycle. Caller is responsible for closing the connection.

The db_path is queried from the database for CLI use. Pass db_path explicitly if the pragma query doesn't work for your setup.

Build vocabulary from indexed content

Extracts unique words from all stored texts, embeds them in batches, and stores in the vocab table for word-level similarity search.

Requires text storage to be enabled.

Options:

• batch_size: Number of words to embed per API call (default 2000)
• clear_existing: Whether to clear existing vocab first (default true)

Returns number of words stored

Example:

memo.build_vocab()
results = memo.like("database")

Track whether vocab building is enabled

Clear completed items from the queue

Removes successfully processed items (status = 0). Returns number of items removed.

Clear all items from the queue

Removes all items regardless of status. Returns number of items removed.

Clear vocabulary for current service

Close database connection

Should be called when done with service to free resources. Safe to call multiple times.

Note: If service was initialized with an existing db connection, close is a no-op (caller owns the connection).

Create a new service configuration

Creates a named service configuration that can be used later with Service.new(service: "name", ...).

Example:

memo.create_service(
  name: "azure-prod",
  format: "openai",
  base_url: "https://mycompany.openai.azure.com/",
  model: "text-embedding-ada-002",
  dimensions: 1536,
  max_tokens: 8191
)

Get the default service configuration

Delete all chunks for a source

Removes all chunks with the given source_id (and optionally source_type). Orphaned embeddings (not referenced by any chunk) are also cleaned up.

Returns number of chunks deleted.

source_id: External source ID (Int64 or String). source_type: Optional filter to only delete chunks with matching source_type. If nil and source_id is Int64, searches integer IDs across all types. If nil and source_id is String, searches string IDs across all types.

Delete a service configuration

By default, fails if the service has any associated embeddings. Use force: true to delete the service and all associated data.

Returns true if deleted, false if not found.

Enqueue a document for later embedding

Adds the document to the embed_queue table without embedding it. Use process_queue to embed queued items.

If the source is already in the queue, the text is updated.

Enqueue a document (Document overload)

Enqueue multiple documents for later embedding

More efficient than calling enqueue() multiple times. Resolves all external IDs to internal IDs in a single transaction.

Count indexed files

Get file record by path

Get file record by content hash

Get file record by source ID

Get a service configuration by name

Returns nil if not found.

Get source text by internal source_id

Index a document

Enqueues the document and processes it immediately with retry support. Returns number of chunks successfully stored.

Supports both integer and string source IDs:

• Int64: Time-based, sortable IDs (e.g., Unix timestamps)
• String: UUIDs and other text identifiers

When source_id is nil, memo creates and manages the source internally. Useful for CLI and cases where external ID correlation isn't needed.

Index a document (Document overload)

Convenience method that accepts a Document struct.

Index multiple documents in a batch

Enqueues all documents and processes them with retry support. More efficient than calling index() multiple times.

Returns total number of documents successfully processed.

Index files from a directory

Walks directory, respects ignore files, skips binary files, and indexes text content. Tracks file metadata for incremental updates.

Options:

• root: Directory to index
• ignore_file: Ignore file name (default ".gitignore")
• incremental: Skip unchanged files based on mtime (default true)
• dry_run: List files without indexing (default false)

Returns tuple of (indexed_count, skipped_count, total_files)

Example:

indexed, skipped, total = memo.index_files("/path/to/project")

Index files without progress callback

Find words semantically similar to the query

Searches the vocabulary table for words with similar embeddings. Requires vocabulary to be built first with build_vocab().

Returns array of VocabResult with word, score, and frequency.

Example:

results = memo.like("database")
results.each do |r|
  puts "#{r.word}: #{r.score}"
end

List indexed files

List all service configurations

Returns array of service info, ordered by creation time (newest first).

List service configurations by format

Returns array of service info for the specified API format.

Mark chunks as read (increment read_count)

Process queued items

Embeds pending items from the queue using the service's batch_size. Returns number of items successfully processed.

Failed items have their status set to the error code and can be retried up to max_retries times.

NOTE Queue items are not atomically claimed. This is intentional - single-worker processing is the expected use case since we're hitting one API endpoint with batched requests. Parallel workers would hit rate limits and add complexity without benefit.

Process queued items asynchronously

Spawns a fiber to process the queue and returns immediately. Use queue_stats to check progress.

Get queue statistics

Returns counts of pending and failed items.

Re-index all content of a given source type

Deletes existing embeddings and queues text for re-embedding. Requires text storage to be enabled.

Returns number of items queued for re-indexing.

Re-index all content of a given source type using a block to fetch text

Use this when text storage is disabled. The block receives the external source_id and should return the text to embed.

Returns number of items queued for re-indexing.

Example:

memo.reindex("article") do |source_id|
  app.get_article_text(source_id)
end
memo.process_queue

Search for semantically similar chunks

Automatically generates query embedding and searches.

Returns array of search results ranked by similarity.

source_id: Filter by external source ID (Int64 or String).

like: LIKE pattern(s) to filter by text content. Single string or array of strings for AND filtering. Example: like: "%cats%" or like: ["%cats%", "%dogs%"] Only works when text storage is enabled.

match: FTS5 full-text search query. Supports AND, OR, NOT, prefix*, "phrases". Example: match: "cats OR dogs", match: "quick brown*" Only works when text storage is enabled.

sql_where: Raw SQL fragment for filtering chunks. Used with ATTACH to filter by external database tables. Example: "c.source_id IN (SELECT id FROM main.artifact WHERE kind = 'goal')" Note: c.source_id here is the internal ID, not external.

include_text: If true, includes text content in search results. Only works when text storage is enabled.

Get usage statistics for a service

Set a service as the default

Returns true if successful, false if service not found.

Get statistics about indexed content

Returns counts of embeddings, chunks, and unique sources.

Track whether text storage is enabled

Update a service configuration

Can update base_url and max_tokens. Returns the updated service info, or nil if not found.

Switch to a different service

Changes the current provider and service configuration. The api_key is required for non-mock services.

Example:

memo.use_service("azure-prod", api_key: ENV["AZURE_API_KEY"])

Get vocabulary statistics