struct Memo::Config::Chunking

Overview

Chunking configuration for text segmentation

The app's preferred chunk sizes (must be <= service max_tokens)

Defined in:

memo/config.cr

Constructors

Instance Method Summary

Constructor Detail

def self.new(min_tokens : Int32 = 200, max_tokens : Int32 = 2000, no_chunk_threshold : Int32 = 1500, tokens_per_byte : Float64 = 0.25) #

[View source]

Instance Method Detail

def max_tokens : Int32 #

[View source]
def min_tokens : Int32 #

[View source]
def no_chunk_threshold : Int32 #

[View source]
def tokens_per_byte : Float64 #

[View source]
def with_tokens_per_byte(ratio : Float64) : Chunking #

Return a copy with updated tokens_per_byte ratio


[View source]