module Noir::TopLevelSplit

Overview

Splits a delimited list at the top nesting level, respecting quoted runs and (optionally) backslash escapes.

This replaces ~44 hand-rolled copies of the same loop spread across src/analyzer/analyzers/, src/analyzer/engines/ and src/miniparsers/, 19 of which were byte-identical to a sibling. A quote- or escape-handling bug used to need 44 separate fixes; the copies had already drifted along seven independent axes (see Rules), so "just pick one and inline it" would have changed detection for whichever analyzers lost their variant.

Implementation note — why a single forward pass into a String::Builder and not text[i] / text[start...index]: String#[](Int) and char-range slicing are O(index) the moment a string contains one multi-byte UTF-8 codepoint, because Crystal has to walk the bytes to find the char boundary. Several of the copies being replaced here indexed per char inside a while loop, making them O(n^2) on any source file with a Korean comment or an emoji in a string literal, and at least one carried a comment about materializing .chars up front to work around exactly that. A splitter only ever moves forward, so each_char plus one String::Builder per part is O(n) with no random access at all, no .chars array, and no ASCII-vs-UTF-8 dispatch.

This is deliberately NOT the single_byte? / Bytes-vs-Array(Char) dispatch used by Noir::JSLiteralScanner. That pattern exists there for find_matching_*, which must random-access from an arbitrary index and so genuinely needs an indexable source. A forward-only splitter does not, and paying for the dispatch (plus a full .chars materialization on non-ASCII input) would be strictly slower than just iterating.

Extended Modules

Defined in:

utils/top_level_split.cr

Instance Method Summary

Instance Method Detail

def split(text : String, delimiter : Char, rules : Rules) : Array(String) #

Splits text on every occurrence of delimiter that sits at depth 0 and outside a quoted run.


[View source]
def split(text : String, delimiter : String, rules : Rules) : Array(String) #

Multi-character separator variant. Only a separator run that sits ENTIRELY at depth 0 and outside quotes splits, so ":>" does not fire on a bare ":" and "||" inside f(a || b) is invisible.

LIMITATION — a delimiter with a state-changing proper prefix: matching a multi-character separator means buffering characters while they are still a prefix of it, and a mismatch releases only the first buffered character before retrying the rest. That release is what changes cursor state, so if a proper prefix of the delimiter contains a quote character or an opener of an enabled Nest kind, releasing it opens a quoted run or raises the depth — and from that point the separator can no longer match at depth 0. #split("(((xy", "((x", nest: Paren) therefore yields ["(((xy"]: the only place "((x" could match starts at index 1, and index 0's ( has already taken the depth to 1, so the depth-0 rule forbids the split. Same for quotes: #split("\"\"\"xy", "\"\"x", quotes: "\"") yields ["\"\"\"xy"], the first " having opened a run that swallows the rest.

This suppresses SPLITS only. The characters themselves are never lost or reordered: the moment a release ends the top-level/unquoted state, the rest of the buffer is flushed immediately, so every part holds the input characters of its window in source order.

Every separator actually used in the tree is punctuation outside both sets ("," and ":>" / ":<|>" in haskell/servant.cr, "||" in specification/traefik.cr), so no caller sees the limitation at all. The note in Cursor#consume about a quote-bearing delimiter is this same situation seen from the other end.


[View source]
def split_spans(text : String, delimiter : Char, rules : Rules, start_pos : Int32 = 0, end_pos : Int32 = -1) : Array(Tuple(String, Int32)) #

Splits like #split, but also reports where each part starts.

Returns {part, offset} tuples where offset is an ABSOLUTE CHAR index into text (not a byte index, and not relative to start_pos) pointing at the first non-whitespace character of that part. Callers use it to map an argument back onto the file it was sliced out of, so they need a position in the same coordinate system they passed the window in.

start_pos and end_pos bound the scan, again as char indices; end_pos < 0 means "to the end of text". Both are clamped into 0..text.size, and an inverted window is treated as empty.

The window is applied by testing the index while iterating, NOT by slicing text first: char-range slicing is O(n) on any string holding a multi-byte codepoint, which is the very cost this module exists to avoid.

Offset of an empty or whitespace-only part: the first non-whitespace character at or after where the part began, looked for in the WHOLE of text and not just inside the part — so a part that is only whitespace reports the index of the delimiter that ended it, and a whitespace-only final part can report an index past end_pos. That is what the four hand-rolled copies this replaces did (their skip_whitespace helper was bounded by content.size, never by the part or the window), and callers feed the offset straight back into text, so the position must stay a valid index into text rather than being clamped to the part.

Char delimiter only, deliberately: every caller that needs offsets splits on a single character, and the multi-character path buffers a pending prefix that is only committed to a part once the match fails (see the String overload's limitation note) — so a part could begin with characters read well before the scan decided they belonged to it, which would make its offset ambiguous.


[View source]