|
Vince's CSV Parser
|
This document describes the high-level architecture of the CSV parser internals and how major classes interact.
Scope:
For queue synchronization protocol details, see THREADSAFE_DEQUE_DESIGN.md. For a narrative byte-to-field walkthrough, see JOURNEY_OF_A_CSV_FIELD.md. For AI-agent workflow and guardrails, see ../../AGENTS.md and ../../tests/AGENTS.md.
The parser is a streaming producer-consumer pipeline:
Two independent parser paths exist and must be kept behaviorally aligned:
CSVFormat::threading(false).CSVRow objects with keyed access, sparse edit overlays, column views, and row-like writer compatibility.csv, while implementation headers are split under include/internal/data_frame/ to keep each proxy/helper focused.operator* and operator->, as CSVReader does. CSVRow field iteration instead returns temporary field values, as described below.CSVRow storage intact.csv::internals::parser; files in include/internal/parser/ intentionally share that namespace.csv::internals::speculative.CSV_ENABLE_THREADS=0.source_start, the absolute source byte offset for the backing chunk; CSVRow::byte_offset() combines it with the row-local data_start.Parser hierarchy:
Reader + row/data ownership:
Notes:
CSVRow -> CSVField field access:
Implication:
Source bytes -> parser chunk read -> parse loop -> RawCSVData + RawCSVFieldList -> CSVRow enqueue -> CSVReader read_row / iteration -> CSVField access/conversion
Operationally:
CSVFormat::threading(false) is active, the same parse cycle runs synchronously on the caller thread and speculative parsing is disabled. In thread-enabled builds this runtime opt-out still uses ThreadSafeDeque<CSVRow> internally; replacing it with SingleThreadDeque would be a small optimization, not a semantic difference.When a subsystem earns a folder under include/internal/, its namespace should follow the folder path. For example, parser source adapters live in include/internal/parser/ and csv::internals::parser, while speculative helpers live in include/internal/speculative/ and csv::internals::speculative.
Fields spanning chunk boundaries must not be split/corrupted.
Mmap and stream parsers must preserve the same externally observable behavior.
A single-threaded application can still have two threads touching parser state. The consumer may wake after rows have been queued and call metadata accessors while CSVReadScheduler's worker finishes the current source window. Queue synchronization protects row transport; it does not by itself protect state in the source adapter or parse orchestrator.
CSVReader intentionally has no public EOF accessor. Source exhaustion can be observed while parsed rows remain queued, so it is not the same as reader exhaustion. has_next_row() and try_has_next_row() answer whether a logical row can be returned by driving check_for_rows() and taking a synchronized copy of the queue front; a returnable row remains queued. Policy-excluded physical records may be discarded while searching. The source-exhaustion flag in CSVParserDriverBase is therefore a plain, worker-owned bool. MmapParser and StreamParser read and write it on the active parser thread, while consumer-side checks in check_for_rows() and initial_read() occur only after CSVReadScheduler has joined. Preserve that ownership and join ordering instead of exposing the flag to an unsynchronized observer.
Speculative diagnostics have a stronger snapshot requirement. The worker merges seven cumulative counters after a speculative window, while the consumer may call CSVReader::speculative_diagnostics(). The merge and copy use the same short critical section so a returned snapshot belongs to one completed merge. Independent atomic counters would remove individual data races but could still combine values from different merge epochs and violate cross-counter invariants.
The remaining synchronization is deliberately narrow:
speculative_diagnostics() safe against the reader's own background worker. It does not make CSVReader, its iterator, row consumption, destruction, or mutation generally safe across user threads.utf8_bom() and parse_worker_count() expose state resolved during synchronous reader initialization and are not mutated by later worker cycles.Regression coverage lives in tests/test_threadsafe_deque_race.cpp ([issue_353]). It observes diagnostics while parsing a 500,000-row, greater-than-10MB fixture with distinct column values, using separate mmap and stream SECTIONs so ThreadSanitizer exercises both source adapters.
CSVParseOrchestrator checks the first source window once, before dispatching serial or speculative parsing. It rejects UTF-16/UTF-32 with a transcoder error and records a leading UTF-8 BOM for CSVReader::utf8_bom(). Only that first BOM is removed; another BOM remains part of the CSV data. Parser cores and speculative chunks carry no BOM state. Format guessing separately checks and strips the leading BOM once per guess_format() call, before scoring delimiter candidates.
Both adapters' first windows include their pre-read head buffer, up to 500KB; the stream adapter may append more bytes before parsing. A first window too short to classify a BOM therefore occurs only at end of input.
Source adapters advance in original source bytes while parsers receive chunk.substr(skip) at base_offset + skip:
RawCSVData::source_start identifies the source position of the parsed view. A first row after a UTF-8 BOM starts at byte 3, and its raw_str() excludes those three bytes. Field offsets remain relative to the backing view.CSVParseWindowResult::complete_prefix_length includes the skipped bytes, also reported as skipped_prefix_length. Mmap re-reads and stream leftovers must begin after this prefix, even when the window completed no row.completed_row() rather than position arithmetic: skipping a BOM advances the source position without completing a row. A BOM-only source emits no row and still consumes the BOM.Regression coverage lives in tests/test_read_csv.cpp ([read_unicode_bom]) for both adapters, guessed/explicit formats, encoding rejection, tiny inputs, row offsets, and a 600K-row file parsed serially and speculatively. The BOM variant of [issue_337] in tests/test_edge_cases_large_rows.cpp covers a header longer than the mmap head buffer.
The parser realizes doubled-quote fields into RawCSVData::quote_arena; ordinary fields remain views into source bytes. CSVRow applies trimming when creating a field view. CSVField owns scalar classification and typed conversion caching.
Avoid designs that force retaining all parsed chunks globally.
DataFrame is a row-backed editing facade, not a columnar analytics engine. Keep normal row access and streaming-friendly storage simple; do not add columnar indirection to every hot-path field access solely to make structural edit strategies symmetric.
Current structural edit policy:
data_frame::DataFrameStorage centrally selects a concrete CleanDataFrame or DirtyDataFrame through an atomic tagged pointer. Calls remain direct and inlineable, with no vtable or C++17 variant dependency. Clean handling owns physical rows/names; dirty handling owns visible names, sparse overlays, and logical-to-physical mapping. Facade features never inspect the active state themselves.data_frame::RowViewAccessor reuses CSVRow's decoded, trimmed field extraction without constructing a scalar-aware CSVField. Both backends use it for parsed values; dirty overlays retain their own lookup. The measurements and rejected physical-column caching alternative are retained in data_frame/PERFORMANCE.md.rows and keys_ directly and update the handler's row bindings and overlay slots because rows are the native storage unit.DirtyDataFrame, including for a clean input frame. Construct fresh shared backing chunks from visible values, including sparse edits, without CSV serialization/reparsing. This preserves empty fields and zero-column rows and returns the frame to clean handling. Stored keys remain stable row identities even after key-column cell edits. View keys are retargeted into private owned key-byte storage before their parsed backing is released; owning-key types require no additional work.CSVRow storage is intentionally left unchanged.This asymmetry is intentional. The consistent principle is to use the cheapest reliable operation that preserves visible semantics and keeps ordinary row access simple. If repeated soft deletes leave too much hidden physical storage, prefer adding an explicit compaction/materialization operation over making all row access pay for a columnar model.
CSVField is a short-lived, non-owning text view with a local lazy conversion cache. Ordinary reads are logically const: only the type and numeric caches are mutable, so const references can classify and convert without changing text or interpretation. Custom decimal parsing remains non-const because it changes interpretation. Forward and reverse field iterators return fields by value. Navigation does not allocate or construct wrappers; dereferencing creates a wrapper through the same decoding, trimming and optional eager-scalar path as indexed access.
Each iterator owns its original RawCSVDataPtr and captures data_start, fields_start, row length and column position. There is no parent-row pointer: reader reuse, assignment or destruction of the original row cannot redirect the iterator. The returned field itself does not own the bytes; a row or iterator for its chunk must outlive use of the field or its string views.
An iterator-owned reference is unsafe when a standard adaptor dereferences a temporary iterator, and a chunk-owned field cache would retain one wrapper per touched field for the chunk's lifetime and share mutable parsing state. Returning values avoids both costs. Each dereference has an independent conversion cache; retain the field value to reuse its parsing results. Arrow access uses a temporary value-owning proxy without a heap allocation.
The legacy iterator_category is input_iterator_tag: C++11-17 forward iterator requirements demand actual references. C++20 iterator_concept is random_access_iterator_tag, with value-returning indexing and dereferencing. The library provides its own small reverse iterator for all supported standards; C++20 standard reverse iterators and reverse views are also supported. This does not make the parsed row writable: assigning a field value cannot edit CSV bytes. See ../../docs/source/row_iterators.md for user-facing examples and migration.
CSVReader::iterator carries std::input_iterator_tag intentionally — this is a hard architectural constraint, not an oversight:
RawCSVData chunks that are freed as the iterator advances.ForwardIterator would require retaining every chunk for the lifetime of any copy of the iterator, which means a 50 GB CSV would require 50+ GB of resident memory — defeating the entire streaming architecture.ForwardIterator (std::max_element, std::sort, etc.) may appear to work on small files (where only one chunk is ever allocated) but are unsafe in general: accessing an earlier iterator position after the chunk it pointed into has been freed is undefined behavior.Correct pattern when random-access algorithms are needed:
What NOT to do:
std::vector<RawCSVDataPtr> cache to CSVReader::iterator to support multi-pass. That destroys bounded-memory behavior.iterator_category to forward_iterator_tag without first solving the chunk-lifetime problem.This invariant is canonical here and summarized in the root AGENTS.md guidance.
For full testing strategy, checklist, and conventions, see:
csv_error.hpp defines CSVErrorCode, CSVError, and CSVConversionError. Reading, DataFrame validation, serialization and writing share this error model.common.hpp detects compiler exception support. Throwing API declarations and definitions are guarded; explicit-error implementations remain shared by both modes. CSVField::try_get() and C++23 as<T>() use the same typed conversion dispatch, including string, view and long-double cases.checked_field_range.hpp adapts checked row access into a C++11 field range for the shared range serializer in both exception modes. It validates proxies, retains each field's owned storage through serialization, and stops on the first access error. The writer discards the failed row's buffered prefix before returning that error, while retaining previously buffered records.