|
Vince's CSV Parser
|
The all encompassing namespace. More...
Classes | |
| struct | CSVError |
| A diagnostic whose text remains valid for the lifetime of the error. More... | |
| class | CSVField |
| Data type representing individual CSV values. More... | |
| struct | CSVFileInfo |
| Returned by the file inspection utilities. More... | |
| class | CSVFormat |
| Stores information about how to parse a CSV file. More... | |
| struct | CSVGuessResult |
| Stores the inferred format of a CSV file. More... | |
| class | CSVReader |
| Main class for parsing CSVs from files and in-memory sources. More... | |
| class | CSVRow |
| Data structure for representing CSV rows. More... | |
| class | DataFrame |
| class | DataFrameCell |
| class | DataFrameColumn |
| class | DataFrameExecutor |
| class | DataFrameOptions |
| class | DataFrameRow |
| class | DelimWriter |
| Class for writing delimiter separated values files. More... | |
| struct | is_invocable_returning |
| struct | is_invocable_returning_impl |
| struct | is_invocable_returning_impl< F, ReturnType, void_t< invoke_result_t< F, Args... > >, Args... > |
| struct | RowOverlay |
| struct | RowOverlaySlot |
Typedefs | |
| using | string_view = std::string_view |
| The string_view class used by this library. | |
| template<bool B, class T = void> | |
| using | enable_if_t = typename std::enable_if< B, T >::type |
| template<typename F , typename... Args> | |
| using | invoke_result_t = typename std::invoke_result< F, Args... >::type |
| template<typename... Ts> | |
| using | void_t = void |
| using | RowCollection = internals::ThreadSafeDeque< CSVRow > |
| Standard type for storing collection of rows. | |
Enumerations | |
| enum class | CSVConversionError { None = 0 , NotANumber , Overflow , FloatToInt , NegativeToUnsigned } |
| Non-throwing CSVField conversion result. More... | |
| enum class | CSVErrorCode { None , InvalidArgument , InvalidFormat , InvalidChunkSize , OpenFailed , MappingFailed , ReadFailed , WriteFailed , UnsupportedEncoding , RowTooLarge , RowLengthMismatch , ColumnNotFound , IndexOutOfBounds , ConversionFailed , MissingKeyColumn , KeyNotFound , DuplicateKey , UnkeyedFrame , ReadOnlyProxy , InvalidProxy , KeyColumnProtected , UnsupportedOperation , SizeLimitExceeded } |
| Operational failures reported through CSVError. More... | |
| enum class | VariableColumnPolicy { THROW = -1 , IGNORE_ROW = 0 , KEEP = 1 , KEEP_NON_EMPTY = 2 } |
| Determines how to handle rows that are shorter or longer than the majority. More... | |
| enum class | ColumnNamePolicy { EXACT = 0 , CASE_INSENSITIVE = 1 } |
| Determines how column name lookups are performed. More... | |
| enum class | DataType { UNKNOWN = classify_scalar::scalar_invalid , CSV_NULL = classify_scalar::scalar_null , CSV_STRING = classify_scalar::scalar_string , CSV_BOOL = classify_scalar::scalar_bool , CSV_INT8 = classify_scalar::scalar_int8 , CSV_INT16 = classify_scalar::scalar_int16 , CSV_INT32 = classify_scalar::scalar_int32 , CSV_INT64 = classify_scalar::scalar_int64 , CSV_BIGINT = classify_scalar::scalar_bigint , CSV_DOUBLE = classify_scalar::scalar_float , CSV_TIMESTAMP = classify_scalar::scalar_timestamp , scalar_custom_begin = classify_scalar::scalar_custom_begin - 1 } |
| Enumerates the different CSV field types recognized by this library. More... | |
Functions | |
| const char * | csv_conversion_error_message (CSVConversionError error) noexcept |
| Return a stable human-readable description for a CSVConversionError. | |
| CSVRow::operator std::vector< std::string > () const | |
| CSVRow::iterator | operator+ (CSVRow::iterator::difference_type n, const CSVRow::iterator &it) noexcept |
| CSVRow::reverse_iterator | operator+ (CSVRow::reverse_iterator::difference_type n, const CSVRow::reverse_iterator &it) noexcept |
| template<> | |
| std::string | CSVField::get< std::string > () const |
| Retrieve this field's original string. | |
| template<> | |
| CONSTEXPR_14 csv::string_view | CSVField::get< csv::string_view > () const |
| Retrieve a view over this field's string. | |
| template<> | |
| CONSTEXPR_14 bool | CSVField::try_get< csv::string_view > (csv::string_view &out) const noexcept |
| CSVGuessResult | try_guess_format (csv::string_view filename, CSVError &error, const std::vector< char > &delims={ ',', '|', '\t', ';', '^', '~' }) |
| Guess the delimiter, header row, and mode column count of a CSV file. | |
| CSVGuessResult | guess_format (csv::string_view filename, const std::vector< char > &delims={ ',', '|', '\t', ';', '^', '~' }) |
Utility Functions | |
| std::unordered_map< std::string, DataType > | csv_data_types (CSVReader &reader) |
| std::unordered_map< std::string, DataType > | try_csv_data_types (CSVReader &reader, CSVError &error) |
| Infer SQL-friendly column data types from an existing CSVReader. | |
| template<typename Source > | |
| auto | try_csv_data_types (Source &&source, CSVError &error, const CSVFormat &format=CSVFormat::guess_csv()) -> decltype(CSVReader::try_open(std::forward< Source >(source), error, format), std::unordered_map< std::string, DataType >()) |
| Infer types from a filename, borrowed stream, or owned stream, reporting I/O errors. | |
| template<typename... ReaderArgs, csv::enable_if_t< std::is_constructible< CSVReader, ReaderArgs... >::value, int > = 0> | |
| std::unordered_map< std::string, DataType > | csv_data_types (ReaderArgs &&... reader_args) |
| Infer SQL-friendly column data types from any CSVReader constructor input. | |
| template<typename State , typename Fn > | |
| bool | try_chunk_parallel_apply (CSVReader &reader, DataFrameExecutor &executor, std::vector< State > &states, Fn &&fn, CSVError &error, size_t chunk_size=50000) |
| Apply a per-column batch function over a CSVReader using a reusable executor. | |
| template<typename State , typename Fn > | |
| void | chunk_parallel_apply (CSVReader &reader, DataFrameExecutor &executor, std::vector< State > &states, Fn &&fn, size_t chunk_size=50000) |
| template<typename State , typename Fn > | |
| bool | try_chunk_parallel_apply (CSVReader &reader, std::vector< State > &states, Fn &&fn, CSVError &error, size_t chunk_size=50000) |
| Apply a per-column batch function over a CSVReader with a temporary executor. | |
| template<typename State , typename Fn > | |
| void | chunk_parallel_apply (CSVReader &reader, std::vector< State > &states, Fn &&fn, size_t chunk_size=50000) |
| CSVFileInfo | try_get_file_info (const std::string &filename, CSVError &error) |
| Get basic information about a CSV file. | |
| CSVFileInfo | get_file_info (const std::string &filename) |
| std::vector< std::string > | try_get_col_names (csv::string_view filename, CSVError &error, const CSVFormat &format=CSVFormat::guess_csv()) |
| Get the column names of a CSV file using just the first 500KB. | |
| long long | try_get_col_pos (csv::string_view filename, csv::string_view col_name, CSVError &error, const CSVFormat &format=CSVFormat::guess_csv()) |
| Find a column position, returning CSV_NOT_FOUND for an absent name or on failure. | |
| std::vector< std::string > | get_col_names (csv::string_view filename, const CSVFormat &format=CSVFormat::guess_csv()) |
| Get the column names of a CSV file using just the first 500KB. | |
| long long | get_col_pos (csv::string_view filename, csv::string_view col_name, const CSVFormat &format=CSVFormat::guess_csv()) |
| Find the position of a column in a CSV file or CSV_NOT_FOUND otherwise. | |
Shorthand Parsing Functions | |
Convenience functions for parsing small strings | |
| std::unique_ptr< CSVReader > | try_parse (csv::string_view in, CSVError &error, const CSVFormat &format=CSVFormat::guess_csv()) |
| Parse CSV from a string view, copying the input into an owned buffer. | |
| CSVReader | parse (csv::string_view in, const CSVFormat &format=CSVFormat::guess_csv()) |
| std::unique_ptr< CSVReader > | try_parse_unsafe (csv::string_view in, CSVError &error, const CSVFormat &format=CSVFormat::guess_csv()) |
| Parse CSV from an in-memory view with zero copy. | |
| CSVReader | parse_unsafe (csv::string_view in, const CSVFormat &format=CSVFormat::guess_csv()) |
| CSVReader | parse_no_header (csv::string_view in) |
| Parses a CSV string with no headers. | |
| CSVReader | operator""_csv (const char *in, size_t n) |
| Parse a RFC 4180 CSV string. | |
| CSVReader | operator""_csv_no_header (const char *in, size_t n) |
| A shorthand for csv::parse_no_header(). | |
Variables | |
| constexpr int | CSV_NOT_FOUND = -1 |
| Integer indicating a requested column wasn't found. | |
| constexpr unsigned | CHAR_OFFSET = std::numeric_limits<char>::is_signed ? 128 : 0 |
| Offset to convert char into array index. | |
CSV Writing | |
| template<class OutputStream > | |
| using | CSVWriter = DelimWriter< OutputStream, ',', '"'> |
| An alias for csv::DelimWriter for writing standard CSV files. | |
| template<class OutputStream > | |
| using | TSVWriter = DelimWriter< OutputStream, '\t', '"'> |
| Class for writing tab-separated values files. | |
| template<class OutputStream > | |
| CSVWriter< OutputStream > | make_csv_writer (OutputStream &out, bool quote_minimal=true) |
| Return a csv::CSVWriter over the output stream. | |
| template<class OutputStream > | |
| TSVWriter< OutputStream > | make_tsv_writer (OutputStream &out, bool quote_minimal=true) |
| Return a csv::TSVWriter over the output stream. | |
The all encompassing namespace.
| using csv::CSVWriter = typedef DelimWriter<OutputStream, ',', '"'> |
An alias for csv::DelimWriter for writing standard CSV files.
csv::make_csv_writer() to instantiate this class over an actual output stream. Definition at line 979 of file csv_writer.hpp.
| using csv::enable_if_t = typedef typename std::enable_if<B, T>::type |
Definition at line 210 of file common.hpp.
| using csv::invoke_result_t = typedef typename std::invoke_result<F, Args...>::type |
Definition at line 223 of file common.hpp.
| using csv::RowCollection = typedef internals::ThreadSafeDeque<CSVRow> |
The string_view class used by this library.
Definition at line 182 of file common.hpp.
| using csv::TSVWriter = typedef DelimWriter<OutputStream, '\t', '"'> |
Class for writing tab-separated values files.
csv::make_tsv_writer() to instantiate this class over an actual output stream. Definition at line 990 of file csv_writer.hpp.
| using csv::void_t = typedef void |
Definition at line 230 of file common.hpp.
|
strong |
Determines how column name lookups are performed.
| Enumerator | |
|---|---|
| EXACT | Case-sensitive match (default) |
| CASE_INSENSITIVE | Case-insensitive match. |
Definition at line 37 of file csv_format.hpp.
|
strong |
Non-throwing CSVField conversion result.
Returned by CSVField::as() inside std::expected, and used internally by CSVField::get() and CSVField::try_get() to keep throwing and non-throwing conversions on the same rules.
Definition at line 23 of file csv_error.hpp.
|
strong |
Operational failures reported through CSVError.
| Enumerator | |
|---|---|
| None | Success, including end-of-stream without a row. |
| InvalidArgument | Invalid operation input, such as a null stream or zero batch size. |
| InvalidFormat | Conflicting parsing characters or an unresolved delimiter query. |
| InvalidChunkSize | Configured read-window size is outside the supported byte limits. |
| OpenFailed | A native file could not be opened, including initial input mapping. |
| MappingFailed | A subsequent input-file region could not be memory-mapped; see CSVError::system. |
| ReadFailed | The input stream reported an I/O failure rather than ordinary EOF. |
| WriteFailed | Writing, flushing, or closing the output failed. |
| UnsupportedEncoding | A UTF-16/UTF-32 BOM was found; transcode the input to UTF-8 first. |
| RowTooLarge | A complete record could not fit within the configured read window. |
| RowLengthMismatch | A record's field count violates the configured variable-column policy. |
| ColumnNotFound | A requested name is absent from the visible column schema. |
| IndexOutOfBounds | A row/column position exceeds the visible schema or physical record width. |
| ConversionFailed | A DataFrame field/key conversion failed; context describes the cause. |
| MissingKeyColumn | Keyed construction has an empty key-column name or a name absent from the schema. |
| KeyNotFound | A requested row key is absent from the frame's key index. |
| DuplicateKey | A key conflicts with an existing row under the requested insertion/duplicate policy. |
| UnkeyedFrame | A key-based operation was requested on a positional-only frame. |
| ReadOnlyProxy | Mutation was requested through a cell, row, or column without edit permission. |
| InvalidProxy | A proxy is unbound or a column proxy was invalidated by structural mutation. |
| KeyColumnProtected | Removing the column that supplies the frame's stored keys is prohibited. |
| UnsupportedOperation | The operation cannot be performed for this frame, such as inserting into a custom-key frame. |
| SizeLimitExceeded | Combined preserved string-view key bytes exceed std::string::max_size(). |
Definition at line 61 of file csv_error.hpp.
|
strong |
Enumerates the different CSV field types recognized by this library.
Definition at line 14 of file data_type.hpp.
|
strong |
Determines how to handle rows that are shorter or longer than the majority.
Definition at line 29 of file csv_format.hpp.
|
inline |
Definition at line 211 of file csv_utility.hpp.
|
inline |
Definition at line 243 of file csv_utility.hpp.
|
inlinenoexcept |
Return a stable human-readable description for a CSVConversionError.
Definition at line 53 of file csv_error.hpp.
Definition at line 6 of file csv_utility.cpp.
|
inline |
Infer SQL-friendly column data types from any CSVReader constructor input.
This convenience overload forwards its arguments directly to CSVReader, so it supports filenames, std::istream sources, owned streams, and custom CSVFormat combinations without additional wrapper code.
Definition at line 158 of file csv_utility.hpp.
| CONSTEXPR_14 csv::string_view csv::CSVField::get< csv::string_view > | ( | ) | const |
Retrieve a view over this field's string.
Definition at line 886 of file csv_row.hpp.
|
inline |
Retrieve this field's original string.
Definition at line 876 of file csv_row.hpp.
|
noexcept |
Definition at line 903 of file csv_row.hpp.
| csv::CSVRow::operator std::vector< std::string > | ( | ) | const |
Definition at line 67 of file csv_row.cpp.
|
inline |
Get the column names of a CSV file using just the first 500KB.
Definition at line 316 of file csv_utility.hpp.
|
inline |
Find the position of a column in a CSV file or CSV_NOT_FOUND otherwise.
Definition at line 327 of file csv_utility.hpp.
|
inline |
Definition at line 281 of file csv_utility.hpp.
|
inline |
Definition at line 134 of file driver.hpp.
|
inline |
Return a csv::CSVWriter over the output stream.
Definition at line 994 of file csv_writer.hpp.
|
inline |
Return a csv::TSVWriter over the output stream.
Definition at line 1000 of file csv_writer.hpp.
|
inline |
Parse a RFC 4180 CSV string.
String literals have static storage duration, so the zero-copy path is safe here.
Definition at line 99 of file csv_utility.hpp.
|
inline |
A shorthand for csv::parse_no_header().
String literals have static storage duration, so the zero-copy path is safe here.
Definition at line 108 of file csv_utility.hpp.
|
noexcept |
Definition at line 311 of file csv_row.cpp.
|
noexcept |
Definition at line 399 of file csv_row.cpp.
|
inline |
Definition at line 50 of file csv_utility.hpp.
|
inline |
Parses a CSV string with no headers.
Definition at line 85 of file csv_utility.hpp.
|
inline |
Definition at line 75 of file csv_utility.hpp.
|
inline |
Apply a per-column batch function over a CSVReader using a reusable executor.
Reads the source in chunks, promotes each chunk into a temporary DataFrame, and applies fn(column, states[column.index()]).
Callbacks may treat each batch DataFrame as read-mostly, and sparse overlay cell edits are synchronized at row granularity. If you need more involved batch orchestration, use CSVReader::read_chunk() and construct a batch-scoped DataFrame yourself.
| std::invalid_argument | if chunk_size == 0 |
Definition at line 195 of file csv_utility.hpp.
|
inline |
Apply a per-column batch function over a CSVReader with a temporary executor.
This is the convenience overload for the common case where callers do not need to reuse worker threads across multiple reader pipelines.
Definition at line 230 of file csv_utility.hpp.
| std::unordered_map< std::string, DataType > csv::try_csv_data_types | ( | CSVReader & | reader, |
| CSVError & | error | ||
| ) |
Infer SQL-friendly column data types from an existing CSVReader.
This consumes rows from reader using the chunked ETL path and returns one inferred DataType per column name.
Definition at line 14 of file csv_utility.cpp.
|
inline |
Infer types from a filename, borrowed stream, or owned stream, reporting I/O errors.
Definition at line 127 of file csv_utility.hpp.
|
inline |
Get the column names of a CSV file using just the first 500KB.
Definition at line 290 of file csv_utility.hpp.
|
inline |
Find a column position, returning CSV_NOT_FOUND for an absent name or on failure.
An absent name is not an error; inspect error to distinguish an I/O failure.
Definition at line 303 of file csv_utility.hpp.
|
inline |
Get basic information about a CSV file.
Definition at line 258 of file csv_utility.hpp.
|
inline |
Guess the delimiter, header row, and mode column count of a CSV file.
**Heuristic:** For each candidate delimiter, calculate a score based on the most common row length (mode). The delimiter with the highest score wins. **Header Detection:** - If the first row has >= columns than the mode, it's treated as the header - Otherwise, the first row with the mode length is treated as the header This approach handles: - Headers with trailing delimiters or optional columns (wider than data rows) - Comment lines before the actual header (first row shorter than mode) - Standard CSVs where first row is the header
Definition at line 125 of file driver.hpp.
|
inline |
Parse CSV from a string view, copying the input into an owned buffer.
Safe for any string_view regardless of the caller's ownership of the underlying memory.
Definition at line 42 of file csv_utility.hpp.
|
inline |
Parse CSV from an in-memory view with zero copy.
WARNING: Non-owning path. The caller must ensure in's backing memory remains valid and immutable while the reader may request additional rows from the source stream.
Rows already obtained from the reader remain valid, but unread rows still depend on the source view staying alive.
Definition at line 67 of file csv_utility.hpp.
|
constexpr |
Offset to convert char into array index.
Definition at line 505 of file common.hpp.
|
constexpr |
Integer indicating a requested column wasn't found.
Definition at line 502 of file common.hpp.