Introducing sqlite-xsv - a new, incredibly fast SQLite extension for working with CSVs
- The fastest CSV parser for SQLite, even faster reading than DuckDB/datafusion/clickhouse!
- Works with gzip'ed or zstd CSVs directly!
- Read CSVs from blobs or from multiple files!
@alexgarciaxyz So this doesn't just read CSV into a SQLite table - the CSV is read at query time, and re-read on each query? Totally makes sense if CSV is big and query is known - neat! Also any plans re open source license?
@ciphergoth Yup, reads the CSV at query time! Helps when the CSV changes with new data, and gives you control over what data to import (a subset of columns, LIMIT, etc.)
Also fixed that blog link, and added MIT+Apache licenses!
@alexgarciaxyz Brilliant, thanks! Also I guess since it's built on XSV, it'll use csv-index files if they are present? Though now I know what they are, I'm having trouble understanding why they're useful - they're not an "index" in the SQL sense...
@ciphergoth It actually doesn't use csv-index - it's built in top of the csv crate, not xsv (which doesn't appear to have a rust API).
But I haven't heard of csv-index before, I'll see if I could add it to sqlite-xsv! It seems useful for `where rowid = 123` queries. Added an issue to track here: https://github.com/asg017/sqlite-xsv/issues/1