A collection of virtual filesystems for working with scalars
Maintainer(s):
teaguesterling
Installing and Loading
INSTALL scalarfs FROM community;
LOAD scalarfs;
Example
LOAD scalarfs;
-- Read JSON from a variable
SET VARIABLE config = '{"debug": true, "port": 8080}';
SELECT * FROM read_json('variable:config');
-- Read CSV from inline content
SELECT * FROM read_csv('data+varchar:name,score
Alice,95
Bob,87');
-- Use a file path stored in a variable
SET VARIABLE data_path = '/data/reports/monthly.csv';
SELECT * FROM read_csv('pathvariable:data_path');
SET VARIABLE data_paths = ['/data/reports/monthly.csv', '/archive/reports/*.csv'];
SELECT * FROM read_csv('pathvariable:data_paths');
-- Write query results to a variable
COPY (SELECT * FROM my_table WHERE active) TO 'variable:exported' (FORMAT json);
SELECT getvariable('exported');
-- Store query results as native values (not serialized text)
COPY (SELECT path FROM files WHERE active) TO 'variable:paths' (FORMAT variable);
SELECT * FROM read_csv('pathvariable:paths'); -- Read all files from the list
About scalarfs
DuckDB's file functions (read_csv, read_json, COPY TO, etc.) expect file paths. scalarfs bridges the gap when your content is already in memory, allowing the same functions to work with:
Variables — Store data in DuckDB variables and read/write them as files Path Variables — Use file paths stored in variables for dynamic file resolution Inline literals — Embed content directly in your queries without temporary files
| Protocol | Purpose | Mode |
|---|---|---|
| variable: | DuckDB variable as file | Read/Write |
| pathvariable: | File path(s) stored in variable | Read/Write* |
| data: | RFC 2397 data URI (base64/url-encoded) | Read |
| data+varchar: | Raw VARCHAR content as file | Read |
| data+blob: | Escaped BLOB content as file | Read |
- Can only write to a
pathvariable:that's a scalar path (not lists).
For full documentation, see: https://scalarfs.readthedocs.io/
Note: This extension was written primarily using Claude and Claude Code as an exercise in AI-driven development.
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| from_blob_uri | scalar | NULL | NULL | |
| from_data_uri | scalar | NULL | NULL | |
| from_scalarfs_uri | scalar | NULL | NULL | |
| from_varchar_uri | scalar | NULL | NULL | |
| to_blob_uri | scalar | NULL | NULL | |
| to_data_uri | scalar | NULL | NULL | |
| to_scalarfs_uri | scalar | NULL | NULL | |
| to_varchar_uri | scalar | NULL | NULL |