Moving Average | | :-------- | :--------- | --------------------------: | | Boston | 2019-01-02 | 517450.75 | | Boston | 2019-01-03 | 508793.20 | | Boston | 2019-01-04 | 508529.83 | | Boston | 2019-01-05 | 523459.85 | | Boston | 2019-01-06 | 526067.14 | | Boston | 2019-01-07 | 524938.71 | | Boston | 2019-01-08 | 518294.57 | | Boston | 2019-01-09 | 520665.42 | | Boston | 2019-01-10 | 528859.00 | | Boston | 2019-01-11 | 532466.66 | | Boston | 2019-01-12 | 516352.00 | | Boston | 2019-01-13 | 499793.00 | | Worcester | 2019-01-02 | 104768.25 | | Worcester | 2019-01-03 | 102713.00 | | Worcester | 2019-01-04 | 102249.50 | | Worcester | 2019-01-05 | 104621.57 | | Worcester | 2019-01-06 | 103856.71 | | Worcester | 2019-01-07 | 103094.85 | | Worcester | 2019-01-08 | 101345.14 | | Worcester | 2019-01-09 | 102313.85 | | Worcester | 2019-01-10 | 104125.00 | | Worcester | 2019-01-11 | 104823.83 | | Worcester | 2019-01-12 | 102017.80 | | Worcester | 2019-01-13 | 99145.75 | You can request multiple different `OVER` clauses in the same `SELECT`, and each will be computed separately. Often, however, you want to use the same window for multiple functions, and you can do this by using a `WINDOW` clause to define a *named* window: ```sql SELECT "Plant", "Date", avg("MWh") OVER seven AS "MWh 7-day Moving Average" FROM "Generation History" WINDOW seven AS ( PARTITION BY "Plant" ORDER BY "Date" ASC RANGE BETWEEN INTERVAL 3 DAYS PRECEDING AND INTERVAL 3 DAYS FOLLOWING) ORDER BY 1, 2; ``` This would be useful, for example, if one also wanted the 7-day moving `min` and `max` to show the bounds of the data. #### Under the Feathers {#blog:2021-10-13-windowing::under-the-feathers} That is a long list of complicated functionality! Making it all work relatively quickly has many pieces, so lets have a look at how they all get implemented in DuckDB. ##### Pipeline Breaking {#blog:2021-10-13-windowing::pipeline-breaking} The first thing to notice is that windowing is a “pipeline breaker”. That is, the `Window` operator has to read all of its inputs before it can start computing a function. This means that if there is some other way to compute something, it may well be faster to use a different technique. One common analytic task is to find the last value in some group. For example, suppose we want the last recorded power output for each plant. It is tempting to use the `rank()` window function with a reverse sort for this task: ```sql SELECT "Plant", "MWh" FROM ( SELECT "Plant", "MWh", rank() OVER ( PARTITION BY "Plant" ORDER BY "Date" DESC) AS r FROM table) t WHERE r = 1; ``` but this requires materialising the entire table, partitioning it, sorting the partitions, and then pulling out a single row from those partitions. A much faster way to do this is to use a self join to filter the table to contain only the last (` max`) value of the `DATE` field: ```sql SELECT table."Plant", "MWh" FROM table, (SELECT "Plant", max("Date") AS "Date" FROM table GROUP BY 1) lasts WHERE table."Plant" = lasts."Plant" AND table."Date" = lasts."Date"; ``` This join query requires two scans of the table, but the only materialised data is the filtering table (which is probably much smaller than the original table), and there is no sorting at all. This type of query showed up [in a user's blog](https://bwlewis.github.io/duckdb_and_r/last/last.html) and we found that the join query was over 20 times faster on their data set:  Of course most analytic tasks that use windowing *do* require using the `Window` operator, and DuckDB uses a collection of techniques to make the performance as fast as possible. ##### Partitioning and Sorting {#blog:2021-10-13-windowing::partitioning-and-sorting} At one time, windowing was implemented by sorting on both the partition and the ordering fields and then finding the partition boundaries. This is resource intensive, both because the entire relation must be sorted, and because sorting is `O(N log N)` in the size of the relation. Fortunately, there are faster ways to implement this step. To reduce resource consumption, DuckDB uses the partitioning scheme from Leis et al.'s [*Efficient Processing of Window Functions in Analytical SQL Queries*](http://www.vldb.org/pvldb/vol8/p1058-leis.pdf) and breaks the partitions up into 1024 chunks using `O(N)` hashing. The chunks still need to be sorted on all the fields because there may be hash collisions, but each partition can now be 1024 times smaller, which reduces the runtime significantly. Moreover, the partitions can easily be extracted and processed in parallel. Sorting in DuckDB recently got a [big performance boost](#blog:2021-08-27-external-sorting), along with the ability to work on partitions that were larger than memory. This functionality has been also added to the `Window` operator, resulting in a 33% improvement in the last-in-group example:  As a final optimization, even though you can request multiple window functions, DuckDB will collect functions that use the same partitioning and ordering, and share the data layout between those functions. ##### Aggregation {#blog:2021-10-13-windowing::aggregation} Most of the [general-purpose window functions](https://duckdb.org/docs/1.5/sql/functions/window_functions) are straightforward to compute, but windowed aggregate functions can be expensive because they need to look at multiple values for each row. They often need to look at the same value multiple times, or repeatedly look at a large number of values, so over the years several approaches have been taken to improve performance. ###### Naïve Windowed Aggregation {#blog:2021-10-13-windowing::nave-windowed-aggregation} Before explaining how DuckDB implements windowed aggregation, we need to take a short detour through how ordinary aggregates are implemented. Aggregate “functions” are implemented using three required operations and one optional operation: * *Initialize* – Creates a state that will be updated. For `sum`, this is the running total, starting at `NULL` (because a `sum` of zero items is `NULL`, not zero.) * *Update* – Updates the state with a new value. For `sum`, this adds the value to the state. * *Finalize* – Produces the final aggregate value from the state. For `sum`, this just copies the running total. * *Combine* – Combines two states into a single state. Combine is optional, but when present it allows the aggregate to be computed in parallel. For `sum`, this produces a new state with the sum of the two input values. The simplest way to compute an individual windowed aggregate value is to *initialize* a state, *update* the state with all the values in the window frame, and then use *finalize* to produce the value of the windowed aggregate. This naïve algorithm will always work, but it is quite inefficient. For example, a running total will re-add all the values from the start of the partition for each running total, and this has a run time of `O(N^2)`. To improve on this, some databases add additional [“moving state” operations](https://www.postgresql.org/docs/14/sql-createaggregate.html) that can add or remove individual values incrementally. This reduces computation in some common cases, but it can only be used for certain aggregates. For example, it doesn't work for `min`) because you don't know if there are multiple duplicate minima. Moreover, if the frame boundaries move around a lot, it can still degenerate to an `O(N^2)` run time. ###### Segment Tree Aggregation {#blog:2021-10-13-windowing::segment-tree-aggregation} Instead of adding more functions, DuckDB uses the *segment tree* approach from Leis et al. above. This works by building a tree on top of the entire partition with the aggregated values at the bottom. Values are combined into states at nodes above them in the tree until there is a single root:  To compute a value, the algorithm generates states for the ragged ends of the frame, *combines* states in the tree above the values in the frame, and *finalizes* the result from the last remaining state. So in the example above (Figure 5 from Leis et al.) only three values need to be added instead of 7. This technique can be used for all *combinable* aggregates. ###### General Windowed Aggregation {#blog:2021-10-13-windowing::general-windowed-aggregation} The biggest drawback of segment trees is the need to manage a potentially large number of intermediate states. For the simple states used for standard distributive aggregates like `sum`, this is not a problem because the states are small, the tree keeps the number of states logarithmically low, and the state used to compute each value is also cheap. For some aggregates, however, the state is not small. Typically these are so-called *holistic* aggregates, where the value depends on all the values of the frame. Examples of such aggregates are `mode` and `quantile`, where each state may have to contain a copy of *all* the values seen so far. While segment trees *can* be used to implement moving versions of any combinable aggregate, this can be quite expensive for large, complex states – and this was not the original goal of the algorithm. To solve this problem, we use the approach from Wesley and Xu's [*Incremental Computation of Common Windowed Holistic Aggregates*](http://www.vldb.org/pvldb/vol9/p1221-wesley.pdf), which generalises segment trees to aggregate-specific data structures. The aggregate can define a fifth optional *window* operation, which will be passed the bottom of the tree and the bounds of the current and previous frame. The aggregate can then create an appropriate data structure for its implementation. For example, the `mode` function maintains a hash table of counts that it can update efficiently, and the `quantile` function maintains a partially sorted list of frame indexes. Moreover, the `quantile` functions can take an array of quantile values, which further increases performance by sharing the partially ordered results among the different quantile values. Because these aggregates can be used in a windowing context, the moving average example above can be easily modified to produce a moving inter-quartile range: ```sql SELECT "Plant", "Date", quantile_cont("MWh", [0.25, 0.5, 0.75]) OVER seven AS "MWh 7-day Moving IQR" FROM "Generation History" WINDOW seven AS ( PARTITION BY "Plant" ORDER BY "Date" ASC RANGE BETWEEN INTERVAL 3 DAYS PRECEDING AND INTERVAL 3 DAYS FOLLOWING) ORDER BY 1, 2; ``` Moving quantiles like this are [more robust to anomalies](https://blogs.sas.com/content/iml/2021/05/26/running-median-smoother.html), which makes them a valuable tool for data series analysis, but they are not generally implemented in most database systems. There are some approaches that can be used in some query engines, but the lack of a general moving aggregation architecture means that these solutions can be [unnatural](https://docs.oracle.com/cd/E57185_01/HIRUG/ch12s07s08.html) or [complex](https://ndesmo.github.io/blog/oracle-moving-metrics/). DuckDB's implementation uses the standard window notation, which means you don't have to learn new syntax or pull the data out into another tool. ###### Ordered Set Aggregates {#blog:2021-10-13-windowing::ordered-set-aggregates} Window functions are often closely associated with some special [“ordered set aggregates”](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE) defined by the SQL standard. Some databases implement these functions using the `Window` operator, but this is rather inefficient because sorting the data (an `O(N log N)` operation) is not required – it suffices to use Hoare's `O(N)` [`FIND`](https://courses.cs.vt.edu/~cs3114/Summer15/Notes/Supplemental/p321-hoare.pdf) algorithm as used in the STL's [`std::nth_element`](https://en.cppreference.com/w/cpp/algorithm/nth_element). DuckDB translates these ordered set aggregates to use the faster `quantile_cont`, `quantile_disc`, and `mode` regular aggregate functions, thereby avoiding using windowing entirely. ###### Extensions {#blog:2021-10-13-windowing::extensions} This architecture also means that any new aggregates we add can benefit from the existing windowing infrastructure. DuckDB is an open source project, and we welcome submissions of useful aggregate functions – or you can create your own domain-specific ones in your own fork. At some point we hope to have a UDF architecture that will allow plug-in aggregates, and the simplicity and power of the interface will let these plugins leverage the notational simplicity and run time performance that the internal functions enjoy. #### Conclusion {#blog:2021-10-13-windowing::conclusion} DuckDB's windowing implementation uses a variety of techniques to speed up what can be the slowest part of an analytic query. It is well integrated with the sorting subsystem and the aggregate function architecture, which makes expressing advanced moving aggregates both natural and efficient. DuckDB is a free and open-source database management system (MIT licensed). It aims to be the SQLite for Analytics, and provides a fast and efficient database system with zero external dependencies. It is available not just for Python, but also for C/C++, R, Java, and more. ## DuckDB-Wasm: Efficient Analytical SQL in the Browser {#blog:2021-10-29-duckdb-wasm} **Publication date:** 2021-10-29 **Authors:** André Kohn, Dominik Moritz **TL;DR:** [DuckDB-Wasm](https://github.com/duckdb/duckdb-wasm) is an in-process analytical SQL database for the browser. It is powered by WebAssembly, speaks Arrow fluently, reads Parquet, CSV and JSON files backed by Filesystem APIs or HTTP requests and has been tested with Chrome, Firefox, Safari and Node.js. You can try it at [shell.duckdb.org](https://shell.duckdb.org) or on [Observable](https://observablehq.com/@cmudig/duckdb).  *DuckDB-Wasm is fast! If you're here for performance numbers, head over to our benchmarks at [shell.duckdb.org/versus](https://shell.duckdb.org/versus).* #### Efficient Analytics in the Browser {#blog:2021-10-29-duckdb-wasm::efficient-analytics-in-the-browser} The web browser has evolved to a universal computation platform that even runs in your car. Its rise has been accompanied by increasing requirements for the browser programming language JavaScript. JavaScript was, first and foremost, designed to be very flexible which comes at the cost of a reduced processing efficiency compared to native languages like C++. This becomes particularly apparent when considering the execution times of more complex data analysis tasks that often fall behind the native execution by orders of magnitude. In the past, such analysis tasks have therefore been pushed to servers that tie any client-side processing to additional round-trips over the internet and introduce their own set of scalability problems. The processing capabilities of browsers were boosted tremendously 4 years ago with the introduction of WebAssembly: > WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications. > > The Wasm stack machine is designed to be encoded in a size- and load-time efficient binary format. WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms. > > (ref: [https://webassembly.org/](https://webassembly.org/)) Four years later, the WebAssembly revolution is in full progress with first implementations being shipped in four major browsers. It has already brought us game engines, [entire IDEs](https://blog.stackblitz.com/posts/introducing-webcontainers/) and even a browser version of [Photoshop](https://web.dev/ps-on-the-web/). Today, we join the ranks with a first release of the npm library [@duckdb/duckdb-wasm](https://www.npmjs.com/package/@duckdb/duckdb-wasm). As an in-process analytical database, DuckDB has the rare opportunity to significantly speed up OLAP workloads in the browser. We believe that there is a need for a comprehensive and self-contained data analysis library. DuckDB-wasm automatically offloads your queries to dedicated worker threads and reads Parquet, CSV and JSON files from either your local filesystem or HTTP servers driven by plain SQL input. In this blog post, we want to introduce the library and present challenges on our journey towards a browser-native OLAP database. *DuckDB-Wasm is not yet stable. You will find rough edges and bugs in this release. Please share your thoughts with us [on GitHub](https://github.com/duckdb/duckdb-wasm/discussions).* #### How to Get Data In? {#blog:2021-10-29-duckdb-wasm::how-to-get-data-in} Let's dive into examples. DuckDB-Wasm provides a variety of ways to load your data. First, raw SQL value clauses like `INSERT INTO sometable VALUES (1, 'foo'), (2, 'bar')` are easy to formulate and only depend on plain SQL text. Alternatively, SQL statements like `CREATE TABLE foo AS SELECT * FROM 'somefile.parquet'` consult our integrated web filesystem to resolve `somefile.parquet` locally, remotely or from a buffer. The methods `insertCSVFromPath` and `insertJSONFromPath` further provide convenient ways to import CSV and JSON files using additional typed settings like column types. And finally, the method `insertArrowFromIPCStream` (optionally through `insertArrowTable`, `insertArrowBatches` or `insertArrowVectors`) copies raw IPC stream bytes directly into a WebAssembly stream decoder. The following example presents different options how data can be imported into DuckDB-Wasm: ```ts // Data can be inserted from an existing arrow.Table await c.insertArrowTable(existingTable, { name: "arrow_table" }); // ..., from Arrow vectors await c.insertArrowVectors({ col1: arrow.Int32Vector.from([1, 2]), col2: arrow.Utf8Vector.from(["foo", "bar"]), }, { name: "arrow_vectors" }); // ..., from a raw Arrow IPC stream const c = await db.connect(); const streamResponse = await fetch(` someapi`); const streamReader = streamResponse.body.getReader(); const streamInserts = []; while (true) { const { value, done } = await streamReader.read(); if (done) break; streamInserts.push(c.insertArrowFromIPCStream(value, { name: "streamed" })); } await Promise.all(streamInserts); // ..., from CSV files // (interchangeable: registerFile{Text,Buffer,URL,Handle}) await db.registerFileText(` data.csv`, "1|foo\n2|bar\n"); // ... with typed insert options await db.importCSVFromPath('data.csv', { schema: 'main', name: 'foo', detect: false, header: false, delimiter: '|', columns: { col1: new arrow.Int32(), col2: new arrow.Utf8(), } }); // ..., from JSON documents in row-major format await db.registerFileText("rows.json", `[ { "col1": 1, "col2": "foo" }, { "col1": 2, "col2": "bar" }, ]`); // ... or column-major format await db.registerFileText("columns.json", `{ "col1": [1, 2], "col2": ["foo", "bar"] }`); // ... with typed insert options await db.importJSONFromPath('rows.json', { name: 'rows' }); await db.importJSONFromPath('columns.json', { name: 'columns' }); // ..., from Parquet files const pickedFile: File = letUserPickFile(); await db.registerFileHandle("local.parquet", pickedFile); await db.registerFileURL("remote.parquet", "https://origin/remote.parquet"); // ..., by specifying URLs in the SQL text await c.query(` CREATE TABLE direct AS SELECT * FROM 'https://origin/remote.parquet' `); // ..., or by executing raw insert statements await c.query(` INSERT INTO existing_table VALUES (1, "foo"), (2, "bar")`); ``` #### How to Get Data Out? {#blog:2021-10-29-duckdb-wasm::how-to-get-data-out} Now that we have the data loaded, DuckDB-Wasm can run queries on two different ways that differ in the result materialization. First, the method `query` runs a query to completion and returns the results as single `arrow.Table`. Second, the method `send` fetches query results lazily through an `arrow.RecordBatchStreamReader`. Both methods are generic and allow for typed results in Typescript: ```ts // Either materialize the query result await conn.query<{ v: arrow.Int32 }>(` SELECT * FROM generate_series(1, 100) t(v) `); // ..., or fetch the result chunks lazily for await (const batch of await conn.send<{ v: arrow.Int32 }>(` SELECT * FROM generate_series(1, 100) t(v) `)) { // ... } ``` Alternatively, you can prepare statements for parameterized queries using: ```ts // Prepare query const stmt = await conn.prepare<{ v: arrow.Int32 }>( `SELECT (v + ?) AS v FROM generate_series(0, 10000) t(v);` ); // ... and run the query with materialized results await stmt.query(234); // ... or result chunks for await (const batch of await stmt.send(234)) { // ... } ``` #### Looks like Arrow to Me {#blog:2021-10-29-duckdb-wasm::looks-like-arrow-to-me} DuckDB-Wasm uses [Arrow](https://arrow.apache.org) as data protocol for the data import and all query results. Arrow is a database-friendly columnar format that is organized in chunks of column vectors, called record batches and that support zero-copy reads with only a small overhead. The npm library `apache-arrow` implements the Arrow format in the browser and is already used by other data processing frameworks, like [Arquero](https://github.com/uwdata/arquero). Arrow therefore not only spares us the implementation of the SQL type logic in JavaScript, it also makes us compatible to existing tools. _Why not use plain Javascript objects?_ WebAssembly is isolated and memory-safe. This isolation is part of it's DNA and drives fundamental design decisions in DuckDB-Wasm. For example, WebAssembly introduces a barrier towards the traditional JavaScript heap. Crossing this barrier is difficult as JavaScript has to deal with native function calls, memory ownership and serialization performance. Languages like C++ make this worse as they rely on smart pointers that are not available through the FFI. They leave us with the choice to either pass memory ownership to static singletons within the WebAssembly instance or maintain the memory through C-style APIs in JavaScript, a language that is too dynamic for sound implementations of the [RAII idiom](https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization). The memory-isolation forces us to serialize data before we can pass it to the WebAssembly instance. Browsers can serialize JavaScript objects natively to and from JSON using the functions `JSON.stringify` and `JSON.parse` but this is slower compared to, for example, copying raw native arrays. #### Web Filesystem {#blog:2021-10-29-duckdb-wasm::web-filesystem} DuckDB-Wasm integrates a dedicated filesystem for WebAssembly. DuckDB itself is built on top of a virtual filesystem that decouples higher level tasks, such as reading a Parquet file, from low-level filesystem APIs that are specific to the operating system. We leverage this abstraction in DuckDB-Wasm to tailor filesystem implementations to the different WebAssembly environments. The following figure shows our current web filesystem in action. The sequence diagram presents a user running a SQL query that scans a single Parquet file. The query is first offloaded to a dedicated web worker through a JavaScript API. There, it is passed to the WebAssembly module that processes the query until the execution hits the `parquet_scan` table function. This table function then reads the file using a buffered filesystem which, in turn, issues paged reads on the web filesystem. This web filesystem then uses an environment-specific runtime to read the file from several possible locations.
Depending on the context, the Parquet file may either reside on the local device, on a remote server or in a buffer that was registered by the user upfront. We deliberately treat all three cases equally to unify the retrieval and processing of external data. This does not only simplify the analysis, it also enables more advanced features like partially consuming structured file formats. Parquet files, for example, consist of multiple row groups that store data in a column-major fashion. As a result, we may not need to download the entire file for a query but only required bytes. A query like `SELECT count(*) FROM parquet_scan(...)`, for example, can be evaluated on the file metadata alone and will finish in milliseconds even on remote files that are several terabytes large. Another more general example are paging scans with `LIMIT` and `OFFSET` qualifiers such as `SELECT * FROM parquet_scan(...) LIMIT 20 OFFSET 40`, or queries with selective filter predicates where entire row groups can be skipped based on metadata statistics. These partial file reads are no groundbreaking novelty and could be implemented in JavaScript today, but with DuckDB-Wasm, these optimizations are now driven by the semantics of SQL queries instead of fine-tuned application logic. *Note: The common denominator among the available File APIs is unfortunately not large. This limits the features that we can provide in the browser. For example, local persistency of DuckDB databases would be a feature with significant impact but requires a way to either read and write synchronously into user-provided files or IndexedDB. We might be able to bypass these limitations in the future but this is subject of ongoing research.* #### Advanced Features {#blog:2021-10-29-duckdb-wasm::advanced-features} WebAssembly 1.0 has landed in all major browsers. The WebAssembly Community Group fixed the design of this first version back in November 2017, which is now referred to as WebAssembly MVP. Since then, the development has been ongoing with eight additional features that have been added to the standard and at least five proposals that are currently in progress. The rapid pace of this development presents challenges and opportunities for library authors. On the one hand, the different features find their way into the browsers at different speeds which leads to a fractured space of post-MVP functionality. On the other hand, features can bring flat performance improvements and are therefore indispensable when aiming for a maximum performance. The most promising feature for DuckDB-Wasm is [exception handling](https://github.com/WebAssembly/exception-handling/blob/main/proposals/exception-handling/Exceptions.md) which is already enabled by default in Chrome 95. DuckDB and DuckDB-Wasm are written in C++ and use exceptions for faulty situations. DuckDB does not use exceptions for general control flow but to automatically propagate errors upwards to the top-level plan driver. In native environments, these exceptions are implemented as “zero-cost exceptions” as they induce no overhead until they are thrown. With the WebAssembly MVP, however, that is no longer possible as the compiler toolchain Emscripten has to emulate exceptions through JavaScript. Without WebAssembly exceptions, DuckDB-Wasm calls throwing functions through a JavaScript hook that can catch exceptions emulated through JavaScript `aborts`. An example for these hook calls is shown in the following figure. Both stack traces originate from a single paged read of a Parquet file in DuckDB-Wasm. The left side shows a stack trace with the WebAssembly MVP and requires multiple calls through the functions `wasm-to-js-i*` . The right stack trace uses WebAssembly exceptions without any hook calls.
This fractured feature space is a temporary challenge that will be resolved once high-impact features like exception handling, SIMD and bulk-memory operations are available everywhere. In the meantime, we will ship multiple WebAssembly modules that are compiled for different feature sets and adaptively pick the best bundle for you using dynamic browser checks. The following example shows how the asynchronous version of DuckDB-Wasm can be instantiated using either manual or JsDelivr bundles: ```ts // Import the ESM bundle (supports tree-shaking) import * as duckdb from '@duckdb/duckdb-wasm/dist/duckdb-esm.js'; // Either bundle them manually, for example as Webpack assets import duckdb_wasm from '@duckdb/duckdb-wasm/dist/duckdb.wasm'; import duckdb_wasm_next from '@duckdb/duckdb-wasm/dist/duckdb-next.wasm'; import duckdb_wasm_next_coi from '@duckdb/duckdb-wasm/dist/duckdb-next-coi.wasm'; const WEBPACK_BUNDLES: duckdb.DuckDBBundles = { asyncDefault: { mainModule: duckdb_wasm, mainWorker: new URL('@duckdb/duckdb-wasm/dist/duckdb-browser-async.worker.js', import.meta.url).toString(), }, asyncNext: { mainModule: duckdb_wasm_next, mainWorker: new URL('@duckdb/duckdb-wasm/dist/duckdb-browser-async-next.worker.js', import.meta.url).toString(), }, asyncNextCOI: { mainModule: duckdb_wasm_next_coi, mainWorker: new URL( '@duckdb/duckdb-wasm/dist/duckdb-browser-async-next-coi.worker.js', import.meta.url, ).toString(), pthreadWorker: new URL( '@duckdb/duckdb-wasm/dist/duckdb-browser-async-next-coi.pthread.worker.js', import.meta.url, ).toString(), }, }; // ..., or load the bundles from jsdelivr const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles(); // Select a bundle based on browser checks const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES); // Instantiate the asynchronous version of DuckDB-Wasm const worker = new Worker(bundle.mainWorker!); const logger = new duckdb.ConsoleLogger(); const db = new duckdb.AsyncDuckDB(logger, worker); await db.instantiate(bundle.mainModule, bundle.pthreadWorker); ``` *You can also test the features and selected bundle in your browser using the web shell command `.features` .* #### Multithreading {#blog:2021-10-29-duckdb-wasm::multithreading} In 2018, the Spectre and Meltdown vulnerabilities sent crippling shockwaves through the internet. Today, we are facing the repercussions of these events, in particular in software that runs arbitrary user code – such as web browsers. Shortly after the publications, all major browser vendors restricted the use of `SharedArrayBuffers` to prevent dangerous timing attacks. `SharedArrayBuffers` are raw buffers that can be shared among web workers for global state and an alternative to the browser-specific message passing. These restrictions had detrimental effects on WebAssembly modules since `SharedArrayBuffers` are necessary for the implementation of POSIX threads in WebAssembly. Without `SharedArrayBuffers`, WebAssembly modules can run in a dedicated web worker to unblock the main event loop but won't be able to spawn additional workers for parallel computations within the same instance. By default, we therefore cannot unleash the parallel query execution of DuckDB in the web. However, browser vendors have recently started to reenable `SharedArrayBuffers` for websites that are [cross-origin-isolated](https://web.dev/coop-coep/). A website is cross-origin-isolated if it ships the main document with the following HTTP headers: ```text Cross-Origin-Embedder-Policy: require-corp Cross-Origin-Opener-Policy: same-origin ``` These headers will instruct browsers to A) isolate the top-level document from other top-level documents outside its own origin and B) prevent the document from making arbitrary cross-origin requests unless the requested resource explicitly opts in. Both restrictions have far reaching implications for a website since many third-party data sources won't yet provide the headers today and the top-level isolation currently hinders the communication with, for example, OAuth pop up's ([there are plans to lift that](https://github.com/whatwg/html/issues/6364)). *We therefore assume that DuckDB-Wasm will find the majority of users on non-isolated websites. We are, however, experimenting with dedicated bundles for isolated sites using the suffix `-next-coi`) and will closely monitor the future need of our users.* #### Web Shell {#blog:2021-10-29-duckdb-wasm::web-shell} We further host a web shell powered by DuckDB-Wasm alongside the library release at [shell.duckdb.org](https://shell.duckdb.org). Use the following shell commands to query remote TPC-H files at scale factor 0.01. When querying your own, make sure to properly set CORS headers since your browser will otherwise block these requests. You can alternatively use the `.files` command to register files from the local filesystem. ```sql .timer on SELECT count(*) FROM 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/lineitem.parquet'; SELECT count(*) FROM 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/customer.parquet'; SELECT avg(c_acctbal) FROM 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/customer.parquet'; SELECT * FROM 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/orders.parquet' LIMIT 10; SELECT n_name, avg(c_acctbal) FROM 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/customer.parquet', 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/nation.parquet' WHERE c_nationkey = n_nationkey GROUP BY n_name; SELECT * FROM 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/region.parquet', 'https://blobs.duckdb.org/data/tpch-sf0.01-parquet/nation.parquet' WHERE r_regionkey = n_regionkey; ``` #### Evaluation {#blog:2021-10-29-duckdb-wasm::evaluation} The following table teases the execution times of some TPC-H queries at scale factor 0.5 using the libraries [DuckDB-Wasm](https://www.npmjs.com/package/@duckdb/duckdb-wasm), [sql.js](https://github.com/sql-js/sql.js/), [Arquero](https://github.com/uwdata/arquero) and [Lovefield](https://github.com/google/lovefield). You can find a more in-depth discussion with all TPC-H queries, additional scale factors and microbenchmarks on the [“DuckDB-Wasm versus X” page](https://shell.duckdb.org/versus). | Query | DuckDB-Wasm | sql.js | Arquero | Lovefield | | ----: | ----------: | ------: | -------: | --------: | | 1 | **0.855 s** | 8.441 s | 24.031 s | 12.666 s | | 3 | **0.179 s** | 1.758 s | 16.848 s | 3.587 s | | 4 | **0.151 s** | 0.384 s | 6.519 s | 3.779 s | | 5 | **0.197 s** | 1.965 s | 18.286 s | 13.117 s | | 6 | **0.086 s** | 1.294 s | 1.379 s | 5.253 s | | 7 | **0.319 s** | 2.677 s | 6.013 s | 74.926 s | | 8 | **0.236 s** | 4.126 s | 2.589 s | 18.983 s | | 10 | **0.351 s** | 1.238 s | 23.096 s | 18.229 s | | 12 | **0.276 s** | 1.080 s | 11.932 s | 10.372 s | | 13 | **0.194 s** | 5.887 s | 16.387 s | 9.795 s | | 14 | **0.086 s** | 1.194 s | 6.332 s | 6.449 s | | 16 | **0.137 s** | 0.453 s | 0.294 s | 5.590 s | | 19 | **0.377 s** | 1.272 s | 65.403 s | 9.977 s | #### Future Research {#blog:2021-10-29-duckdb-wasm::future-research} We believe that WebAssembly unveils hitherto dormant potential for shared query processing between clients and servers. Pushing computation closer to the client can eliminate costly round-trips to the server and thus increase interactivity and scalability of in-browser analytics. We further believe that the release of DuckDB-Wasm could be the first step towards a more universal data plane spanning across multiple layers including traditional database servers, clients, CDN workers and computational storage. As an in-process analytical database, DuckDB might be the ideal driver for distributed query plans that increase the scalability and interactivity of SQL databases at low costs. ## Fast Moving Holistic Aggregates {#blog:2021-11-12-moving-holistic} **Publication date:** 2021-11-12 **Author:** Richard Wesley **TL;DR:** DuckDB, a free and open-source analytical data management system, has a windowing API that can compute complex moving aggregates like interquartile ranges and median absolute deviation much faster than the conventional approaches. In a [previous post](#blog:2021-10-13-windowing), we described the DuckDB windowing architecture and mentioned the support for some advanced moving aggregates. In this post, we will compare the performance various possible moving implementations of these functions and explain how DuckDB's performant implementations work. #### What Is an Aggregate Function? {#blog:2021-11-12-moving-holistic::what-is-an-aggregate-function} When people think of aggregate functions, they typically have something simple in mind such as `SUM` or `AVG`. But more generally, what an aggregate function does is _summarise_ a set of values into a single value. Such summaries can be arbitrarily complex, and involve any data type. For example, DuckDB provides aggregates for concatenating strings (` STRING_AGG`) and constructing lists (` LIST`). In SQL, aggregated sets come from either a `GROUP BY` clause or an `OVER` windowing specification. ##### Holistic Aggregates {#blog:2021-11-12-moving-holistic::holistic-aggregates} All of the basic SQL aggregate functions like `SUM` and `MAX` can be computed by reading values one at a time and throwing them away. But there are some functions that potentially need to keep track of all the values before they can produce a result. These are called _holistic_ aggregates, and they require more care when implementing. For some aggregates (like `STRING_AGG`) the order of the values can change the result. This is not a problem for windowing because `OVER` clauses can specify an ordering, but in a `GROUP BY` clause, the values are unordered. To handle this, order-sensitive aggregates can include a `WITHIN GROUP(ORDER BY| q1 | median | q3 |
|---|---|---|
| 1.04 | 1.74 | 3.13 |
| quantile_disc(0.99 ORDER BY trip_distance) |
|---|
| 19.7 |
`SELECT * FROM st_drivers();`
| short_name | long_name | can_create | can_copy | can_open | help_url | | -------------- | --------------------------------------------------- | ---------- | -------- | -------- | -------------------------------------------------------------------- | | ESRI Shapefile | ESRI Shapefile | true | false | true |Wait, what's all this about coordinate reference systems and projections?
> The earth is not flat, but sometimes it is useful to pretend it is for the sake of simplicity by “projecting” the coordinates onto a flat surface. The “parameters” of a projection – e.g., where the “origin” is located, what unit coordinates are in, or how the earth's shape is approximated – are encapsulated by a “Spatial Reference System” or “Coordinate Reference System” (CRS) which is usually referenced by a shorthand identifier composed of an authority and a code, e.g., “EPSG:4326” or “ESRI:102718”. Projections are always lossy, so its important to use a CRS that is well suited for the “area of interest” your data is in. The spatial extension uses the [PROJ](https://proj.org/) library to handle coordinate reference systems and projections.`SELECT * FROM rides LIMIT 10;`
| vendor_id | pickup_datetime | dropoff_datetime | passenger_count | trip_distance | pickup_longitude | pickup_latitude | rate_code | store_and_fwd_flag | dropoff_longitude | dropoff_latitude | payment_type | fare_amount | surcharge | mta_tax | tip_amount | tolls_amount | total_amount | | --------- | ------------------- | ------------------- | --------------- | ------------------ | ------------------ | --------------- | --------- | ------------------ | ------------------ | ---------------- | ------------ | ----------- | --------- | ------- | ---------- | ------------ | ------------ | | VTS | 2010-01-01 00:00:17 | 2010-01-01 00:00:17 | 3 | 0.0 | -73.87105699999998 | 40.773522 | 1 | | -73.871048 | 40.773545 | CAS | 45.0 | 0.0 | 0.5 | 0.0 | 0.0 | 45.5 | | VTS | 2010-01-01 00:00:20 | 2010-01-01 00:00:20 | 1 | 0.05 | -73.97512999999998 | 40.789973 | 1 | | -73.97498799999998 | 40.790598 | CAS | 2.5 | 0.5 | 0.5 | 0.0 | 0.0 | 3.5 | | CMT | 2010-01-01 00:00:23 | 2010-01-01 00:00:25 | 1 | 0.0 | -73.999431 | 40.71216 | 1 | 0 | -73.99915799999998 | 40.712421 | No | 2.5 | 0.5 | 0.5 | 0.0 | 0.0 | 3.5 | | CMT | 2010-01-01 00:00:33 | 2010-01-01 00:00:55 | 1 | 0.0 | -73.97721699999998 | 40.749633 | 1 | 0 | -73.97732899999998 | 40.749629 | Cas | 2.5 | 0.5 | 0.5 | 0.0 | 0.0 | 3.5 | | VTS | 2010-01-01 00:01:00 | 2010-01-01 00:01:00 | 1 | 0.0 | -73.942313 | 40.784332 | 1 | | -73.942313 | 40.784332 | Cre | 10.0 | 0.0 | 0.5 | 2.0 | 0.0 | 12.5 | | VTS | 2010-01-01 00:01:06 | 2010-01-01 00:01:06 | 2 | 0.38 | -73.97463 | 40.756687 | 1 | | -73.979872 | 40.759143 | CAS | 3.7 | 0.5 | 0.5 | 0.0 | 0.0 | 4.7 | | VTS | 2010-01-01 00:01:07 | 2010-01-01 00:01:07 | 2 | 0.23 | -73.987358 | 40.718475 | 1 | | -73.98518 | 40.720468 | CAS | 2.9 | 0.5 | 0.5 | 0.0 | 0.0 | 3.9 | | CMT | 2010-01-01 00:00:02 | 2010-01-01 00:01:08 | 1 | 0.1 | -73.992807 | 40.741418 | 1 | 0 | -73.995799 | 40.742596 | No | 2.9 | 0.5 | 0.5 | 0.0 | 0.0 | 3.9 | | VTS | 2010-01-01 00:01:23 | 2010-01-01 00:01:23 | 1 | 0.6099999999999999 | -73.98003799999998 | 40.74306 | 1 | | -73.974862 | 40.750387 | CAS | 3.7 | 0.5 | 0.5 | 0.0 | 0.0 | 4.7 | | VTS | 2010-01-01 00:01:34 | 2010-01-01 00:01:34 | 1 | 0.02 | -73.954122 | 40.801173 | 1 | | -73.95431499999998 | 40.800897 | CAS | 45.0 | 0.0 | 0.5 | 0.0 | 0.0 | 45.5 |`SELECT * FROM zones LIMIT 10;`
| zone | LocationID | borough | geom | | ----------------------- | ---------- | ------------- | ------------------ | | Newark Airport | 1 | EWR | POLYGON (...) | | Jamaica Bay | 2 | Queens | MULTIPOLYGON (...) | | Allerton/Pelham Gardens | 3 | Bronx | POLYGON (...) | | Alphabet City | 4 | Manhattan | POLYGON (...) | | Arden Heights | 5 | Staten Island | POLYGON (...) | | Arrochar/Fort Wadsworth | 6 | Staten Island | POLYGON (...) | | Astoria | 7 | Queens | POLYGON (...) | | Astoria Park | 8 | Queens | POLYGON (...) | | Auburndale | 9 | Queens | POLYGON (...) | | Baisley Park | 10 | Queens | POLYGON (...) |`SELECT * FROM joined USING SAMPLE 10 ROWS;`
| pickup_point | dropoff_point | start_zone | end_zone | trip_distance | time | | ------------------------------------ | ------------------------------------ | ------------------------ | ----------------------------- | ------------- | -------- | | POINT (40.722223 -73.98385299999998) | POINT (40.715507 -73.992438) | East Village | Lower East Side | 10.3 | 00:19:16 | | POINT (40.648687 -73.783522) | POINT (40.649567 -74.005812) | JFK Airport | Sunset Park West | 23.57 | 00:28:00 | | POINT (40.761603 -73.96661299999998) | POINT (40.760232 -73.96344499999998) | Upper East Side South | Sutton Place/Turtle Bay North | 17.6 | 00:27:05 | | POINT (40.697212 -73.937495) | POINT (40.652377 -73.93983299999998) | Stuyvesant Heights | East Flatbush/Farragut | 13.55 | 00:24:00 | | POINT (40.721462 -73.993583) | POINT (40.774205 -73.90441699999998) | Lower East Side | Steinway | 28.75 | 01:03:00 | | POINT (40.716955 -74.004328) | POINT (40.754688 -73.991612) | TriBeCa/Civic Center | Garment District | 18.4 | 00:46:12 | | POINT (40.740052 -73.994918) | POINT (40.75439 -73.98587499999998) | Flatiron | Garment District | 24.2 | 00:35:25 | | POINT (40.763017 -73.95949199999998) | POINT (40.763615 -73.959182) | Lenox Hill East | Lenox Hill West | 18.4 | 00:33:46 | | POINT (40.865663 -73.927458) | POINT (40.86537 -73.927352) | Washington Heights North | Washington Heights North | 10.47 | 00:27:00 | | POINT (40.738408 -73.980345) | POINT (40.696038 -73.955493) | Gramercy | Bedford | 16.4 | 00:21:47 |`head -n 10 joined.geojsonseq`
```json { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:52:00" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.789923, 40.643515 ], [ -73.97608, 40.680395 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:35:00" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.776445, 40.645422 ], [ -73.98427, 40.670782 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:45:42" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.776878, 40.645065 ], [ -73.992153, 40.662571 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:36:00" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.788028, 40.641508 ], [ -73.97584, 40.670927 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:47:58" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.781855, 40.644749 ], [ -73.980129, 40.663663 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:32:10" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.787494, 40.641559 ], [ -73.974694, 40.673479 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:36:59" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.790138, 40.643342 ], [ -73.982721, 40.662379 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:32:00" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.786952, 40.641248 ], [ -73.97421, 40.676237 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:33:21" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.783892, 40.648514 ], [ -73.979283, 40.669721 ] ] } } { "type": "Feature", "properties": { "start_zone": "JFK Airport", "end_zone": "Park Slope", "trip_time": "00:35:45" }, "geometry": { "type": "LineString", "coordinates": [ [ -73.776643, 40.645272 ], [ -73.978873, 40.66723 ] ] } } ```episode_num) | max(trek_facts.
aired_date) | max(trek_facts.
cnt_kirk_hookups) | ... | max(trek_facts.
bool_enterprise_saved_the_day) | | :------------------------------ | :----------------------------- | :----------------------------------- | :--- | :------------------------------------------------ | | 29 | 1967-04-13 | 2 | ... | 1 | The `REPLACE` syntax is also useful when applied to a dynamic set of columns. In this example, we want to convert the dates into timestamps prior to finding the maximum value in each column. Previously this would have required an entire subquery or CTE to pre-process just that single column! ```sql SELECT max(COLUMNS(* REPLACE aired_date::timestamp AS aired_date)) FROM trek_facts; ``` | max(trek_facts.
season_num) | max(trek_facts.
episode_num) | max(aired_date :=
CAST(aired_date AS TIMESTAMP)) | ... | max(trek_facts.
bool_enterprise_saved_the_day) | | :----------------------------- | :------------------------------ | :--------------------------------------------------- | :--- | :------------------------------------------------ | | 1 | 29 | 1967-04-13 00:00:00 | ... | 1 | ##### `COLUMNS()` with Lambda Functions {#blog:2023-08-23-even-friendlier-sql::columns-with-lambda-functions} The most flexible way to query a dynamic set of columns is through a [lambda function](https://duckdb.org/docs/1.5/sql/functions/nested#lambda-functions). This allows for any matching criteria to be applied to the names of the columns, not just regular expressions. See more details about lambda functions below. For example, if using the `LIKE` syntax is more comfortable, we can select columns matching a `LIKE` pattern rather than with a regular expression. ```sql SELECT episode_num, COLUMNS(lambda col: col LIKE '%warp%') FROM trek_facts WHERE COLUMNS(lambda col: col LIKE '%warp%') >= 2; ``` | episode_num | cnt_warp_speed_orders | highest_warp_speed_issued | | :---------- | :-------------------- | :------------------------ | | 14 | 3 | 7 | | 17 | 2 | 7 | | 18 | 2 | 8 | | 29 | 2 | 8 | ##### Automatic JSON to Nested Types Conversion {#blog:2023-08-23-even-friendlier-sql::automatic-json-to-nested-types-conversion} The first installment in the series mentioned JSON dot notation references as future work. However, the team has gone even further! Instead of referring to JSON-typed columns using dot notation, JSON can now be [automatically parsed](#blog:2023-03-03-json) into DuckDB’s native types for significantly faster performance, compression, as well as that friendly dot notation! First, install and load the `httpfs` and `json` extensions if they don't come bundled with the client you are using. Then query a remote JSON file directly as if it were a table! ```sql INSTALL httpfs; LOAD httpfs; INSTALL json; LOAD json; SELECT starfleet[10].model AS starship FROM 'https://raw.githubusercontent.com/vlad-saling/star-trek-ipsum/master/src/content/content.json'; ``` | starship | | :------------------------------------------------------------------------------------------------------------------------------------------------------------ | | USS Farragut - NCC-1647 - Ship on which James Kirk served as a phaser station operator. Attacked by the Dikironium Cloud Creature, killing half the crew. ad. | Now for some new SQL capabilities beyond the ideas from the prior post! #### `FROM` First in `SELECT` Statements {#blog:2023-08-23-even-friendlier-sql::from-first-in-select-statements} When building a query, the first thing you need to know is where your data is coming `FROM`. Well then why is that the second clause in a `SELECT` statement?? No longer! DuckDB is building SQL as it should have always been – putting the `FROM` clause first! This addresses one of the longest standing complaints about SQL, and the DuckDB team implemented it in 2 days. ```sql FROM my_table SELECT my_column; ``` Not only that, the `SELECT` statement can be completely removed and DuckDB will assume all columns should be `SELECT`ed. Taking a look at a table is now as simple as: ```sql FROM my_table; -- SELECT * FROM my_table ``` Other statements like `COPY` are simplified as well. ```sql COPY (FROM trek_facts) TO 'phaser_filled_facts.parquet'; ``` This has an additional benefit beyond saving keystrokes and staying in a development flow state: autocomplete will have much more context when you begin to choose columns to query. Give the AI a helping hand! Note that this syntax is completely optional, so your `SELECT * FROM` keyboard shortcuts are safe, even if they are obsolete... 🙂 #### Function Chaining {#blog:2023-08-23-even-friendlier-sql::function-chaining} Many SQL blogs advise the use of CTEs instead of subqueries. Among other benefits, they are much more readable. Operations are compartmentalized into discrete chunks and they can be read in order top to bottom instead of forcing the reader to work their way inside out. DuckDB enables the same interpretability improvement for every scalar function! Use the dot operator to chain functions together, just like in Python. The prior expression in the chain is used as the first argument to the subsequent function. ```sql SELECT ('Make it so') .upper() .string_split(' ') .list_aggr('string_agg','.') .concat('.') AS im_not_messing_around_number_one; ``` | im_not_messing_around_number_one | | :------------------------------- | | MAKE.IT.SO. | Now compare that with the old way... ```sql SELECT concat( list_aggr( string_split( upper('Make it stop'), ' '), 'string_agg','.'), '.') AS oof; ``` | oof | | :------------ | | MAKE.IT.STOP. | #### Union by Name {#blog:2023-08-23-even-friendlier-sql::union-by-name} DuckDB aims to blend the best of databases and dataframes. This new syntax is inspired by the [concat function in Pandas](https://pandas.pydata.org/docs/reference/api/pandas.concat.html). Rather than vertically stacking tables based on column position, columns are matched by name and stacked accordingly. Simply replace `UNION` with `UNION BY NAME` or `UNION ALL` with `UNION ALL BY NAME`. For example, we had to add some new alien species proverbs in The Next Generation: ```sql CREATE TABLE proverbs AS SELECT 'Revenge is a dish best served cold' AS klingon_proverb UNION ALL BY NAME SELECT 'You will be assimilated' AS borg_proverb, 'If winning is not important, why keep score?' AS klingon_proverb; FROM proverbs; ``` | klingon_proverb | borg_proverb | | :------------------------------------------- | :---------------------- | | Revenge is a dish best served cold | NULL | | If winning is not important, why keep score? | You will be assimilated | This approach has additional benefits. As seen above, not only can tables with different column orders be combined, but so can tables with different numbers of columns entirely. This is helpful as schemas migrate, and is particularly useful for DuckDB’s [multi-file reading capabilities](https://duckdb.org/docs/1.5/data/multiple_files/combining_schemas#union-by-name). #### Insert by Name {#blog:2023-08-23-even-friendlier-sql::insert-by-name} Another common situation where column order is strict in SQL is when inserting data into a table. Either the columns must match the order exactly, or all of the column names must be repeated in two locations within the query. Instead, add the keywords `BY NAME` after the table name when inserting. Any subset of the columns in the table in any order can be inserted. ```sql INSERT INTO proverbs BY NAME SELECT 'Resistance is futile' AS borg_proverb; SELECT * FROM proverbs; ``` | klingon_proverb | borg_proverb | | :------------------------------------------- | :---------------------- | | Revenge is a dish best served cold | NULL | | If winning is not important, why keep score? | You will be assimilated | | NULL | Resistance is futile | #### Dynamic `PIVOT` and `UNPIVOT` {#blog:2023-08-23-even-friendlier-sql::dynamic-pivot-and-unpivot} Historically, databases are not well-suited for pivoting operations. However, DuckDB’s `PIVOT` and `UNPIVOT` clauses can create or stack dynamic column names for a truly flexible pivoting capability! In addition to that flexibility, DuckDB also provides both the SQL standard syntax and a friendlier shorthand. For example, let’s take a look at some procurement forecast data just as the Earth-Romulan war was beginning: ```sql CREATE TABLE purchases (item VARCHAR, year INTEGER, count INTEGER); INSERT INTO purchases VALUES ('phasers', 2155, 1035), ('phasers', 2156, 25039), ('phasers', 2157, 95000), ('photon torpedoes', 2155, 255), ('photon torpedoes', 2156, 17899), ('photon torpedoes', 2157, 87492); FROM purchases; ``` | item | year | count | | :--------------- | :--- | :---- | | phasers | 2155 | 1035 | | phasers | 2156 | 25039 | | phasers | 2157 | 95000 | | photon torpedoes | 2155 | 255 | | photon torpedoes | 2156 | 17899 | | photon torpedoes | 2157 | 87492 | It is easier to compare our phaser needs to our photon torpedo needs if each year’s data is visually close together. Let’s pivot this into a friendlier format! Each year should receive its own column (but each year shouldn’t need to be specified in the query!), we want to sum up the total `count`, and we still want to keep a separate group (row) for each `item`. ```sql CREATE TABLE pivoted_purchases AS PIVOT purchases ON year USING sum(count) GROUP BY item; FROM pivoted_purchases; ``` | item | 2155 | 2156 | 2157 | | :--------------- | :--- | :---- | :---- | | phasers | 1035 | 25039 | 95000 | | photon torpedoes | 255 | 17899 | 87492 | Looks like photon torpedoes went on sale... Now imagine the reverse situation. Scotty in engineering has been visually analyzing and manually constructing his purchases forecast. He prefers things pivoted so it’s easier to read. Now you need to fit it back into the database! This war may go on for a bit, so you may need to do this again next year. Let’s write an `UNPIVOT` query to return to the original format that can handle any year. The `COLUMNS` expression will use all columns except `item`. After stacking, the column containing the column names from `pivoted_purchases` should be renamed to `year`, and the values within those columns represent the `count`. The result is the same dataset as the original. ```sql UNPIVOT pivoted_purchases ON COLUMNS(* EXCLUDE item) INTO NAME year VALUE count; ``` | item | year | count | | :--------------- | :--- | :---- | | phasers | 2155 | 1035 | | phasers | 2156 | 25039 | | phasers | 2157 | 95000 | | photon torpedoes | 2155 | 255 | | photon torpedoes | 2156 | 17899 | | photon torpedoes | 2157 | 87492 | More examples are included as a part of our [DuckDB 0.8.0 announcement post](#blog:2023-05-17-announcing-duckdb-080::new-sql-features), and the [`PIVOT`](https://duckdb.org/docs/1.5/sql/statements/pivot) and [`UNPIVOT`](https://duckdb.org/docs/1.5/sql/statements/unpivot) documentation pages highlight more complex queries. Stay tuned for a future post to cover what is happening behind the scenes! #### List Lambda Functions {#blog:2023-08-23-even-friendlier-sql::list-lambda-functions} List lambdas allow for operations to be applied to each item in a list. These do not need to be pre-defined – they are created on the fly within the query. In this example, a lambda function is used in combination with the `list_transform` function to shorten each official ship name. ```sql SELECT (['Enterprise NCC-1701', 'Voyager NCC-74656', 'Discovery NCC-1031']) .list_transform(lambda x: x.string_split(' ')[1]) AS short_name; ``` | ship_name | | :------------------------------- | | [Enterprise, Voyager, Discovery] | Lambdas can also be used to filter down the items in a list. The lambda returns a list of booleans, which is used by the `list_filter` function to select specific items. The `contains` function is using the [function chaining](#blog:2023-08-23-even-friendlier-sql::function-chaining) described earlier. ```sql SELECT (['Enterprise NCC-1701', 'Voyager NCC-74656', 'Discovery NCC-1031']) .list_filter(lambda x: x.contains('1701')) AS the_original; ``` | the_original | | :-------------------- | | [Enterprise NCC-1701] | #### List Comprehensions {#blog:2023-08-23-even-friendlier-sql::list-comprehensions} What if there was a simple syntax to both modify and filter a list? DuckDB takes inspiration from Python’s approach to list comprehensions to dramatically simplify the above examples. List comprehensions are syntactic sugar – these queries are rewritten into lambda expressions behind the scenes! Within brackets, first specify the transformation that is desired, then indicate which list should be iterated over, and finally include the filter criteria. ```sql SELECT [x.string_split(' ')[1] FOR x IN ['Enterprise NCC-1701', 'Voyager NCC-74656', 'Discovery NCC-1031'] IF x.contains('1701')] AS ready_to_boldly_go; ``` | ready_to_boldly_go | | :----------------- | | [Enterprise] | #### Exploding Struct.* {#blog:2023-08-23-even-friendlier-sql::exploding-struct} A struct in DuckDB is a set of key/value pairs. Behind the scenes, a struct is stored with a separate column for each key. As a result, it is computationally easy to explode a struct into separate columns, and now it is also syntactically simple as well! This is another example of allowing SQL to handle dynamic column names. ```sql WITH damage_report AS ( SELECT {'gold_casualties':5, 'blue_casualties':15, 'red_casualties': 10000} AS casualties ) FROM damage_report SELECT casualties.*; ``` | gold_casualties | blue_casualties | red_casualties | | :-------------- | :-------------- | :------------- | | 5 | 15 | 10000 | #### Automatic Struct Creation {#blog:2023-08-23-even-friendlier-sql::automatic-struct-creation} DuckDB exposes an easy way to convert any table into a single-column struct. Instead of `SELECT`ing column names, `SELECT` the table name itself. ```sql WITH officers AS ( SELECT 'Captain' AS rank, 'Jean-Luc Picard' AS name UNION ALL SELECT 'Lieutenant Commander', 'Data' ) FROM officers SELECT officers; ``` | officers | | :------------------------------------------- | | {'rank': Captain, 'name': Jean-Luc Picard} | | {'rank': Lieutenant Commander, 'name': Data} | #### Union Data Type {#blog:2023-08-23-even-friendlier-sql::union-data-type} DuckDB utilizes strong typing to provide high performance and enforce data quality. However, DuckDB is also as forgiving as possible using approaches like implicit casting to avoid always having to cast between data types. Another way DuckDB enables flexibility is the new `UNION` data type. A `UNION` data type allows for a single column to contain multiple types of values. This can be thought of as an “opt-in” to SQLite’s flexible data typing rules (the opposite direction of SQLite’s recently announced [strict tables](https://www.sqlite.org/stricttables.html)). By default DuckDB will seek the common denominator of data types when combining tables together. The below query results in a `VARCHAR` column: ```sql SELECT 'The Motion Picture' AS movie UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 'First Contact'; ``` | movie | | :----------------- | | The Motion Picture | | First Contact | | 6 | | 5 | | 4 | | 3 | | 2 | However, if a `UNION` type is used, each individual row retains its original data type. A `UNION` is defined using key-value pairs with the key as a name and the value as the data type. This also allows the specific data types to be pulled out as individual columns: ```sql CREATE TABLE movies ( movie UNION(num INTEGER, name VARCHAR) ); INSERT INTO movies VALUES ('The Motion Picture'), (2), (3), (4), (5), (6), ('First Contact'); FROM movies SELECT movie, union_tag(movie) AS type, movie.name, movie.num; ``` | movie | type | name | num | | :----------------- | :--- | :----------------- | :--- | | The Motion Picture | name | The Motion Picture | | | 2 | num | | 2 | | 3 | num | | 3 | | 4 | num | | 4 | | 5 | num | | 5 | | 6 | num | | 6 | | First Contact | name | First Contact | | #### Additional Friendly Features {#blog:2023-08-23-even-friendlier-sql::additional-friendly-features} Several other friendly features are worth mentioning and some are powerful enough to warrant their own blog posts. DuckDB takes a nod from the [`describe` function in Pandas](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html) and implements a `SUMMARIZE` keyword that will calculate a variety of statistics about each column in a dataset for a quick, high-level overview. Simply prepend `SUMMARIZE` to any table or `SELECT` statement. Have a look at the [correlated subqueries post](#blog:2023-05-26-correlated-subqueries-in-sql) to see how to use subqueries that refer to each others’ columns. DuckDB’s advanced optimizer improves correlated subquery performance by orders of magnitude, allowing for queries to be expressed as naturally as possible. What was once an anti-pattern for performance reasons can now be used freely! DuckDB has added more ways to `JOIN` tables together that make expressing common calculations much easier. Some like `LATERAL`, `ASOF`, `SEMI`, and `ANTI` joins are present in other systems, but have high-performance implementations in DuckDB. DuckDB also adds a new `POSITIONAL` join that combines by the row numbers in each table to match the commonly used Pandas capability of joining on row number indexes. See the [`JOIN` documentation](https://duckdb.org/docs/1.5/sql/query_syntax/from) for details, and look out for a blog post describing DuckDB’s state of the art `ASOF` joins! #### Summary and Future Work {#blog:2023-08-23-even-friendlier-sql::summary-and-future-work} DuckDB aims to be the easiest database to use. Fundamental architectural decisions to be in-process, have zero dependencies, and have strong typing contribute to this goal, but the friendliness of its SQL dialect has a strong impact as well. By extending the industry-standard PostgreSQL dialect, DuckDB aims to provide the simplest way to express the data transformations you need. These changes range from altering the ancient clause order of the `SELECT` statement to begin with `FROM`, allowing a fundamentally new way to use functions with chaining, to advanced nested data type calculations like list comprehensions. Each of these features are available in the 0.8.1 release. Future work for friendlier SQL includes: * Lambda functions with more than 1 argument, like `list_zip` * Underscores as digit separators (Ex: `1_000_000` instead of `1000000`) * Extension user experience, including autoloading * Improvements to file globbing * Your suggestions! Please let us know what areas of SQL can be improved! We welcome your feedback on [Discord](https://discord.duckdb.org/) or [GitHub](https://github.com/duckdb/duckdb/discussions). Live long and prosper! 🖖 ## DuckDB's AsOf Joins: Fuzzy Temporal Lookups {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups} **Publication date:** 2023-09-15 **Author:** Richard Wesley **TL;DR:** DuckDB supports AsOf Joins – a way to match nearby values. They are especially useful for searching event tables for temporal analytics. Do you have time series data that you want to join, but the timestamps don't quite match? Or do you want to look up a value that changes over time using the times in another table? And did you end up writing convoluted (and slow) inequality joins to get your results? Then this post is for you! #### What Is an AsOf Join? {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::what-is-an-asof-join} Time series data is not always perfectly aligned. Clocks may be slightly off, or there may be a delay between cause and effect. This can make connecting two sets of ordered data challenging. AsOf Joins are a tool for solving this and other similar problems. One of the problems that AsOf Joins are used to solve is finding the value of a varying property at a specific point in time. This use case is so common that it is where the name came from: _Give me the value of the property **as of this time.**_ More generally, however, AsOf joins embody some common temporal analytic semantics, which can be cumbersome and slow to implement in standard SQL. ##### Portfolio Example {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::portfolio-example} Let's start with a concrete example. Suppose we have a table of stock [`prices`](https://duckdb.org/data/prices.csv) with timestamps: | ticker | when | price | | :----- | :------------------ | ----: | | APPL | 2001-01-01 00:00:00 | 1 | | APPL | 2001-01-01 00:01:00 | 2 | | APPL | 2001-01-01 00:02:00 | 3 | | MSFT | 2001-01-01 00:00:00 | 1 | | MSFT | 2001-01-01 00:01:00 | 2 | | MSFT | 2001-01-01 00:02:00 | 3 | | GOOG | 2001-01-01 00:00:00 | 1 | | GOOG | 2001-01-01 00:01:00 | 2 | | GOOG | 2001-01-01 00:02:00 | 3 | We have another table containing portfolio [`holdings`](https://duckdb.org/data/holdings.csv) at various points in time: | ticker | when | shares | | :----- | :------------------ | -----: | | APPL | 2000-12-31 23:59:30 | 5.16 | | APPL | 2001-01-01 00:00:30 | 2.94 | | APPL | 2001-01-01 00:01:30 | 24.13 | | GOOG | 2000-12-31 23:59:30 | 9.33 | | GOOG | 2001-01-01 00:00:30 | 23.45 | | GOOG | 2001-01-01 00:01:30 | 10.58 | | DATA | 2000-12-31 23:59:30 | 6.65 | | DATA | 2001-01-01 00:00:30 | 17.95 | | DATA | 2001-01-01 00:01:30 | 18.37 | We can compute the value of each holding at that point in time by finding the most recent price before the holding's timestamp by using an AsOf Join: ```sql SELECT h.ticker, h.when, price * shares AS value FROM holdings h ASOF JOIN prices p ON h.ticker = p.ticker AND h.when >= p.when; ``` This attaches the value of the holding at that time to each row: | ticker | when | value | | :----- | :------------------ | ----: | | APPL | 2001-01-01 00:00:30 | 2.94 | | APPL | 2001-01-01 00:01:30 | 48.26 | | GOOG | 2001-01-01 00:00:30 | 23.45 | | GOOG | 2001-01-01 00:01:30 | 21.16 | It essentially executes a function defined by looking up nearby values in the `prices` table. Note also that missing `ticker` values do not have a match and don't appear in the output. ##### Outer AsOf Joins {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::outer-asof-joins} Because AsOf produces at most one match from the right hand side, the left side table will not grow as a result of the join, but it could shrink if there are missing times on the right. To handle this situation, you can use an *outer* AsOf Join: ```sql SELECT h.ticker, h.when, price * shares AS value FROM holdings h ASOF LEFT JOIN prices p ON h.ticker = p.ticker AND h.when >= p.when ORDER BY ALL; ``` As you might expect, this will produce `NULL` prices and values instead of dropping left side rows when there is no ticker or the time is before the prices begin. | ticker | when | value | | :----- | :------------------ | ----: | | APPL | 2000-12-31 23:59:30 | | | APPL | 2001-01-01 00:00:30 | 2.94 | | APPL | 2001-01-01 00:01:30 | 48.26 | | GOOG | 2000-12-31 23:59:30 | | | GOOG | 2001-01-01 00:00:30 | 23.45 | | GOOG | 2001-01-01 00:01:30 | 21.16 | | DATA | 2000-12-31 23:59:30 | | | DATA | 2001-01-01 00:00:30 | | | DATA | 2001-01-01 00:01:30 | | ##### Windowing Alternative {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::windowing-alternative} Standard SQL can implement this kind of join, but you need to use a window function and an inequality join. These can both be fairly expensive operations, but the query would look like this: ```sql WITH state AS ( SELECT ticker, price, "when", lead("when", 1, 'infinity') OVER (PARTITION BY ticker ORDER BY "when") AS end FROM prices ) SELECT h.ticker, h.when, price * shares AS value FROM holdings h INNER JOIN state s ON h.ticker = s.ticker AND h.when >= s.when AND h.when < s.end; ``` The default value of `infinity` is used to make sure there is an end value for the last row that can be compared. Here is what the `state` CTE looks like for our example: | ticker | price | when | end | | :----- | ----: | :------------------ | :------------------ | | APPL | 1 | 2001-01-01 00:00:00 | 2001-01-01 00:01:00 | | APPL | 2 | 2001-01-01 00:01:00 | 2001-01-01 00:02:00 | | APPL | 3 | 2001-01-01 00:02:00 | infinity | | GOOG | 1 | 2001-01-01 00:00:00 | 2001-01-01 00:01:00 | | GOOG | 2 | 2001-01-01 00:01:00 | 2001-01-01 00:02:00 | | GOOG | 3 | 2001-01-01 00:02:00 | infinity | | MSFT | 1 | 2001-01-01 00:00:00 | 2001-01-01 00:01:00 | | MSFT | 2 | 2001-01-01 00:01:00 | 2001-01-01 00:02:00 | | MSFT | 3 | 2001-01-01 00:02:00 | infinity | In the case where there is no equality condition, the planner would have to use an inequality join, which can be very expensive. And even in the equality condition case, the resulting hash join may end up with long chains of identical `ticker` keys that will all match and need pruning. #### Why AsOf? {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::why-asof} If SQL can compute AsOf joins already, why do we need a new join type? There are two big reasons: expressibility and performance. The windowing alternative is more verbose and harder to understand than the AsOf syntax, so making it easier to say what you are doing helps others (or even you!) understand what is happening. The syntax also makes it easier for DuckDB to understand what you want and produce your results faster. The window and inequality join version loses the valuable information that the intervals do not overlap. It also prevents the query optimizer from moving the join because SQL insists that windowing happens *after* joins. By treating the operation *as a join* with *known data constraints*, DuckDB can move the join for performance and use a tailored join algorithm. The algorithm we use is to sort the right side table and then do a kind of merge join with the left side values. But unlike a standard merge join, AsOf can stop searching when it finds the first match because there is at most one match. ##### State Tables {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::state-tables} You may be wondering why the Common Table Expression in the `WITH` clause was called *state*. This is because the `prices` table is really an example of what in temporal analytics is called an *event table* . The rows of an event table contain timestamps and what happened at that time (i.e., events). The events in the `prices` table are changes to the price of a stock. Another common example of an event table is a structured log file: Each row of the log records when something “happened” – usually a change to a part of the system. Event tables are difficult to work with because each fact only has the start time. In order to know whether the fact is still true (or true at a specific time) you need the end time as well. A table with both the start and end time is called a *state table*. Converting event tables to state tables is a common temporal data preparation task, and the windowing CTE above shows how to do it in general using SQL. ##### Sentinel Values {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::sentinel-values} One limitation of the windowing approach is that the ordering type needs to have sentinel value that can be used if it does not support `infinity`, either an unused value or `NULL`. Both of these choices are potentially problematic. In the first case, it may not be easy to determine an upper sentinel value (suppose the ordering was a string column?) In the second case, you would need to write the condition as `h.when < s.end OR s.end IS NULL` and using an `OR` like this in a join condition makes comparisons slow and hard to optimize. Moreover, if the ordering column is already using `NULL` to indicate missing values, this option is not available. For most state tables, there are suitable choices (e.g., large dates) but one of the advantages of AsOf is that it can avoid having to design a state table if it is not needed for the analytic task. ##### Event Table Variants {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::event-table-variants} So far we have been using a standard type of event table where the timestamps are assumed to be the start of the state transitions. But AsOf can now use any inequality, which allows it to handle other types of event tables. To explore this, let's use two very simple tables with no equality conditions. The build side will just have four integer “timestamps” with alphabetic values: | Time | Value | | ---: | ----: | | 1 | a | | 2 | b | | 3 | c | | 4 | d | The probe table will just be the time values plus the midpoints, and we can make a table showing what value each probe time matches for greater than or equal to: | Probe | >= | | ----: | --- | | 0.5 | | | 1.0 | a | | 1.5 | a | | 2.0 | b | | 2.5 | b | | 3.0 | c | | 3.5 | c | | 4.0 | d | | 4.5 | d | This shows us that the interval a probe value matches is in the half-open interval `[Tn, Tn+1)`. Now let's see what happens if use strictly greater than as the inequality: | Probe | > | | ----: | --- | | 0.5 | | | 1.0 | | | 1.5 | a | | 2.0 | a | | 2.5 | b | | 3.0 | b | | 3.5 | c | | 4.0 | c | | 4.5 | d | Now we can see that the interval a probe value matches is in the half-open interval `(Tn, Tn+1]`. The only difference is that the interval is closed at the end instead of the beginning. This means that for this inequality type, the time is not part of the interval. What if the inequality goes in the other direction, say less than or equal to? | Probe | <= | | ----: | --- | | 0.5 | a | | 1.0 | a | | 1.5 | b | | 2.0 | b | | 2.5 | c | | 3.0 | c | | 3.5 | d | | 4.0 | d | | 4.5 | | Again, we have half-open intervals, but this time we are matching the _previous_ interval `(Tn-1, Tn]`. One way to interpret this is that the times in the build table are the _end_ of the interval, instead of the beginning. Also, unlike greater than or equal to, the interval is closed at the end instead of the beginning. Adding this to what we found for strictly greater than, we can interpret this as meaning that the lookup times are part of the interval when non-strict inequalities are used. We can check this by looking at the last inequality: strictly less than: | Probe | < | | ----: | --- | | 0.5 | a | | 1.0 | b | | 1.5 | b | | 2.0 | c | | 2.5 | c | | 3.0 | d | | 3.5 | d | | 4.0 | | | 4.5 | | In this case the matching intervals are `[Tn-1, Tn)`. This is a strict inequality, so the table time is not in the interval, and it is a less than, so the time is the end of the interval. To sum up, here is the full list: | Inequality | Interval | | ---------- | ---------- | | > | (Tn, Tn+1] | | >= | [Tn, Tn+1) | | <= | (Tn-1, Tn] | | < | [Tn-1, Tn) | We now have two natural interpretations of what the inequalities mean: * The greater (resp. less) than inequalities mean the time is the beginning (resp. end) of the interval. * The strict (resp. non-strict) inequalities mean the time is excluded from (resp. included in) the interval. So if we know whether the time marks the start or the end of the event, and whether the time is include or excluded, we can choose the appropriate AsOf inequality. ##### Usage {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::usage} So far we have been explicit about specifying the conditions for AsOf, but SQL also has a simplified join condition syntax for the common case where the column names are the same in both tables. This syntax uses the `USING` keyword to list the fields that should be compared for equality. AsOf also supports this syntax, but with two restrictions: * The last field is the inequality * The inequality is `>=` (the most common case) Our first query can then be written as: ```sql SELECT ticker, h.when, price * shares AS value FROM holdings h ASOF JOIN prices p USING(ticker, "when"); ``` Be aware that if you don't explicitly list the columns in the `SELECT`, the ordering field value will be the probe value, not the build value. For a natural join, this is not an issue because all the conditions are equalities, but for AsOf, one side has to be chosen. Since AsOf can be viewed as a lookup function, it is more natural to return the “function arguments” than the function internals. ##### Under the Hood {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::under-the-hood} What an AsOf Join is really doing is allowing you to treat an event table as a state table for join operations. By knowing the semantics of the join, it can avoid creating a full state table and be more efficient than a general inequality join. Let's start by looking at how the windowing version works. Remember that we used this query to convert the event table to a state table: ```sql WITH state AS ( SELECT ticker, price, "when", lead("when", 1, 'infinity') OVER (PARTITION BY ticker ORDER BY "when") AS end FROM prices ); ``` The state table CTE is created by hash partitioning the table on `ticker`, sorting on `when` and then computing another column that is just `when` shifted down by one. The join is then implemented with a hash join on `ticker` and two comparisons on `when`. If there was no `ticker` column (e.g., the prices were for a single item) then the join would be implemented using our inequality join operator, which would materialise and sort both sides because it doesn't know that the ranges are disjoint. The AsOf operator uses all three operator pipeline APIs to consolidate and collect rows. During the `sink` phase, AsOf hash partitions and sorts the right hand side to make a temporary state table. (In fact it uses the same code as Window, but without unnecessarily materialising the end column.) During the `operator` phase, it filters out (or returns) rows that cannot match because of `NULL` values in the predicate expressions, and then hash partitions and sorts the remaining rows into a cache. Finally, during the `source` phase, it matches hash partitions and then merge joins the sorted values within each hash partition. #### Benchmarks {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::benchmarks} Because AsOf joins can be implemented in various ways using standard SQL queries, benchmarking is really about comparing the various alternatives. One alternative is a debugging `PRAGMA` for AsOf called `debug_asof_iejoin`, which implements the join using Window and IEJoin. This allows us to easily toggle between the implementations and compare runtimes. Other alternatives combine equi-joins and window functions. The equi-join is used to implement the equality matching conditions, and the window is used to select the closest inequality. We will now look at two different windowing techniques and compare their performance. If you wish to skip this section, the bottom line is that while they are sometimes a bit faster, the AsOf join has the most consistent behavior of all the algorithms. ##### Window as State Table {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::window-as-state-table} The first benchmark compares a hash join with a state table. It probes a 5M row table of values built from 100K timestamps and 50 partitioning keys using a self-join where only 50% of the keys are present and the timestamps have been shifted to be halfway between the originals: ```sql CREATE OR REPLACE TABLE build AS ( SELECT k, '2001-01-01 00:00:00'::TIMESTAMP + INTERVAL (v) MINUTE AS t, v FROM range(0, 100_000) vals(v), range(0, 50) keys(k) ); CREATE OR REPLACE TABLE probe AS ( SELECT k * 2 AS k, t - INTERVAL (30) SECOND AS t FROM build ); ``` The `build` table looks like this: | k | t | v | | --- | ------------------- | --- | | 0 | 2001-01-01 00:00:00 | 0 | | 0 | 2001-01-01 00:01:00 | 1 | | 0 | 2001-01-01 00:02:00 | 2 | | 0 | 2001-01-01 00:03:00 | 3 | | ... | ... | ... | and the probe table looks like this (with only even values for k): | k | t | | --- | ------------------- | | 0 | 2000-12-31 23:59:30 | | 0 | 2001-01-01 00:00:30 | | 0 | 2001-01-01 00:01:30 | | 0 | 2001-01-01 00:02:30 | | 0 | 2001-01-01 00:03:30 | | ... | ... | The benchmark just does the join and sums up the `v` column: ```sql SELECT sum(v) FROM probe ASOF JOIN build USING(k, t); ``` The debugging `PRAGMA` does not allow us to use a hash join, but we can create the state table in a CTE again and use an inner join: ```sql -- Hash Join implementation WITH state AS ( SELECT k, t AS begin, v, lead(t, 1, 'infinity'::TIMESTAMP) OVER (PARTITION BY k ORDER BY t) AS end FROM build ) SELECT sum(v) FROM probe p INNER JOIN state s ON p.t >= s.begin AND p.t < s.end AND p.k = s.k; ``` This works because the planner assumes that equality conditions are more selective than inequalities and generates a hash join with a filter. Running the benchmark, we get results like this: | Algorithm | Median of 5 | | :--------- | ----------: | | AsOf | 0.425 s | | IEJoin | 3.522 s | | State Join | 192.460 s | The runtime improvement of AsOf over IEJoin here is about 9×. The horrible performance of the Hash Join is caused by the long (100K) bucket chains in the hash table. The second benchmark tests the case where the probe side is about 10× smaller than the build side: ```sql CREATE OR REPLACE TABLE probe AS SELECT k, '2021-01-01T00:00:00'::TIMESTAMP + INTERVAL (random() * 60 * 60 * 24 * 365) SECOND AS t, FROM range(0, 100_000) tbl(k); CREATE OR REPLACE TABLE build AS SELECT r % 100_000 AS k, '2021-01-01T00:00:00'::TIMESTAMP + INTERVAL (random() * 60 * 60 * 24 * 365) SECOND AS t, (random() * 100_000)::INTEGER AS v FROM range(0, 1_000_000) tbl(r); SELECT sum(v) FROM probe p ASOF JOIN build b ON p.k = b.k AND p.t >= b.t -- Hash Join Version WITH state AS ( SELECT k, t AS begin, v, lead(t, 1, 'infinity'::TIMESTAMP) OVER (PARTITION BY k ORDER BY t) AS end FROM build ) SELECT sum(v) FROM probe p INNER JOIN state s ON p.t >= s.begin AND p.t < s.end AND p.k = s.k; ``` | Algorithm | Median of 5 runs | | :--------- | ---------------: | | State Join | 0.065 s | | AsOf | 0.077 s | | IEJoin | 49.508 s | Now the runtime improvement of AsOf over IEJoin is huge (~500×) because it can leverage the partitioning to eliminate almost all of the equality mismatches. The Hash Join implementation does much better here because the optimizer notices that the probe side is smaller and builds the hash table on the “probe” table. Also, the probe values here are unique, so the hash table chains are minimal. ##### Window with Ranking {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::window-with-ranking} Another way to use the window operator is to: * Join the tables on the equality predicates * Filter to pairs where the build time is before the probe time * Partition the result on both the equality keys _and_ the probe timestamp * Sort the partitions on the build timestamp _descending_ * Filter out all value except rank 1 (i.e., the largest build time <= the probe time) The query looks like: ```sql WITH win AS ( SELECT p.k, p.t, v, rank() OVER (PARTITION BY p.k, p.t ORDER BY b.t DESC) AS r FROM probe p INNER JOIN build b ON p.k = b.k AND p.t >= b.t QUALIFY r = 1 ) SELECT k, t, v FROM win; ``` The advantage of this windowing query is that it does not require sentinel values, so it can work with any data type. The disadvantage is that it creates many more partitions because it includes both timestamps, which requires more complex sorting. Moreover, because it applies the window _after_ the join, it can produce huge intermediates that can result in external sorting and expensive out-of-memory operations. For this benchmark, we will be using three build tables, and two probe tables, all containing 10K integer equality keys. The probe tables have either 1 or 15 timestamps per key: ```sql CREATE OR REPLACE TABLE probe15 AS SELECT k, t FROM range(10_000) cs(k), range('2022-01-01'::TIMESTAMP, '2023-01-01'::TIMESTAMP, INTERVAL 26 DAY) ts(t); CREATE OR REPLACE TABLE probe1 AS SELECT k, '2022-01-01'::TIMESTAMP t FROM range(10_000) cs(k); ``` The build tables are much larger and have approximately 10/100/1000× the number of entries as the 15 element tables: ```sql -- 10:1 CREATE OR REPLACE TABLE build10 AS SELECT k, t, (random() * 1000)::DECIMAL(7, 2) AS v FROM range(10_000) ks(k), range('2022-01-01'::TIMESTAMP, '2023-01-01'::TIMESTAMP, INTERVAL 59 HOUR) ts(t); -- 100:1 CREATE OR REPLACE TABLE build100 AS SELECT k, t, (random() * 1000)::DECIMAL(7, 2) AS v FROM range(10_000) ks(k), range('2022-01-01'::TIMESTAMP, '2023-01-01'::TIMESTAMP, INTERVAL 350 MINUTE) ts(t); -- 1000:1 CREATE OR REPLACE TABLE build1000 AS SELECT k, t, (random() * 1000)::DECIMAL(7, 2) AS v FROM range(10_000) ks(k), range('2022-01-01'::TIMESTAMP, '2023-01-01'::TIMESTAMP, INTERVAL 35 MINUTE) ts(t); ``` The AsOf join queries are: ```sql -- AsOf/IEJoin SELECT p.k, p.t, v FROM probe p ASOF JOIN build b ON p.k = b.k AND p.t >= b.t ORDER BY 1, 2; -- Rank WITH win AS ( SELECT p.k, p.t, v, rank() OVER (PARTITION BY p.k, p.t ORDER BY b.t DESC) AS r FROM probe p INNER JOIN build b ON p.k = b.k AND p.t >= b.t QUALIFY r = 1 ) SELECT k, t, v FROM win ORDER BY 1, 2; ``` The results are shown here:  (Median of 5 except for Rank/15/1000). * For all ratios with 15 probes, AsOf is the most performant. * For small ratios with 15 probes, Rank beats IEJoin (both with windowing), but by 100:1 it is starting to explode. * For single element probes, Rank is most effective, but even there, its edge over AsOf is only about 50% at scale. This shows that AsOf could be possibly be improved upon, but predicting where that happens would be tricky, and getting it wrong would have enormous costs. #### Future Work {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::future-work} DuckDB can now execute AsOf joins for all inequality types with reasonable performance. In some cases, the performance gain is several orders of magnitude over the standard SQL versions – even with our fast inequality join operator. While the current AsOf operator is completely general, there are a couple of planning optimisations that could be applied here. * When there are selective equality conditions, it is likely that a hash join with filtering against a materialised state table would be significantly faster. If we can detect this and suitable sentinel values are available, the planner could choose to use a hash join instead of the default AsOf implementation. * There are also use cases where the probe table is much smaller than the build table, along with equality conditions, and performing a hash join against the *probe* table could yield significant performance improvements. Nevertheless, remember that one of the advantages of SQL is that it is a declarative language: You specify *what* you want and leave it up to the database to figure out *how*. Now that we have defined the semantics of the AsOf join, you the user can write queries saying this is *what* you want – and we are free to keep improving the *how*! #### Happy Joining! {#blog:2023-09-15-asof-joins-fuzzy-temporal-lookups::happy-joining} One of the most interesting parts of working on DuckDB is that it stretches the traditional SQL model of unordered data. DuckDB makes it easy to query *ordered* data sets such as data frames and Parquet files, and when you have data like that, you expect to be able to do ordered analysis! Implementing Fast Sorting, Fast Windowing and Fast AsOf joins is how we are making this expectation a reality. ## Announcing DuckDB 0.9.0 {#blog:2023-09-26-announcing-duckdb-090} **Publication date:** 2023-09-26 **Authors:** Mark Raasveldt, Hannes Mühleisen  The DuckDB team is happy to announce the latest DuckDB release (0.9.0). This release is named Undulata after the [Yellow-billed duck](https://en.wikipedia.org/wiki/Yellow-billed_duck) native to Africa. To install the new version, please visit the [installation guide](https://duckdb.org/install/). The full release notes can be found on [GitHub](https://github.com/duckdb/duckdb/releases/tag/v0.9.0). #### What's New in 0.9.0 {#blog:2023-09-26-announcing-duckdb-090::whats-new-in-090} There have been too many changes to discuss them each in detail, but we would like to highlight several particularly exciting features! * Out-of-Core Hash Aggregate * Storage Improvements * Index Improvements * DuckDB-Wasm Extensions * Extension Auto-Loading * Improved AWS Support * Iceberg Support * Azure Support * PySpark-Compatible API Below is a summary of those new features with examples, starting with a change in our SQL dialect that is designed to produce more intuitive results by default. #### Breaking SQL Changes {#blog:2023-09-26-announcing-duckdb-090::breaking-sql-changes} [**Struct Auto-Casting**](https://github.com/duckdb/duckdb/pull/8942). Previously the names of struct entries were ignored when determining auto-casting rules. As a result, struct field names could be silently renamed. Starting with this release, this will result in an error instead. ```sql CREATE TABLE structs (s STRUCT(i INTEGER)); INSERT INTO structs VALUES ({'k': 42}); ``` ```console Mismatch Type Error: Type STRUCT(k INTEGER) does not match with STRUCT(i INTEGER). Cannot cast STRUCTs with different names ``` Unnamed structs constructed using the `ROW` function can still be inserted into struct fields. ```sql INSERT INTO structs VALUES (ROW(42)); ``` #### Core System Improvements {#blog:2023-09-26-announcing-duckdb-090::core-system-improvements} **[Out-of-Core Hash Aggregates](https://github.com/duckdb/duckdb/pull/7931)** and **[Hash Aggregate Performance Improvements.](https://github.com/duckdb/duckdb/pull/8475)** When working with large data sets, memory management is always a potential pain point. By using a streaming execution engine and buffer manager, DuckDB supports many operations on larger than memory data sets. DuckDB also aims to support queries where *intermediate* results do not fit into memory by using disk-spilling techniques. In this release, support for disk-spilling techniques is further extended through the support for out-of-core hash aggregates. Now, hash tables constructed during `GROUP BY` queries or `DISTINCT` operations that do not fit in memory due to a large number of unique groups will spill data to disk instead of throwing an out-of-memory exception. Due to the clever use of radix partitioning, performance degradation is gradual, and performance cliffs are avoided. Only the subset of the table that does not fit into memory will be spilled to disk. The performance of our hash aggregate has also improved in general, especially when there are many groups. For example, we compute the number of unique rows in a data set with 30 million rows and 15 columns by using the following query: ```sql SELECT count(*) FROM (SELECT DISTINCT * FROM tbl); ``` If we keep all the data in memory, the query should use around 6 GB. However, we can still complete the query if less memory is available. In the table below, we can see how the runtime is affected by lowering the memory limit: | memory limit | v0.8.1 | v0.9.0 | | -----------: | -----: | -----: | | 10 GB | 8.52 s | 2.91 s | | 9 GB | 8.52 s | 3.45 s | | 8 GB | 8.52 s | 3.45 s | | 7 GB | 8.52 s | 3.47 s | | 6 GB | OOM | 3.41 s | | 5 GB | OOM | 3.67 s | | 4 GB | OOM | 3.87 s | | 3 GB | OOM | 4.20 s | | 2 GB | OOM | 4.39 s | | 1 GB | OOM | 4.91 s | **[Compressed Materialization.](https://github.com/duckdb/duckdb/pull/7644)** DuckDB's streaming execution engine has a low memory footprint, but more memory is required for operations such as grouped aggregation. The memory footprint of these operations can be reduced by compression. DuckDB already uses [many compression techniques in its storage format](#blog:2022-10-28-lightweight-compression), but many of these techniques are too costly to use during query execution. However, certain lightweight compression techniques are so cheap that the benefit of the reducing memory footprint outweight the cost of (de)compression. In this release, we add support for compression of strings and integer types right before data goes into the grouped aggregation and sorting operators. By using statistics, both types are compressed to the smallest possible integer type. For example, if we have the following table: ```text ┌───────┬─────────┐ │ id │ name │ │ int32 │ varchar │ ├───────┼─────────┤ │ 300 │ alice │ │ 301 │ bob │ │ 302 │ eve │ │ 303 │ mallory │ │ 304 │ trent │ └───────┴─────────┘ ``` The `id` column uses a 32-bit integer. From our statistics we know that the minimum value is 300, and the maximum value is 304. We can subtract 300 and cast to an 8-bit integer instead, reducing the width from 4 bytes down to 1. The `name` column uses our internal string type, which is 16 bytes wide. However, our statistics tell us that the longest string here is only 7 bytes. We can fit this into a 64-bit integer like so: ```text alice -> alice005 bob -> bob00003 eve -> eve00003 mallory -> mallory7 trent -> trent005 ``` This reduces the width from 16 bytes down to 8. To support sorting of compressed strings, we flip the bytes on big-endian machines so that our comparison operators are still correct: ```text alice005 -> 500ecila bob00003 -> 30000bob eve00003 -> 30000eve mallory7 -> 7yrollam trent005 -> 500tnert ``` By reducing the size of query intermediates, we can prevent/reduce spilling data to disk, reducing the need for costly I/O operations, thereby improving query performance. **Window Function Performance Improvements ([#7831](https://github.com/duckdb/duckdb/pull/7831), [#7996](https://github.com/duckdb/duckdb/pull/7996), [#8050](https://github.com/duckdb/duckdb/pull/8050), [#8491](https://github.com/duckdb/duckdb/pull/8491)).** This release features many improvements to the performance of Window functions due to improved vectorization of the code, more re-use of partial aggregates and improved parallelism through work stealing of tasks. As a result, performance of [Window functions has improved significantly, particularly in scenarios where there are no or few partitions](https://github.com/duckdb/duckdb/issues/7809#issuecomment-1679387022). ```sql SELECT sum(driver_pay) OVER ( ORDER BY dropoff_datetime ASC RANGE BETWEEN INTERVAL 3 DAYS PRECEDING AND INTERVAL 0 DAYS FOLLOWING ) FROM tripdata; ``` | Version | Run time | | ------: | -------: | | v0.8.0 | 33.8 s | | v0.9.0 | 3.8 s | #### Storage Improvements {#blog:2023-09-26-announcing-duckdb-090::storage-improvements} [**Vacuuming of Deleted Row Groups**](https://github.com/duckdb/duckdb/pull/7794). Starting with this release, when deleting data using `DELETE` statements, entire row groups that are deleted will be automatically cleaned up. Support is also added to [truncate the database file on checkpoint](https://github.com/duckdb/duckdb/pull/7824) which allows the database file to be reduced in size after data is deleted. Note that this only occurs if the deleted row groups are located at the end of the file. The system does not yet move around data in order to reduce the size of the file on disk. Instead, free blocks earlier on in the file are re-used to store later data. **Index Storage Improvements ([#7930](https://github.com/duckdb/duckdb/pull/7930), [#8112](https://github.com/duckdb/duckdb/pull/8112), [#8437](https://github.com/duckdb/duckdb/pull/8437), [#8703](https://github.com/duckdb/duckdb/pull/8703))**. Many improvements have been made to both the in-memory footprint, and the on-disk footprint of ART indexes. In particular for indexes created to maintain `PRIMARY KEY`, `UNIQUE` or `FOREIGN KEY` constraints the storage and in-memory footprint is drastically reduced. ```sql CREATE TABLE integers (i INTEGER PRIMARY KEY); INSERT INTO integers FROM range(10000000); ``` | Version | Size | | ------- | -----: | | v0.8.0 | 278 MB | | v0.9.0 | 78 MB | In addition, due to improvements in the manner in which indexes are stored on disk they can now be written to disk incrementally instead of always requiring a full rewrite. This allows for much quicker checkpointing for tables that have indexes. #### Extensions {#blog:2023-09-26-announcing-duckdb-090::extensions} [**Extension Auto-Loading**](https://github.com/duckdb/duckdb/pull/8732). Starting from this release, DuckDB supports automatically installing and loading of trusted extensions. As many workflows rely on core extensions that are not bundled, such as `httpfs`, many users found themselves having to remember to load the required extensions up front. With this change, the extensions will instead be automatically loaded (and optionally installed) when used in a query. For example, in Python the following code snippet now works without needing to explicitly load the `httpfs` or `json` extensions. ```python import duckdb duckdb.sql("FROM 'https://raw.githubusercontent.com/duckdb/duckdb/main/data/json/example_n.ndjson'") ``` The set of autoloadable extensions is limited to official extensions distributed by DuckLabs, and can be [found here](https://github.com/duckdb/duckdb/blob/8feb03d274892db0e7757cd62c145b18dfa930ec/scripts/generate_extensions_function.py#L298). The behavior can also be disabled using the `autoinstall_known_extensions` and `autoload_known_extensions` settings, or through the more general `enable_external_access` setting. See the [configuration options](https://duckdb.org/docs/1.5/configuration/overview). [**DuckDB-Wasm Extensions**](https://github.com/duckdb/duckdb-wasm/pull/1403). This release adds support for loadable extensions to DuckDB-Wasm. Previously, any extensions that you wanted to use with the Wasm client had to be baked in. With this release, extensions can be loaded dynamically instead. When an extension is loaded, the Wasm bundle is downloaded and the functionality of the extension is enabled. Give it a try in our [Wasm shell](https://shell.duckdb.org). ```sql LOAD inet; SELECT '127.0.0.1'::INET; ``` [**AWS Extension**](https://github.com/duckdb/duckdb-aws). This release marks the launch of the DuckDB AWS extension. This extension contains AWS related features that rely on the AWS SDK. Currently, the extension contains one function, `LOAD_AWS_CREDENTIALS`, which uses the AWS [Credential Provider Chain](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html#credentialProviderChain) to automatically fetch and set credentials: ```sql CALL load_aws_credentials(); SELECT * FROM 's3://some-bucket/that/requires/authentication.parquet'; ``` [See the documentation for more information](https://duckdb.org/docs/1.5/core_extensions/aws). [**Experimental Iceberg Extension**](https://github.com/duckdb/duckdb-iceberg). This release marks the launch of the DuckDB Iceberg extension. This extension adds support for reading tables stored in the [Iceberg format](https://iceberg.apache.org). ```sql SELECT count(*) FROM iceberg_scan('data/iceberg/lineitem_iceberg', allow_moved_paths = true); ``` [See the documentation for more information](https://duckdb.org/docs/1.5/core_extensions/iceberg/overview). [**Experimental Azure Extension**](https://github.com/duckdb/duckdb-azure). This release marks the launch of the DuckDB Azure extension. This extension allows for DuckDB to natively read data stored on Azure, in a similar manner to how it can read data stored on S3. ```sql SET azure_storage_connection_string = '
First we will create an example data table. `col1` is unique on each row, but the other columns are various groupings of the rows.
```sql CREATE OR REPLACE TABLE example AS SELECT x % 11 AS col1, x % 5 AS col2, x % 2 AS col3, 1 AS col4 FROM range(1, 11) t(x); FROM example; ```
Remember that there are pointers within the fixed-size rows pointing to variable-size data. The MetaData describes which fixed-size rows point to which Var Page and the last known address of the Var Page. For example, MetaData 1 describes 5 rows stored in Row Page 1 at offset 0, with variable-size data stored in Var Page 1, which had an address of `0x42`. Let’s say the buffer manager decides to offload Var Page 1. When we request Var Page 1 again, it’s loaded into address `0x500`. The pointers within those 5 rows are now invalid. For example, one of the rows contains the pointer `0x48`, which means that it is stored at offset `0x48 - 0x42 = 6` in Var Page 1. We can recompute the pointer by adding the offset to the new address of the page: `0x500 + 6 = 0x506`. Pointer recomputation is done for rows with their strings stored on the same Row and Var Page, so we create a new MetaData every time a Row Page or Var Page is full. The advantage of pointer recomputation over (de-)serialization is that it can be done lazily. We can check whether the Var Page was offloaded by comparing the pointer in the MetaData with the current pointer to the page. We don’t have to recompute the pointers if they are the same. #### External Aggregation {#blog:2024-03-29-external-aggregation::external-aggregation} Now that we’ve figured out how to deal with temporary data, it’s finally time to talk about hash aggregation. The first big challenge is to perform the aggregation in parallel. DuckDB uses [Morsel-Driven Parallelism](https://db.in.tum.de/~leis/papers/morsels.pdf) to parallelize query execution, which essentially means that query operators, such as aggregation, must be parallelism-aware. This differs from [plan-driven parallelism](https://dl.acm.org/doi/pdf/10.1145/93605.98720), keeping operators unaware of parallelism. To briefly summarize [our first blog post on aggregation](#blog:2022-03-07-aggregate-hashtable): In DuckDB, all active threads have their own thread-local hash table, which they sink input data into. This will keep threads busy until all input data has been read. Multiple threads will likely have the _exact same group_ in their hash table. Therefore, the thread-local hash tables must be combined to complete the grouped aggregation. This can be done in parallel by partitioning the hash tables and assigning each thread to combine the data from each partition. For the most part, we still use this same approach. You’ll see this in the image below, which illustrates our new implementation.
We call the first phase _Thread-Local Pre-Aggregation_. The input data are _morsels_, chunks of around 100,000 rows. These are assigned to active threads, which sink them into their thread-local hash table until all input data has been read. We use _linear probing_ to resolve collisions and _salt_ to reduce the overhead of dealing with said collisions. This is explained in [our first blog post on aggregation](#blog:2022-03-07-aggregate-hashtable), so I won’t repeat it here. Now that we’ve explained what _hasn’t_ changed, we can talk about what _has_ changed. The first difference compared to last time is the way that we partition. Before, if we had, for example, 32 threads, each thread would create 32 hash tables, one for each partition. This totals a whopping 1024 hash tables, which did not scale well when even more threads were active. Now, each thread has one hash table, _but the data within each hash table is partitioned_. The data is also stored on the specialized page layout we presented earlier so that it can easily be offloaded to storage. The second difference is that the hash tables are not _resized_ during Thread-Local Pre-Aggregation. We keep the hash tables’ size small, reducing the amount of cache misses during this phase. This means that the hash table will be full at some point. When it’s full, we reset it and start over. We can do this because we’ll finish the aggregation later in the second phase. When we reset the hash table, we “unpin” the pages that store the actual data, which tells our buffer manager it can write them to storage when it needs to free up memory. Together, these two changes result in a low memory requirement during the first phase. Each thread only needs to keep a small hash table in memory. We may collect a lot of data by filling up the hash table many times, but the buffer manager can offload almost all of it if needed. For the second phase, _Partition-Wise Aggregation_, the thread-local partitioned data is exchanged, and each thread combines the data of a single partition into a hash table. This phase is mostly the same as before, except that we now sometimes create many more partitions than threads. Why? The hash table for one partition might fit in memory, but 8 threads could be combining a partition simultaneously, and we might not be able to fit 8 partitions in memory. The easy solution to this problem is to _over-partition_. If we make more partitions than threads, for example, 32 partitions, the size of the partitions will be smaller, and the 8 threads will combine only 8 out of the 32 partitions simultaneously, which won’t require nearly as much memory. #### Experiments {#blog:2024-03-29-external-aggregation::experiments} Aggregations that result in only a few unique groups can easily fit in memory. To evaluate our external hash aggregation implementation, we need aggregations that have many unique groups. For this purpose, we will use the [H2O.ai database-like ops benchmark](https://duckdblabs.github.io/db-benchmark/), which [we've resurrected](#blog:2023-04-14-h2oai), and [now maintain](#blog:2023-11-03-db-benchmark-update). Specifically, we will use the `G1_1e9_2e0_0_0.csv.zst` file, which is 50 GB uncompressed. The source code for the H2O.ai benchmark can be found on [GitHub](https://github.com/duckdblabs/db-benchmark). You can download the file yourself from[`pop.csv`](https://duckdb.org/data/cli/pop.csv), the population of each of the top-10 most populous cities.
```csv city,province,population Amsterdam,North Holland,905234 Rotterdam,South Holland,656050 The Hague,South Holland,552995 Utrecht,Utrecht,361924 Eindhoven,North Brabant,238478 Groningen,Groningen,234649 Tilburg,North Brabant,224702 Almere,Flevoland,218096 Breda,North Brabant,184716 Nijmegen,Gelderland,179073 ```[`area.csv`](https://duckdb.org/data/cli/area.csv), the area of each of the top-10 most populous cities.
```csv city,area Amsterdam,219.32 Rotterdam,324.14 The Hague,98.13 Utrecht,99.21 Eindhoven,88.92 Groningen,197.96 Tilburg,118.13 Almere,248.77 Breda,128.68 Nijmegen,57.63 ```[`cities-airports.csv`](https://duckdb.org/data/cli/cities-airports.csv), the [IATA codes](https://en.wikipedia.org/wiki/IATA_airport_code) of civilian airports serving given cities.
```csv city,IATA Amsterdam,AMS Haarlemmermeer,AMS Eindhoven,EIN Groningen,GRQ Eelde,GRQ Maastricht,MST Beek,MST Rotterdam,RTM The Hague,RTM ```[`airport-names.csv`](https://duckdb.org/data/cli/airport-names.csv), the airport names belonging to given IATA codes.
```csv IATA,airport name AMS,Amsterdam Airport Schiphol EIN,Eindhoven Airport GRQ,Groningen Airport Eelde MST,Maastricht Aachen Airport RTM,Rotterdam The Hague Airport ```4. [DuckLabs](https://ducklabs.com) sponsored the [Hack4Her event](https://hack4her.github.io/), a female-focused student hackathon in the Netherlands. During the DuckDB Challenge of the event, teams built a community-driven app providing safe walking routes in Amsterdam using DuckDB and its [geospatial library](https://duckdb.org/docs/1.5/core_extensions/spatial/overview). 
#### Looking Ahead {#blog:2024-06-22-github-20k-stars::looking-ahead} There are several interesting events lined up for the summer. First, two books about DuckDB are expected to be released: * [**Getting Started with DuckDB**](https://www.packtpub.com/product/getting-started-with-duckdb/9781803241005), authored by Simon Aubury and Ned Letcher, and published by Packt Publishing * [**DuckDB in Action**](https://www.manning.com/books/duckdb-in-action), authored by Mark Needham, Michael Hunger and Michael Simons, and published by Manning Publications Second, we are holding our next user community meeting, [DuckCon #5](https://duckdb.org/events/2024/08/15/duckcon5/) in Seattle on August 15 with the regular “State of the Duck” update as well as three regular talks and several lightning talks.  Third, we will improve DuckDB's extension ecosystem and streamline the publication process for community extensions. Finally, we have a series of blog posts lined up for publication. These will discuss DuckDB's performance over time, the results of the user survey we conducted during the spring, DuckDB's storage format, and many more. Stay tuned! We are looking forward to the next part of our journey and, of course, the next 10 000 stars on GitHub. ## Benchmarking Ourselves over Time at DuckDB {#blog:2024-06-26-benchmarks-over-time} **Publication date:** 2024-06-26 **Author:** Alex Monahan **TL;DR:** In the last 3 years, DuckDB has become 3-25× faster and can analyze ~10× larger datasets all on the same hardware. A big part of DuckDB's focus is on the developer experience of working with data. However, performance is an important consideration when investigating data management systems. Fairly comparing data processing systems using benchmarks is [very difficult](https://mytherin.github.io/papers/2018-dbtest.pdf). Whoever creates the benchmark is likely to know one system better than the rest, influencing benchmark selection, how much time is spent tuning parameters, and more. Instead, this post focuses on benchmarking *our own* performance over time. This approach avoids many comparison pitfalls, and also provides several valuable data points to consider when selecting a system. * **How fast is it improving?** Learning a new tool is an investment. Picking a vibrant, rapidly improving database ensures your choice pays dividends for years to come. Plus, if you haven't experimented with a tool in a while, you can see how much faster it has become since you last checked! * **What is it especially good at?** The choice of benchmark is an indicator of what types of workloads a tool is useful for. The higher the variety of analyses in the benchmark, the more broadly useful the tool can be. * **What scale of data can it handle?** Many benchmarks are deliberately smaller than typical workloads. This allows the benchmark to complete in a reasonable amount of time when run with many configurations. However, an important question to answer when selecting a system is whether the size of your data can be handled within the size of your compute resources. There are some limitations when looking at the performance of a system over time. If a feature is brand new, there is no prior performance to compare to! As a result, this post focuses on fundamental workloads rather than DuckDB's ever-increasing set of integrations with different lakehouse data formats, cloud services, and more. The code used to run the benchmark also avoids many of DuckDB's [Friendlier SQL](https://duckdb.org/docs/1.5/sql/dialect/friendly_sql) additions, as those have also been added more recently. (When writing these queries, it felt like going back in time!) #### Benchmark Design Summary {#blog:2024-06-26-benchmarks-over-time::benchmark-design-summary} This post measures DuckDB's performance over time using the [H2O.ai benchmark](https://duckdblabs.github.io/db-benchmark/), plus some new benchmarks added for importing, exporting, and using window functions. Please see our previous [blog](#blog:2023-04-14-h2oai) [posts](#blog:2023-11-03-db-benchmark-update) for details on why we believe the H2O.ai benchmark is a good approach! The full details of the benchmark design are in the appendix. * H2O.ai, plus import/export and window function tests * Python instead of R * 5 GB scale for everything, plus 50 GB scale for group bys and joins * Median of 3 runs * Using a MacBook Pro M1 with 16 GB RAM * DuckDB versions 0.2.7 through 1.0.0 * Nearly 3 years, from 2021-06-14 to 2024-06-03 * Default settings * Pandas pre-version 0.5.1, Apache Arrow 0.5.1+ #### Overall Benchmark Results {#blog:2024-06-26-benchmarks-over-time::overall-benchmark-results} The latest DuckDB can complete one run of the full benchmark suite in under 35 seconds, while version 0.2.7 required nearly 500 seconds for the same task in June 2021. **That is 14 times faster, in only 3 years!** ##### Performance over Time {#blog:2024-06-26-benchmarks-over-time::performance-over-time} > **Note.** These graphs are interactive, thanks to [Plotly.js](https://plotly.com/javascript/)! > Feel free to filter the various series (single click to hide, double click to show only that series) and click-and-drag to zoom in. > Individual benchmark results are visible on hover. The above plot shows the median runtime in seconds for all tests. Due to the variety of uses for window functions, and their relative algorithmic complexity, the 16 window function tests require the most time of any category. This plot normalizes performance to the latest version of DuckDB to show relative improvements over time. If you look at the point in time when you most recently measured DuckDB performance, that number will show you how many times faster DuckDB is now! A portion of the overall improvement is DuckDB's addition of multi-threading, which became the default in November 2021 with version 0.3.1. DuckDB also moved to a push-based execution model in that version for additional gains. Parallel data loading boosted performance in December 2022 with version 0.6.1, as did improvements to the core `JOIN` algorithm. We will explore other improvements in detail later in the post. However, we see that all aspects of the system have seen improvements, not just raw query performance! DuckDB focuses on the entire data analysis workflow, not just aggregate or join performance. CSV parsing has seen significant gains, import and export have improved significantly, and window functions have improved the most of all. What was the slight regression from December 2022 to June 2023? Window functions received additional capabilities and experienced a slight performance degradation in the process. However, from June 2023 onward we see substantial performance improvement across the board for window functions. If window functions are filtered out of the chart, we see a smoother trend. You may also notice that starting with version 0.9 in September 2023, the performance appears to plateau. What is happening here? First, don't forget to zoom in! Over the last year, DuckDB has still improved over 3×! More recently, the DuckLabs team focused on scalability by developing algorithms that support larger-than-memory calculations. We will see the fruits of those labors in the scale section later on! In addition, DuckDB focused exclusively on bug fixes in versions 0.10.1, 0.10.2, and 0.10.3 in preparation for an especially robust DuckDB 1.0. Now that those two major milestones (larger than memory calculations and DuckDB 1.0) have been accomplished, performance improvements will resume! It is worth noting that the boost from moving to multi-threading will only occur once, but there are still many opportunities moving forward. ##### Performance by Version {#blog:2024-06-26-benchmarks-over-time::performance-by-version} We can also recreate the overall plot by version rather than by time. This demonstrates that DuckDB has been doing more frequent releases recently. See [DuckDB's release calendar](https://duckdb.org/release_calendar) for the full version history. If you remember the version that you last tested, you can compare how much faster things are now with 1.0! #### Results by Workload {#blog:2024-06-26-benchmarks-over-time::results-by-workload} ##### CSV Reader {#blog:2024-06-26-benchmarks-over-time::csv-reader} DuckDB has invested substantially in building a [fast and robust CSV parser](#blog:2023-10-27-csv-sniffer). This is often the first task in a data analysis workload, and it tends to be undervalued and underbenchmarked. DuckDB has **improved CSV reader performance by nearly 3×**, while adding the ability to handle many more CSV dialects automatically. ##### Group By {#blog:2024-06-26-benchmarks-over-time::group-by} Group by or aggregation operations are critical steps in OLAP workloads, and have therefore received substantial focus in DuckDB, **improving over 12× in the last 3 years**. In November 2021, version 0.3.1 enabled multithreaded aggregation by default, providing a significant speedup. In December 2022, data loads into tables were parallelized with the release of version 0.6.1. This is another example of improving the entire data workflow, as this group by benchmark actually stressed the insertion performance substantially. Inserting the results was taking the majority of the time! Enums were also used in place of strings for categorical columns in version 0.6.1. This means that DuckDB was able to use integers rather than strings when operating on those columns, further boosting performance. Despite what appears at first glance to be a performance plateau, zooming in to 2023 and 2024 reveals a ~20% improvement. In addition, aggregations have received significant attention in the most recent versions to enable larger-than-memory aggregations. You can see that this was achieved while continuing to improve performance for the smaller-than-memory case. ##### Join {#blog:2024-06-26-benchmarks-over-time::join} Join operations are another area of focus for analytical databases, and DuckDB in particular. Join speeds have **improved by 4× in the last 3 years**! Version 0.6.1 in December 2022 introduced improvements to the out-of-core hash join that actually improved the smaller-than-memory case as well. Parallel data loading from 0.6.1 also helps in this benchmark as well, as some results are the same size as the input table. In recent versions, joins have also been upgraded to support larger-than-memory capabilities. This focus has also benefitted the smaller-than-memory case and has led to the improvements in 0.10, launched in February 2024. ##### Window Functions {#blog:2024-06-26-benchmarks-over-time::window-functions} Over the time horizon studied, window functions have **improved a dramatic 25×**! Window function performance was improved substantially with the 0.9.0 release in September 2023. [14 different performance optimizations contributed](https://github.com/duckdb/duckdb/issues/7809#issuecomment-1679387022). Aggregate computation was vectorized (with special focus on the [segment tree data structure](https://www.vldb.org/pvldb/vol8/p1058-leis.pdf)). Work stealing enabled multi-threaded processing and sorting was adapted to run in parallel. Care was also taken to pre-allocate memory in larger batches. DuckDB's window functions are also capable of processing larger-than-memory datasets. We leave benchmarking that feature for future work! ##### Export {#blog:2024-06-26-benchmarks-over-time::export} Often DuckDB is not the final step in a workflow, so export performance has an impact. Exports are **10× faster now!** Until recently, the DuckDB format was not backward compatible, so the recommended long term persistence format was Parquet. Parquet is also critical to interoperability with many other systems, especially data lakes. DuckDB works well as a workflow engine, so exporting to other in-memory formats is quite common as well. In the September 2022 release (version 0.5.1) we see significant improvements driven by switching from Pandas to Apache Arrow as the recommended in-memory export format. DuckDB's underlying data types share many similarities with Arrow, so data transfer is quite quick. Parquet export performance has improved by 4–5× over the course of the benchmark, with dramatic improvements in versions 0.8.1 (June 2023) and 0.10.2 (April 2024). Version 0.8.1 added [parallel Parquet writing](https://github.com/duckdb/duckdb/pull/7375) while continuing to preserve insertion order. The change driving the improvement in 0.10.2 was more subtle. When exporting strings with high cardinality, DuckDB decides whether or not to do dictionary compression depending on if it reduces file size. From 0.10.2 onward, the [compression ratio is tested after a sample of the values are inserted into the dictionary](https://github.com/duckdb/duckdb/pull/11461), rather than after all values are added. This prevents substantial unnecessary processing for high-cardinality columns where dictionary compression is unhelpful. ###### Exporting Apache Arrow vs. Pandas vs. Parquet {#blog:2024-06-26-benchmarks-over-time::exporting-apache-arrow-vs-pandas-vs-parquet} This plot shows the performance of all three export formats over the entire time horizon (rather than picking the winner between Pandas and Arrow). It allows us to see at what point Apache Arrow passes Pandas in performance. Pandas export performance has improved substantially over the course of the benchmark. However, Apache Arrow has proven to be the more efficient data format, so Arrow is now preferred for in-memory exports. Interestingly, DuckDB's Parquet export is now so efficient that it is faster to write a persistent Parquet file than it is to write to an in-memory Pandas dataframe! It is even competitive with Apache Arrow. ##### Scan Other Formats {#blog:2024-06-26-benchmarks-over-time::scan-other-formats} In some use cases, DuckDB does not need to store the raw data, but instead should simply read and analyze it. This allows DuckDB to fit seamlessly into other workflows. This benchmark measures how fast DuckDB can scan and aggregate various data formats. To enable comparisons over time, we switch from Pandas to Arrow at version 0.5.1 as mentioned. DuckDB is **over 8× faster in this workload**, and the absolute time required is very short. DuckDB is a great fit for this type of work! ###### Scanning Apache Arrow vs. Pandas vs. Parquet {#blog:2024-06-26-benchmarks-over-time::scanning-apache-arrow-vs-pandas-vs-parquet} Once again, we examine all three formats over the entire time horizon. When scanning data, Apache Arrow and Pandas are more comparable in performance. As a result, while Arrow is clearly preferable for exports, DuckDB will happily read Pandas with similar speed. However, in this case, the in-memory nature of both Arrow and Pandas allow them to perform 2–3× faster than Parquet. In absolute terms, the time required to complete this operation is a very small fraction of the benchmark, so other operations should be the deciding factor. #### Scale tests {#blog:2024-06-26-benchmarks-over-time::scale-tests} Analyzing larger-than-memory data is a superpower for DuckDB, allowing it to be used for substantially larger data analysis tasks than were previously possible. In version 0.9.0, launched in September 2023, [DuckDB's hash aggregate was enhanced to handle out-of-core (larger than memory) intermediates](https://github.com/duckdb/duckdb/pull/7931). The details of the algorithm, along with some benchmarks, are available in [this blog post](#blog:2024-03-29-external-aggregation). This allows for DuckDB to aggregate one billion rows of data (50 GB in size) on a MacBook Pro with only 16 GB of RAM, even when the number of unique groups in the group by is large. This represents at least a 10× improvement in aggregate processing scale over the course of the 3 years of the benchmark. DuckDB's hash join operator has supported larger-than-memory joins since version 0.6.1 in December 2022. However, the scale of this benchmark (coupled with the limited RAM of the benchmarking hardware), meant that this benchmark could still not complete successfully. In version 0.10.0, launched in February 2024, [DuckDB's memory management received a significant upgrade](https://github.com/duckdb/duckdb/pull/10147) to handle multiple concurrent operators all requiring significant memory. The [0.10.0 release blog post](#blog:2024-02-13-announcing-duckdb-0100::temporary-memory-manager) shares additional details about this feature. As a result, by version 0.10.0 DuckDB was able to handle calculations on data that is significantly larger than memory, even if the intermediate calculations are large in size. All operators are supported, including sorting, aggregating, joining, and windowing. Future work can further test the boundaries of what is possible with DuckDB's out-of-core support, including window functions and even larger data sizes. ##### Hardware Capabilities over Time {#blog:2024-06-26-benchmarks-over-time::hardware-capabilities-over-time} DuckDB's performance on the same hardware has improved dramatically, and at the same time, the capabilities of hardware are increasing rapidly as well. {: width="450" } {: width="450" } Source: [Our World in Data](https://ourworldindata.org/grapher/historical-cost-of-computer-memory-and-storage?yScale=linear&time=2021..latest&facet=metric&uniformYAxis=0) The price of RAM has declined by 2.2× and the price of SSD storage has decreased by 2.7× from 2021 to 2023 alone. Thanks to the combination of DuckDB enhancements and hardware prices, the scale of analysis possible on a single node has increased by substantially more than an order of magnitude in just 3 years! #### Analyzing the Results Yourself {#blog:2024-06-26-benchmarks-over-time::analyzing-the-results-yourself} A DuckDB 1.0 database containing the results of these benchmarks is available at
First we will create an example data table. We are a duck product distributor, and we are tracking our fowl finances.
```sql CREATE OR REPLACE TABLE business_metrics ( product_line VARCHAR, product VARCHAR, year INTEGER, quarter VARCHAR, revenue INTEGER, cost INTEGER ); INSERT INTO business_metrics VALUES ('Waterfowl watercraft', 'Duck boats', 2022, 'Q1', 100, 100), ('Waterfowl watercraft', 'Duck boats', 2022, 'Q2', 200, 100), ('Waterfowl watercraft', 'Duck boats', 2022, 'Q3', 300, 100), ('Waterfowl watercraft', 'Duck boats', 2022, 'Q4', 400, 100), ('Waterfowl watercraft', 'Duck boats', 2023, 'Q1', 500, 100), ('Waterfowl watercraft', 'Duck boats', 2023, 'Q2', 600, 100), ('Waterfowl watercraft', 'Duck boats', 2023, 'Q3', 700, 100), ('Waterfowl watercraft', 'Duck boats', 2023, 'Q4', 800, 100), ('Duck Duds', 'Duck suits', 2022, 'Q1', 10, 10), ('Duck Duds', 'Duck suits', 2022, 'Q2', 20, 10), ('Duck Duds', 'Duck suits', 2022, 'Q3', 30, 10), ('Duck Duds', 'Duck suits', 2022, 'Q4', 40, 10), ('Duck Duds', 'Duck suits', 2023, 'Q1', 50, 10), ('Duck Duds', 'Duck suits', 2023, 'Q2', 60, 10), ('Duck Duds', 'Duck suits', 2023, 'Q3', 70, 10), ('Duck Duds', 'Duck suits', 2023, 'Q4', 80, 10), ('Duck Duds', 'Duck neckties', 2022, 'Q1', 1, 1), ('Duck Duds', 'Duck neckties', 2022, 'Q2', 2, 1), ('Duck Duds', 'Duck neckties', 2022, 'Q3', 3, 1), ('Duck Duds', 'Duck neckties', 2022, 'Q4', 4, 1), ('Duck Duds', 'Duck neckties', 2023, 'Q1', 5, 1), ('Duck Duds', 'Duck neckties', 2023, 'Q2', 6, 1), ('Duck Duds', 'Duck neckties', 2023, 'Q3', 7, 1), ('Duck Duds', 'Duck neckties', 2023, 'Q4', 8, 1), ; FROM business_metrics; ```[`schema.sql`](https://github.com/pdet/taxi-benchmark/blob/0.1/sql/schema.sql).
```sql CREATE TABLE trips ( trip_id BIGINT, vendor_id VARCHAR, pickup_datetime TIMESTAMP, dropoff_datetime TIMESTAMP, store_and_fwd_flag VARCHAR, rate_code_id BIGINT, pickup_longitude DOUBLE, pickup_latitude DOUBLE, dropoff_longitude DOUBLE, dropoff_latitude DOUBLE, passenger_count BIGINT, trip_distance DOUBLE, fare_amount DOUBLE, extra DOUBLE, mta_tax DOUBLE, tip_amount DOUBLE, tolls_amount DOUBLE, ehail_fee DOUBLE, improvement_surcharge DOUBLE, total_amount DOUBLE, payment_type VARCHAR, trip_type VARCHAR, pickup VARCHAR, dropoff VARCHAR, cab_type VARCHAR, precipitation BIGINT, snow_depth BIGINT, snowfall BIGINT, max_temperature BIGINT, min_temperature BIGINT, average_wind_speed BIGINT, pickup_nyct2010_gid BIGINT, pickup_ctlabel VARCHAR, pickup_borocode BIGINT, pickup_boroname VARCHAR, pickup_ct2010 VARCHAR, pickup_boroct2010 BIGINT, pickup_cdeligibil VARCHAR, pickup_ntacode VARCHAR, pickup_ntaname VARCHAR, pickup_puma VARCHAR, dropoff_nyct2010_gid BIGINT, dropoff_ctlabel VARCHAR, dropoff_borocode BIGINT, dropoff_boroname VARCHAR, dropoff_ct2010 VARCHAR, dropoff_boroct2010 BIGINT, dropoff_cdeligibil VARCHAR, dropoff_ntacode VARCHAR, dropoff_ntaname VARCHAR, dropoff_puma VARCHAR); ```Vanilla query plan (operators and expected cardinalities)
```text ┌───────────────────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~5000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ HASH_GROUP_BY │ │ ──────────────────── │ │ ~5000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~10000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~10000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ RIGHT_DELIM_JOIN │ │ ──────────────────── │ │ ~10000000 Rows ├──────────────┐ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ SEQ_SCAN ││ HASH_JOIN │ │ ──────────────────── ││ ──────────────────── │ │ ~10000 Rows ││ ~10000000 Rows ├──────────────┐ └───────────────────────────┘└─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ PROJECTION ││ DUMMY_SCAN │ │ ──────────────────── ││ │ │ ~10000000 Rows ││ │ └─────────────┬─────────────┘└───────────────────────────┘ ┌─────────────┴─────────────┐ │ FILTER │ │ ──────────────────── │ │ ~10000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~50000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ WINDOW │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~50000000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ CROSS_PRODUCT ├──────────────┐ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ SEQ_SCAN ││ DELIM_SCAN │ │ ──────────────────── ││ ──────────────────── │ │ ~10000 Rows ││ ~5000 Rows │ └───────────────────────────┘└───────────────────────────┘ ```Query plan with HNSW index (operators and expected cardinalities)
```text ┌───────────────────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~50000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ HASH_GROUP_BY │ │ ──────────────────── │ │ ~50000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~50000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~50000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ~50000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ HNSW_INDEX_JOIN │ │ ──────────────────── │ │ ~50000 Rows │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ SEQ_SCAN │ │ ──────────────────── │ │ ~10000 Rows │ └───────────────────────────┘ ````QUALIFY` query:
```sql FROM lineitem SELECT *, row_number() OVER (PARTITION BY l_suppkey ORDER BY l_shipdate DESC) AS my_ranking QUALIFY my_ranking <= 3; ````max_by` query:
```sql FROM lineitem SELECT unnest( max_by(lineitem, l_shipdate, 3), recursive := 1 ) GROUP BY l_suppkey; ```Python Benchmarking Script
```python import duckdb import pandas as pd from threading import Thread from time import sleep from datetime import datetime from os import remove def check_memory(stop_function, filepath, sleep_seconds, results_dict): print("Starting background thread") background_con = duckdb.connect(filepath) max_memory = 0 max_temporary_storage = 0 while True: if stop_function(): break # Profile the memory memory_profile = background_con.sql(""" FROM duckdb_memory() SELECT tag, round(memory_usage_bytes / (1000000), 0)::bigint AS memory_usage_mb, round(temporary_storage_bytes / (1000000), 0)::bigint AS temporary_storage_mb; """).df() print(memory_profile) total_memory = background_con.sql(""" FROM memory_profile select sum(memory_usage_mb) AS total_memory_usage_mb, sum(temporary_storage_mb) AS total_temporary_storage_mb """).fetchall() print('Current memory:', total_memory[0][0]) print('Current temporary_storage:', total_memory[0][1]) if total_memory[0][0] > max_memory: max_memory = total_memory[0][0] if total_memory[0][1] > max_temporary_storage: max_temporary_storage = total_memory[0][1] print('Maximum memory:', max_memory) print('Maximum temporary_storage:', max_temporary_storage) sleep(sleep_seconds) results_dict["max_memory"] = max_memory results_dict["max_temporary_storage"] = max_temporary_storage background_con.close() return def query_and_profile(filepath, sql): con = duckdb.connect(filepath) con.sql("set max_memory='36GB'") results_dict = {} stop_threads = False background_memory_thread = Thread(target=check_memory, args=(lambda : stop_threads, filepath, 0.1, results_dict, )) background_memory_thread.start() print("Starting query:") start_time = datetime.now() results_df = con.sql(sql).df() results_dict["total_time_seconds"] = (datetime.now() - start_time).total_seconds() print(results_df.head(10)) stop_threads = True background_memory_thread.join() con.close() return results_dict filepath = './arg_max_check_duckdb_memory_v3.duckdb' con = duckdb.connect(filepath) print("Begin initial tpch load") con.sql("""call dbgen(sf=1);""") con.close() sql = """ FROM lineitem SELECT UNNEST( max_by(lineitem, l_shipdate, 3), recursive := 1 ) GROUP BY l_suppkey ;""" max_by_results = query_and_profile(filepath, sql) sql = """ FROM lineitem SELECT *, row_number() OVER (PARTITION BY l_suppkey ORDER BY l_shipdate DESC) AS my_ranking QUALIFY my_ranking <= 3 ;""" qualify_results = query_and_profile(filepath, sql) print('max_by_results:', max_by_results) print('qualify_results:', qualify_results) remove(filepath) ```
`tpch-queries.sql`
```sql PRAGMA version; SET enable_progress_bar = false; LOAD tpch; .timer on PRAGMA tpch(1); PRAGMA tpch(2); PRAGMA tpch(3); PRAGMA tpch(4); PRAGMA tpch(5); PRAGMA tpch(6); PRAGMA tpch(7); PRAGMA tpch(8); PRAGMA tpch(9); PRAGMA tpch(10); PRAGMA tpch(11); PRAGMA tpch(12); PRAGMA tpch(13); PRAGMA tpch(14); PRAGMA tpch(15); PRAGMA tpch(16); PRAGMA tpch(17); PRAGMA tpch(18); PRAGMA tpch(19); PRAGMA tpch(20); PRAGMA tpch(21); PRAGMA tpch(22); ```Query runtimes (in seconds)
| Query | SF100 / microSD | SF100 / NVMe | SF300 / microSD | SF300 / NVMe |
|---|---|---|---|---|
| Q1 | 81.1 | 15.6 | 242.0 | 55.1 |
| Q2 | 7.9 | 2.4 | 27.8 | 7.9 |
| Q3 | 31.5 | 11.8 | 218.9 | 52.7 |
| Q4 | 40.2 | 11.4 | 157.5 | 40.9 |
| Q5 | 32.2 | 12.3 | 215.9 | 54.1 |
| Q6 | 1.6 | 1.4 | 155.9 | 32.7 |
| Q7 | 12.1 | 12.3 | 255.2 | 69.6 |
| Q8 | 25.0 | 19.2 | 298.0 | 77.8 |
| Q9 | 74.0 | 50.1 | 337.2 | 147.7 |
| Q10 | 54.7 | 24.3 | 234.9 | 82.6 |
| Q11 | 7.8 | 2.3 | 34.0 | 14.9 |
| Q12 | 43.1 | 13.6 | 202.9 | 50.5 |
| Q13 | 59.2 | 51.7 | 207.4 | 177.2 |
| Q14 | 33.0 | 9.7 | 269.7 | 59.0 |
| Q15 | 11.1 | 7.1 | 157.2 | 39.6 |
| Q16 | 8.7 | 8.7 | 33.4 | 27.1 |
| Q17 | 8.3 | 7.6 | 249.4 | 66.2 |
| Q18 | 73.9 | 40.9 | 374.7 | 177.1 |
| Q19 | 66.0 | 17.8 | 317.9 | 73.8 |
| Q20 | 22.4 | 8.4 | 273.1 | 56.0 |
| Q21 | 66.9 | 35.2 | 569.5 | 172.5 |
| Q22 | 9.2 | 8.4 | 34.1 | 26.7 |
| Geomean | 23.8 | 11.7 | 171.9 | 55.2 |
| Total | 769.9 | 372.3 | 4,866.5 | 1,561.8 |
](/images/blog/windowing/vectorization-improvements.png){kind=link}
](/images/blog/windowing/streaming-lead.png){kind=link}
](/images/blog/windowing/parallel-partitions.png){kind=link}
](/images/blog/windowing/partition-major.png){kind=link}
The process to acquire this key file has a number of steps, outlined below. Luckily they only need to be done once! This is also available in the [repo README](https://github.com/evidence-dev/duckdb_gsheets/blob/main/docs/pages/index.md).
To connect DuckDB to Google Sheets via an access token, you’ll need to create a Service Account through the Google API Console. The GSheets extension will use it to generate an access token periodically. 1. Navigate to the [Google API Console](https://console.developers.google.com/apis/library). 2. Create a new project. 3. Search for the Google Sheets API and enable it. 4. In the left-hand navigation, go to the **Credentials** tab. 5. Click **+ Create Credentials** and select **Service Account**. 6. Name the Service Account and assign it the **Owner** role for your project. Click **Done** to save. 7. From the **Service Accounts** page, click on the Service Account you just created. 8. Go to the **Keys** tab, then click **Add Key** > **Create New Key**. 9. Choose **JSON**, then click **Create**. The JSON file will download automatically. 10. Open your Google Sheet and share it with the Service Account email.
The DuckDB UI uses interactive notebooks to define SQL scripts and show the results of queries.
However, its capabilities go far beyond this.
Let’s go over its main features.
> The DuckDB UI runs all your queries locally: your queries and data never leave your computer.
> If you would like to use [MotherDuck](https://motherduck.com/) through the UI, you have to [opt-in explicitly](#blog:2025-03-12-duckdb-ui::motherduck-integration).
#### Features {#blog:2025-03-12-duckdb-ui::features}
##### Databases {#blog:2025-03-12-duckdb-ui::databases}
Your attached databases are shown on the left.
This list includes in-memory databases plus any files and URLs you’ve loaded.
You can explore tables and views by expanding databases and schemas.
##### Table Summary {#blog:2025-03-12-duckdb-ui::table-summary}
Click on a table or view to show a summary below.
The UI shows the number of rows, the name and type of each column, and a profile of the data in each column.
Select a column to see a more detailed summary of its data.
You can use the _“Preview data”_ button near the top right to inspect the first 100 rows.
You can also find the SQL definition of the table or view here.
##### Notebooks {#blog:2025-03-12-duckdb-ui::notebooks}
You can organize your work into named notebooks.
Each cell of the notebook can execute one or more SQL statements.
The UI supports syntax highlighting and autocomplete to assist with writing your queries.
You can run the whole cell, or just a selection,
then sort, filter, or further transform the results using the provided controls.
##### Column Explorer {#blog:2025-03-12-duckdb-ui::column-explorer}
The right panel contains the [Column Explorer](https://motherduck.com/blog/introducing-column-explorer/), which shows a summary of your results.
You can dive into each column to gain insights.
##### MotherDuck Integration {#blog:2025-03-12-duckdb-ui::motherduck-integration}
If you would like to connect to [MotherDuck](https://motherduck.com/), you can sign into MotherDuck to persist files and tables to a [cloud data warehouse](https://motherduck.com/docs/getting-started/) crafted for using DuckDB at scale and sharing data with your team.
##### …And More! {#blog:2025-03-12-duckdb-ui::and-more}
The DuckDB UI is under active development. Expect additions and improvements!
#### Footprint {#blog:2025-03-12-duckdb-ui::footprint}
Like the DuckDB CLI, the DuckDB UI creates some files in the `.duckdb` directory in your home directory.
The UI puts its files in a sub-directory, `extension_data/ui`:
* Your notebooks and some other state are stored in a DuckDB database, `ui.db`.
* When you export data to the clipboard or a file (using the controls below the results), some tiny intermediate files (e.g., `ui_export.csv`) are generated.
Your data is cleared from these files after the export is completed, but some near-empty files remain, one per file type.
#### Internals {#blog:2025-03-12-duckdb-ui::internals}
Support for the UI is implemented in a [DuckDB extension](https://duckdb.org/docs/1.5/core_extensions/overview).
The extension embeds a localhost HTTP server, which serves the UI browser application, and also exposes an API for communication with DuckDB.
In this way, the UI leverages the native DuckDB instance from which it was started, enabling full access to your local memory, compute, and file system.
Results are returned in an efficient binary form closely matching DuckDB’s in-memory representation ([DataChunk](https://github.com/duckdb/duckdb/blob/v1.2.1/src/include/duckdb/common/types/data_chunk.hpp)).
[Server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) enable prompt notification of updates such as attaching databases.
These techniques and others make for a low-latency experience that keeps you in your flow.
See the [UI extension documentation](https://duckdb.org/docs/1.5/core_extensions/ui) for more details.
#### Summary {#blog:2025-03-12-duckdb-ui::summary}
In this blog post, we presented the new DuckDB UI, a powerful web interface for DuckDB.
The DuckDB UI shares many of its design principles with the DuckDB database.
It’s simple, fast, feature-rich, and portable, and runs locally on your computer.
The DuckDB UI extension is also open source: visit the [`duckdb/duckdb-ui` repository](https://github.com/duckdb/duckdb-ui) if you want to dive in deeper into the extension's code.
> The repository does not contain the source code for the frontend, which is currently not available as open-source.
> Releasing it as open-source is under consideration.
For help or to share feedback, please [file an issue](https://github.com/duckdb/duckdb-ui/issues/new), join the `#ui` channel in either the [DuckDB Discord](https://discord.duckdb.org/) or the [MotherDuck Community Slack](https://slack.motherduck.com/).
Happy quacking!
#### Update (2025-03-20) {#blog:2025-03-12-duckdb-ui::update-2025-03-20}
We are very happy to see the overwhelmingly positive response to the announcement of the DuckDB Local UI. In this short update, we’d like to address some of the questions raised by the community:
* Regarding the **licensing of the frontend code** (HTML, JavaScript, etc.), MotherDuck is currently reviewing licensing options for the UI code and assets and will clarify shortly.
* Regarding **offline (“air-gapped”) operations,** we are working towards supporting offline usage in a future version, and are currently exploring how to provide version notifications as upgrades become available.
* Regarding the **client-server protocol for the UI,** we’re going to work on opening up the client-server protocol so that other user interfaces can use the “back-end” part of the UI extension to run queries in DuckDB.
## Preview: Amazon S3 Tables in DuckDB {#blog:2025-03-14-preview-amazon-s3-tables}
**Publication date:** 2025-03-14
**Authors:** Sam Ansmink, Tom Ebergen, Gábor Szárnyas
**TL;DR:** We are happy to announce a new preview feature that adds support for Apache Iceberg REST Catalogs, enabling DuckDB users to connect to Amazon S3 Tables and Amazon SageMaker Lakehouse with ease.
> The AWS Storage Blog also published a post on this feature, see [Streamlining access to tabular datasets stored in Amazon S3 Tables with DuckDB](https://aws.amazon.com/blogs/storage/streamlining-access-to-tabular-datasets-stored-in-amazon-s3-tables-with-duckdb/).
#### Iceberg Ahead! {#blog:2025-03-14-preview-amazon-s3-tables::iceberg-ahead}
In recent years, the [Iceberg open table format](https://iceberg.apache.org/) has become increasingly popular. Major data warehouse platforms such as
[Databricks](https://www.databricks.com/company/newsroom/press-releases/databricks-agrees-acquire-tabular-company-founded-original-creators),
[Snowflake](https://docs.snowflake.com/en/release-notes/2024/other/2024-10-18-snowflake-open-catalog-ga),
[Google BigQuery](https://cloud.google.com/blog/products/data-analytics/biglake-support-for-building-apache-iceberg-lakehouses-is-now-ga)
and
[AWS](https://aws.amazon.com/blogs/big-data/read-and-write-s3-iceberg-table-using-aws-glue-iceberg-rest-catalog-from-open-source-apache-spark/)
have all announced or already implemented support for Iceberg tables. These platforms also support Iceberg [catalogs](https://iceberg.apache.org/terms/#catalog), which are responsible for tracking current metadata for a collection of Iceberg tables grouped by namespaces.
DuckDB has supported reading Iceberg tables [since September 2023](#blog:2023-09-26-announcing-duckdb-090) via the [`iceberg` extension](https://duckdb.org/docs/1.5/core_extensions/iceberg/overview). Today, we are happy to introduce a new preview feature in this extension, which allows attaching to [Iceberg REST catalogs](https://www.tabular.io/apache-iceberg-cookbook/getting-started-catalog-background/). This preview release coincides with two AWS announcements yesterday: [support for Iceberg tables in Amazon S3 Tables](https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-s3-tables-apache-iceberg-rest-catalog-apis/) and the [GA release of the integration between S3 Tables and SageMaker Lakehouse (AWS Glue Data Catalog)](https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-sagemaker-lakehouse-integration-s3-tables-generally-available/). In practice, these developments mean that DuckDB now provides an end-to-end solution for reading Iceberg tables in [S3 Tables](https://duckdb.org/docs/1.5/core_extensions/iceberg/catalogs#amazon-s3-tables) and [SageMaker Lakehouse](https://duckdb.org/docs/1.5/core_extensions/iceberg/catalogs#aws-glue-amazon-sagemaker-lakehouse).
> DuckDB's support for Iceberg REST Catalog endpoints in Amazon S3 Tables is the result of a collaboration between AWS and DuckLabs.
#### Using Apache Iceberg REST Catalogs in DuckDB {#blog:2025-03-14-preview-amazon-s3-tables::using-apache-iceberg-rest-catalogs-in-duckdb}
##### Steps for Installing {#blog:2025-03-14-preview-amazon-s3-tables::steps-for-installing}
To connect to Apache Iceberg REST Catalogs in DuckDB,
make sure you are running the **latest stable** DuckDB release (version 1.2.1).
For our example steps, we'll use the DuckDB [CLI client](https://duckdb.org/docs/1.5/clients/overview).
You can obtain this client from the [installation page](https://duckdb.org/install) and start it with:
```bash
duckdb
```
Next, we need to install the “bleeding edge” versions of the required extensions from the [`core_nightly` repository](https://duckdb.org/docs/1.5/extensions/installing_extensions#extension-repositories).
```sql
FORCE INSTALL aws FROM core_nightly;
FORCE INSTALL httpfs FROM core_nightly;
FORCE INSTALL iceberg FROM core_nightly;
```
> For more information on using the `core_nightly` repository, please see the [notes](#blog:2025-03-14-preview-amazon-s3-tables::footnotes) at the end of the post.
With these extensions installed, your DuckDB is now capable of using Apache Iceberg REST Catalogs.
Let's find some data.
##### Setting up an Amazon S3 Table Bucket {#blog:2025-03-14-preview-amazon-s3-tables::setting-up-an-amazon-s3-table-bucket}
(If you already have Iceberg tables in Amazon S3 Tables, you can skip to the [“Reading Iceberg Catalogs with DuckDB” section](#blog:2025-03-14-preview-amazon-s3-tables::reading-amazon-s3-tables-with-duckdb).)
In this post, we demonstrate how to read data from Amazon S3 Tables.
To follow along, make sure that your account has [`s3tables` permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-setting-up.html)
and create a new [S3 table bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets.html).
Note that Amazon S3 Tables is currently only supported in [selected AWS regions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-regions-quotas.html).
##### Populating an Amazon S3 Table Bucket {#blog:2025-03-14-preview-amazon-s3-tables::populating-an-amazon-s3-table-bucket}
If you don't have an S3 table bucket with tables already, we found the easiest way to get going is to create a table using Amazon Athena.
See their [instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-getting-started.html#s2-tables-tutorial-EMR-cluster).
For our example, we created a simple table with three columns using the Athena query editor:
```sql
CREATE TABLE duck_species (
id INT,
english_name STRING,
latin_name STRING
) TBLPROPERTIES ('table_type' = 'ICEBERG');
```
Let's insert some data to the table:
```sql
INSERT INTO duck_species VALUES
(0, 'Anas nivis', 'Snow duck');
```
##### Reading Amazon S3 Tables with DuckDB {#blog:2025-03-14-preview-amazon-s3-tables::reading-amazon-s3-tables-with-duckdb}
Querying S3 Tables with DuckDB is really easy.
The first step is to get your AWS credentials into DuckDB.
You can achieve this in two ways.
First, you can let DuckDB detect your AWS credentials and configuration based on the default profile in your `~/.aws` directory by creating the following secret using the [Secrets Manager](https://duckdb.org/docs/1.5/configuration/secrets_manager):
```sql
CREATE SECRET (
TYPE s3,
PROVIDER credential_chain
);
```
Alternatively, you can set the AWS key, secret, and region values manually. For example:
```sql
CREATE SECRET (
TYPE s3,
KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩',
SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩',
REGION '⟨us-east-1⟩'
);
```
> **Tip.** To see the secrets in your session, run `FROM duckdb_secrets();`{:.language-sql .highlight}
Next, point DuckDB to your S3 table bucket.
You can do so by copy-pasting the S3 Tables ARN value directly from the AWS Management Console and using it in the `ATTACH` command:
```sql
ATTACH 'arn:aws:s3tables:⟨us-east-1⟩:⟨111122223333⟩:bucket/⟨bucket-name⟩'
AS my_s3_tables_catalog (
TYPE iceberg,
ENDPOINT_TYPE s3_tables
);
```
And that's all! Now, DuckDB is connected to Amazon S3 Tables.
To show the available tables, run:
```sql
SHOW ALL TABLES;
```
```text
┌──────────────────────┬─────────┬───────────────┬──────────────┬──────────────┬───────────┐
│ database │ schema │ name │ column_names │ column_types │ temporary │
│ varchar │ varchar │ varchar │ varchar[] │ varchar[] │ boolean │
├──────────────────────┼─────────┼───────────────┼──────────────┼──────────────┼───────────┤
│ my_s3_tables_catalog │ ducks │ duck_species │ [__] │ [INTEGER] │ false │
└──────────────────────┴─────────┴───────────────┴──────────────┴──────────────┴───────────┘
```
You can query tables as if they were ordinary DuckDB tables:
```sql
FROM my_s3_tables_catalog.ducks.duck_species;
```
```text
┌───────┬──────────────┬────────────┐
│ id │ english_name │ latin_name │
│ int32 │ varchar │ varchar │
├───────┼──────────────┼────────────┤
│ 0 │ Anas nivis │ Snow duck │
└───────┴──────────────┴────────────┘
```
You also have an alternative option to connect to S3 Tables using the Amazon SageMaker Lakehouse (AWS Glue Data Catalog) Iceberg REST Catalog endpoint.
To do so, run:
```sql
ATTACH '⟨account_id⟩:s3tablescatalog/⟨namespace_name⟩'
AS (
TYPE iceberg,
ENDPOINT_TYPE glue
);
```
> **Tip.** If you need basic read access to tabular data in a single S3 table bucket, use the `s3_tables` endpoint type.
> If you want a unified view across all of your tabular data in AWS, use the `glue` endpoint type.
##### Schema Evolution {#blog:2025-03-14-preview-amazon-s3-tables::schema-evolution}
A key feature of the Iceberg format is [schema evolution](https://iceberg.apache.org/docs/1.7.1/evolution/),
i.e., the ability to follow changes in the table's schema.
To demonstrate this, we go back to the Athena query editor and add a new column to the `duck_species` table:
```sql
ALTER TABLE duck_species
ADD COLUMNS (conservation_status STRING);
```
Then, we insert a few more duck species:
```sql
INSERT INTO duck_species VALUES
(1, 'Anas eatoni', 'Eaton''s pintail', 'Vulnerable'),
(2, 'Histrionicus histrionicus', 'Harlequin duck', 'Least concern');
```
Let's run the query again from DuckDB:
```sql
FROM my_s3_tables_catalog.ducks.duck_species;
```
The query now returns a table with the additional fourth column, which has a `NULL` value in the row inserted before the change in the schema
– as expected.
```text
┌───────┬───────────────────────────┬─────────────────┬─────────────────────┐
│ id │ english_name │ latin_name │ conservation_status │
│ int32 │ varchar │ varchar │ varchar │
├───────┼───────────────────────────┼─────────────────┼─────────────────────┤
│ 1 │ Anas eatoni │ Eaton's pintail │ Vulnerable │
│ 2 │ Histrionicus histrionicus │ Harlequin duck │ Least concern │
│ 0 │ Anas nivis │ Snow duck │ NULL │
└───────┴───────────────────────────┴─────────────────┴─────────────────────┘
```
#### Conclusion {#blog:2025-03-14-preview-amazon-s3-tables::conclusion}
The latest preview release of the DuckDB `iceberg` extension enables directly reading tables using Iceberg REST endpoints.
This allows you to query Amazon S3 Tables and Amazon SageMaker Lakehouse (AWS Glue Data Catalog) with ease.
As of today, the extension is in an experimental state and is under active development.
We will publish a stable release later this year.
#### Footnotes {#blog:2025-03-14-preview-amazon-s3-tables::footnotes}
##### Cleaning Up {#blog:2025-03-14-preview-amazon-s3-tables::cleaning-up}
If you created a new S3 table bucket to follow the examples,
don't forget to clean up by [deleting your S3 table bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-delete.html).
##### Using the `core_nightly` Repository {#blog:2025-03-14-preview-amazon-s3-tables::using-the-core_nightly-repository}
The extensions used in this blog post are currently experimental, and hence they are distributed through the [`core_nightly` repository](https://duckdb.org/docs/1.5/extensions/installing_extensions#extension-repositories). If you want to switch back to using extensions from the `core` repository, follow the [extension documentation](https://duckdb.org/docs/1.5/extensions/installing_extensions#force-installing-to-upgrade-extensions).
Note that DuckDB does not support reloading extensions. Therefore, if you experience any issues, try restarting DuckDB after updating the extensions.
## Maximizing Your Delta Scan Performance in DuckDB {#blog:2025-03-21-maximizing-your-delta-scan-performance}
**Publication date:** 2025-03-21
**Author:** Sam Ansmink
**TL;DR:** We released a new version of the [`delta` extension](/docs/current/core_extensions/delta), which includes several new features and performance improvements. In this blog post, we’ll put the `delta` extension through its paces with some benchmarks and take a deep dive into some of the new performance-related features.
#### Overview {#blog:2025-03-21-maximizing-your-delta-scan-performance::overview}
In our [previous](#blog:2024-06-10-delta) post, we talked about what the Delta Lake table format is all about, and how DuckDB’s `delta` extension leverages the [Delta Kernel](https://github.com/delta-io/delta-kernel-rs) library to offer native support. In this blog post, we will focus on how to get the most performance out of reading Delta tables from DuckDB. We’ll start with a small recap of Delta, then demonstrate the performance gains that were made over the past few releases. Finally, we demonstrate three key features available in the latest Delta release that will ensure you get the most out of your Delta reading performance: metadata caching, file skipping and partition information pushdown.
#### The Delta Open Table Format {#blog:2025-03-21-maximizing-your-delta-scan-performance::the-delta-open-table-format}
Let’s start off with a small recap of Delta to get back up to speed. [Delta Lake](https://delta.io/) is an [Open Table Format](https://delta.io/blog/open-table-formats/) similar to [Apache Iceberg](https://iceberg.apache.org/) and [Apache Hudi](https://hudi.apache.org/). Open Table Formats are best understood as “a collection of data and metadata files” that aim to provide the flexibility of data lakes while providing some of the consistency guarantees of traditional data warehouses. In the case of Delta, the format consists of Parquet files for data and a mix of Parquet, JSON and binary files for metadata. Besides providing an improved level of consistency, the additional metadata provided by Open Table Formats allows various types of performance optimizations through things like column statistics and file skipping. For a more in-depth explanation, we refer to the [previous Delta blog post](#blog:2024-06-10-delta).
#### The `delta` Extension {#blog:2025-03-21-maximizing-your-delta-scan-performance::the-delta-extension}
DuckDB natively supports reading Delta Tables through the [`delta` extension](https://duckdb.org/docs/1.5/core_extensions/delta). This extension is one of the [core DuckDB extensions](https://duckdb.org/docs/1.5/core_extensions/overview) with >70k weekly downloads. Using this extension to read from a Delta Table is really simple. Since DuckDB v1.2.0, the `delta` extension will be automatically installed upon first use and loaded when invoking the `delta_scan` function.
So for example, to read a local Delta table, simply open any DuckDB client and run:
```sql
SELECT * FROM delta_scan('./⟨path_to_your_delta_table⟩');
```
Is your Delta table on someone else’s machine, perhaps in AWS? DuckDB can also query straight from S3! To have DuckDB automatically load your AWS credentials and query a remote Delta table, run:
```sql
CREATE SECRET (TYPE s3, PROVIDER credential_chain);
SELECT * FROM delta_scan('s3://⟨your-bucket⟩/⟨your_delta_table⟩');
```
For other cloud providers such as Azure or Google Cloud, check the [extension’s documentation page](https://duckdb.org/docs/1.5/core_extensions/delta#usage).
#### Performance Improvements between `delta` v0.1.0 and 0.3.0 {#blog:2025-03-21-maximizing-your-delta-scan-performance::performance-improvements-between-delta-v010-and-030}
While the first release (v0.1.0) of the `delta` extension already came with various performance-related features such as projection pushdown and constant filter pushdown, the features added since then have massively improved the performance of `delta_scan`. To illustrate this, our first benchmark will use the industry standard [TPC-DS](https://duckdb.org/docs/1.5/core_extensions/tpcds) benchmark with the scale factor 1 data set (SF1).
##### Benchmark Setup {#blog:2025-03-21-maximizing-your-delta-scan-performance::benchmark-setup}
For this benchmark, we started up an [AWS c6id.4xlarge instance](https://instances.vantage.sh/aws/ec2/c6id.4xlarge) (16 vCPUs, 32 GB RAM) and wrote the TPC-DS SF1 dataset to an S3 bucket in the same region (eu-west-1) using [PySpark](https://github.com/duckdb/duckdb-delta/blob/026345b9cf9092e3dd5ae42cc501ec8ed45ca09b/scripts/data_generator/generate_test_data.py).
Each benchmark is run a total of 6 times with the result being the median runtime of the last 5 runs with the first being considered a cold run. The aggregated results are shown in the following table.
| Result | Total runtime | Min runtime | Max runtime | Median runtime | Queries timed out |
| ---------------------------------- | ------------: | ----------: | ----------: | -------------: | ----------------: |
| `delta` extension v0.1.0 | 444.76 | 0.48 | 21.31 | 3.63 | 4 |
| `delta` extension v0.3.0 | 151.06 | 0.46 | 6.15 | 1.22 | 0 |
The detailed results of the benchmark are shown in the foldout:
Detailed TPC-DS SF1 benchmark results, `delta` extension v0.1.0 vs. v0.3.0
](/images/blog/delta-performance-fig1.svg){kind=link}
Detailed TPC-DS SF1 benchmark results with different configurations
](/images/blog/delta-performance-fig2.svg){kind=link}
##### Top 5 Busiest Train Stations per Month {#blog:2025-03-28-using-duckdb-in-streamlit::top-5-busiest-train-stations-per-month}
To extract the 5 busiest train stations per month, we start from the above query and add a `row_number` calculation with a final filter of the ranking to be less or equal to 5:
```python
stations_query, _ = get_stations_services_query(get_duckdb_conn())
top_5_query = (
stations_query.aggregate("""
station_name,
service_month: monthname(service_date),
service_month_id: month(service_date),
num_services: sum(num_services)
""")
.select("""
station_name,
service_month,
service_month_id,
num_services,
rn: row_number()
over (
partition by service_month
order by num_services desc
)
""")
.filter("rn <= 5")
.order("service_month_id, station_name")
)
```
Streamlit provides a series of charts, such as line and bar, but, unfortunately, they do not offer many personalization possibilities. Therefore, the above chart was generated with [Plotly](https://plotly.com/python/), an open source graphing library, integrated with Streamlit by calling `st.plotly_chart`.
##### Railway Network Utilization Analysis {#blog:2025-03-28-using-duckdb-in-streamlit::railway-network-utilization-analysis}
The data about the train rides through the train stations of The Netherlands, at day level, is a perfect input for analyzing the utilization of the railway network across the country and during the year.
###### Railway Network Utilization across the Country {#blog:2025-03-28-using-duckdb-in-streamlit::railway-network-utilization-across-the-country}
To analyze the network utilization across the country, we use `density_map` chart from Plotly, which will generate a heat map on a map chart. Because heat maps work best with pre-aggregated data, we first aggregate the number of train services and the geolocation of the station:
```python
stations_df = stations_query.aggregate(
"geo_lat, geo_lng, num_services: sum(num_services)"
).df()
```
It is interesting to observe that the north-east of the country is barely covered by railway and that, while Groningen and The Hague appear to be used during the year, neither of their train stations was in the top 5 busiest train stations per month.
####### Animating the Heat Map {#blog:2025-03-28-using-duckdb-in-streamlit::animating-the-heat-map}
From the above static chart, we can easily generate an animation with Plotly, by providing the animation frame parameter.
In our case, we generate an animation with the railway network utilization at service day level during July 2024:
###### Railway Network Utilization during the Year {#blog:2025-03-28-using-duckdb-in-streamlit::railway-network-utilization-during-the-year}
From a year point of view, we are curious to discover how busy the network is in each month and each day of the week, which is better displayed through a heat map. We prepare the input for the chart, by pivoting the number of services by the month name of the service date and by grouping by the ISO day name of the week. At the moment, there is no pivot method in the Python relational API, but we can use the SQL query generated by the relational API as source in the pivot raw SQL:
```python
@st.cache_data(ttl=3600, max_entries=100)
def get_stations_services_data(_duckdb_conn):
query = _duckdb_conn.sql("from services").aggregate("""
service_day: dayname("Service:Date"),
service_day_isodow: isodow("Service:Date"),
service_month: monthname("Service:Date"),
num_services: count(distinct "Service:RDT-ID")
""")
return (
_duckdb_conn.sql(f"""
pivot ({query.sql_query()})
on service_month
using sum(num_services)
group by service_day, service_day_isodow
order by service_day_isodow
""")
.select(
"January",
"February",
"March",
"April",
"May",
"June",
"July",
"August",
"September",
"October",
"November",
"December",
"service_day",
)
.df()
)
```
We have decided to create a function, in order to highlight [`cache_data` functionality in Streamlit](https://docs.streamlit.io/develop/concepts/architecture/caching#stcache_data). While the above query is executed in 300 ms on a MacBook Pro with 12 GB memory, we want to cache the results because the result of the query does not change often. `cache_data` is used in Streamlit to cache the results of a function and just as `cache_resource`, it has the attributes time to live and max entries. The two caching mechanisms require that the input parameters passed to the function to be [hashable](https://docs.streamlit.io/develop/concepts/architecture/caching#excluding-input-parameters). Since the DuckDB connection object is not hashable, we can tell Streamlit to ignore it by adding the prefix `_`.
> It is important to notice that the order of columns and values matters in a heat map chart and that we need to reset the index of the pivoted data before plotting.
The code of implementing interactive maps with Plotly in Streamlit is available on [GitHub](https://github.com/duckdb/duckdb-blog-examples/blob/main/duckdb_streamlit/pages/railway_network_utilization.py).
#### Finding the Closest 5 Stations with Folium {#blog:2025-03-28-using-duckdb-in-streamlit::finding-the-closest-5-stations-with-folium}
One other use-case for the railway network data is to find the closest train stations to a location selected by the user on the map. While Streamlit and Plotly have the functionality to register click events on the chart, they work only when the user is selecting points which are part of the chart data. Therefore, we will use `streamlit-folium`, a Streamlit component wrapped around [Folium](https://python-visualization.github.io/folium/latest/), which can register the clicks on any point of the map.
To work with geo data in DuckDB we have to install and load the spatial extension:
```python
duckdb_conn.sql("install spatial;")
duckdb_conn.sql("load spatial;")
```
We then define a function to retrieve the query, which returns the closest five train stations to a location, composed of latitude and longitude.
```python
def get_closest_stations_query(duckdb_conn, lat, lng):
stations_selection = duckdb_conn.sql("""
select name_long as station_name, geo_lat, geo_lng, code
from stations st
where exists (
select count(*)
from services sv
where st.code = sv."Stop:Station code"
having count(*) > 100
)
""")
return (
stations_selection.project(f"""
code as station_code,
station_name,
geo_lat,
geo_lng,
station_geo_point: st_point(geo_lng, geo_lat),
clicked_geo_point: st_point({lng}, {lat}),
distance_in_m: st_distance_sphere(
st_point(geo_lng, geo_lat),
clicked_geo_point
),
distance_in_km: round(distance_in_m/1000,2)
""")
.order("distance_in_km")
.limit(5)
)
```
In the above query we use `st_point` to create a point type field and `st_distance_sphere` to get the distance, in meters, between two points.
> It is important to mention that while we usually refer to a geo point as `[latitude, longitude]`, when working with spatial extensions we usually create the point as `[longitude, latitude]`. This applies to [PostGIS](https://postgis.net/) as well.
When a user clicks on the map, we store the clicks in the session state and rerun Streamlit to display the map with the new user selection. When rerun happens, Streamlit will rerun the entire application for the current session, therefore understanding the session state and caching mechanisms is essential when building Streamlit applications.
The code of implementing interactive maps with Folium in Streamlit is available on [GitHub](https://github.com/duckdb/duckdb-blog-examples/blob/main/duckdb_streamlit/pages/closest_train_stations.py).
#### Conclusion {#blog:2025-03-28-using-duckdb-in-streamlit::conclusion}
In this post, we illustrated how DuckDB can be used in Streamlit, how the Python relational API can make our code cleaner by chaining queries and how to implement interactive maps with Plotly and Folium, while leveraging the Streamlit caching mechanism.
## Fully Local Data Transformation with dbt and DuckDB {#blog:2025-04-04-dbt-duckdb}
**Publication date:** 2025-04-04
**Author:** Petrica Leuca
**TL;DR:** In this post, we implement data transformation and reverse ETL pipelines with DuckDB and dbt using the `dbt-duckdb` adapter.
#### Introduction {#blog:2025-04-04-dbt-duckdb::introduction}
The [Data Build Tool](https://www.getdbt.com/), `dbt`, is an open-source transformation framework, which enables data teams to adopt software engineering best practices in the code they deliver, such as Git workflow and unit testing. Other notable features of `dbt` are data lineage, documentation and data testing as part of the execution pipeline.
In this article we will demonstrate how to perform fully local data transformations with dbt and DuckDB.
To this end, we'll use the [`dbt-duckdb`](https://github.com/duckdb/dbt-duckdb) adapter, which integrates the `dbt` standard and with DuckDB's processing power.
##### Data Model {#blog:2025-04-04-dbt-duckdb::data-model}
We use two open datasets in this post: railway services, provided by the team behind the [Rijden de Treinen *(Are the trains running?)* application](https://www.rijdendetreinen.nl/en/about) and cartography information about the Netherlands provided by [cartomap](https://github.com/cartomap/nl).
The datasets are organized into:
- a persisted DuckDB database, in which we store the railway services data from 2024, hosted on Cloudflare;
- a provinces GeoJSON file, containing the geographic information about the Dutch provinces, hosted on GitHub;
- a municipalities GeoJSON file, containing the geographic information about Dutch municipalities, stored together with the code.
After doing an initial exploration of the above data, we can observe that a province can have one or many municipalities, a municipality can have none or many train stations and a train service record is connected to one and only one train station. Therefore we decide on the following data model:
- a dimension table, `dim_nl_provinces`, to hold information about the Dutch provinces;
- a dimension table, `dim_nl_municipalities`, to hold information about the Dutch municipalities, linked to `dim_nl_provinces`;
- a dimension table, `dim_nl_train_stations`, to hold information about the train stations in the Netherlands, linked to `dim_nl_municipalities`;
- a fact table, `fact_services`, to hold information about the train services, linked to `dim_nl_train_stations`.
The purpose of processing the train services and the Netherlands cartography data is to organize the data in a structure which can easily be used in future railway data analysis use cases. Such curated data structures are often called [_data marts_](https://en.wikipedia.org/wiki/Data_mart).
> **Tip.** The following code is available on [GitHub](https://github.com/duckdb/duckdb-blog-examples/tree/main/dbt_duckdb).
#### Processing Data with DuckDB and dbt {#blog:2025-04-04-dbt-duckdb::processing-data-with-duckdb-and-dbt}
After we have [initialized](https://docs.getdbt.com/reference/commands/init) our project, we configure the connection details for DuckDB in the `profiles.yml` file. Along with specifying if the database should be in memory or persisted to disk, we also specify:
- which extensions are required for the data processing, e.g., [spatial](https://duckdb.org/docs/1.5/core_extensions/spatial/overview);
- external databases, attached from the local disk or other storage solutions.
```yaml
dutch_railway_network:
outputs:
dev:
type: duckdb
path: data/dutch_railway_network.duckdb
extensions:
- spatial
- httpfs
threads: 5
attach:
- path: 'https://blobs.duckdb.org/nl-railway/train_stations_and_services.duckdb'
type: duckdb
alias: external_db
target: dev
```
We then configure the `sources.yml` file under the `models` directory by
specifying external sources (such as files) and table definitions
from the attached database(s):
```yaml
version: 2
sources:
- name: geojson_external
tables:
- name: nl_provinces
config:
external_location: "https://cartomap.github.io/nl/wgs84/provincie_2025.geojson"
- name: nl_municipalities
config:
external_location: "seeds/gemeente_2025.geojson"
- name: external_db
database: external_db
schema: main
tables:
- name: stations
- name: services
```
> The `external_location` can point to CSV, Parquet or JSON files.
> Both the local file system and remote endpoints (e.g., HTTP or S3) are supported.
With both the profile and source defined, we can now load the data.
##### Loading Data to DuckDB {#blog:2025-04-04-dbt-duckdb::loading-data-to-duckdb}
In `dbt` the way of storing the data in the target system is called `materialization`. The `dbt-duckdb` adapter provides the following materialization options:
- `table`, replacing the target table at each run;
- `incremental`, with `append` and `delete+insert` options, modifying the data in the table, but not the table itself (if it exists);
- `snapshot`, implementing a [Slowly Changing Dimension Type 2](https://en.wikipedia.org/wiki/Slowly_changing_dimension) table with time validity intervals;
- `view`, replacing the target view at each run.
Another feature of using the above adapter is that the SQL dialect used in the data processing scripts has all the [friendly SQL extensions](https://duckdb.org/docs/1.5/sql/dialect/friendly_sql) DuckDB has to offer.
To refresh the data in the `dim_nl_provinces` table, we use the `st_read` spatial function, which reads and parses automatically the `nl_provinces` GeoJSON file, defined in `sources.yml`.
```sql
{{ config(materialized='table') }}
SELECT
{{ dbt_utils.generate_surrogate_key(['id']) }} AS province_sk,
id AS province_id,
statnaam AS province_name,
geom AS province_geometry,
{{ common_columns() }}
FROM st_read({{ source("geojson_external", "nl_provinces") }}) AS src;
```
Similarly, we fully refresh the data for `dim_nl_municipalities` and `fact_services`.
To build the relation between a train station location and a municipality,
we use the `st_contains` spatial function, which returns `true` when a geometry contains another geometry:
```sql
{{ config(materialized='table') }}
SELECT
{{ dbt_utils.generate_surrogate_key(['tr_st.code']) }} AS station_sk,
tr_st.id AS station_id,
tr_st.code AS station_code,
tr_st.name_long AS station_name,
tr_st.type AS station_type,
st_point(tr_st.geo_lng, tr_st.geo_lat) AS station_geo_location,
coalesce(dim_mun.municipality_sk, 'unknown') AS municipality_sk,
{{ common_columns() }}
FROM {{ source("external_db", "stations") }} AS tr_st
LEFT JOIN {{ ref ("dim_nl_municipalities") }} AS dim_mun
ON st_contains(
dim_mun.municipality_geometry,
st_point(tr_st.geo_lng, tr_st.geo_lat)
)
WHERE tr_st.country = 'NL';
```
>To read from the external sources, we reference the source by providing the source and table names.
#### Exporting Data from DuckDB {#blog:2025-04-04-dbt-duckdb::exporting-data-from-duckdb}
One major benefit of using DuckDB for data processing is the ability to export data to files (such as CSV, JSON and Parquet) and to refresh data directly in PostgreSQL or MySQL databases.
##### External Files {#blog:2025-04-04-dbt-duckdb::external-files}
The feature of exporting data to files is enabled by the `dbt-duckdb` adapter with the [`external` materialization](https://github.com/duckdb/dbt-duckdb#readme). With the `external` materialization, we are able to export data to CSV, JSON and Parquet file types to a specified storage location (local or external). The load type is `full refresh`, therefore existing files are overwritten.
In the following processing step, we export aggregated train service data at month level, to a Parquet file, partitioned by year and month:
```sql
{{
config(
materialized='external',
location="data/exports/nl_train_services_aggregate",
options={
"partition_by": "service_year, service_month",
"overwrite": True
}
)
}}
SELECT
year(service_date) AS service_year,
month(service_date) AS service_month,
service_type,
service_company,
tr_st.station_sk,
tr_st.station_name,
m.municipality_sk,
m.municipality_name,
p.province_sk,
p.province_name,
count(*) AS number_of_rides
FROM {{ ref ("fact_services") }} AS srv
INNER JOIN {{ ref("dim_nl_train_stations") }} AS tr_st
ON srv.station_sk = tr_st.station_sk
INNER JOIN {{ ref("dim_nl_municipalities") }} AS m
ON tr_st.municipality_sk = m.municipality_sk
INNER JOIN {{ ref("dim_nl_provinces") }} AS p
ON m.province_sk = p.province_sk
WHERE service_year = {{ var('execution_year') }}
GROUP BY ALL
```
The exported files are placed in [Hive partitioned directory structure](https://duckdb.org/docs/1.5/data/partitioning/hive_partitioning).
```text
./service_year=2024/service_month=1:
49255 Apr 2 14:54 data_0.parquet
...
./service_year=2024/service_month=12:
48031 Apr 2 14:54 data_0.parquet
```
##### PostgreSQL {#blog:2025-04-04-dbt-duckdb::postgresql}
After we process the data into our railway services data mart, we can generate from it a daily aggregate at train station level, organized into a star schema model, by having the dimension keys part of the data:
```sql
{{
config(
materialized='incremental',
incremental_strategy='delete+insert',
unique_key="""
service_date,
service_type,
service_company,
station_sk
"""
)
}}
SELECT
service_date,
service_type,
service_company,
srv.station_sk,
mn.municipality_sk,
province_sk,
count(*) AS number_of_rides,
{{ common_columns() }}
FROM {{ ref ("fact_services") }} AS srv
INNER JOIN {{ ref("rep_dim_nl_train_stations") }} AS tr_st
ON srv.station_sk = tr_st.station_sk
INNER JOIN {{ ref("rep_dim_nl_municipalities") }} AS mn
ON tr_st.municipality_sk = mn.municipality_sk
WHERE NOT service_arrival_cancelled
{% if is_incremental() %}
AND srv.invocation_id = (
SELECT invocation_id
FROM {{ ref("fact_services") }}
ORDER BY last_updated_dt DESC
LIMIT 1
)
{% endif %}
GROUP BY ALL
```
Thanks to DuckDB's ability to connect and write to a PostgreSQL database, we can add the above processing step to our `dbt` project, under the directory `models/reverse_etl`.
To connect to a PostgreSQL database, we need to specify in `profiles.yml`:
- the [`postgres` extension](https://duckdb.org/docs/1.5/core_extensions/postgres/overview);
- the PostgreSQL connection string in the `attach` section.
```yaml
dutch_railway_network:
outputs:
dev:
type: duckdb
path: data/dutch_railway_network.duckdb
extensions:
- ...
- postgres
threads: 5
attach:
- ...
- path: "postgresql://postgres:{{ env_var('DBT_DUCKDB_PG_PWD') }}@localhost:5466/postgres"
type: postgres
alias: postgres_db
target: dev
```
We also have to configure the model's database details in `dbt_project.yml`:
```yaml
models:
dutch_railway_network:
transformation:
schema: main
+docs:
node_color: 'silver'
reverse_etl:
database: postgres_db
schema: public
+docs:
node_color: '#d5b85a'
```
With this configuration, all the models from the `transformation` directory will be executed on the `main_main` schema, while the models from `reverse_etl` will be executed on the `main_public` schema.
After executing the models with `dbt run --model +reverse_etl`, the data is available to query from PostgreSQL:
```bash
psql -U postgres
```
```sql
SELECT count(*), sum(number_of_rides)
FROM main_public.rep_fact_train_services_daily_agg;
```
```text
count | sum
--------+----------
240826 | 17438151
```
> It is important to mention that while PostgreSQL is the target database and `dbt` provides `merge` as an incremental strategy for it; the execution of the above pipeline is happening in DuckDB, therefore the incremental load can be done only with `append` or `delete+insert` strategies.
#### Execution Details {#blog:2025-04-04-dbt-duckdb::execution-details}
The above implementation consists of 10 models and 20 data tests, processing 400 MB of data from the attached DuckDB database, together with small data organized in GeoJSON files. The total execution time, on a single thread and on a MacBook Pro of 12 GB, is between 40 and 45 seconds. From the total execution time, approximately 30 seconds are spent to process the train services data and 4 seconds to write the aggregated data to PostgreSQL:
```text
05:48:07 Running with dbt=1.9.3
05:48:08 Registered adapter: duckdb=1.9.2
05:48:08 Found 10 models, 20 data tests, 4 sources, 565 macros
05:48:08
05:48:08 Concurrency: 1 threads (target='dev')
...
05:48:45 19 of 30 OK created sql table model main_main.fact_services .................... [OK in 32.60s]
...
05:48:50 26 of 30 OK created sql incremental model postgres_db.main_public.rep_fact_train_services_daily_agg [OK in 3.74s]
...
05:48:51 Finished running 2 external models, 1 incremental model, 7 table models, 20 data tests in 0 hours 0 minutes and 42.63 seconds (42.63s).
05:48:51
05:48:51 Completed successfully
05:48:51
05:48:51 Done. PASS=30 WARN=0 ERROR=0 SKIP=0 TOTAL=30
```
#### Conclusion {#blog:2025-04-04-dbt-duckdb::conclusion}
In this post we have demonstrated how DuckDB integrates with `dbt` and how it is part of the data processing ecosystem by showcasing data mart creation, file exports and reverse ETL to a PostgreSQL database.
## DuckDB's CSV Reader and the Pollock Robustness Benchmark: Into the CSV Abyss {#blog:2025-04-16-duckdb-csv-pollock-benchmark}
**Publication date:** 2025-04-16
**Authors:** Pedro Holanda, Gábor Szárnyas, Gerardo Vitagliano
**TL;DR:** DuckDB ships with a fast and robust CSV reader, which – we believe – can consume most CSV files found in the wild. To empirically evaluate this, we used the Pollock Benchmark, a state-of-the-art test suite designed to measure how well CSV readers can operate on non-standard files, and found that DuckDB ranks #1.
#### The Sorry State of the CSV Landscape {#blog:2025-04-16-duckdb-csv-pollock-benchmark::the-sorry-state-of-the-csv-landscape}
It is a well-known fact that CSV files come in all shapes and forms.
While there is a [well-defined standard](https://www.ietf.org/rfc/rfc4180.txt), it is common for systems to export data without following even the basic rules of CSV construction.
For example, Rabobank, one of the biggest banks in the Netherlands, exports the financial data of its clients with unescaped quotes in quoted values.
If a big financial institute cannot write proper CSV files, imagine the errors occurring in CSV files exported by decades-old legacy software, in hand-written CSV files, files exported from spreadsheet software, and so on!
These sorts of errors are so prevalent that spreadsheet software, [dataframe libraries](https://news.ycombinator.com/item?id=39665312) and specialized CSV tools go above and beyond to salvage these files and hammer them into tables that users can work with.
#### DuckDB's CSV Parser {#blog:2025-04-16-duckdb-csv-pollock-benchmark::duckdbs-csv-parser}
When we designed the DuckDB CSV parser, we strived to ensure that it's not only easy to use and fast, but also extremely reliable – to the point it can read most CSVs thrown at it.
In this blog post, we study this from two angles.
First, we explore the main options that can help you read non-standard CSV files and show their effects on parsing with some simple examples.
Second, we evaluate how robust DuckDB's CSV parser is by putting it to the test with the [Pollock Benchmark](https://www.vldb.org/pvldb/vol16/p1870-vitagliano.pdf).
> There is a lot of discourse on the internet about whether a system should go above and beyond to read files that do not follow the standard. And yes, non-standard files are frustrating. We can all complain about them on the internet or decide to ignore them. But we strongly believe that ignoring reality comes at a cost. Being able to read non-standard files – even if it might result in ambiguity – should be the user's decision.
##### Running Example {#blog:2025-04-16-duckdb-csv-pollock-benchmark::running-example}
As our running example, we'll use a CSV file called `cafes.csv`:
```csv
ZIP,Name,Country
1014,"Cafe ""Gedoogt""",Netherlands
1015,"Cafe "De Tuin" Amsterdam",Netherlands
1095,Joost
1872,Cafe Gezellig,Netherlands,5
```
Despite its small size, the file contains many different errors.
Let's inspect it line by line:
| Line | Row | Comment |
| ---: | --------------------------------------------- | ----------------------------------------------------------------------------- |
| 1 | `ZIP,Name,Country` | The header of the file. |
| 2 | `1014,"Cafe ""Gedoogt""",Netherlands` | A well-defined line, with a quoted value that contains escaped quotes (` ""`). |
| 3 | `1015,"Cafe "De Tuin" Amsterdam",Netherlands` | The value for `Name` contains unescaped quotes. |
| 4 | `1095,Joost` | This line is missing the value for the `Country`. |
| 5 | `1872,Cafe Gezellig,Netherlands,5` | Has an extra column with the value `5`. |
##### Loading the Example Naïvely {#blog:2025-04-16-duckdb-csv-pollock-benchmark::loading-the-example-navely}
If we naïvely try to read our example, we get a table with only a single line:
```sql
FROM read_csv('cafes.csv');
```
```text
┌─────────┬───────────────┬─────────────┬─────────┐
│ column0 │ column1 │ column2 │ column3 │
│ int64 │ varchar │ varchar │ int64 │
├─────────┼───────────────┼─────────────┼─────────┤
│ 1872 │ Cafe Gezellig │ Netherlands │ 5 │
└─────────┴───────────────┴─────────────┴─────────┘
```
We can see that the reader ignored all lines except the last one.
Let's query the [CSV sniffer](#blog:2023-10-27-csv-sniffer) to see the dialect it detected:
```sql
SELECT Delimiter, Quote, Escape, SkipRows
FROM sniff_csv('cafes.csv');
```
```text
┌───────────┬─────────┬─────────┬──────────┐
│ Delimiter │ Quote │ Escape │ SkipRows │
│ varchar │ int16 │ varchar │ uint16 │
├───────────┼─────────┼─────────┼──────────┤
│ , │ \0 │ \0 │ 4 │
└───────────┴─────────┴─────────┴──────────┘
```
We can see that the sniffer decided that quotes and escape characters are not present in the file (represented as `\0` values).
Consequently, it decided to skip the first 4 rows (including the header) in order to read the file according to the dialect.
> **Tip.** DuckDB allows users to retrieve all errors that occur when reading a CSV file using the `reject_errors` table.
> This is beyond the scope of this blog post, but you can find more details in the [“Reading Faulty CSV Files” documentation page](https://duckdb.org/docs/1.5/data/csv/reading_faulty_csv_files).
##### Loading the Example with a User-Defined CSV Dialect {#blog:2025-04-16-duckdb-csv-pollock-benchmark::loading-the-example-with-a-user-defined-csv-dialect}
Let's attempt to read the faulty CSV files with a user-defined CSV dialect.
To this end, we'll set the `auto_detect` option to `false` and manually specify the presence of the `header`, the values of the `delim`(iter), `quote`, and `escape` options, as well as the schema with the `columns` option.
```sql
FROM read_csv('cafes.csv',
auto_detect = false,
header = true,
delim = ',',
quote = '"',
escape = '"',
columns = {'ZIP': 'INT16', 'Name': 'VARCHAR', 'Country': 'VARCHAR'}
);
```
Running this SQL statement will result in an error due to unescaped quotes in line 3: the parser can't determine whether the first quote in `"De Tuin"` marks the end of the quoted value or not:
```console
Invalid Input Error:
CSV Error on Line: 3
Original Line: 1015,"Cafe "De Tuin" Amsterdam",Netherlands
Value with unterminated quote found.
Possible fixes:
* Disable the parser's strict mode (strict_mode=false) to allow reading rows that do not comply with the CSV standard.
* Enable ignore errors (ignore_errors=true) to skip this row
* Set quote to empty or to a different value (e.g., quote='')
```
The error message informs us that there was a value with an _unterminated quote,_ caused by the unescaped quote character.
It also suggests some possible fixes, which will guide our attempts in the next sections.
##### Disabling Strict Mode {#blog:2025-04-16-duckdb-csv-pollock-benchmark::disabling-strict-mode}
As the error message informed us, one of the ways to relax the constraints of the DuckDB CSV reader to read this file is by utilizing the `strict_mode` option. Namely, disabling the `strict_mode` option will allow the CSV Reader to go over the following common errors:
1. Unescaped quoted values, such as the one in line 3.
2. Rows that have too many columns. For example, for the row `1872,Cafe Gezellig,Netherlands,5`, the last column is ignored to fit our schema of three columns.
3. A mix of newline delimiters, e.g., files having both `\n` and `\r\n` as newline delimiters. This error does not occur in our running example.
By default, `strict_mode` is set to `true`, which implies that DuckDB will not attempt to parse rows that do not fit the dialect.
However, setting it to `false` allows the CSV reader to attempt to read the file even if it cannot be correctly read under the given configuration.
Let's go back to our example and turn off strict mode:
```sql
FROM read_csv('cafes.csv',
auto_detect = false,
header = true,
strict_mode = false,
delim = ',',
quote = '"',
escape = '"',
columns = {'ZIP': 'INT16', 'Name': 'VARCHAR', 'Country': 'VARCHAR'}
);
```
We can see that the reader now only errors at line 4, indicating the previous lines were correctly read up to that point:
```console
Invalid Input Error:
CSV Error on Line: 4
Original Line: 1095,Joost
Expected Number of Columns: 3 Found: 2
Possible fixes:
* Enable null padding (null_padding=true) to replace missing values with NULL
* Enable ignore errors (ignore_errors=true) to skip this row
```
> It is important to note two things for the `strict_mode = false` option:
>
> * When using this option, there is no guarantee that the result is correct.
> This is because it's impossible to define what constitutes a “correct result” when parsing a non-standard CSV file.
>
> * DuckDB runs a best-effort parser, as it will try only fail if it encounters an error to which it can't find any reasonable guess of what the correct data should look like (e.g., imagine a column of integers, where some integers are represented as a spelled-out string, this will cause a casting error).
> While it is possible that, with a given combination of errors, our parser may still fail, we are striving to make it even more robust as we face the abyss.
##### Ignoring Errors {#blog:2025-04-16-duckdb-csv-pollock-benchmark::ignoring-errors}
The other option suggested by the error message was to set `ignore_errors = true`.
This option simply means that any rows which do not fit the chosen dialect or schema will be skipped from the result.
In practice, if we add it to our query:
```sql
FROM read_csv('cafes.csv',
auto_detect = false,
header = true,
strict_mode = false,
delim = ',',
quote = '"',
escape = '"',
columns = {'ZIP': 'INT16', 'Name': 'VARCHAR', 'Country': 'VARCHAR'},
ignore_errors = true
);
```
We get an almost complete result! Only missing line number 4 `1095,Joost` since it is missing a column.
```text
┌───────┬────────────────────────┬─────────────┐
│ ZIP │ Name │ Country │
│ int16 │ varchar │ varchar │
├───────┼────────────────────────┼─────────────┤
│ 1014 │ Cafe "Gedoogt" │ Netherlands │
│ 1015 │ Cafe De Tuin Amsterdam │ Netherlands │
│ 1872 │ Cafe Gezellig │ Netherlands │
└───────┴────────────────────────┴─────────────┘
```
> The `ignore_errors` option also affects the sniffer's behavior, as it will ignore lines that produce errors during sniffing.
> In general, the configuration that results in the fewest errors will be preferred.
##### Null Padding for Missing Values {#blog:2025-04-16-duckdb-csv-pollock-benchmark::null-padding-for-missing-values}
The last option that can help when reading non-standard CSV files is the `null_padding` option.
This option is useful when lines in the CSV file have an inconsistent number of values, with some missing values for given columns.
Let's read the file with the `null_padding = true` option.
```sql
FROM read_csv('cafes.csv',
auto_detect = false,
header = true,
strict_mode = false,
delim = ',',
quote = '"',
escape = '"',
columns = {'ZIP': 'INT16', 'Name': 'VARCHAR', 'Country': 'VARCHAR'},
null_padding = true
);
```
This will produce the following result:
```text
┌───────┬────────────────────────┬─────────────┐
│ ZIP │ Name │ Country │
│ int16 │ varchar │ varchar │
├───────┼────────────────────────┼─────────────┤
│ 1014 │ Cafe "Gedoogt" │ Netherlands │
│ 1015 │ Cafe De Tuin Amsterdam │ Netherlands │
│ 1095 │ Joost │ NULL │
│ 1872 │ Cafe Gezellig │ Netherlands │
└───────┴────────────────────────┴─────────────┘
```
##### Minimal Configuration to Read the Example File {#blog:2025-04-16-duckdb-csv-pollock-benchmark::minimal-configuration-to-read-the-example-file}
Whew. We loaded the file but the `read_csv` call is about 200 characters long!
Luckily, we can do better than that: if we set both `strict_mode = false` and `null_padding = true`, we only need to specify the `quote` and `escape` values.
This call only requires about 80 characters and gets a valid result from this CSV file:
```sql
FROM read_csv('cafes.csv',
strict_mode = false,
null_padding = true,
quote = '"',
escape = '"'
);
```
```text
┌───────┬────────────────────────┬─────────────┬─────────┐
│ ZIP │ Name │ Country │ column3 │
│ int64 │ varchar │ varchar │ int64 │
├───────┼────────────────────────┼─────────────┼─────────┤
│ 1014 │ Cafe "Gedoogt" │ Netherlands │ NULL │
│ 1015 │ Cafe De Tuin Amsterdam │ Netherlands │ NULL │
│ 1095 │ Joost │ NULL │ NULL │
│ 1872 │ Cafe Gezellig │ Netherlands │ 5 │
└───────┴────────────────────────┴─────────────┴─────────┘
```
With this configuration, the sniffer actually creates a new column (` column3`) to accommodate the extra value in the last line.
This example shows that even on a CSV file exhibiting several issues, DuckDB's CSV sniffer can still make reasonably good decisions on the file's dialect and schema.
> **Tip.** You can drop the last column by e.g. using `SELECT #1, #2, #3 FROM ...` in the query.
#### The Pollock Benchmark {#blog:2025-04-16-duckdb-csv-pollock-benchmark::the-pollock-benchmark}
The [Pollock Benchmark](https://hpi.de/naumann/projects/data-preparation/pollock.html) is a CSV data loading benchmark designed to evaluate how robust a CSV system is when reading non-standard CSV files. The benchmark was published at [VLDB 2023](https://www.vldb.org/pvldb/vol16/p1870-vitagliano.pdf) and is completely open-source, with a DuckDB entry recently added to its [repository](https://github.com/HPI-Information-Systems/Pollock).
The authors of the Pollock benchmark analyzed over 245,000 public CSV datasets to understand the most common ways the RFC-4180 standard is violated in CSV files in the wild. After identifying the most common error types, they created a CSV file generator with a pollution mechanism, which inserts these errors. In total, the benchmark tool generates over 2,200 polluted files. It also generates the correct dialect configuration that should be used to read these files, along with a clean version of the file.
To evaluate the robustness of systems, the benchmark tool reads the polluted file with the configuration set for a given system-under-test.
It then uses the system to write a new file with the answer and compares it with the clean version.
This comparison yields a score that reflects how accurately the system has read the file.
There are mainly two scores that the benchmark produces: the simple score and the weighted score, with the weighted score accounting for how commonly that type of error appears in real life.
In this section, we will describe the most common errors depicted in the paper and share the results of the benchmark after adding DuckDB.
##### Common Errors {#blog:2025-04-16-duckdb-csv-pollock-benchmark::common-errors}
The paper depicts many of the common errors that appear in CSV files.
In this section, we will briefly discuss some of the most common errors, but refer to the paper for a complete overview:
1. **Inconsistent Number of Cells in Rows:** RFC-4180 requires that all rows, including the header, have a consistent number of columns, but many files have too many or too few delimiters per row.
2. **Non-Standard Newline Sequences:** RFC-4180 requires that files use a carriage return followed by a line feed (` \r\n`), but many files use only line feeds, carriage returns, or a mix of them.
3. **Multiple Header Lines:** The standard allows for an optional single header line but many files have multiple headers.
4. **Incorrectly Quoted or Unescaped Cells:** This error occurs when quoted values have unescaped quotes in them. See the [`file_escape_char_0x00.csv`](https://github.com/HPI-Information-Systems/Pollock/blob/main/polluted_files/csv/file_escape_char_0x00.csv#L11) for an example. Note how GitHub's table renderer, which can read the [clean version of this file](https://github.com/HPI-Information-Systems/Pollock/blob/main/polluted_files/clean/file_escape_char_0x00.csv) fails to parse it.
5. **Files with Multibyte Delimiters:** The standard requires files to be comma-separated. Although single-byte delimiters are widely accepted, multibyte delimiters are common.
6. **Files with Multiple Tables:** When a CSV file stores multiple tables with different schemas in the same file.
##### Methodology {#blog:2025-04-16-duckdb-csv-pollock-benchmark::methodology}
The benchmark includes a wide variety of systems being tested, such as CSV parsing frameworks (e.g., CleverCSV), relational database systems (e.g., PostgreSQL), spreadsheet software (e.g., LibreOffice Calc), data visualization tools (e.g., Dataviz), and dataframe libraries (e.g., Pandas).
An important aspect of the benchmark is that it provides the dialect and schema for each file. Without this, systems that do not have a sniffer (e.g., PostgreSQL) would not be able to read the file. There is no specific rule or differentiation in how much each system utilizes that information. For example, Pandas only takes partial advantage of these settings.
To incorporate these differences in our evaluation of DuckDB, we decided to add two different configurations:
1. **DuckDB (benchmark config).**
Under this configuration, all the options in the configuration file that are relevant to DuckDB – such as CSV dialect and schema – are passed to the reader. In addition, we also set all the options described in the previous section (i.e., `null_padding = true, strict_mode = false, ignore_errors = true`). This essentially tells us how much we can read from these files if the users manually set the necessary options.
2. **DuckDB (auto-detect only).**
Under this configuration, we do not take advantage of providing custom configuration files. The only options set are those that allow for non-standard file reading (i.e., `null_padding = true, strict_mode = false, ignore_errors = true`). Hence, this option also evaluates the full power of our sniffer in scenarios of uncertainty.
##### Pollock Scores {#blog:2025-04-16-duckdb-csv-pollock-benchmark::pollock-scores}
The following table presents the results with DuckDB included. We also limited the original table to show only the systems with the best scores in each category (e.g., parsing frameworks, relational systems, and so on). For both the simple and weighted scores, DuckDB with all options set (the default configuration for the Pollock benchmark) is the clear winner.
In terms of the simple score, this means that DuckDB correctly read `99.61%` of the data from all the files.
It also correctly handles the most common errors, as reflected in the weighted score.
Pollock scores sorted by weighted score (out of 10):
| System under test | Pollock score (weighted) | Pollock score (simple) |
| ------------------------------- | -----------------------: | ---------------------: |
| _DuckDB 1.2 (benchmark config)_ | 9.599 | 9.961 |
| “SpreadDesktop” | 9.597 | 9.929 |
| Pandas 1.4.3 | 9.431 | 9.895 |
| “SpreadWeb” | 9.431 | 9.721 |
| SQLite 3.39.0 | 9.375 | 9.955 |
| _DuckDB 1.2 (auto-detect only)_ | 8.439 | 9.075 |
| UniVocity 2.9.1 | 7.936 | 9.939 |
| LibreOffice Calc 7.3.6 | 7.833 | 9.925 |
| Dataviz | 5.152 | 5.003 |
> According to the [benchmark's website](https://hpi.de/naumann/projects/data-preparation/pollock.html),
> “SpreadDesktop” is a desktop-based commercial spreadsheet software and “SpreadWeb” is a web-based spreadsheet software.
> Their real names were omitted due to licensing.
Click here to see the full result table
| System under test | Pollock score (weighted) | Pollock score (simple) |
|---|---|---|
| DuckDB 1.2 (benchmark config) | 9.599 | 9.961 |
| “SpreadDesktop” | 9.597 | 9.929 |
| CleverCSV 0.7.4 | 9.453 | 9.193 |
| Python native csv 3.10.5 | 9.436 | 9.721 |
| Pandas 1.4.3 | 9.431 | 9.895 |
| “SpreadWeb” | 9.431 | 9.721 |
| SQLite 3.39.0 | 9.375 | 9.955 |
| CSVCommons 1.9.0 | 9.253 | 6.647 |
| DuckDB 1.2 (auto-detect only) | 8.439 | 9.075 |
| UniVocity 2.9.1 | 7.936 | 9.939 |
| LibreOffice Calc 7.3.6 | 7.833 | 9.925 |
| OpenCSV 5.6 | 7.746 | 6.632 |
| MySQL 8.0.31 | 7.484 | 9.587 |
| MariaDB 10.9.3 | 7.483 | 9.585 |
| PostgreSQL 15.0 | 6.961 | 0.136 |
| R native csv 4.2.1 | 6.405 | 7.792 |
| Dataviz | 5.152 | 5.003 |
| Hypoparsr 0.1.0 | 4.372 | 3.888 |
Another approach is to use the [`time_bucket` function](https://duckdb.org/docs/1.5/sql/functions/timestamp#time_bucketbucket_width-timestamp-offset), which will truncate the timestamp to the bucket width provided, starting from the specified offset. For example, we calculate the number of services each quarter of an hour, starting with `00`:
```sql
SELECT
time_bucket(
INTERVAL 15 MINUTE, -- bucket width
station_service_time,
INTERVAL 0 MINUTE -- offset
) AS window_start,
window_start + INTERVAL 15 MINUTE as window_end,
count(*) AS number_of_services
FROM ams_traffic_v
WHERE year(station_service_time) = 2024
GROUP BY ALL
ORDER BY 1;
```
```text
┌─────────────────────┬─────────────────────┬────────────────────┐
│ window_start │ window_end │ number_of_services │
│ timestamp │ timestamp │ int64 │
├─────────────────────┼─────────────────────┼────────────────────┤
│ 2024-01-01 01:30:00 │ 2024-01-01 01:45:00 │ 1 │
│ 2024-01-01 01:45:00 │ 2024-01-01 02:00:00 │ 1 │
│ 2024-01-01 02:15:00 │ 2024-01-01 02:30:00 │ 2 │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ 2024-12-31 20:45:00 │ 2024-12-31 21:00:00 │ 2 │
│ 2024-12-31 21:00:00 │ 2024-12-31 21:15:00 │ 1 │
│ 2024-12-31 23:45:00 │ 2025-01-01 00:00:00 │ 2 │
├─────────────────────┴─────────────────────┴────────────────────┤
│ 32932 rows (6 shown) 3 columns │
└────────────────────────────────────────────────────────────────┘
```
> The time bucket function is generating the buckets from the timestamp column itself, there could be gaps in the time series data.
> As seen in the above result, the first record is `2024-01-01 01:30:00`, because there are no records before that timestamp.
Given that tumbling windows are non-overlapping intervals, we can calculate summaries, such as the average number of services during a 15 minute interval. It is interesting to observe that the number of train services is quite stable during the day but it's much lower during the night – even in Amsterdam.
#### Hopping Windows {#blog:2025-05-02-stream-windowing-functions::hopping-windows}
[_Hopping windows_](https://learn.microsoft.com/en-us/stream-analytics-query/hopping-window-azure-stream-analytics) are fixed-size time intervals, but, contrary to tumbling windows, are overlapping. A hopping window is defined by:
- how much time should elapse between the window start time, called **hopping size**;
- how much time should a window contain, called **window size**.
One use case for hopping windows is to identify the five busiest 15-minute periods (window size) during 2024, starting every 5 minutes (hopping size). We start by generating artificial hopping windows for all the dates we are interested in:
```sql
WITH time_range AS (
SELECT
range AS window_start,
window_start + INTERVAL 15 MINUTE AS window_end
FROM range(
'2024-01-01 00:00:00'::TIMESTAMP,
'2025-01-01 00:00:00'::TIMESTAMP,
INTERVAL 5 MINUTE -- hopping size
)
)
```
```text
┌─────────────────────┬─────────────────────┐
│ window_start │ window_end │
│ timestamp │ timestamp │
├─────────────────────┼─────────────────────┤
│ 2024-01-01 00:00:00 │ 2024-01-01 00:15:00 │
│ 2024-01-01 00:05:00 │ 2024-01-01 00:20:00 │
│ 2024-01-01 00:10:00 │ 2024-01-01 00:25:00 │
│ · │ · │
│ · │ · │
│ · │ · │
│ 2024-12-31 23:45:00 │ 2025-01-01 00:00:00 │
│ 2024-12-31 23:50:00 │ 2025-01-01 00:05:00 │
│ 2024-12-31 23:55:00 │ 2025-01-01 00:10:00 │
├─────────────────────┴─────────────────────┤
│ 105408 rows (6 shown) 2 columns │
└───────────────────────────────────────────┘
```
We then join the above intervals with the train service data in order to calculate the number of services for each **[left-closed, right-open)** interval:
```sql
SELECT
window_start,
window_end,
count(service_sk) AS number_of_services
FROM ams_traffic_v
INNER JOIN time_range AS ts
ON station_service_time >= ts.window_start
AND station_service_time < ts.window_end
GROUP BY ALL
ORDER BY 3 DESC, 1 ASC
LIMIT 5;
```
resulting in:
```text
┌─────────────────────┬─────────────────────┬────────────────────┐
│ window_start │ window_end │ number_of_services │
│ timestamp │ timestamp │ int64 │
├─────────────────────┼─────────────────────┼────────────────────┤
│ 2024-02-17 10:25:00 │ 2024-02-17 10:40:00 │ 28 │
│ 2024-02-17 11:25:00 │ 2024-02-17 11:40:00 │ 28 │
│ 2024-02-17 16:25:00 │ 2024-02-17 16:40:00 │ 28 │
│ 2024-02-17 09:25:00 │ 2024-02-17 09:40:00 │ 27 │
│ 2024-02-17 12:25:00 │ 2024-02-17 12:40:00 │ 27 │
└─────────────────────┴─────────────────────┴────────────────────┘
```
Can you imagine how it must have been like in the control room when within 15 minutes, 28 trains were arriving or departing in a station with 15 tracks?
> By applying a `RIGHT OUTER JOIN` in the above query, gaps are filled with 0 number of services.
#### Sliding Windows {#blog:2025-05-02-stream-windowing-functions::sliding-windows}
[_Sliding windows_](https://learn.microsoft.com/en-us/stream-analytics-query/sliding-window-azure-stream-analytics) are overlapping intervals, but, compared to hopping windows, they are dynamically generated from the time column analyzed, therefore changing when new records are inserted.
Sliding windows can be implemented by using the [`RANGE` window framing](https://duckdb.org/docs/1.5/sql/functions/window_functions#framing):
```sql
SELECT
station_service_time - INTERVAL 15 MINUTE AS window_start, -- window size
station_service_time AS window_end,
count(service_sk) OVER (
ORDER BY station_service_time
RANGE
BETWEEN INTERVAL 15 MINUTE PRECEDING -- window size
AND CURRENT ROW
) AS number_of_services
FROM ams_traffic_v
ORDER BY 3 DESC, 1
LIMIT 5;
```
```text
┌─────────────────────┬─────────────────────┬────────────────────┐
│ window_start │ window_end │ number_of_services │
│ timestamp │ timestamp │ int64 │
├─────────────────────┼─────────────────────┼────────────────────┤
│ 2024-02-17 11:25:00 │ 2024-02-17 11:40:00 │ 29 │
│ 2024-02-17 10:24:00 │ 2024-02-17 10:39:00 │ 28 │
│ 2024-02-17 11:18:00 │ 2024-02-17 11:33:00 │ 28 │
│ 2024-02-17 11:18:00 │ 2024-02-17 11:33:00 │ 28 │
│ 2024-02-17 11:23:00 │ 2024-02-17 11:38:00 │ 28 │
└─────────────────────┴─────────────────────┴────────────────────┘
```
> Because the current row is included in the calculation, the sliding windows are **[left-closed, right-closed]**.
#### Session Windows {#blog:2025-05-02-stream-windowing-functions::session-windows}
A [_session window_](https://learn.microsoft.com/en-us/stream-analytics-query/session-window-azure-stream-analytics) groups events that happen close together in time, separated by inactivity gaps. A new session starts when the time between two events exceeds a defined timeout. The most common use case of session windows is to detect gaps in the timestamped data.
We continue the data analysis by identifying the days in which there were periods of time larger than 10 minutes in which no train was arriving/departing in/from the Amsterdam Centraal Station. In this context, a session window is the period of time in which train services run without a service inactivity gap longer than 10 minutes.
We start by calculating, for each record, the previous service time, by using the [`lag` window function](https://duckdb.org/docs/1.5/sql/functions/window_functions#lagexpr-offset-default-order-by-ordering-ignore-nulls). We observed above that there is almost no traffic during the night, therefore we include only services between 6 AM and 11 PM:
```sql
SELECT
service_sk,
station_service_time,
lag(station_service_time) OVER (
PARTITION BY station_service_time::DATE
ORDER BY station_service_time
) AS previous_service_time,
date_diff('minute', previous_service_time, station_service_time) AS gap_minutes
FROM ams_traffic_v
WHERE hour(station_service_time) BETWEEN 6 AND 23
```
In the above query we also calculate the gap, in minutes, between the current service and the previous service, with [`date_diff`](https://duckdb.org/docs/1.5/sql/functions/time#date_diffpart-starttime-endtime). If there is no previous service, the column will be `NULL`, depicting the first service session in the day:
```text
┌──────────────────────┬───────────────────────┬─────────────┐
│ station_service_time │ previous_service_time │ gap_minutes │
│ timestamp │ timestamp │ int64 │
├──────────────────────┼───────────────────────┼─────────────┤
│ 2024-01-09 06:00:00 │ NULL │ NULL │
│ 2024-01-16 06:00:00 │ NULL │ NULL │
│ 2024-01-22 06:00:00 │ NULL │ NULL │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ 2024-11-28 06:01:00 │ NULL │ NULL │
│ 2024-12-05 06:01:00 │ NULL │ NULL │
│ 2024-12-23 06:00:00 │ NULL │ NULL │
├──────────────────────┴───────────────────────┴─────────────┤
│ 366 rows (6 shown) 3 columns │
└────────────────────────────────────────────────────────────┘
```
> **Tip.** Because `gap_minutes` is computed based on a window function, we can filter on it with `QUALIFY`, e.g.: `QUALIFY gap_minutes IS NULL`
We then mark if the current record is in the same session as the previous one, by comparing the minutes elapsed to a timeout, in our case 10 minutes:
```sql
IF(gap_minutes >= 10 OR gap_minutes IS NULL, 1, 0) AS new_session
```
By applying a moving sum, at day level, over the `new_session` attribute, we assign an identifier to the session:
```sql
sum(new_session) OVER (
PARTITION BY station_service_date
ORDER BY station_service_time ROWS UNBOUNDED PRECEDING
) AS session_id_in_day
```
Bringing it all together, we can now retrieve the dates which had at least one inactivity gap of 10 minutes during the 18 hours day service time (the number of hours between 6 AM and 11 PM):
```sql
WITH ams_daily_traffic AS (
SELECT
service_sk,
station_service_time,
lag(station_service_time) OVER (
PARTITION BY station_service_time::DATE
ORDER BY station_service_time
) AS previous_service_time,
date_diff('minute', previous_service_time, station_service_time) AS gap_minutes
FROM ams_traffic_v
WHERE hour(station_service_time) BETWEEN 6 AND 23
), window_calculation AS (
SELECT
service_sk,
station_service_time,
station_service_time::DATE AS station_service_date,
gap_minutes,
IF(gap_minutes >= 10 OR gap_minutes IS NULL, 1, 0) new_session,
sum(new_session) OVER (
PARTITION BY station_service_date
ORDER BY station_service_time ROWS UNBOUNDED PRECEDING
) AS session_id_in_day
FROM ams_daily_traffic
), session_window AS (
SELECT
station_service_date,
session_id_in_day,
max(gap_minutes) AS gap_minutes,
min(station_service_time) AS window_start,
max(station_service_time) AS window_end,
count(service_sk) AS number_of_services
FROM window_calculation
GROUP BY ALL
)
SELECT
station_service_date,
max(ceil(date_diff('minute', window_start, window_end) / 60)) AS number_of_hours_without_gap,
count(*) AS number_of_sessions,
sum(number_of_services) as number_of_services,
FROM session_window
GROUP BY ALL
HAVING number_of_hours_without_gap < 18
ORDER BY 2, 1;
```
```text
┌──────────────────────┬─────────────────────────────┬────────────────────┬────────────────────┐
│ station_service_date │ number_of_hours_without_gap │ number_of_sessions │ number_of_services │
│ date │ double │ int64 │ int128 │
├──────────────────────┼─────────────────────────────┼────────────────────┼────────────────────┤
│ 2024-04-29 │ 7.0 │ 12 │ 521 │
│ 2024-12-31 │ 14.0 │ 6 │ 946 │
│ 2024-01-01 │ 16.0 │ 6 │ 847 │
│ 2024-04-30 │ 16.0 │ 7 │ 645 │
│ 2024-04-14 │ 17.0 │ 3 │ 1289 │
│ 2024-05-01 │ 17.0 │ 5 │ 788 │
│ 2024-05-02 │ 17.0 │ 3 │ 729 │
│ 2024-05-03 │ 17.0 │ 5 │ 699 │
│ 2024-05-04 │ 17.0 │ 3 │ 907 │
│ 2024-05-19 │ 17.0 │ 3 │ 837 │
│ 2024-10-28 │ 17.0 │ 2 │ 748 │
│ 2024-10-29 │ 17.0 │ 2 │ 785 │
│ 2024-10-30 │ 17.0 │ 2 │ 783 │
│ 2024-11-02 │ 17.0 │ 2 │ 654 │
├──────────────────────┴─────────────────────────────┴────────────────────┴────────────────────┤
│ 14 rows 4 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
```
Something must have happened on 29 April 2024! We observe that, during the 18 hours of service, there were 12 session windows, which means that, for at least 10 times, no train arrived or departed during a period of 10 minutes. A reason for this could be that a regular train service was not running on that day. And indeed maintenance work started between [Amsterdam and Utrecht](https://www.rijdendetreinen.nl/werkzaamheden/13161-29-april-2024-werkzaamheden-amsterdam-utrecht-amsterdam-weesp).
> **Tip.** Time windows are visualized with [Plotly timeline charts](https://plotly.com/python/gantt/#gantt-charts-and-timelines-with-plotlyexpress), a type of Gantt charts.
#### Conclusion {#blog:2025-05-02-stream-windowing-functions::conclusion}
In this post we have demonstrated how stream windowing functions can be implemented on historical timestamped data in DuckDB, offering a starting point in time (series) data analysis. We also recommend [“Catching up with Windowing”](#blog:2025-02-10-window-catchup), a post about DuckDB's windowing features, which can be adopted in the functions presented herein.
## Sorting on Insert for Fast Selective Queries {#blog:2025-05-14-sorting-for-fast-selective-queries}
**Publication date:** 2025-05-14
**Author:** Alex Monahan
**TL;DR:** Sorting data when loading can speed up selective read queries by an order of magnitude, thanks to DuckDB's automatic min-max indexes (also known as zone maps). This approach applies to most columnar file formats and databases as well. This post unpacks the DuckDB file structure as an example of a columnar data format and gives practical advice for using sorting to improve the speed of queries.
The fastest way to read data is to not read data.
It's as simple as that!
This post is all about how to read as little data as possible to answer selective read queries.
> Feel free to skip straight to the [best practices](#blog:2025-05-14-sorting-for-fast-selective-queries::sorting-best-practices)!
#### Use Cases {#blog:2025-05-14-sorting-for-fast-selective-queries::use-cases}
The cases where these techniques are most useful are when read performance is more critical than write performance.
In these cases, we can deliberately add ordering steps that will slow down our writes, but will greatly speed up our reads.
This is often the case when data is pre-processed in the background, but read queries are customer facing in a dashboard or app.
Using sorting will be helpful in any of these situations:
- Your dataset is large and doesn't fit entirely in memory
- You only want to read a portion of your dataset for each query
- You access your data via HTTP(S)
- Your data lives in the cloud on object storage such as AWS S3
This overview describes the DuckDB file format, but thanks to DuckDB's [partial reading support](https://duckdb.org/docs/1.5/core_extensions/httpfs/https#partial-reading), these techniques can be generally applied to nearly any columnar file format or database.
This is a great way to speed up querying Apache Parquet files on remote endpoints – including data lakes!
> Stay tuned for a subsequent post covering advanced multi-column sorting!
#### How Does the DuckDB Format Help? {#blog:2025-05-14-sorting-for-fast-selective-queries::how-does-the-duckdb-format-help}
Let's build up some intuition around the structure of the DuckDB file format!
DuckDB stores data in a single file.
Each file is called a _database_ (the DuckDB library is the _database engine_).
Each database file can store multiple tables, views, functions, indexes, and primary/foreign key relationships all in the same file.
This has some key advantages for portability, but it also allows for the storage of more than just data – metadata can be stored as well.
This metadata allows for the DuckDB engine to selectively read just portions of the DuckDB file as they are requested, which is key for handling larger than memory datasets and for the performance optimizations in this post!
DuckDB stores data in a columnar fashion (meaning that values within a column are stored together in the same set of blocks).
However, columnar does not mean storing the entire column contiguously!
Before storing data, DuckDB breaks tables up into chunks of rows called row groups.
Each row group is 122 880 rows by default.
Next, we zoom in on `Row Group 1` from `Table 1` in the first diagram:
Within each row group, the data related to a single column is stored contiguously together on disk in one or more blocks.
DuckDB compresses this data to reduce file size.
##### Using Zone Maps to Skip Reading Data {#blog:2025-05-14-sorting-for-fast-selective-queries::using-zone-maps-to-skip-reading-data}
So, how does storing data in columns help with selective read queries?
By itself, it does not!
However, at the start of the column data stored within each row group, DuckDB also stores metadata about the column data being stored.
This includes the minimum and maximum values of that column within that row group.
We call these _zone maps_ or _min-max indexes._
When DuckDB receives a SQL query that contains a filter, before reading the column segments off of disk, it checks the metadata first.
Could this filter value possibly fall within the minimum/maximum range of this column segment?
If it is not possible, then DuckDB can skip reading the data in that entire row group.
For example, `Column N` in this example is storing the abbreviations for US states.
If we are querying data for the state of `VA` (Virginia), could this row group contain any data for `VA`?
Our query could be:
```sql
FROM "Table 1"
WHERE "Column N" = 'VA';
```
> DuckDB's [friendly SQL](https://duckdb.org/docs/1.5/sql/dialect/friendly_sql) makes the traditional SQL `SELECT *` optional!
DuckDB first checks the zone map (labeled `Column N Metadata` in the diagram).
Does `VA` fall within the range (alphabetically) of `AZ` to `PA`?
No it does not!
We can skip this entire row group.
If instead we were filtering for the state of `NM` (New Mexico), the zone map indicates that it is possible that `NM` data exists within this row group.
As a result, DuckDB will retrieve the entire column of data (possibly from a remote location) and check each row to see if any `NM` data exists.
According to the diagram, there is no `NM` data in this row group, so retrieving it and checking each row for a match would be unnecessary effort.
All 122 880 rows would need to be checked!
If only the zone map index did not cover such a large range of state abbreviations...
##### Strategically Skipping Data {#blog:2025-05-14-sorting-for-fast-selective-queries::strategically-skipping-data}
Our goal is to sort our data so that the zone maps are as selective as possible for the columns we are interested in filtering.
If the min-max index of this row group only extended from `AZ` to `CA`, then the data would be far more likely to be skipped.
Said another way, each subset of the data we want to retrieve should only be stored in a few row groups.
We don't want to pull every row group and check every row!
Hopefully states that are early in the alphabet are all grouped together in the same row group, for example.
When executing selective queries, you could aim for pulling just a single row group!
However, since DuckDB's multithreading model is based on row groups, you should still see high performance as long as the number of row groups is less than the number of threads (≈CPU cores) DuckDB is using.
#### Sorting Best Practices {#blog:2025-05-14-sorting-for-fast-selective-queries::sorting-best-practices}
There are many rules of thumb for how to best utilize min-max indexes, since their effectiveness is highly data and workload dependent.
These are useful both in DuckDB as well as in other columnar formats like Apache Parquet or other columnar databases.
Here are several approaches to consider!
When deciding which columns to sort, it is critical to examine the `WHERE` clauses of any read workloads.
Often the most important factor is which columns are used to filter most often.
A basic approach would be to sort by all columns that are used as filters, beginning with the columns used most often.
Another option if the workload involves filtering by multiple different columns is to consider sorting first by columns with the lowest cardinality (the fewest unique values).
For example, it may be helpful to sort by a broad `customer_type` prior to a unique `customer_id`.
It is often useful to only read recent data, so ensuring recent data is not spread throughout the table can improve performance.
However, when ordering by timestamps, be mindful that timestamps are frequently very high cardinality.
As a result, sorting by lower cardinality columns first may be helpful.
You may be tempted to sort by timestamp first – **resist that temptation!** (or at least test out alternatives).
It may be more beneficial to sort by a timestamp rounded to the nearest week, month, or year and then by other columns.
> To benefit from min-max indexes, a `WHERE` clause must filter directly on a specific column, not on a calculated expression.
> If an expression is used, it must be evaluated for each row, so no row groups can be skipped!
#### Additional Techniques {#blog:2025-05-14-sorting-for-fast-selective-queries::additional-techniques}
There are more ways to take full advantage of the min-max indexes in DuckDB!
##### Avoid Small Inserts {#blog:2025-05-14-sorting-for-fast-selective-queries::avoid-small-inserts}
If a workload is inserting data in small batches or a single row at a time, there is not an opportunity to effectively sort the data when it is being inserted.
Instead, the data will be sorted largely by insertion time, which will only provide effective pruning for time-based filters.
If possible, bulk inserts or batching will allow the sorting to work more effectively for other columns.
As an alternative, there can be a periodic re-sorting job, which is analogous to a re-indexing task in transactional systems.
##### Sort in Chunks {#blog:2025-05-14-sorting-for-fast-selective-queries::sort-in-chunks}
Sorting can be a computationally expensive operation for large tables.
One way to reduce the amount of memory (or disk spill) required when sorting is to process the table in pieces by looping through multiple SQL statements, each filtered to a specific chunk.
Since SQL does not have a looping construct, this would be handled by a host language (Python, Jinja templating, etc.).
The pseudocode would be to:
```sql
CREATE OR REPLACE TABLE sorted_table AS
FROM unsorted_table
WITH NO DATA;
for chunk in chunks:
INSERT INTO sorted_table
FROM unsorted_table
WHERE chunking_column = chunk
ORDER BY other_columns...;
```
This will have the effect of sorting initially by the chunking column, and then by the `other_columns`.
It may also take longer to run (since the data must be scanned once per chunk), but memory use is likely to be much lower.
##### Sort the First Few Characters of Strings {#blog:2025-05-14-sorting-for-fast-selective-queries::sort-the-first-few-characters-of-strings}
Approximate sorting works well for improving read performance.
In the zone maps of `VARCHAR` columns, DuckDB stores just the first 8 bytes of the min and max string values.
As a result, there is no need to sort more than the first 8 bytes (8 ASCII characters)!
This has an added benefit of faster sorting, as the runtime of [DuckDB's radix sort algorithm](#blog:2021-08-27-external-sorting) is sensitive to the length of strings (by design!).
The time complexity of the algorithm is `O(nk)`, where `n` is the number of rows, and `k` is the width of the sorting key.
Sorting by just the first few characters of a `VARCHAR` can be quicker and less compute intensive while achieving similar read performance.
DuckDB's `VARCHAR` data type also inlines strings when they are under 12 bytes, so working with short strings is faster for that reason as well.
For example:
```sql
CREATE OR REPLACE TABLE sorted_table AS
FROM unsorted_table
ORDER BY varchar_column_to_sort[:8];
```
> DuckDB's [friendly SQL](https://duckdb.org/docs/1.5/sql/dialect/friendly_sql#functions-and-expressions) allows bracket notation for string slicing!
##### Filter by More Columns {#blog:2025-05-14-sorting-for-fast-selective-queries::filter-by-more-columns}
Adding filters to a `WHERE` clause can be helpful if those columns being filtered have any kind of approximate order.
For example, instead of just filtering by `customer_id`, if the table is sorted by `customer_type`, include that in the query also.
Often, if the `customer_id` is known at query time, it is possible to know other metadata as well.
##### Adjust the Row Group Size {#blog:2025-05-14-sorting-for-fast-selective-queries::adjust-the-row-group-size}
One parameter that can be tuned for specific workloads is the number of rows within a row group (the `ROW_GROUP_SIZE`).
If there are many unique values within a column being filtered on, then a smaller number of rows per row group could reduce the total number of rows that must be scanned.
However, there is an overhead of checking metadata more often when row groups are smaller, so there is a tradeoff.
A larger row group size may actually be preferable if a table is particularly large and queries are very selective.
For example, if querying a large fact table with years of history, but filtering to only the last week of data.
Larger row group sizes reduce the number of metadata checks that are necessary to reach the recent data.
However, each row group is larger, so there is a tradeoff there as well.
To adjust the row group size, pass in a parameter when attaching a database.
Note that a row group size should be a power of 2.
The minimum row group size is 2 048, the vector size of DuckDB.
```sql
ATTACH './smaller_row_groups.duckdb' (ROW_GROUP_SIZE 8_192);
```
#### Sneak Peak of the Next Post in the Series {#blog:2025-05-14-sorting-for-fast-selective-queries::sneak-peak-of-the-next-post-in-the-series}
A subsequent post in this series will cover advanced sorting techniques.
These can boost performance if multiple reader workloads with different filters are querying the same dataset.
For example, what if I need to filter by US state for some queries, but by customer last name for others?
Additionally, sorting by rounded time buckets (day, month, or year) and then by other columns can be helpful when only recent data is needed.
Microbenchmarks will show the 10× performance benefits that were mentioned at the start of this post!
The post will also provide ways to measure how effectively a table is sorted along various columns.
#### Conclusion {#blog:2025-05-14-sorting-for-fast-selective-queries::conclusion}
Ordering data upon insert can significantly speed up read queries that include filters.
Once your dataset becomes large or you are storing it remotely, consider applying these techniques.
Plus, these approaches can be used in nearly any columnar file format or database!
Happy analyzing!
## Machine Learning Prototyping with DuckDB and scikit-learn {#blog:2025-05-16-scikit-learn-duckdb}
**Publication date:** 2025-05-16
**Author:** Petrica Leuca
**TL;DR:** In this post, we prototype a machine learning workflow using DuckDB for data handling and scikit-learn for modeling.
#### Introduction {#blog:2025-05-16-scikit-learn-duckdb::introduction}
Machine learning prototyping often involves juggling datasets, preprocessing steps, and performance constraints, which can make the process both complex and time-consuming. [scikit-learn](https://scikit-learn.org/stable/) is one of Python's most popular and powerful libraries for machine learning, offering a large set of utilities for building and evaluating models. In this post, we will explore how DuckDB complements `scikit-learn` in the model development life cycle, by implementing a penguin species prediction model on Penguins observations in the Palmer Archipelago.
> The following implementation is executed in a [marimo Python notebook](https://marimo.io/), which is available on GitHub, in our [examples repository](https://github.com/duckdb/duckdb-blog-examples/tree/main/scikit_learn_duckdb).
#### Data Preparation {#blog:2025-05-16-scikit-learn-duckdb::data-preparation}
We start by loading the [Palmer Penguins dataset](https://github.com/allisonhorst/palmerpenguins/) and use DuckDB's [`COLUMNS(*)` expression](https://duckdb.org/docs/1.5/sql/expressions/star#columns) to filter out any record which contains `NA` or `NULL`:
```python
duckdb_conn.read_csv(
"https://blobs.duckdb.org/data/penguins.csv"
).filter(
"columns(*)::text != 'NA'"
).filter(
"columns(*) is not null"
).select(
"*, row_number() over () as observation_id"
).to_table(
"penguins_data"
)
```
Even if the `NA` values are removed from the dataset, DuckDB has inferred the column type as `VARCHAR` at the moment of `read_csv`. Therefore, we modify the column type to a numeric type, `DECIMAL(5, 2)`, using the [`ALTER TABLE`](https://duckdb.org/docs/1.5/sql/statements/alter_table#set-data-type) statement:
```python
duckdb_conn.sql(
"alter table penguins_data alter bill_length_mm set data type decimal(5, 2)"
)
```
> **Tip.** One can also [define the schema](https://duckdb.org/docs/1.5/data/csv/tips#override-the-types-of-specific-columns) at import time.
We can now plot the data and check for species specific clusters. For example, using a [scatter plot](https://plotly.com/python/line-and-scatter/) we identify clusters at the combination between bill depth and bill length:
We also observe that there are a few descriptive columns in the dataset, such as `species` and `island`. In a machine learning workflow, one common preprocessing step is to transform such values into numerical values, assigning them a unique identifier; a process called _label encoding_. While scikit-learn offers a [LabelEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) utility, this process is very similar to how we work with [reference tables](https://roelantvos.com/blog/designing-reference-tables-for-the-data-warehouse/) in a data warehouse. Therefore, we define a function which will create, for each column (feature), a reference table, leveraging the DuckDB Python [relational API](https://duckdb.org/docs/1.5/clients/python/relational_api):
```python
def process_reference_data(duckdb_conn):
for feature in ["species", "island"]:
duckdb_conn.sql(f"drop table if exists {feature}_ref")
(
duckdb_conn.table("penguins_data")
.select(feature)
.unique(feature)
.row_number(
window_spec=f"over (order by {feature})",
projected_columns=feature
)
.select(f"{feature}, #2 - 1 as {feature}_id")
.to_table(f"{feature}_ref")
)
duckdb_conn.table(f"{feature}_ref").show()
```
After the execution of the above function, two tables which will contain the distinct values of the category and a unique identifier are created, e.g., `species_ref`:
```text
┌───────────┬────────────┐
│ species │ species_id │
│ varchar │ int64 │
├───────────┼────────────┤
│ Adelie │ 0 │
│ Chinstrap │ 1 │
│ Gentoo │ 2 │
└───────────┴────────────┘
```
The last step in our data preparation step is to define a selection query variable, which selects the data from the initial dataset and the reference tables:
```python
selection_query = (
conn.table("penguins_data")
.join(conn.table("island_ref"), condition="island")
.join(conn.table("species_ref"), condition="species")
)
```
> `selection_query` is lazy evaluated, therefore, no data is retrieved at the definition time.
#### Model Training {#blog:2025-05-16-scikit-learn-duckdb::model-training}
Our scope is to predict a penguin's specie based on its characteristics (features), such as bill length and depth, flipper length, body mass and island. Such a model is called a classification model because it predicts the category to which the input data belongs.`scikit-learn` offers [multiple classification models](https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html). For our dataset, we have chosen to implement a [random forest classifier](https://en.wikipedia.org/wiki/Random_forest), based on decision trees:
We start by splitting the data into train and test data, with the [`train_test_split` utility](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) from `scikit-learn`:
```python
def train_split_data(selection_query):
X_df = selection_query.select("""
bill_length_mm,
bill_depth_mm,
flipper_length_mm,
body_mass_g,
island_id,
observation_id,
species_id
""").order("observation_id").df()
y_df = [
x[0]
for x in selection_query.order("observation_id").select("species_id").fetchall()
]
num_test = 0.30
return train_test_split(X_df, y_df, test_size=num_test)
X_train, X_test, y_train, y_test = train_split_data(selection_query)
```
Splitting the data is a common step in a machine learning workflow and it returns:
- `X_train`, containing the input data based on which we want to assign a species category;
- `y_train`, containing the species for each record in `X_train`;
- `X_test`, containing the input data based on which we will test the model;
- `y_test`, containing the species for each record in `X_test`.
We then define the [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) and fit the `X_train` and `y_train` data to it. We save the model in a [pickle](https://docs.python.org/3/library/pickle.html) file, such that we can use the model without retraining:
```python
model = RandomForestClassifier(n_estimators=1, max_depth=2, random_state=5)
model.fit(X_train.drop(["observation_id", "species_id"], axis=1).values, y_train)
pickle.dump(model, open("./model/penguin_model.sav", "wb"))
```
We can now check the model accuracy score:
```python
model.score(
X_test.drop(["observation_id", "species_id"], axis=1).values, y_test
)
```
```text
0.98
```
> We are working with `pickle` due to the small size of our model. Other persistence methods are detailed on [the `scikit-learn` documentation page](https://scikit-learn.org/stable/model_persistence.html).
#### Inference with DuckDB {#blog:2025-05-16-scikit-learn-duckdb::inference-with-duckdb}
Inference is the process of using the model to get predictions on (new) data:
```python
model = pickle.load(open("./model/penguin_model.sav", "rb"))
model.predict(...)
```
When working with DuckDB there are three approaches to retrieve the predicted category from the data. We can use Pandas, or a DuckDB Python UDF with or without batching. In the following, we present these approaches.
##### With Pandas {#blog:2025-05-16-scikit-learn-duckdb::with-pandas}
The most common method of retrieving predictions is to load the predictions into a new column in a Pandas dataframe:
```python
predicted_df = selection_query.select("""
bill_length_mm,
bill_depth_mm,
flipper_length_mm,
body_mass_g,
island_id,
observation_id,
species_id
""").df()
predicted_df["predicted_species_id"] = model.predict(
predicted_df.drop(["observation_id", "species_id"], axis=1).values
)
```
With DuckDB, the dataframe can be [queried with SQL](https://duckdb.org/docs/1.5/guides/python/import_pandas):
```python
(
duckdb_conn.table("predicted_df")
.select("observation_id", "species_id", "predicted_species_id")
.filter("species_id != predicted_species_id")
)
```
resulting in:
```text
┌────────────────┬────────────┬──────────────────────┐
│ observation_id │ species_id │ predicted_species_id │
│ int64 │ int64 │ int64 │
├────────────────┼────────────┼──────────────────────┤
│ 13 │ 1 │ 2 │
│ 15 │ 1 │ 2 │
│ 39 │ 1 │ 2 │
│ 44 │ 1 │ 2 │
│ 68 │ 1 │ 2 │
│ 70 │ 1 │ 2 │
│ 76 │ 1 │ 2 │
│ 90 │ 1 │ 2 │
│ 94 │ 1 │ 2 │
│ 104 │ 1 │ 2 │
│ 106 │ 1 │ 2 │
│ 110 │ 1 │ 2 │
│ 124 │ 1 │ 2 │
│ 126 │ 1 │ 2 │
│ 243 │ 3 │ 2 │
│ 296 │ 2 │ 1 │
├────────────────┴────────────┴──────────────────────┤
│ 16 rows 3 columns │
└────────────────────────────────────────────────────┘
```
> **Warning.** If there is a table which has the same name as the dataframe, then `register` can be used to give another table name to the dataframe, e.g., `duckdb_conn.register("table_name", predicted_df)`
##### DuckDB Python UDF, Row by Row {#blog:2025-05-16-scikit-learn-duckdb::duckdb-python-udf-row-by-row}
DuckDB offers the possibility to register [User Defined Functions (UDFs)](https://duckdb.org/docs/1.5/clients/python/function) from Python functions. Because the UDF is executed row by row, our prediction function will return the predicted species id for each row:
```python
def get_prediction_per_row(
bill_length_mm: Decimal,
bill_depth_mm: Decimal,
flipper_length_mm: int,
body_mass_g: int,
island_id: int
) -> int:
model = pickle.load(open("./model/penguin_model.sav", "rb"))
return int(
model.predict(
[
[
bill_length_mm,
bill_depth_mm,
flipper_length_mm,
body_mass_g,
island_id,
]
]
)[0]
)
```
In the above Python function we provide the required features as input for the model and return the predicted value (an `integer`). With this information, we create the DuckDB function:
```python
duckdb_conn.create_function(
"predict_species_per_row", get_prediction_per_row, return_type=int
)
```
We can now use the UDF in the SQL `SELECT` clause:
```python
selection_query.select("""
observation_id,
species_id,
predict_species_per_row(
bill_length_mm,
bill_depth_mm,
flipper_length_mm,
body_mass_g,
island_id
) as predicted_species_id
""").filter("species_id != predicted_species_id")
```
This returns the same results as above.
##### DuckDB Python UDF, Batch Style {#blog:2025-05-16-scikit-learn-duckdb::duckdb-python-udf-batch-style}
While the row-by-row prediction can be useful when the scope is to retrieve the prediction for a limited number of rows, it will be less performant when working with more data. Therefore, by aggregating the data into arrays (which preserve order), we can simulate a mass retrieval (or batch style) of the prediction.
We first create a Python function to get the predictions for a JSON object, which contains the columnar representation of the features needed as model input:
```python
def get_prediction_per_batch(input_data: dict[str, list[Decimal | int ]]) -> np.ndarray:
"""
input_data example:
{
"bill_length_mm": [40.5],
"bill_depth_mm": [41.5],
"flipper_length_mm: [250],
"body_mass_g": [3000],
"island_id": [1]
}
"""
model = pickle.load(open("./model/penguin_model.sav", "rb"))
input_data_parsed = orjson.loads(input_data)
input_data_converted_to_numpy = np.stack(tuple(input_data_parsed.values()), axis=1)
return model.predict(input_data_converted_to_numpy)
duckdb_conn.create_function(
"predict_species_per_batch",
get_prediction_per_batch,
return_type=duckdb.typing.DuckDBPyType(list[int]),
)
```
> While DuckDB has `MAP` and `STRUCT` data types, which convert automatically to a dict, they have a slower execution time compared to `json_object` (including the `orjson` deserialization time).
With `json_object` we extract the columnar representation of the features, in the format `'feature_name': array[feature]`:
```python
json_object(
'bill_length_mm', array_agg(bill_length_mm),
'bill_depth_mm', array_agg(bill_depth_mm),
'flipper_length_mm', array_agg(flipper_length_mm),
'body_mass_g', array_agg(body_mass_g),
'island_id', array_agg(island_id)
) as input_data,
```
We then pack in a [`STRUCT`](https://duckdb.org/docs/1.5/sql/data_types/struct), the predictions together with other columns we are interested in:
```python
struct_pack(
observation_id := array_agg(observation_id),
species_id := array_agg(species_id),
predicted_species_id := predict_species_per_batch(input_data)
) as output_data
```
Last, we [`unnest`](https://duckdb.org/docs/1.5/sql/query_syntax/unnest) the results, in order to flatten the lists into tables:
```python
.select("""
unnest(output_data.observation_id) as observation_id,
unnest(output_data.species_id) as species_id,
unnest(output_data.predicted_species_id) as predicted_species_id
""")
```
The above query is wrapped into a Python function, named `get_selection_query_for_batch`, with which we can chain queries, e.g., for mass retrieval of incorrect predictions:
```python
get_selection_query_for_batch(selection_query).filter("species_id != predicted_species_id")
```
A batch approach can be implemented by using [`LIMIT` and `OFFSET`](https://duckdb.org/docs/1.5/sql/query_syntax/limit) to loop through the data:
```python
for i in range(4):
(
get_selection_query_for_batch(
selection_query
.order("observation_id")
.limit(100, offset=100*i)
.select("*")
)
.filter("species_id != predicted_species_id")
).show()
```
> `LIMIT` and `OFFSET` are executed last, therefore, they should be applied before the prediction selection.
##### Performance Considerations {#blog:2025-05-16-scikit-learn-duckdb::performance-considerations}
To obtain performance data on a larger scale dataset, we have generated a dummy dataset with approximately 59 million records. On a [sample](https://duckdb.org/docs/1.5/sql/samples) of 10%, on a 16 GB MacBook Pro, the batch processing ranges between 3 and 4 seconds, while the Pandas implementation is executed under 1 second. This is because the Python UDF includes multiple conversion steps which affect performance:
- parsing the input data as JSON;
- converting the JSON to numpy array;
- unnesting the array to rows.
We have still chosen to show-case the Python UDFs, because they are a powerful asset when working within a Python environment and the performance difference is negligible when we speak of small data.
#### Conclusion {#blog:2025-05-16-scikit-learn-duckdb::conclusion}
In this post, we have demonstrated how DuckDB complements `scikit-learn` in the machine learning development life cycle, with a focus on the data preparation and inference phases. The inference results on the dummy data are poor; is this because of the model or the dummy data? We leave this question open to our readers, as a means to explore how DuckDB can be used in the model evaluation and model optimization phases.
## The Lost Decade of Small Data? {#blog:2025-05-19-the-lost-decade-of-small-data}
**Publication date:** 2025-05-19
**Author:** Hannes Mühleisen
**TL;DR:** We benchmark DuckDB on a 2012 MacBook Pro to decide: did we lose a decade chasing distributed architectures for data analytics?
Much has been said, not in the very least by ourselves, about how [data is actually not that “Big”](https://youtu.be/-wCzn9gKoUk?si=npV0awLO3nQZvrlS&t=2637) and how the speed of hardware innovation is outpacing the growth of useful datasets. We may have gone so far to [predict a data singularity in the near future](https://youtu.be/GELhdezYmP0?si=Rj3JZfoKW8JQDaci&t=2494), where 99% of useful datasets can be comfortably queried on a single node. As it was [recently shown](https://www.fivetran.com/blog/how-do-people-use-snowflake-and-redshift), the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think.
But we started wondering, when did this development really start? When did personal computers like the ubiquitous MacBook Pro, usually condemned to running Chrome, become the data processing powerhouses that they really are today?
Let's turn our attention to the **2012** [Retina MacBook Pro](https://en.wikipedia.org/wiki/MacBook_Pro#Retina_(2012–2016)), a computer many people (myself included) bought at the time because of its gorgeous “Retina” display. Millions were sold. Despite being unemployed at the time, I had even splurged for the 16 GB RAM upgrade. But there was another often-forgotten revolutionary change in this machine: it was the first MacBook with a built-in Solid-State Disk (SSD) _and_ a competitive [4-core 2.6 GHz “Core i7” CPU](https://everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-i7-2.6-15-mid-2012-retina-display-specs.html). It's funny to watch the [announcement](https://www.youtube.com/watch?v=QNcbZDQKwSQ) again, where they _do_ stress the performance aspect of the “all-flash architecture” as well.
Side note: the MacBook Air was actually the first MacBook with an (optional) built-in SSD already back in 2008. But it did not have the CPU firepower of the Pro, sadly.
Coincidentally, I still have this laptop in the [DuckLabs](https://ducklabs.com/) office, currently used by my kids to type their names in a massive font size or watch Bluey on YouTube when they're around. But can this relic still run modern-day DuckDB? How will its performance compare to modern MacBooks? And could we have had the data revolution that we are seeing now already back in 2012? Let's find out!
#### Software {#blog:2025-05-19-the-lost-decade-of-small-data::software}
First, what about the operating system? In order to make the comparison fair(er) to the decades, we actually _downgraded_ the operating system on the Retina to OS X 10.8.5 “Mountain Lion”, the operating system version that shipped just a few weeks after the laptop itself in July 2012. Even though the Retina can actually run 10.15 (Catalina), we felt a true 2012 comparison should also use an operating system from the era. Below is a screenshot of the user interface for those of us who sometimes feel a little old.
Moving on to DuckDB itself: here at DuckDB we are more than a little religious about portability and dependencies – or rather the lack thereof. This means that very little had to happen to make DuckDB run on the ancient Mountain Lion: the stock DuckDB binary is built with by default with backwards-compatibility to OS X 11.0 (Big Sur), but simply changing the flag and recompiling turned out to be enough to make DuckDB 1.2.2 run on Mountain Lion. We would have loved to also use a 2012 compiler to build DuckDB, but, alas, C++ 11 was unsurprisingly simply _too new_ in 2012 to be fully supported by compilers. Either way, the binary runs fine and could have been also produced by working around the compiler bugs. Or we could have just hand-coded Assembly like [others have done](https://www.youtube.com/watch?v=zsvafrKvTTY&t=54s).
#### Benchmarks {#blog:2025-05-19-the-lost-decade-of-small-data::benchmarks}
But we're not interested in synthetic CPU scores, we're interested in [synthetic SQL scores](https://hannes.muehleisen.org/publications/DBTEST2018-performance-testing.pdf) instead! To see how the old machine is holding up when performing _serious_ data crunching, we used the _at this point rather tired but well-known TPC-H benchmark_ at scale factor 1000. This means that the two main tables, `lineitem` and `orders` contain 6 and 1.5 billion rows, respectively. When stored as a DuckDB database, the database has a size of ca. 265 GB.
From the [audited results on the TPC website](https://www.tpc.org/tpch/results/tpch_advanced_sort_V35.asp?PRINTVER=false&VERSION=3&FLTCOL1=h_sf&FLTCOLOPR1=%3D&FLTCHO1=1000&ADDFILTERROW=&filterRowCount=1&SRTCOL1=h_sponsor&SRTDIR1=ASC&ADDSORTROW=&sortRowCount=1&DISPRES=100++++PERCENT&include_withdrawn_results=none&include_historic_results=yes&include_specification_revision=ON&include_server_cpu=ON&include_total_system_price=ON&include_cluster_info=ON), we can see that running the benchmark on this scale factor on a single node seems to require hardware costing hundreds of thousands of Dollars.
We ran each of the 22 benchmark queries five times, and took the median runtime to remove noise. However, because the amount of RAM (16 GB) is very much less than the database size (265 GB), no significant amount of the input data can be cached in the buffer manager, so those are not really what people sometimes call “hot” runs.
Below are the per-query results in seconds:
| query | latency |
| ----: | ------: |
| 1 | 142.2 |
| 2 | 23.2 |
| 3 | 262.7 |
| 4 | 167.5 |
| 5 | 185.9 |
| 6 | 127.7 |
| 7 | 278.3 |
| 8 | 248.4 |
| 9 | 675.0 |
| 10 | 1266.1 |
| 11 | 33.4 |
| 12 | 161.7 |
| 13 | 384.7 |
| 14 | 215.9 |
| 15 | 197.6 |
| 16 | 100.7 |
| 17 | 243.7 |
| 18 | 2076.1 |
| 19 | 283.9 |
| 20 | 200.1 |
| 21 | 1011.9 |
| 22 | 57.7 |
But what do those cold numbers actually mean? The hidden sensation is that we actually _have_ numbers, this old computer could actually complete all benchmark queries using DuckDB! If we look at the time a bit closer, we see the queries take anywhere between a minute and half an hour. Those are not unreasonable waiting times for analytical queries on that sort of data in any way. Heck, you would have been waiting way longer back in 2012 for Hadoop YARN to pick up your job in the first place only to spew stack traces at you at some point.
#### 2023 Improvements {#blog:2025-05-19-the-lost-decade-of-small-data::2023-improvements}
But how do those results stack up against a modern MacBook? As a comparison point, we used a modern ARM-based [M3 Max MacBook Pro](https://everymac.com/systems/apple/macbook_pro/specs/macbook-pro-m3-max-16-core-cpu-40-core-gpu-14-late-2023-specs.html), which happened to be sitting on the same desk. But between them, the two MacBooks represent more than a decade of hardware development.
Looking at [GeekBench 5 benchmark scores](https://everymac.com/ultimate-mac-comparison-chart/?compare=all-macs&highlight=0&prod1=MacBookPro159&prod2=MacBookPro063&prod3=) alone, we see a ca. 7× difference in raw CPU speed when using all cores, and ca. factor 3 difference in single-core speed. Of course there are also big differences in RAM and SSD speeds. Funnily, the display size and resolution are almost unchanged.
Here are the results side-by-side:
| query | latency_old | latency_new | speedup |
| ----: | ----------: | ----------: | ------: |
| 1 | 142.2 | 19.6 | 7.26 |
| 2 | 23.2 | 2.0 | 11.60 |
| 3 | 262.7 | 21.8 | 12.05 |
| 4 | 167.5 | 11.1 | 15.09 |
| 5 | 185.9 | 15.5 | 11.99 |
| 6 | 127.7 | 6.6 | 19.35 |
| 7 | 278.3 | 14.9 | 18.68 |
| 8 | 248.4 | 14.5 | 17.13 |
| 9 | 675.0 | 33.3 | 20.27 |
| 10 | 1266.1 | 23.6 | 53.65 |
| 11 | 33.4 | 2.2 | 15.18 |
| 12 | 161.7 | 10.1 | 16.01 |
| 13 | 384.7 | 24.4 | 15.77 |
| 14 | 215.9 | 9.2 | 23.47 |
| 15 | 197.6 | 8.2 | 24.10 |
| 16 | 100.7 | 4.1 | 24.56 |
| 17 | 243.7 | 15.3 | 15.93 |
| 18 | 2076.1 | 47.6 | 43.62 |
| 19 | 283.9 | 23.1 | 12.29 |
| 20 | 200.1 | 10.9 | 18.36 |
| 21 | 1011.9 | 47.8 | 21.17 |
| 22 | 57.7 | 4.3 | 13.42 |
We do see significant speedups, from 7 up to as much as 53. The [geometric mean](https://en.wikipedia.org/wiki/Geometric_mean) of the timings improved from 218 to 12, a ca. 20× improvement.
#### Reproducibility {#blog:2025-05-19-the-lost-decade-of-small-data::reproducibility}
The binary, scripts, queries, and results are [available on GitHub](https://github.com/hannes/old-macbook-tpch) for inspection. We also made the [TPC-H SF1000 database file](https://blobs.duckdb.org/data/tpch-sf1000.db) available for download so you don't have to generate it. But be warned, it's a large file.
#### Discussion {#blog:2025-05-19-the-lost-decade-of-small-data::discussion}
We have seen how the decade-old MacBook Pro Retina has been able to complete a complex analytical benchmark. A newer laptop was able to significantly improve on those times.
But absolute speedup numbers are a bit pointless here. The difference is purely _quantitative_, not _qualitative_.
From a user perspective, it matters much more that those queries complete in somewhat reasonable time, not if it took 10 or 100 seconds to do so. We can tackle almost the same kind of data problems with both laptops, we just have to be willing to wait a little longer. This is especially true given DuckDB's [out-of-core capability](https://duckdb.org/docs/1.5/guides/performance/how_to_tune_workloads), which allows it to spill query intermediates to disks if required.
What is perhaps more interesting is that back in 2012, it would have been completely feasible to have a single-node SQL engine like DuckDB that could run complex analytical SQL queries against a database of 6 billion rows in manageable time – and we did not even have to [immerse it in dry ice this time](https://youtu.be/H1N2Jr34jwU?si=xCxQqZcLs8WA_5Yl&t=1763).
History is full of “what if”s, what if something like DuckDB had existed in 2012? The main ingredients were there, vectorized query processing had [already been invented in 2005](https://www.cidrdb.org/cidr2005/papers/P19.pdf). Would the now somewhat-silly-looking move to distributed systems for data analysis have ever happened? The dataset size of our benchmark database was awfully close to that 99.9% percentile of input data volume for analytical queries in 2024. And while the retina MacBook Pro was a high-end machine in 2012, by 2014 many other vendors shifted to offering laptops with built-in SSD storage and larger amounts of memory became more widespread.
So, yes, we really did lose a full decade.
## Announcing DuckDB 1.3.0 {#blog:2025-05-21-announcing-duckdb-130}
**Publication date:** 2025-05-21
**Author:** The DuckDB team
**TL;DR:** The DuckDB team is happy to announce that today we're releasing DuckDB version 1.3.0, codenamed “Ossivalis”.
> To install the new version, please visit the [installation guide](https://duckdb.org/install/). Note that it can take a few hours to days to release some client libraries (e.g., Go, R, Java) and extensions (e.g., the UI) due to the extra changes and review rounds required.
We are proud to release DuckDB 1.3.0. This release of DuckDB is named “Ossivalis” after Bucephala Ossivalis, an ancestor of the [Goldeneye duck](https://en.wikipedia.org/wiki/Common_goldeneye) that lived millions of years ago.
In this blog post, we cover the most important features of the new release. DuckDB is moving rather quickly, and we could cover only a small fraction of the changes in this release. For the complete release notes, see the [release page on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.3.0).
#### Breaking Changes and Deprecations {#blog:2025-05-21-announcing-duckdb-130::breaking-changes-and-deprecations}
##### Old Linux glibc Deprecation {#blog:2025-05-21-announcing-duckdb-130::old-linux-glibc-deprecation}
Now that all mainstream Linux distributions use [glibc 2.28](https://lists.gnu.org/archive/html/info-gnu/2018-08/msg00000.html) or newer, DuckDB's official Linux binaries require at least glibc 2.28 or newer. The release is [built using the `manylinux_2_28` image from Python](https://github.com/duckdb/duckdb/pull/16956), which combines an older glibc with a newer compiler. This change also implies that extensions are no longer distributed for the `linux_amd64_gcc4` platform.
We highly value [portability](https://duckdb.org/why_duckdb#portable), so it is of course still possible to [build DuckDB from source](https://duckdb.org/docs/1.5/dev/building/overview) for older versions of glibc.
##### Lambda Function Syntax {#blog:2025-05-21-announcing-duckdb-130::lambda-function-syntax}
Previously, lambda functions in DuckDB could be specified using the single arrow syntax: `x -> x + 1`.
The single arrow operator is also used by the JSON extension to express [JSON extraction](https://duckdb.org/docs/1.5/data/json/json_functions#json-extraction-functions) using the syntax `->'field'`.
The two meanings of the single arrow operator are treated the same by the binder, thus they share the same (low) precedence, necessitating extra parentheses in JSON expression with equality checks:
```sql
SELECT (JSON '{"field": 42}')->'field' = 42;
-- throws a Binder Error:
-- No function matches the given name and argument types 'json_extract(JSON, BOOLEAN)
SELECT ((JSON '{"field": 42}')->'field') = 42;
-- return true
```
This often caused confusion among users, therefore, the new release deprecates the old arrow lambda syntax and replaces it with Python-style lambda syntax:
```sql
SELECT list_transform([1, 2, 3], lambda x: x + 1);
```
To make the transition smoother, the deprecation will happen in several steps over the next year.
First, DuckDB 1.3.0 [introduces a new setting to configure the lambda syntax](https://github.com/duckdb/duckdb/pull/17235):
```sql
SET lambda_syntax = 'default';
SET lambda_syntax = 'enable_single_arrow';
SET lambda_syntax = 'disable_single_arrow';
```
Currently, `default` enables both syntax styles, i.e., the old single arrow syntax and the Python-style syntax.
We will gradually phase out the old syntax.
DuckDB 1.5.0 will be the last release supporting the single arrow syntax without explicitly enabling it.
Following versions will first disable the single arrow syntax by default,
then ultimately remove the `lambda_syntax` flag and fully deprecate the single arrow syntax,
so the old behavior will no longer be accessible.
##### Serializing Strings in List with Escapes {#blog:2025-05-21-announcing-duckdb-130::serializing-strings-in-list-with-escapes}
Starting with the new version, DuckDB escapes characters such as `'` in strings serialized in nested data structures to allow round-tripping between the serialized string and the nested representation.
For example:
```sql
SELECT ['hello ''my'' world'] AS s;
```
DuckDB version 1.2.2 returns `[hello 'my' world]` while DuckDB 1.3.0 returns `['hello \'my\' world']`.
To serialize a list of strings with the old behavior, use the [`array_to_string` function](https://duckdb.org/docs/1.5/sql/functions/list#array_to_string):
```sql
SELECT printf('[%s]', array_to_string(
['hello ''my'' world', 'hello ''cruel'' world'], ', '
)) AS s;
```
```text
┌─────────────────────────────────────────┐
│ s │
│ varchar │
├─────────────────────────────────────────┤
│ [hello 'my' world, hello 'cruel' world] │
└─────────────────────────────────────────┘
```
##### Minor SQL Parser Changes {#blog:2025-05-21-announcing-duckdb-130::minor-sql-parser-changes}
* The term `AT` now needs quotes to be used as an identifier as it is used for [time travel in Iceberg](https://github.com/duckdb/duckdb-iceberg/pull/225).
* `lambda` is now a reserved keyword due to the change in lambda syntax.
* `GRANT` is no longer a reserved keyword.
#### New Features {#blog:2025-05-21-announcing-duckdb-130::new-features}
The new DuckDB release again contains a lot of exciting new features:
##### External File Cache {#blog:2025-05-21-announcing-duckdb-130::external-file-cache}
DuckDB is used a lot to read from remote files, e.g., Parquet files stored on HTTP servers or blob storage. Previous versions would always fully re-read file data. With this release, we [added a cache for data from external files](https://github.com/duckdb/duckdb/pull/16463). This cache is subject to the overall DuckDB memory limit. If space is available, it will be used to dynamically cache data from external files. This should greatly improve re-running queries on remote data. For example:
```sql
.timer on
.mode trash -- do not show query result
FROM 'https://blobs.duckdb.org/data/shakespeare.parquet';
Run Time (s): real ⟨1.456⟩ user 0.037920 sys 0.028510
FROM 'https://blobs.duckdb.org/data/shakespeare.parquet';
Run Time (s): real ⟨0.360⟩ user 0.029188 sys 0.007620
```
We can see that the query is much faster the second time around due to the cache. In previous versions, the runtime would have been the same.
The cache contents can be queried using the `duckdb_external_file_cache()` table function like so:
```sql
.mode duckbox -- re-enable output
FROM duckdb_external_file_cache();
```
```text
┌───────────────────────────────────────────────────┬──────────┬──────────┬─────────┐
│ path │ nr_bytes │ location │ loaded │
│ varchar │ int64 │ int64 │ boolean │
├───────────────────────────────────────────────────┼──────────┼──────────┼─────────┤
│ https://blobs.duckdb.org/data/shakespeare.parquet │ 1697483 │ 4 │ true │
│ https://blobs.duckdb.org/data/shakespeare.parquet │ 16384 │ 1681808 │ true │
└───────────────────────────────────────────────────┴──────────┴──────────┴─────────┘
```
The cache is enabled by default but can be disabled with:
```sql
SET enable_external_file_cache = false;
```
##### Directly Query Data Files with the CLI {#blog:2025-05-21-announcing-duckdb-130::directly-query-data-files-with-the-cli}
DuckDB's command line interface (CLI) gained the capability to [directly query Parquet, CSV or JSON files](https://github.com/duckdb/duckdb/pull/17415). This works by just using e.g. a Parquet file instead of the database file. This will expose a view that can be queried. For example, say we have a Parquet file called `region.parquet`, this will work:
```bash
duckdb region.parquet -c 'FROM region;'
```
```text
┌─────────────┐
│ r_name │
│ varchar │
├─────────────┤
│ AFRICA │
│ AMERICA │
│ ASIA │
│ EUROPE │
│ MIDDLE EAST │
└─────────────┘
```
When using the CLI like this, what actually happens is that we launch a temporary in-memory DuckDB database, and create two views over the given file:
* `file` – this view is always named the same, regardless of the name of the file.
* `[base_file_name]` – this view depends on the name of the file, e.g., for `region.parquet` this is `region`.
Both views can be queried and will give the same result.
The main advantage of this feature is usability: we can use the regular shell to navigate to a file, and then use DuckDB to open that file without having to refer to the path of the file at the SQL level.
##### `TRY` Expression {#blog:2025-05-21-announcing-duckdb-130::try-expression}
DuckDB already supported [`TRY_CAST`](https://duckdb.org/docs/1.5/sql/expressions/cast#TRY_CAST), which was trying to cast a value but did not fail the query if this was not possible. For example:
```sql
SELECT TRY_CAST('asdf' AS INTEGER);
```
returns `NULL`. This release [generalizes this functionality](https://github.com/duckdb/duckdb/pull/15939) beyond casting to arbitrary expressions that can error using `TRY`. For example, the logarithm of 0 [is undefined](https://www.quora.com/What-is-log-0), and `log(0)` will throw an exception and tell you that it “cannot take logarithm of zero”. With the new `TRY`, this will return `NULL` instead, e.g.:
```sql
SELECT TRY(log(0));
```
```text
NULL
```
Again, this will work for arbitrary expressions. We recommend to use `TRY` sparingly however if an error is expected often because there is going to be a performance impact. If any *batch* of rows causes an error, we switch to row-by-row execution of the expression to figure out exactly which row had an error and which did not. This is slower.
##### Updatings Structs {#blog:2025-05-21-announcing-duckdb-130::updatings-structs}
Starting with the new release, [it's possible to update the sub-schema of structs using the `ALTER TABLE` clause](https://github.com/duckdb/duckdb/pull/17003). You can add, drop, and rename fields:
```sql
CREATE TABLE test (s STRUCT(i INTEGER, j INTEGER));
INSERT INTO test VALUES (ROW(1, 1)), (ROW(2, 2));
ALTER TABLE test DROP COLUMN s.i;
ALTER TABLE test ADD COLUMN s.k INTEGER;
ALTER TABLE test RENAME COLUMN s.j TO l;
```
```text
┌──────────────────────────────┐
│ s │
│ struct(l integer, k integer) │
├──────────────────────────────┤
│ {'l': 1, 'k': NULL} │
│ {'l': 2, 'k': NULL} │
└──────────────────────────────┘
```
Altering structs is also supported [inside `LIST` and `MAP` columns](https://github.com/duckdb/duckdb/pull/17462).
##### Swapping in New Databases {#blog:2025-05-21-announcing-duckdb-130::swapping-in-new-databases}
The [`ATTACH OR REPLACE` clause](https://github.com/duckdb/duckdb/pull/15355) allows you to replace a database,
so you can swap a database on the fly.
For example:
```sql
ATTACH 'taxi_v1.duckdb' AS taxi;
USE taxi;
ATTACH OR REPLACE 'taxi_v2.duckdb' AS taxi;
```
This feature was implemented by external contributor [`xevix`](https://github.com/xevix).
##### UUID v7 Support {#blog:2025-05-21-announcing-duckdb-130::uuid-v7-support}
> **Warning.** **Update (2025-05-23).**
> The [UUID v7 implementation in DuckDB v1.3.0 is not consistent with the UUID standard](https://github.com/duckdb/duckdb/issues/17611), causing the timestamp values to be off.
> This means that the timestamps in the generated UUID v7 values will only be correct if they are used exclusively within DuckDB.
> Importing or exporting UUID v7 values from/to systems will yield incorrect timestamps.
> We [patched this bug](https://github.com/duckdb/duckdb/pull/17612) and the fix will soon be available in the preview builds and the [upcoming 1.3.1 patch release](https://duckdb.org/release_calendar#upcoming-releases).
DuckDB now [supports](https://github.com/duckdb/duckdb/pull/15819) [UUID v7](https://uuid7.com/), which is a newer version of UUIDs. `UUIDv7` combines a Unix timestamp in milliseconds and random bits, offering both uniqueness and sortability. This is useful to e.g. order UUIDs by age or to combine the ubiquitous `ID` and `TIMESTAMP` columns in many tables into a single `UUIDv7` column.
New UUIDs can be created using the `uuidv7()` scalar function. For example:
```sql
SELECT uuidv7();
```
```text
┌──────────────────────────────────────┐
│ uuidv7() │
│ uuid │
├──────────────────────────────────────┤
│ 8196f1f6-e3cf-7a74-bc0e-c89ac1ea1e19 │
└──────────────────────────────────────┘
```
There are also additional functions to determine the UUID version (` uuid_extract_version()`) and to extract the internal timestamp (` uuid_extract_timestamp()`), for example:
```sql
SELECT uuid_extract_version(uuidv7());
```
```text
┌────────────────────────────────┐
│ uuid_extract_version(uuidv7()) │
│ uint32 │
├────────────────────────────────┤
│ 7 │
└────────────────────────────────┘
```
```sql
SELECT uuid_extract_timestamp(uuidv7());
```
```text
┌──────────────────────────────────┐
│ uuid_extract_timestamp(uuidv7()) │
│ timestamp with time zone │
├──────────────────────────────────┤
│ 2025-05-21 08:32:14.61+00 │
└──────────────────────────────────┘
```
This feature was implemented by external contributor [`dentiny`](https://github.com/dentiny).
##### Expression Support in `CREATE SECRET` {#blog:2025-05-21-announcing-duckdb-130::expression-support-in-create-secret}
DuckDB has an internal “secret” management facility for things like S3 credentials. With this release, it is [possible to use scalar expressions](https://github.com/duckdb/duckdb/pull/15801) in the creation of the secret. This allows for secret contents to not be specified in query text, which makes them easier to keep out of log files, etc. For example:
```sql
SET VARIABLE my_bearer_token = 'hocus pocus this token is bogus';
CREATE SECRET http (
TYPE http,
BEARER_TOKEN getvariable('my_bearer_token')
);
```
You can see that the `BEARER_TOKEN` field in the secret is set from the `getvariable` function in `CREATE SECRET`. In the CLI, this is also possible through *environment variables* using `getenv()`. For example, this is now possible:
```bash
MY_SECRET=asdf duckdb -c \
"CREATE SECRET http (TYPE http, BEARER_TOKEN getenv('MY_SECRET'))"
```
##### Unpacking Columns {#blog:2025-05-21-announcing-duckdb-130::unpacking-columns}
DuckDB v1.3.0 brings a further boost to the popular [`COLUMNS(*)` expression](https://duckdb.org/docs/1.5/sql/expressions/star#columns-expression).
Previously, _unpacking_ the entities into a list was possible by adding a leading `*` character:
```sql
CREATE TABLE tbl AS SELECT 21 AS a, 1.234 AS b;
SELECT [*COLUMNS(*)] AS col_exp FROM tbl;
```
```text
┌─────────────────┐
│ col_exp │
│ decimal(13,3)[] │
├─────────────────┤
│ [21.000, 1.234] │
└─────────────────┘
```
However, this syntax could not be used in tandem with other expressions such as casting:
```sql
SELECT [*COLUMNS(*)::VARCHAR] AS col_exp FROM tbl;
```
```console
Binder Error:
*COLUMNS() can not be used in this place
```
The new `UNPACK` keyword removes this limitation. The following expression
```sql
SELECT [UNPACK(COLUMNS(*)::VARCHAR)] AS col_exp FROM tbl;
```
is equivalent to:
```sql
SELECT [a::VARCHAR, b::VARCHAR] AS col_exp FROM tbl;
```
```text
┌─────────────┐
│ col_exp │
│ varchar[] │
├─────────────┤
│ [21, 1.234] │
└─────────────┘
```
##### Spatial `JOIN` Operator {#blog:2025-05-21-announcing-duckdb-130::spatial-join-operator}
We added a new specialized join operator as part of the [`spatial` extension](https://duckdb.org/docs/1.5/core_extensions/spatial/overview), which greatly improves the efficiency of _spatial joins_, that is, queries that `JOIN` two geometry columns using specific spatial predicate functions, such as `ST_Intersects` and `ST_Contains`.
Similarly to a `HASH_JOIN`, the `SPATIAL_JOIN` builds a temporary lookup data-structure for the smaller side of the join, except it's an R-Tree, instead of a hash table. What this means for you is that you don't need to create an index first, or do any other pre-processing to optimize your spatial joins. It's all handled by the join operator internally.
While the query optimizer will try to instantiate this new operator for `LEFT`, `OUTER`, `INNER` and `RIGHT` spatial joins, one limitation currently is that the join can only contain a single join condition, or the optimizer will fall back to use a less efficient join strategy.
The following example illustrates how the `SPATIAL_JOIN` operator becomes part of the query plan. It's a relatively small query, but on my machine it executes almost 100× faster than it used to do in DuckDB v1.2.2!
```sql
LOAD spatial;
-- generate random points
CREATE TABLE points AS
SELECT
ST_Point(x, y) AS geom,
(y * 50) + x // 10 AS id
FROM
generate_series(0, 1000, 5) r1(x),
generate_series(0, 1000, 5) r2(y);
-- generate random polygons
CREATE TABLE polygons AS
SELECT
ST_Buffer(ST_Point(x, y), 5) AS geom,
(y * 50) + x // 10 AS id
FROM
generate_series(0, 500, 10) r1(x),
generate_series(0, 500, 10) r2(y);
-- inspect the join plan
EXPLAIN
SELECT *
FROM polygons
JOIN points ON ST_Intersects(points.geom, polygons.geom);
```
```text
...
┌─────────────┴─────────────┐
│ SPATIAL_JOIN │
│ ──────────────────── │
│ Join Type: INNER │
│ Conditions: ├──────────────┐
│ ST_Intersects(geom, geom) │ │
│ ~40401 Rows │ │
└─────────────┬─────────────┘ │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ SEQ_SCAN ││ SEQ_SCAN │
│ ──────────────────── ││ ──────────────────── │
│ Table: points ││ Table: polygons │
│ Type: Sequential Scan ││ Type: Sequential Scan │
│ ~40401 Rows ││ ~2601 Rows │
└───────────────────────────┘└───────────────────────────┘
```
For the curious, there are more details [in the PR](https://github.com/duckdb/duckdb-spatial/pull/545).
#### Internal Changes {#blog:2025-05-21-announcing-duckdb-130::internal-changes}
For this release there have also been a large number of internal changes.
We have completed an [almost complete re-implementation](https://github.com/duckdb/duckdb/pulls?q=is%3Apr+is%3Aclosed+closed%3A%3E2025-02-05+parquet) of DuckDB's *Parquet reader and writer*. This should greatly improve Parquet performance and reliability, and also expanded [Parquet feature support](https://parquet.apache.org/docs/file-format/implementationstatus/) for obscure logical types like `UNKNOWN` and `FLOAT16`.
We have also done a large amount of internal changes around the [reading of multiple files](https://github.com/duckdb/duckdb/pulls?q=is%3Apr+is%3Aclosed+closed%3A%3E2025-02-05+multifilereader+) (e.g., a folder of Parquet files) in an API called the `MultiFileReader`. We have unified the handling of multiple files across many of our file readers, e.g., Parquet, CSV, JSON, Avro, etc. This allows DuckDB to handle e.g. schema differences between multiple files in a unified way.
We have also added a new string compression method, `DICT_FSST`. Before, DuckDB supported *either* [dictionary encoding](https://en.wikipedia.org/wiki/Dictionary_coder) *or* [FSST compression](https://github.com/cwida/fsst) (“Fast Static Symbol Table”) for strings. Those compression methods could not be mixed within a storage block (265 kB by default). However, we observed a lot of real-world data where part of the block would benefit from dictionary encoding, and another part would benefit from FSST. FSST does not by default duplicate-eliminate strings. This release [combines both methods](https://github.com/duckdb/duckdb/pull/15637) into a new compression method, `DICT_FFST`. This *first* runs dictionary encoding and *then* compresses the dictionary using FSST. Dictionary encoding and FSST-only encoding are also still available. We have also [optimized storing validity masks](https://github.com/duckdb/duckdb/pull/15591) (“which rows are NULL?”) in this release, some compression methods (like the new `DICT_FSST`) can handle NULLs internally and this obviates the need for a separate validity mask. Combined, those new features should greatly reduce the storage space required especially for strings. Note that the compression method is automatically picked by DuckDB based on actually observed compression ratios so users will not have to explicitly set this.
#### Final Thoughts {#blog:2025-05-21-announcing-duckdb-130::final-thoughts}
These were a few highlights – but there are many more features and improvements in this release. There have been over 3,000 commits by over 75 contributors since we released v1.2.2. The full release notes can be [found on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.3.0). We would like to thank our community for providing detailed issue reports and feedback. And our special thanks goes to external contributors, who directly landed features in this release!
## Arrow IPC Support in DuckDB {#blog:2025-05-23-arrow-ipc-support-in-duckdb}
**Publication date:** 2025-05-23
**Authors:** Pedro Holanda, Ian Cook, Dewey Dunnington, Bryce Mecum
**TL;DR:** DuckDB now supports consuming and producing the Arrow IPC Serialization Format through the `arrow` community extension.
#### DuckDB and Arrow {#blog:2025-05-23-arrow-ipc-support-in-duckdb::duckdb-and-arrow}
In recent years, the [Apache Arrow project](https://arrow.apache.org/docs/) has gained a lot of traction in the data world, thanks to its columnar format that allows for easy interchange of data between different systems – mostly with a zero-copy approach.
Apache Arrow powers, among others, the integration between DuckDB and Polars.
In practice, when DuckDB produces or consumes a Polars DataFrame, it is actually using the [Arrow columnar format](https://arrow.apache.org/docs/format/Columnar.html) underneath.
The importance of having such a format is also one of the main reasons DuckDB was among the pioneers in [integrating with Arrow](#blog:2021-12-03-duck-arrow) and implementing an [Arrow Database Connectivity (ADBC)](https://arrow.apache.org/adbc/) [interface](https://duckdb.org/docs/1.5/clients/adbc) – particularly because Arrow makes this possible with no additional dependencies, thanks to its [C data interface](https://arrow.apache.org/docs/format/CDataInterface.html).
But one limitation of Arrow's C data interface is that it exchanges data using pointers (memory addresses). This limits the possibilities if you want to exchange Arrow data between different processes or systems. To overcome this limitation, the Arrow project also specifies the [Arrow IPC format](https://arrow.apache.org/docs/format/Columnar.html#format-ipc), which allows users to efficiently serialize Arrow columnar data and pass it between processes or over a network. This data can be consumed as a stream, either directly from a memory buffer or from a file.
We're thrilled to announce that DuckDB is now able to consume and produce these Arrow streams via the new [`arrow` community extension](https://duckdb.org/community_extensions/extensions/arrow). In this post, we will describe the Arrow IPC serialization format in more detail, show how to install the new `arrow` community extension for DuckDB, and give a demo showing how to use it.
#### Arrow Interprocess Communication (Arrow IPC) {#blog:2025-05-23-arrow-ipc-support-in-duckdb::arrow-interprocess-communication-arrow-ipc}
The Arrow IPC format provides a way of serializing (and optionally compressing) Arrow-formatted data, enabling you to transfer data over a network or store it on disk while keeping it in Arrow format, avoiding the overhead of converting it to a different format. Arrow IPC supports LZ4 and Zstd compression, and when stored as a file, it also supports a file footer that can be used to speed up retrieval and processing by allowing parts of the data to be skipped (similar to the approach used by the Parquet format). When compared to Parquet, the Arrow IPC format has two main benefits:
1. **Ease of implementation:** Writing a low-level Arrow IPC consumer/producer is less complex than writing a Parquet one, especially if the system already integrates with the Arrow format.
2. **Faster encoding and decoding:** The process of encoding and decoding (serializing and deserializing) Arrow data is much simpler and faster than with Parquet. This can yield faster processing times—especially if you are streaming data that does not need to be stored on disk afterwards.
Arrow and Parquet are complementary formats. Parquet's sophisticated system of encoding and compression options typically yields much smaller files, making Parquet a better choice for archival storage. Arrow's ability to eliminate encoding and decoding overheads typically yields faster and more efficient data interchange, making Arrow a better choice for query result transfer and ephemeral caching.
To give you an illustration of just how simple the Arrow IPC format is, consider the following illustration. In the Arrow IPC format, a table is serialized as a sequence of record batches (a collection of records organized in a columnar layout), preceded by their shared schema:
*Figure from [Apache Arrow Blog: How the Apache Arrow Format Accelerates Query Result Transfer](https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer).*
Note that in realistic scenarios, record batches are much larger and the figure above is simplified for illustrative purposes.
#### The Arrow Community Extension {#blog:2025-05-23-arrow-ipc-support-in-duckdb::the-arrow-community-extension}
DuckDB has included an integration with the Arrow IPC format for many years, via the [(now-deprecated) Arrow core extension](https://github.com/duckdb/arrow). However, the main purpose of this support was to allow DuckDB interoperability with JavaScript, hence it was designed only to read in-memory serialized buffers, and not Arrow IPC files. The extension's code complexity and maintainability were very high, because working with Arrow IPC required having the entire Arrow C++ library as a dependency, as we did not want to write our own serialization and deserialization code for the Arrow IPC format.
More recently, a much smaller Arrow C++ implementation started to gain popularity as a way to interact with Arrow IPC data: the [nanoarrow library](https://arrow.apache.org/nanoarrow/). Using nanoarrow, we completely redesigned the old DuckDB Arrow extension to have a much smaller dependency, a cleaner codebase, and the ability to scan Arrow IPC files. We also took the opportunity to shift the Arrow DuckDB extension from a core extension to a community extension. This change was made for two main reasons. The first is to enable the Arrow developer and user community to be more involved in building and supporting the extension. The second is to have a release schedule that is not tied to the DuckDB release schedule. In practice, this means that members of the core Arrow developer community can decide when a new version of the extension will be released.
Installing and loading the new Arrow extension is very simple:
```sql
INSTALL arrow FROM community;
LOAD arrow;
```
#### Demo {#blog:2025-05-23-arrow-ipc-support-in-duckdb::demo}
In this demo, we will use the new Arrow DuckDB extension to generate the `lineitem` TPC-H table with scale factor 10 as an Arrow IPC file. While our demo will focus on Arrow IPC data stored in a file, the extension itself also allows you to consume and produce the Arrow IPC format directly as buffers. You can find detailed examples of usage and accepted parameters in the extension's [README](https://github.com/paleolimbot/duckdb-nanoarrow/blob/main/README.md).
We start off by loading the `arrow` extension and generating our TPC-H tables.
```sql
LOAD arrow;
CALL dbgen(sf = 10);
```
To generate the Arrow IPC files we can simply use the `COPY ... TO ...` clause, as follows. We use the [recommended file extension `.arrows`](https://arrow.apache.org/faq/#mime-types-iana-media-types-for-arrow-data) since this file is in the Arrow IPC stream format.
```sql
COPY lineitem TO 'lineitem.arrows';
```
> In this demo, for simplicity, we wrote our table in a single file. However, our Arrow `COPY ... TO ...` clause allows us to set the `chunk_size` and the number of `row_groups` per file. These options allow us to produce data optimized for the best possible performance for your use case. For example, a smaller `chunk_size` may reduce overall performance but benefit streaming scenarios.
We can now run TPC-H query 6 directly on our file using the `read_arrow` function:
```sql
SELECT
sum(l_extendedprice * l_discount) AS revenue
FROM
read_arrow('lineitem.arrows')
WHERE
l_shipdate >= CAST('1994-01-01' AS date)
AND l_shipdate < CAST('1995-01-01' AS date)
AND l_discount BETWEEN 0.05
AND 0.07
AND l_quantity < 24;
```
which prints:
```text
┌─────────────────┐
│ revenue │
│ decimal(38,4) │
├─────────────────┤
│ 1230113636.0101 │
│ (1.23 billion) │
└─────────────────┘
```
Thanks to [replacement scans](https://duckdb.org/docs/1.5/sql/dialect/friendly_sql#data-import), you can omit the function `read_arrow` if the filename ends with `.arrow` or `.arrows`. For example:
```sql
SELECT count(*) FROM 'lineitem.arrows';
```
which prints:
```text
┌─────────────────┐
│ count_star() │
│ int64 │
├─────────────────┤
│ 59986052 │
│ (59.99 million) │
└─────────────────┘
```
> For simplicity, we focus on the scenario of reading a single file, but our reader supports multi-file reading, with functionality on par with the DuckDB Parquet reader.
What if you want to fetch an Arrow IPC stream directly from a server into DuckDB? To demonstrate this, we can start an HTTP file server in the same directory where we saved `lineitem.arrows`. We use the Node.js package [serve](https://www.npmjs.com/package/serve) to do this (instead of Python's built-in `http.server`) because it supports [HTTP range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests):
```bash
npx serve -l 8008
```
Then you can use [DuckDB's `httpfs` extension](https://duckdb.org/docs/1.5/core_extensions/httpfs/https) to query the Arrow data over the HTTP(S) protocol:
```sql
INSTALL httpfs;
LOAD httpfs;
LOAD arrow;
SELECT count(*) FROM read_arrow('http://localhost:8008/lineitem.arrows');
```
which prints the same result:
```text
┌─────────────────┐
│ count_star() │
│ int64 │
├─────────────────┤
│ 59986052 │
│ (59.99 million) │
└─────────────────┘
```
Alternatively, you can use a tool like `curl` to fetch Arrow IPC data from a server and pipe it to DuckDB in the terminal:
```bash
URL="http://localhost:8008/lineitem.arrows"
SQL="LOAD arrow; FROM read_arrow('/dev/stdin') SELECT count(*);"
curl -s "$URL" | duckdb -c "$SQL"
```
which prints the same result. For other demos of the `arrow` extension, see our [arrow-ipc demo](https://github.com/pdet/arrow-ipc-demo) repository.
#### Bonus: Cool Use Cases for Arrow IPC {#blog:2025-05-23-arrow-ipc-support-in-duckdb::bonus-cool-use-cases-for-arrow-ipc}
Running a DuckDB query against data in Arrow IPC format like in the demo above is a pretty neat trick, and it works so well because DuckDB and Arrow are a natural pair due to both using a columnar data layout. However, you may be wondering what else you can do with Arrow IPC data. One of Arrow's main goals is interoperability, and by saving our data in Arrow IPC format, we've opened up many options for connecting with other tools.
For example, we can now work with our data with [PyArrow](https://arrow.apache.org/docs/python/):
```python
import pyarrow as pa
with open('lineitem.arrows', 'rb') as source:
stream = pa.ipc.open_stream(source)
tbl = stream.read_all()
```
or [Polars](https://pola.rs):
```python
import polars as pl
tbl = pl.read_ipc_stream("lineitem.arrows")
```
or [ClickHouse](https://clickhouse.com):
```sql
CREATE TABLE
lineitem
ENGINE MergeTree()
ORDER BY tuple()
AS
SELECT * FROM file('lineitem.arrows', 'ArrowStream');
```
or any of the numerous other Arrow libraries (available in a dozen different languages) or Arrow-compatible systems.
The benefits of Arrow IPC don't stop there: Arrow IPC is also ideal for larger-than-memory use cases. Using [PyArrow](https://arrow.apache.org/docs/python/), we can [memory-map](https://en.wikipedia.org/wiki/Memory-mapped_file) our `lineitem.arrows` file and work with it without reading the entire thing into memory:
```python
import pyarrow as pa
with pa.memory_map('lineitem.arrows', 'rb') as source:
stream = pa.ipc.open_stream(source)
tbl = stream.read_all()
tbl.num_rows
# => 59986052
```
Then, we can check that PyArrow didn't have to allocate any buffers to hold the data because it all lives on disk:
```python
pa.total_allocated_bytes()
# => 0
```
Now we can perform the same query we did in the demo above and show we get the same result:
```python
import datetime
import pyarrow.compute as pc
subset = tbl.filter(
(pc.field("l_shipdate") >= datetime.datetime(1994, 1, 1)) &
(pc.field("l_shipdate") < datetime.datetime(1995, 1, 1)) &
(pc.field("l_discount") >= 0.05) &
(pc.field("l_discount") <= 0.07) &
(pc.field("l_quantity") < 24.)
)
pc.sum(pc.multiply(subset.column("l_extendedprice"), subset.column("l_discount")))
# => (2, 1, 2) | | 2 | 2 * 2 = 4 | (2, 1, 2) | (2, 2, 4) | (2, 0, 1)
(2, 1, 2)
(2, 2, 4) | | 3 | 2 * 4 = 8 | (2, 2, 4) | (2, 3, 8) | (2, 0, 1)
...
(2, 3, 8) | | 4 | 2 * 8 = 16 | (2, 3, 8) | (2, 4, 16) | (2, 0, 1)
...
(2, 4, 16) | | 5 | 2 * 16 = 32 | (2, 4, 16) | (2, 5, 32) | (2, 0, 1)
...
(2, 5, 32) | | 6 | 2 * 32 = 64 | (2, 5, 32) | (2, 6, 64) | (2, 0, 1)
...
(2, 6, 64) | | 7 | 2 * 64 = 128 | (2, 6, 64) | ∅ _stop!_ | (2, 0, 1)
...
(2, 6, 64) | When the recursive CTE completes, the union table holds the entire result set, including all intermediate rows from every iteration: ```text ┌───────┬───────┬───────┐ │ a │ b │ c │ -- a^b = c │ int32 │ int32 │ int32 │ ├───────┼───────┼───────┤ │ 2 │ 0 │ 1 │ │ 2 │ 1 │ 2 │ │ 2 │ 2 │ 4 │ │ 2 │ 3 │ 8 │ │ 2 │ 4 │ 16 │ │ 2 │ 5 │ 32 │ │ 2 │ 6 │ 64 │ └───────┴───────┴───────┘ ``` The union table provides a complete record of the history of the power computation. This can lead to unnecessary overhead, especially if all we need is the last row – the final result of the recursion. Storing every intermediate value can be inefficient, in particular when the intermediate results aren't required or when working with large datasets (that involve many rows or wide rows, e.g., in the presence of array-typed columns). #### Recursive CTEs Suffer from Amnesia {#blog:2025-05-23-using-key::recursive-ctes-suffer-from-amnesia} While the union table holds the just mentioned history _once the computation is complete,_ recursive CTEs suffer from a case of “amnesia” _while the iteration is going on:_ the recursive part only ever sees the intermediate results of the _immediately preceding_ iteration. This can be limiting and many of us will have seen how query authors work around this limitation in terms of manually maintained arrays (or similar container structures) that hold information about previous iterations. This may be costly. Then again, enabling the recursive part to access the union table with all its accumulated – potentially sizable – intermediate results may easily incur performance problems while we iterate. A real conundrum. #### `USING` a `KEY` to Cure Amnesia {#blog:2025-05-23-using-key::using-a-key-to-cure-amnesia} Can we afford to allow the recursive part to access the union table? Yes, but we need a means to control its size. Operating the union table in _append-only mode_ thus is a no-go. Instead, let the recursive part optionally _overwrite_ existing rows in the union table with fresh information that has been computed in the current iteration. This can reduce the union table's size significantly (see our experiments below). Starting with version 1.3, DuckDB features a [`USING KEY` variant](https://duckdb.org/docs/1.5/sql/query_syntax/with#recursive-ctes-with-using-key) of recursive CTEs that incorporates just this idea. > If you would like to read more about its origins, check out [CIDR 2023](https://db.cs.uni-tuebingen.de/publications/2023/a-fix-for-the-fixation-on-fixpoints/a-fix-for-the-fixation-on-fixpoints.pdf). For further information about implementation, see [SIGMOD 2025](https://db.cs.uni-tuebingen.de/publications/2025/using-key/how-duckdb-is-using-key-to-unlock-recursive-query-performance.pdf). The variant introduces two major differences compared to traditional recursive CTEs: 1. It provides access to the working table (as always) as well as **access to the union table** – which we now call the **_recurring table_**. 2. Instead of simply appending new rows to the recurring table, the table now functions more like a dictionary (similar to a Python `dict`), allowing **key-based updates**. To use this new feature, add the `USING KEY (...)` clause to your recursive CTE: ```sql WITH RECURSIVE power(a, b, c) USING KEY (a) -- key: a, payload: b, c AS ( ... ); ``` With `USING KEY`, the schema of the recursive CTE is divided into **key columns** and **payload columns**. The key columns are specified using the `USING KEY (⟨column names⟩)`{:.language-sql .highlight} clause, while the remaining columns are treated as payload. This division affects how the recurring table behaves. Instead of stubbornly appending new rows on each iteration, it acts more like a dictionary: if the recursive part returns a row that hasn't been seen before, it is added to the recurring table as usual. But if a row shares a key with an existing entry in the recurring table, the payload is updated – replacing the previous values for that key in the recurring table. > If multiple rows with the same key are produced in a single iteration, only the last one is retained. Therefore, you may wish to use the `ORDER BY` clause in the recursive part to control which row is kept. This approach allows recursive queries to maintain and update state more efficiently, especially for algorithms in which keeping the latest (or “best”) value for a given key is crucial. ##### Overwriting Old Intermediate Results {#blog:2025-05-23-using-key::overwriting-old-intermediate-results} Let us pick up our recursive example query from above. Now we compute the powers of bases 2 and 3, as long as these are less than 100: ```sql WITH RECURSIVE power(a, b, c) USING KEY (a) AS ( FROM (VALUES (2, 0, 1), (3, 0, 1)) -- 2^0 = 1, 3^0 = 1 UNION SELECT a, b+1, a * c -- a^(b+1) = a * a^b FROM power -- reads the working table (contains two rows) WHERE a * c < 100 ) FROM power; -- reads the recurring table (contains two rows) ``` We start with two rows in the **non-recursive part**: one with a key (base) of `2` and another with a key of `3`. In the **recursive part**, we multiply the intermediate power by the base. This produces two rows, again with the keys `2` and `3`. Unlike traditional recursive CTEs, we **do not append** these two new rows to the recurring table. Instead, we **update the existing rows** with keys `2` and `3` in the recurring table, overwriting their payload values. We thus keep only two rows throughout the entire computation, each holding the current power value for its key (or base) in column `a`. | Iteration | Output of recursive step | Working table | Intermediate table | Recurring table | | --------- | ---------------------------------- | -------------------------- | -------------------------- | -------------------------- | | 0 | SELECT 2, 0, 1
SELECT 3, 0, 1 | ∅ (no rows) | (2, 0, 1)
(3, 0, 1) | (2, 0, 1)
(3, 0, 1) | | 1 | 2 * 1 = 2
3 * 1 = 3 | (2, 0, 1)
(3, 0, 1) | (2, 1, 2)
(3, 1, 3) | (2, 1, 2)
(3, 1, 3) | | 2 | 2 * 2 = 4
3 * 3 = 9 | (2, 1, 2)
(3, 1, 3) | (2, 2, 4)
(3, 2, 9) | (2, 2, 4)
(3, 2, 9) | | 3 | 2 * 4 = 8
3 * 9 = 27 | (2, 2, 4)
(3, 2, 9) | (2, 3, 8)
(3, 3, 27) | (2, 3, 8)
(3, 3, 27) | | 4 | 2 * 8 = 16
3 * 27 = 81 | (2, 3, 8)
(3, 3, 27) | (2, 4, 16)
(3, 4, 81) | (2, 4, 16)
(3, 4, 81) | | 5 | 2 * 16 = 32
3 * 81 = 243 | (2, 4, 16)
(3, 4, 81) | (2, 5, 32) | (2, 5, 32)
(3, 4, 81) | | 6 | 2 * 32 = 64 | (2, 5, 32) | (2, 6, 64) | (2, 6, 64)
(3, 4, 81) | | 7 | 2 * 64 = 128 | (2, 6, 64) | ∅ _stop!_ | (2, 6, 64)
(3, 4, 81) | As we can see, the size of the recurring table remains constant. Irrelevant computation history is overwritten which leads to **reduced memory usage**. The final recurring table reads: ```text ┌───────┬───────┬───────┐ │ a │ b │ c │ -- a^b = c │ int32 │ int32 │ int32 │ ├───────┼───────┼───────┤ │ 2 │ 6 │ 64 │ │ 3 │ 4 │ 81 │ └───────┴───────┴───────┘ ``` This behavior is especially useful with algorithms in which we are interested in the **latest**, **best**, or **smallest** value for a given key. The maximum number of rows in the recurring table is now **bounded by the number of unique keys**. Since the number of distinct keys in use is under control of the recursive part, this can be a powerful advantage when working with sizable datasets. ##### A Change of Key {#blog:2025-05-23-using-key::a-change-of-key} Now, _should_ you happen to be interested in the history of the computation and are willing to invest the memory space, all that is required is a change of key. With ```sql WITH RECURSIVE power(a, b, c) USING KEY (a, b) AS ( -- formerly: USING KEY (a) ... ) FROM power ORDER BY a, b; ``` the iteration counter (or exponent) in column `b` is considered to be part of the key, too. The recurring table will thus track unique `(a, b)` combinations (i.e., base, exponent) and we are able to trace what was going on during iteration: ```text ┌───────┬───────┬───────┐ │ a │ b │ c │ │ int32 │ int32 │ int32 │ ├───────┼───────┼───────┤ │ 2 │ 0 │ 1 │ │ 2 │ 1 │ 2 │ │ 2 │ 2 │ 4 │ │ 2 │ 3 │ 8 │ │ 2 │ 4 │ 16 │ │ 2 │ 5 │ 32 │ │ 2 │ 6 │ 64 │ │ 3 │ 0 │ 1 │ │ 3 │ 1 │ 3 │ │ 3 │ 2 │ 9 │ │ 3 │ 3 │ 27 │ │ 3 │ 4 │ 81 │ └───────┴───────┴───────┘ ``` ##### Accessing Relevant History {#blog:2025-05-23-using-key::accessing-relevant-history} Another major difference to vanilla recursive CTEs: now that the size of the recurring table is under control, we can afford to reference it directly in the recursive part of the CTE. This allows us to access _any intermediate result_ that has not been overwritten yet – regardless of which iteration computed these results. No amnesia anymore! To access the recurring table, simply prefix the CTE name with the pseudo-schema name `recurring`: ```sql WITH RECURSIVE t(...) USING KEY (...) AS ( ... FROM recurring.t -- reads the recurring table while we iterate ) ...; ``` #### `USING KEY` Can Unlock Performance Advantages {#blog:2025-05-23-using-key::using-key-can-unlock-performance-advantages} To further highlight the differences between vanilla and key-based CTEs, let's consider a more complex example using a graph dataset – a social network graph. > The graphs were derived from the [LDBC Social Network Benchmark (SNB)](https://ldbcouncil.org/), which provides a generator for synthetic social network data. To make the graphs more manageable for use with the `WITH RECURSIVE` query, we further filtered them by person name, reducing their density. In this dataset, nodes represent people while edges correspond to relationships between them. The tables we’re working with are `Person(id)`, which contains all existing ids in the network and `knows(person1id, person2id)` in which each row holds a pair of people knowing each other. > If you would like to try this out for yourself, begin by attaching the database to any DuckDB session. ```sql ATTACH 'https://blobs.duckdb.org/data/using-key-graph.duckdb'; USE 'using-key-graph'; ``` Our goal is to compute the _shortest path between all pairs of people_ in the social network. Since this is a problem with inherent quadratic complexity, we need to closely eye runtime and memory requirements. For each pair, we start by adding a row that has one person as the start node and the other as the target node. We then iteratively explore all individuals known by the start node, continuing the traversal until the target person is reached. Paths through the network are encoded by `via` nodes: to reach the target from the start node, first proceed to the `via` node (an immediate neighbor of the start node) – once there, use that node's `via` entry to continue your traversal. A recursive CTE that implements this approach reads as follows: ```sql WITH RECURSIVE paths(here, current, via, len, there, completed, found) AS ( SELECT n1.id AS here, n1.id AS current, NULL::BIGINT AS via, 0 AS len, n2.id AS there, false AS completed, false AS found FROM Person AS n1 JOIN Person AS n2 ON (n2.id <> n1.id) UNION ALL SELECT paths.here, person2id AS current, coalesce(paths.via, knows.person2id) AS via, paths.len+1 AS len, paths.there, bool_or(knows.person2id = paths.there) OVER (PARTITION BY paths.here, paths.there ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS completed, knows.person2id = paths.there AS found FROM paths JOIN knows ON (paths.current = knows.person1id AND NOT paths.completed) ) SELECT here, there, via, len FROM paths WHERE found ORDER BY len, here, there; ``` > The working table of recursive CTE `paths` has seven columns: > > - `here` refers to the person where the traversal starts, > - `there` refers to the target person we aim to reach, > - `via` indicates the immediate neighbor of the start node of the current path, > - `len` is the length of the current path, which is incremented by one for each step taken in the traversal. > > The remaining three columns control the traversal logic and optimize the computation. > > - `current` refers to the node that is currently being explored during the traversal, > - `found` indicates whether the current path has successfully reached the target person, > - `completed` tracks whether any path with the same `here` and `there` has already reached the target. This prevents further traversal for that pair once a shortest path has been found, thus avoiding the exploration of longer paths. If we search in a large graph with many edges, the **union table** in a vanilla recursive CTE can grow large, potentially exceeding memory limits. This not only causes significant **performance issues** but can even lead to [**query crashes**](https://duckdb.org/docs/1.5/guides/troubleshooting/oom_errors) in extreme cases. The new key-based CTEs can avoid this problem by changing how the state of the search is maintained. This enables a range of new algorithms to be expressed efficiently in SQL, including algorithms for finding the shortest paths in large graphs. One such algorithm is [**Distance Vector Routing (DVR)**](https://en.wikipedia.org/wiki/Distance-vector_routing_protocol), a method for computing the shortest paths in a network based on node-local routing table that indicate where to “hop next”. - In DVR, each node maintains a **routing table** that records the length of the paths that reach other nodes. - We use the **recurring table** to store these routing tables for _all_ nodes in the network: a row `(here, there, via, len)` indicates that the first hop on the path from node `here` to node `there` is node `via`. The length of the overall path is `len`. - When a shorter path to a target node is found, the corresponding routing table entry is updated. - Routing updates found in the last iteration are distributed to neighboring nodes through the **working table**. To check if a newly incoming routing update improves a currently known path, we perform a lookup in the recurring table. If the new length is smaller than that of the known path, we update the entry and propagate the update to our immediate neighbors. This mechanism allows for efficient path finding even in very large graphs – see below how DVR outperforms the above approach based on a vanilla recursive CTEs. ```sql WITH RECURSIVE dvr(here, there, via, len) USING KEY (here, there) AS ( -- initialize routing tables for all nodes, only the routes to -- immediate neighbors are known at this time SELECT n.person1id AS here, n.person2id AS there, n.person2id AS via, 1::DOUBLE AS len FROM knows AS n UNION (SELECT n.person1id AS here, dvr.there, dvr.here AS via, 1 + dvr.len AS len FROM dvr -- working table - routing updates shared by neighbors JOIN knows AS n ON (n.person2id = dvr.here AND -- send update only to immediate neighbors n.person1id <> dvr.there) -- no need to store a route to myself LEFT JOIN recurring.dvr AS rec ON -- recurring table (current routing tables) (rec.here = n.person1id AND rec.there = dvr.there) -- identify affected routing table entry WHERE 1 + dvr.len < coalesce(rec.len, 'Infinity'::DOUBLE) -- does the routing update improve the entry in the routing table? ORDER BY len -- shortest path first ) ) FROM dvr ORDER BY len, here, there; ``` The larger the social network graphs (A to G), the more pronounced is the performance gap between vanilla recursive CTEs (REC) and the new `USING KEY` variant (KEY). The table below reports **the number of rows processed during each iteration**: | graph | nodes | edges | KEY | REC | | :---: | ----: | -----: | ------: | ----------: | | A | 184 | 233 | 744 | 352,906 | | B | 322 | 903 | 8,232 | 40,732,577 | | C | 424 | 1,446 | 19,213 | 605,859,791 | | D | 484 | 2,049 | 30,871 | ❌ | | E | 1,119 | 8,809 | 255,425 | ❌ | | F | 1,481 | 14,256 | 491,880 | ❌ | | G | 1,618 | 16,619 | 607,926 | ❌ | Even for the smallest graph, Graph A, the difference is significant: the REC CTE produces around 350,000 rows, while KEY generates only 744 rows. As the graph size increases, the gap becomes even more striking. In Graph C, with 424 nodes and 1,446 edges, the REC approach processes nearly 1 billion rows, while the KEY method handles less than 20,000 rows. Although this isn't the largest graph in our benchmark, the REC approach is already approaching out-of-memory conditions (OOM, ❌). This substantial difference in memory usage is only part of the story. The performance of the REC approach also degrades quickly. While both CTEs perform similarly on small graphs, REC becomes significantly slower as the graph grows – eventually crashing – whereas KEY continues to scale smoothly:
And that’s the beauty of the new `USING KEY` CTEs.
It enables more efficient expression of complex iterative algorithms by
providing key-based control over the size of intermediate results that are
passed from iteration to iteration – memory pressure lessens, run time
performance can go up significantly. If you're working with recursive CTEs
in DuckDB, be sure to take advantage of this powerful addition – it may
make a significant difference in your queries.
## DuckLake: SQL as a Lakehouse Format {#blog:2025-05-27-ducklake}
**Publication date:** 2025-05-27
**Authors:** Mark Raasveldt, Hannes Mühleisen
**TL;DR:** DuckLake simplifies lakehouses by using a standard SQL database for all metadata, instead of complex file-based systems, while still storing data in open formats like Parquet. This makes it more reliable, faster, and easier to manage.
## Faster Dashboards with Multi-Column Approximate Sorting {#blog:2025-06-06-advanced-sorting-for-fast-selective-queries}
**Publication date:** 2025-06-06
**Author:** Alex Monahan
**TL;DR:** With any columnar data format, using advanced multi-column sorting when loading data can improve performance on a wide variety of selective read queries. Space filling curve encodings like Morton (Z-Order) and Hilbert approximately sort multiple columns together. Sorting by rounded timestamps adds additional benefits when filtering on recent data.
An animated Hilbert space filling curve by TimSauder – Own work, CC BY-SA 4.0
It is rare to have a dashboard with a single chart.
It is even more rare to only query your data one way.
It is rarer still to query your data with no filters at all!
When queries read a subset of the total rows, sorting data while loading provides significant benefits.
Please have a look at [the prior post in this series](#blog:2025-05-14-sorting-for-fast-selective-queries) to understand how this approach works and for practical tips!
In order to see these benefits in a wider variety of real world cases, we have to get a bit more creative.
These advanced techniques help when one of these situations occur:
- Queries filter on different columns
- Query patterns are not perfectly predictable
- Queries filter by time and at least one other column
In this post, we describe several advanced sorting strategies, compare them with some experiments (microbenchmarks), and then calculate a metric to measure their effectiveness.
#### The Strategy {#blog:2025-06-06-advanced-sorting-for-fast-selective-queries::the-strategy}
Instead of sorting precisely by one or a small number of columns, we want to sort approximately by a larger number of columns.
That will allow queries with different `WHERE` clauses to all benefit from DuckDB's [min-max indexes (zone maps)](https://duckdb.org/docs/1.5/sql/indexes#min-max-index-zonemap).
This post introduces two high-level approaches with several examples of each: sorting by space filling curves, and sorting by truncated timestamps.
##### Space Filling Curves {#blog:2025-06-06-advanced-sorting-for-fast-selective-queries::space-filling-curves}
Both Morton and Hilbert are space filling curve algorithms that are designed to combine multiple columns into an order that preserves some approximate ordering for both columns.
One application of space filling curves is in geospatial analytics and it is a helpful illustration.
If a dataset contained the latitude and longitude coordinates of every café on earth (one row per café), but we wanted to sort so that cafés that are physically close to one another are near each other in the list, we could use a space filling curve.
Cafés that are somewhat close in both latitude and longitude will receive a similar Morton or Hilbert encoding value.
This will allow us to quickly execute queries like “Find all cafés within this rectangular region on a map”.
(A rectangular region like that is called a bounding box in geospatial-land!)
The [GIF at the top of this post](#blog:2025-06-06-advanced-sorting-for-fast-selective-queries::hilbert-curve) shows various levels of granularity of a Hilbert encoding – imagine if the x axis were longitude and the y axis were latitude.
The “zig-zag” line of the Hilbert algorithm is the list of cafés, sorted approximately.
Both Morton and Hilbert operate on integers or floats, but this post outlines a way to use them to sort `VARCHAR` columns as well.
A SQL macro is used to convert the first few characters of a `VARCHAR` column into an integer as a pre-processing step.
##### Truncated Timestamps {#blog:2025-06-06-advanced-sorting-for-fast-selective-queries::truncated-timestamps}
More recent data tends to be more useful data, so frequently queries filter on a time column.
However, often queries filter on time and on other columns as well.
**Don't just sort on the timestamp column!**
You will miss out on performance benefits.
This is because timestamps tend to be so granular, that in practical terms the data is only sorted by timestamp.
How many rows of your data were inserted at exactly `2025-01-01 01:02:03.456789`?
Probably just one!
To sort on multiple columns as well as a time column, first sort by a truncated timestamp (truncated to the start of the day, week, year, etc.) and then on the other columns.
#### The “On-Time Flights” Dataset {#blog:2025-06-06-advanced-sorting-for-fast-selective-queries::the-on-time-flights-dataset}
Ever wondered how likely you will be delayed when you are getting ready to fly?
If you are traveling in the US, you can use a government dataset to see how on-time each route is!
You can [get more details on the dataset at its official site](https://www.transtats.bts.gov/DatabaseInfo.asp?QO_VQ=EFD%20&Yv0x=D), but I grabbed a [Parquet version of the data from Kaggle](https://www.kaggle.com/datasets/robikscube/flight-delay-dataset-20182022).
It includes almost 5 years of data on US flights.
Combined, the Parquet files are about 1.1 GB.
This is fairly small when accessed locally but can be more of a challenge over a slow remote connection.
In the rest of the post, we'll run different benchmarks to demonstrate how pruning can help reducing the amount of network traffic required to answer certain queries.
Our first experiment is a hypothetical use case to serve a dashboard where people can explore a single `origin`, a single `dest` (destination), or an `origin`–`dest` pair.
The `origin` and `dest` columns are three-letter airport codes (like `LAX` or `JFK`).
We assume once you have picked one of those filters, you want all the rows that match and a few arbitrary columns.
Our second experiment will add in a time component where our hypothetical users could additionally filter to the latest 1, 13, or 52 weeks of data in the dataset.
#### Experimental Design {#blog:2025-06-06-advanced-sorting-for-fast-selective-queries::experimental-design}
We will show a few approaches to sort data approximately to speed up retrieval.
The control groups are:
- `random`: Sort by a hash – the worst case scenario!
- `origin`: Single-column sort on `origin`
- `origin-dest`: Sort on `origin`, then on `destination`
Our alternative sorting approaches are:
- `zipped_varchar`: Sort by one letter at a time, alternating between `origin` and `destination`.
– For example, an `origin` of `ABC` and a `destination` of `XYZ` would become `AXBYCZ`, which would then be sorted.
- `morton`: Convert `origin` and `destination` into integers, then order by [Morton encoding (Z-order)](https://en.wikipedia.org/wiki/Z-order_curve)
- `hilbert`: Convert `origin` and `destination` into integers, then order by [Hilbert encoding](https://en.wikipedia.org/wiki/Hilbert_curve)
> The `zipped_varchar` algorithm is implemented using a SQL macro so no extensions are needed.
> It also handles arbitrary length strings.
> The Morton and Hilbert encoding functions come from the [`lindel` DuckDB community extension](https://duckdb.org/community_extensions/extensions/lindel), contributed by [Rusty Conover](https://github.com/rustyconover).
> Thank you to Rusty and the folks who have built the [`lindel` Rust crate](https://crates.io/crates/lindel) upon which the DuckDB extension is based!
> The [`spatial` extension](https://duckdb.org/docs/1.5/core_extensions/spatial/overview) also contains an [`ST_Hilbert` function](https://duckdb.org/docs/1.5/core_extensions/spatial/functions#st_hilbert) that works similarly.
> Thanks to [Max Gabrielsson](https://github.com/Maxxen) and the GDAL community!
These plots display query runtime when pulling from a DuckDB file hosted on S3.
These same techniques can also be successfully applied to the [DuckLake](https://ducklake.select/) integrated data lake and catalog format!
DuckLake is the modern evolution of the cloud data lakehouse – take a minute to check out the [launch post](#blog:2025-05-27-ducklake) if you haven't yet!
DuckLake has an [additional concept of a partition](https://ducklake.select/docs/stable/duckdb/advanced_features/partitioning), which enables entire files to be skipped.
To take full advantage of DuckLake, first partition your data (by time or otherwise) and then apply the techniques in this post when loading your data.
All experiments were run on an M1 MacBook Pro with DuckDB v1.2.2.
The tests were conducted with the DuckDB Python client with results returned as Apache Arrow tables.
In between each query, the DuckDB connection is closed and recreated (this time is not measured as a part of the results).
This is to better simulate a single user's experience accessing the dashboard in our hypothetical use case.
Expand to see the basic sorting queries
For reproducibility, here are the very standard queries used to initially load the data from Parquet, sort randomly, sort by `origin`, and sort by `origin` and then `destination`. The Parquet files were downloaded [from Kaggle](https://www.kaggle.com/datasets/robikscube/flight-delay-dataset-20182022). ```sql CREATE TABLE IF NOT EXISTS flights AS FROM './Combined_Flights*.parquet'; -- The hash function is used instead of random -- for consistency across re-runs CREATE TABLE IF NOT EXISTS flights_random AS FROM flights ORDER BY hash(rowid + 42); CREATE TABLE IF NOT EXISTS flights_origin AS FROM flights ORDER BY origin; CREATE TABLE IF NOT EXISTS flights_origin_dest AS FROM flights ORDER BY origin, dest; ```Expand to see the Zipped `VARCHAR` sorting queries
As an example of an approach that does not require an extension, this SQL macro roughly approximates a space filling curve approach, but using alphanumeric characters instead of integers. The outcome is a dataset that is somewhat sorted by one column and somewhat sorted by another. ```sql CREATE OR REPLACE FUNCTION main.zip_varchar(i, j, num_chars := 6) AS ( -- By default using 6 characters from each string so that -- if data is ASCII, we can fit it all in 12 bytes so that it is stored inline -- rather than requiring a pointer [ list_value(z[1], z[2]) FOR z IN list_zip( substr(i, 1, num_chars).rpad(num_chars, ' ').string_split(''), substr(j, 1, num_chars).rpad(num_chars, ' ').string_split('') ) ].flatten().array_to_string('') ); CREATE TABLE IF NOT EXISTS flights_zipped_varchar AS FROM flights ORDER BY main.zip_varchar(origin, dest, num_chars := 3); ``` Here is an example of the output that the `zip_varchar` function produces: ```sql SELECT 'ABC' AS origin, 'XYZ' AS dest, main.zip_varchar(origin, dest, num_chars := 3) AS zipped_varchar; ``` | origin | dest | zipped_varchar | | ------ | ---- | -------------- | | ABC | XYZ | AXBYCZ |Expand to see the Morton and Hilbert sorting queries
The goal of a space filling curve is to map multiple dimensions (in our case, two: `origin` and `destination`) down to a single dimension, but to preserve the higher dimension locality between data points. Morton and Hilbert encodings are designed to accept integers or floating point numbers. However, in our examples, we want to apply these techniques to `VARCHAR` columns. Strings actually encode a substantial amount of data per length of the string. This is because numbers can only have 10 values per digit (in our base 10 numbering system), but a string can have many more (all lowercase letters, capital letters, or symbols). As a result, we are not able to encode very long strings into integers – only the first few characters. This will still work for our approximate sorting use case! This SQL function can convert a `VARCHAR` containing ASCII characters (up to 8 characters in length) into a `UBIGINT`. It splits the `VARCHAR` up into individual characters, calculates the ASCII number for that character, converts that to bits, concatenates the bits together, then converts to a `UBIGINT`. ```sql CREATE OR REPLACE FUNCTION main.varchar_to_ubigint(i, num_chars := 8) AS ( -- The maximum number of characters that will fit in a UBIGINT is 8 -- and a UBIGINT is the largest type that the lindel community extension accepts for Hilbert or Morton encoding list_reduce( [ ascii(my_letter)::UTINYINT::BIT::VARCHAR FOR my_letter IN (i[:num_chars]).rpad(num_chars, ' ').string_split('') ], (x, y) -> x || y )::BIT::UBIGINT ); ``` The `morton_encode` and `hilbert_encode` functions from the [`lindel` community extension](https://duckdb.org/community_extensions/extensions/lindel) can then be used within the `ORDER BY` clause to sort by the Morton or Hilbert encoding. ```sql INSTALL lindel FROM community; LOAD lindel; CREATE TABLE IF NOT EXISTS flights_morton AS FROM flights ORDER BY morton_encode([ varchar_to_ubigint(origin, num_chars := 3), varchar_to_ubigint(dest, num_chars := 3) ]::UBIGINT[2]); CREATE TABLE IF NOT EXISTS flights_hilbert AS FROM flights ORDER BY hilbert_encode([ varchar_to_ubigint(origin, num_chars := 3), varchar_to_ubigint(dest, num_chars := 3) ]::UBIGINT[2]); ``` Alternatively, the [`spatial` extension](https://duckdb.org/docs/1.5/core_extensions/spatial/overview) can be used to execute a Hilbert encoding. It requires a bounding box to be supplied, as this helps determine the granularity of the encoding for geospatial use cases. It performed similarly to the Hilbert approach included in the experimental results. ```sql SET VARIABLE bounding_box = ( WITH flights_converted_to_ubigint AS ( FROM flights SELECT *, varchar_to_ubigint(origin, num_chars := 3) AS origin_ubigint, varchar_to_ubigint(dest, num_chars := 3) AS dest_ubigint ) FROM flights_converted_to_ubigint SELECT { min_x: min(origin_ubigint), min_y: min(dest_ubigint), max_x: max(origin_ubigint), max_y: max(dest_ubigint) }::BOX_2D ); CREATE OR REPLACE TABLE flights_hilbert_spatial AS FROM flights ORDER BY ST_Hilbert( varchar_to_ubigint(origin, num_chars := 3), varchar_to_ubigint(dest, num_chars := 3), getvariable('bounding_box') ); ```Expand to see the selective read queries used to measure performance
In summary, this microbenchmark tests: 1. Querying tables sorted by the 6 different approaches (3 control and 3 advanced) 1. Filtering using three query patterns: filters on `origin`, `destination`, and `origin` / `destination` 1. Filtering with 4 different `origin`/`dest` pairs: 1. SFO – LAX 1. LAX – SFO 1. ORD – LGA 1. LGA – ORD Each of these 72 unique queries are repeated 3 times for a total of 216 queries. All are included in the resulting plots. ```sql FROM ⟨sorted_table⟩ SELECT flightdate, airline, origin, dest, deptime, arrtime WHERE origin = ⟨origin⟩; FROM ⟨sorted_table⟩ SELECT flightdate, airline, origin, dest, deptime, arrtime WHERE dest = ⟨dest⟩; FROM ⟨sorted_table⟩ SELECT flightdate, airline, origin, dest, deptime, arrtime WHERE origin = ⟨origin⟩ AND dest = ⟨dest⟩; ```Expand to see the approximate time sorting queries
```sql CREATE TABLE IF NOT EXISTS flights_hilbert_day AS FROM flights ORDER BY date_trunc('day', flightdate), hilbert_encode([ varchar_to_ubigint(origin, num_chars := 3), varchar_to_ubigint(dest, num_chars := 3) ]::UBIGINT[2]); CREATE TABLE IF NOT EXISTS flights_hilbert_month AS FROM flights ORDER BY date_trunc('month', flightdate), hilbert_encode([ varchar_to_ubigint(origin, num_chars := 3), varchar_to_ubigint(dest, num_chars := 3) ]::UBIGINT[2]); CREATE TABLE IF NOT EXISTS flights_hilbert_year AS FROM flights ORDER BY date_trunc('year', flightdate), hilbert_encode([ varchar_to_ubigint(origin, num_chars := 3), varchar_to_ubigint(dest, num_chars := 3) ]::UBIGINT[2]); ```Expand to see the selective read queries used to measure performance
In summary, this microbenchmark tests: 1. Querying tables sorted with `flightdate` truncated to three levels of granularity: day, month, and year, then by Hilbert encoding. 1. Filtering on three time ranges: the latest 1 week, 13 weeks, and 52 weeks 1. Filtering on `origin`, `destination`, and `origin` / `destination` 1. Filtering with 4 different `origin`/`dest` pairs: 1. SFO – LAX 1. LAX – SFO 1. ORD – LGA 1. LGA – ORD Each of these 108 unique queries are repeated 3 times for a total of 324 queries. ```sql FROM ⟨sorted_table⟩ SELECT flightdate, airline, origin, dest, deptime, arrtime WHERE origin = ⟨origin⟩ AND flightdate >= ('2022-07-31'::TIMESTAMP – INTERVAL ⟨time_range⟩ WEEKS); FROM ⟨sorted_table⟩ SELECT flightdate, airline, origin, dest, deptime, arrtime WHERE dest = ⟨dest⟩ AND flightdate >= ('2022-07-31'::TIMESTAMP – INTERVAL ⟨time_range⟩ WEEKS); FROM ⟨sorted_table⟩ SELECT flightdate, airline, origin, dest, deptime, arrtime WHERE origin = ⟨origin⟩ AND dest = ⟨dest⟩ AND flightdate >= ('2022-07-31'::TIMESTAMP – INTERVAL ⟨time_range⟩ WEEKS); ```Expand to see the macros for calculating “Number of Row Groups per Value”
```sql -- These are helper functions for writing dynamic SQL -- sq = single quotes -- dq = double quotes -- nq = no quotes CREATE OR REPLACE FUNCTION sq(my_varchar) AS ( '''' || replace(my_varchar,'''', '''''') || '''' ); CREATE OR REPLACE FUNCTION dq(my_varchar) AS ( '"' || replace(my_varchar,'"', '""') || '"' ); CREATE OR REPLACE FUNCTION nq(my_varchar) AS ( replace(my_varchar, ';', 'No semicolons are permitted here') ); CREATE OR REPLACE FUNCTION dq_list(my_list) AS ( list_transform(my_list, (i) -> dq(i)) ); CREATE OR REPLACE FUNCTION nq_list(my_list) AS ( list_transform(my_list, (i) -> nq(i)) ); CREATE OR REPLACE FUNCTION dq_concat(my_list, separator) AS ( list_reduce(dq_list(my_list), (x, y) -> x || separator || y) ); CREATE OR REPLACE FUNCTION nq_concat(my_list, separator) AS ( list_reduce(nq_list(my_list), (x, y) -> x || separator || y) ); -- This function produces the "Number of row groups per Value" boxplot CREATE OR REPLACE FUNCTION rowgroup_counts(table_name, column_list) AS TABLE ( FROM query(' WITH by_rowgroup_id AS ( FROM ' || dq(table_name) || ' SELECT ceiling((count(*) OVER ()) / 122_880) AS total_row_groups, floor(rowid / 122_880) AS rowgroup_id, ' || dq_concat(column_list, ',') || ' ), rowgroup_id_counts AS ( FROM by_rowgroup_id SELECT case ' || nq_concat(list_transform(column_list, (i) -> ' when grouping(' || dq(i) || ') = 0 then alias(' || dq(i) || ') '),' ') || ' end AS column_name, coalesce(*columns(* exclude (rowgroup_id, total_row_groups))) AS column_value, first(total_row_groups) AS total_row_groups, count(distinct rowgroup_id) AS rowgroup_id_count GROUP BY GROUPING SETS ( ' || nq_concat(list_transform(dq_list(column_list), (j) -> '(' || j || ')'), ', ') ||' ) ) FROM rowgroup_id_counts SELECT ' || sq(table_name) || ' AS table_name, * ORDER BY column_name ') ); -- This is an optional function that can summarize the data -- as an alternative to boxplot charts CREATE OR REPLACE FUNCTION summarize_rowgroup_counts(table_name, column_list) AS TABLE ( FROM rowgroup_counts(table_name, column_list) SELECT table_name, column_name, total_row_groups, min(rowgroup_id_count) AS min_cluster_depth, avg(rowgroup_id_count) AS avg_cluster_depth, max(rowgroup_id_count) AS max_cluster_depth, map([0.1, 0.25, 0.5, 0.75, 0.9], quantile_cont(rowgroup_id_count, [0.1, 0.25, 0.5, 0.75, 0.9]))::JSON AS quantiles, histogram(rowgroup_id_count, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 32, 64, 128, 256])::JSON AS histograms, GROUP BY ALL ORDER BY ALL ); ```
#### Keyword Search {#blog:2025-06-13-text-analytics::keyword-search}
*Keyword search* is the most basic form of text retrieval, matching exact words or phrases in text fields using SQL conditions such as `CONTAINS`, `ILIKE`, or other DuckDB [text functions](https://duckdb.org/docs/1.5/sql/functions/text).
It is fast, requires no preprocessing, and works well for structured queries like filtering logs, matching tags, or finding product names.
For example, getting the texts and their emotion label containing the phrase `excited to learn` is a matter of applying `filter` on the relation defined above:
```python
text_emotions_rel.filter("text ilike '%excited to learn%'").select("""
emotion,
substring(
text,
position('excited to learn' in text),
len('excited to learn')
) as substring_text
""")
```
```text
┌─────────┬──────────────────┐
│ emotion │ substring_text │
│ varchar │ varchar │
├─────────┼──────────────────┤
│ sadness │ excited to learn │
│ joy │ excited to learn │
│ joy │ excited to learn │
│ fear │ excited to learn │
│ fear │ excited to learn │
│ joy │ excited to learn │
│ joy │ excited to learn │
│ sadness │ excited to learn │
└─────────┴──────────────────┘
```
One common step in text processing is to split text into tokens (keywords), where raw text is broken down into smaller units (typically words), that can be analyzed or indexed. This process, known as *tokenization*, helps convert unstructured text into a structured form suitable for keyword search. In DuckDB this process can be implemented with the [`regexp_split_to_table` function](https://duckdb.org/docs/1.5/sql/functions/text#regexp_split_to_tablestring-regex), which will split the text based on the provided regex and return each keyword on a row.
> This step is case sensitive, therefore it is important to convert all text to a consistent case (by applying [`lcase`](https://duckdb.org/docs/1.5/sql/functions/text#lcasestring) or [`ucase`](https://duckdb.org/docs/1.5/sql/functions/text#ucasestring)) before processing.
In the below code snippet we select all the keywords by splitting the text on one or more non-word characters (anything except `[a-zA-Z0-9_]`):
```python
text_emotions_tokenized_rel = text_emotions_rel.select("""
text_id,
emotion,
regexp_split_to_table(text, '\\W+') as token
""")
```
In the tokenization step, we usually exclude common words (such as `and`, `the`), called *stopwords*. In DuckDB we implement the exclusion by applying an [`ANTI JOIN`](https://duckdb.org/docs/1.5/sql/query_syntax/from#semi-and-anti-joins) on a curated CSV file hosted on GitHub:
```python
english_stopwords_rel = duckdb_conn.read_csv(
"https://raw.githubusercontent.com/stopwords-iso/stopwords-en/refs/heads/master/stopwords-en.txt",
header=False,
).select("column0 as token")
text_emotions_tokenized_rel.join(
english_stopwords_rel,
condition="token",
how="anti",
).to_table("text_emotion_tokens")
```
Now that we have tokenized and cleaned the text, we can implement keyword search by ranking the match with [similarity functions](https://duckdb.org/docs/1.5/sql/functions/text#text-similarity-functions), such as [Jaccard](https://en.wikipedia.org/wiki/Jaccard_index):
```python
text_token_rel = conn.table(
"text_emotion_tokens"
).select("token, emotion, jaccard(token, 'learn') as jaccard_score")
text_token_rel = text_token_rel.max(
"jaccard_score",
groups="emotion, token",
projected_columns="emotion, token"
)
text_token_rel.order("3 desc").limit(10)
```
```text
┌──────────┬─────────┬────────────────────┐
│ emotion │ token │ max(jaccard_score) │
│ varchar │ varchar │ double │
├──────────┼─────────┼────────────────────┤
│ fear │ learn │ 1.0 │
│ surprise │ learn │ 1.0 │
│ love │ learn │ 1.0 │
│ joy │ lerna │ 1.0 │
│ sadness │ learn │ 1.0 │
│ fear │ learner │ 1.0 │
│ anger │ learn │ 1.0 │
│ joy │ leaner │ 1.0 │
│ fear │ allaner │ 1.0 │
│ anger │ learner │ 1.0 │
├──────────┴─────────┴────────────────────┤
│ 10 rows 3 columns │
└─────────────────────────────────────────┘
```
We can also visualize the data to gain insights. One simple and effective approach is to plot the most frequently used words. By counting token occurrences across the dataset and displaying them in bubble plots, we can quickly identify dominant themes, repeated keywords, or unusual patterns in the text. For example, we plot the data by using a [scatter facet plot](https://plotly.com/python/facet-plots/) per emotion:
From the above plot, we observe repeated keywords, such as `feel - feeling`, `love - loved - loving`. In order to de-duplicate such data, we need to look at the [word stem](https://en.wikipedia.org/wiki/Word_stem) rather than at the word itself. This brings us to full-text search.
#### Full-Text Search {#blog:2025-06-13-text-analytics::full-text-search}
The [Full-Text Search (FTS) DuckDB extension](https://duckdb.org/docs/1.5/core_extensions/full_text_search) is an experimental extension, which implements two main [full-text search](https://cloud.google.com/discover/what-is-full-text-search) functionalities:
- the `stem` function, to retrieve the word stem;
- the `match_bm25` function, to calculate the [Best Match score](https://en.wikipedia.org/wiki/Okapi_BM25).
By applying `stem` to the token column, we can now visualize the most frequently used word stem in our data:
We observe that `feel` and `love` appear only once and new word stems are plotted, such as `support`, `surpris`.
While the `stem` function can be used standalone, the `match_bm25` one requires the build of a FTS index, a special index that allows fast and efficient searching of text by indexing the words (tokens) in a column:
```python
conn.sql("""
PRAGMA create_fts_index(
"text_emotions",
text_id,
"text",
stemmer = 'english',
stopwords = 'english_stopwords',
ignore = '(\\.|[^a-z])+',
strip_accents = 1,
lower = 1,
overwrite = 1
)
""")
```
In the FTS index creation we are using the same list of English stopwords as in the tokenization process, by saving it into a table, named `english_stopwords`. The index is case insensitive due to the `lower` parameter, which will lowercase the text automatically.
> **Warning.** The index can be created only on tables and it requires a unique identifier of the text. It also needs to be rebuilt when the underlying data has been modified.
Once the index has been created, we can rank the match between the `text` column and the phrase `excited to learn`:
```python
text_emotions_rel
.select("""
emotion,
text,
emotion_color,
fts_main_text_emotions.match_bm25(
text_id,
'excited to learn'
)::decimal(3, 2) as bm25_score
""")
.order("bm25_score desc")
.limit(10)
```
Out of the 10 returned texts, displayed above in a [table plot](https://plotly.com/python/table/), 2 are poor matches to our search input; likely due to BM25 scoring being skewed by common terms or differences in document length.
#### Semantic Search {#blog:2025-06-13-text-analytics::semantic-search}
Compared to keyword and full-text search, *semantic search* takes into account the meaning and context of the text. Instead of just looking for exact words, it uses techniques like [vector embeddings](https://github.com/veekaybee/what_are_embeddings) to capture the underlying concepts. Semantic search, which is case insensitive, can be implemented in DuckDB, by making use of the (also experimental) [Vector Similarity Search extension](https://duckdb.org/docs/1.5/core_extensions/vss).
The vector embeddings of a (list of) text can be calculated with the [`sentence-transformers` library](https://sbert.net/) and the [pre-trained model `all-MiniLM-L6-v2`](https://sbert.net/docs/sentence_transformer/pretrained_models.html):
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
def get_text_embedding_list(list_text: list[str]):
"""
Return the list of normalized vector embeddings for list_text.
"""
return model.encode(list_text, normalize_embeddings=True)
```
For example, `get_text_embedding_list(['excited to learn'])` will return:
```text
array([[ 3.14795598e-02, -6.66208193e-02, 1.05058309e-02,
4.12571728e-02, -8.67664907e-03, -1.79746319e-02,
...
-2.50727013e-02, -3.00881546e-03, 1.55055271e-02]], dtype=float32)
```
We register the model inference function as a [Python User Defined Function](https://duckdb.org/docs/1.5/clients/python/function) and create a table with a column of type `FLOAT[384]` to load the embeddings into:
```python
conn.create_function(
"get_text_embedding_list",
get_text_embedding_list,
return_type='FLOAT[384][]'
)
conn.sql("""
create table text_emotion_embeddings (
text_id integer,
text_embedding FLOAT[384]
)
""")
```
With the Python UDF, we save, in batches, the model output in `text_emotion_embeddings`:
```python
for i in range(num_batches):
selection_query = (
duckdb_conn.table("text_emotions")
.order("text_id")
.limit(batch_size, offset=batch_size*i)
.select("*")
)
(
selection_query.aggregate("""
array_agg(text) as text_list,
array_agg(text_id) as id_list,
get_text_embedding_list(text_list) as text_emb_list
""").select("""
unnest(id_list) as text_id,
unnest(text_emb_list) as text_embedding
""")
).insert_into("text_emotion_embeddings")
```
> We wrote about model inference in DuckDB in the post [Machine Learning Prototyping with DuckDB and scikit-learn](#blog:2025-05-16-scikit-learn-duckdb).
We can now perform semantic search by using the [cosine distance](https://en.wikipedia.org/wiki/Cosine_similarity#Cosine_distance) between the vector embedding of our search text, `excited to learn`, and the embedding of the `text` field:
```python
input_text_emb_rel = conn.sql("""
select get_text_embedding_list(['excited to learn'])[1] as input_text_embedding
""")
text_emotions_rel
.join(conn.table("text_emotion_embeddings"), condition="text_id")
.join(input_text_emb_rel, condition="1=1")
.select("""
text,
emotion,
emotion_color,
array_cosine_distance(
text_embedding,
input_text_embedding
)::decimal(3, 2) as cosine_distance_score
""")
.order("cosine_distance_score asc")
.limit(10)
```
Interesting to observe that the phrase `i am excited to learn and feel privileged to be here` didn't make it in the top 10 results in our semantic search!
##### Similarity Joins {#blog:2025-06-13-text-analytics::similarity-joins}
Vector embeddings are most known for their usability in search engines, but they can be used in a variety of text analytics use cases, such as topic grouping, classification or semantic matching between documents. The VSS extension provides [vector similarity joins](https://duckdb.org/docs/1.5/core_extensions/vss#bonus-vector-similarity-search-joins), which can be used to conduct these types of analytics.
For example, we show in the below [heatmap chart](https://plotly.com/python/heatmaps/) the number of texts for each combination of emotion labels, where the x-axis corresponds to the semantic matching between the text and the emotion, the y-axis to the classified emotion, and the color indicates the count of texts assigned to each pair:
It is particularly noticeable that from the 6 emotions, only `sadness` has a strong semantic match with the text classified with the same label. As with full-text search, semantic search is affected by differences in document length (in this case the emotion keyword versus a text).
#### Hybrid Search {#blog:2025-06-13-text-analytics::hybrid-search}
While each type of search has its own applicability, we observed that some results are not as expected:
- keyword search and full-text search do not take word meaning into account;
- semantic search scored synonyms higher than the search text.
In practice, the three search methods are combined and used to perform a “hybrid search”, in order to improve the search relevance and accuracy. We start by calculating the score for each type of search, by implementing custom logic, such as a check on the emotion:
```python
if(
emotion = 'joy' and contains(text, 'excited to learn'),
1,
0
) exact_match_score,
fts_main_text_emotions.match_bm25(
text_id,
'excited to learn'
)::decimal(3, 2) as bm25_score,
array_cosine_similarity(
text_embedding,
input_text_embedding
)::decimal(3, 2) as cosine_similarity_score
```
The BM25 score is ranked in descending order and the cosine distance in ascending order. In hybrid search, we use the `array_cosine_similarity` score to ensure the same sort order (in this case descending).
> cosine similarity = 1 - cosine distance
Because the BM25 score can be, in theory, unbounded, we need [to scale the score](https://en.wikipedia.org/wiki/Feature_scaling) to the interval `[0, 1]` by implementing the min-max normalization:
```python
max(bm25_score) over () as max_bm25_score,
min(bm25_score) over () as min_bm25_score,
(bm25_score - min_bm25_score) / nullif((max_bm25_score - min_bm25_score), 0) as norm_bm25_score
```
The hybrid search score is calculated by applying a weight to the BM25 and cosine similarity scores:
```python
if(
exact_match_score = 1,
exact_match_score,
cast(
0.3 * coalesce(norm_bm25_score, 0) +
0.7 * coalesce(cosine_similarity_score, 0)
as
decimal(3, 2)
)
) as hybrid_score
```
And here are the results! Much better, don't you think?
#### Conclusion {#blog:2025-06-13-text-analytics::conclusion}
In this post, we showed how DuckDB can be used for text analytics by combining keyword, full-text, and semantic search techniques. Using the experimental `fts` and `vss` extensions with the `sentence-transformers` library, we demonstrated how DuckDB can support both traditional and modern text analytics workflows.
## Discovering DuckDB Use Cases via GitHub {#blog:2025-06-27-discovering-w-github}
**Publication date:** 2025-06-27
**Author:** Petrica Leuca
**TL;DR:** In this post, we use the GitHub API to find repositories that mention DuckDB, then use DuckDB itself to parse and query the results efficiently with SQL.
#### Introduction {#blog:2025-06-27-discovering-w-github::introduction}
One of the persistent challenges of maintaining an open-source library is the lack of visibility into how, where, and by whom it's being used. At DuckDB we look at the [`awesome-duckdb` repository](https://github.com/davidgasquez/awesome-duckdb), which contains a curated list of DuckDB libraries, tools and resources. To supplement this, we wanted to automate insights from GitHub in order to uncover new and emerging projects using DuckDB. In this post we write about how we use DuckDB to query the GitHub API, export the data into a Markdown file and visualize the historical data, generated from Git commits.
#### Data Retrieval from GitHub {#blog:2025-06-27-discovering-w-github::data-retrieval-from-github}
GitHub provides a [series of REST API](https://docs.github.com/en/rest?apiVersion=2022-11-28) which can be used to retrieve data from GitHub. Among them, the [search repositories API](https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-repositories) returns a list of public repositories whose name, description or readme matches the search text.
> GitHub also has a search code API in order to match on code dependency, but it does not return all the information the search repository provides. One could use the search code API and for each result to retrieve the repository information from the repository API.
In order to use the GitHub API, we create an [access token](https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-repositories--fine-grained-access-tokens) with read access on public repositories. With the access token, saved as an environment variable, we create a secret in DuckDB to be used by the API request:
```python
import os
import duckdb
def get_duckdb_conn():
conn = duckdb.connect()
conn.sql(f"""
CREATE SECRET http_auth (
TYPE http,
BEARER_TOKEN '{os.getenv("READ_PUBLIC_REPO_TOKEN")}',
SCOPE 'https://api.github.com/search'
);
""")
return conn
```
With the DuckDB Python client's [`read_json`](https://duckdb.org/docs/1.5/clients/python/relational_api#read_json) function we are able to query the API response with SQL. The above secret is passed automatically by DuckDB to the API call:
```python
duckdb_conn = get_duckdb_conn()
api_response = duckdb_conn.read_json(
"https://api.github.com/search/repositories?q=duckdb"
)
```
The API call returns the following response:
```text
┌─────────────┬────────────────────┬──────────────────────────────────────┐
│ total_count │ incomplete_results │ items │
│ int64 │ boolean │ struct(id bigint, node_id varchar,...│
├─────────────┼────────────────────┼──────────────────────────────────────┤
│ 3837 │ false │ [{'id': 138754790, 'node_id': ... │
└─────────────┴────────────────────┴──────────────────────────────────────┘
```
We can analyze the response by:
- getting the list of columns with `api_response.columns`:
```text
['total_count', 'incomplete_results', 'items']
```
- getting the column types `api_response.types`:
```sql
[
BIGINT,
BOOLEAN,
STRUCT(id BIGINT, node_id VARCHAR, "name" VARCHAR, full_name VARCHAR, private BOOLEAN, "owner" STRUCT(login VARCHAR, id BIGINT, node_id VARCHAR, avatar_url VARCHAR, gravatar_id VARCHAR, url VARCHAR, html_url VARCHAR, followers_url VARCHAR, following_url VARCHAR, gists_url VARCHAR, starred_url VARCHAR, subscriptions_url VARCHAR, organizations_url VARCHAR, repos_url VARCHAR, events_url VARCHAR, received_events_url VARCHAR, "type" VARCHAR, user_view_type VARCHAR, site_admin BOOLEAN), html_url VARCHAR, description VARCHAR, fork BOOLEAN, url VARCHAR, forks_url VARCHAR, keys_url VARCHAR, collaborators_url VARCHAR, teams_url VARCHAR, hooks_url VARCHAR, issue_events_url VARCHAR, events_url VARCHAR, assignees_url VARCHAR, branches_url VARCHAR, tags_url VARCHAR, blobs_url VARCHAR, git_tags_url VARCHAR, git_refs_url VARCHAR, trees_url VARCHAR, statuses_url VARCHAR, languages_url VARCHAR, stargazers_url VARCHAR, contributors_url VARCHAR, subscribers_url VARCHAR, subscription_url VARCHAR, commits_url VARCHAR, git_commits_url VARCHAR, comments_url VARCHAR, issue_comment_url VARCHAR, contents_url VARCHAR, compare_url VARCHAR, merges_url VARCHAR, archive_url VARCHAR, downloads_url VARCHAR, issues_url VARCHAR, pulls_url VARCHAR, milestones_url VARCHAR, notifications_url VARCHAR, labels_url VARCHAR, releases_url VARCHAR, deployments_url VARCHAR, created_at TIMESTAMP, updated_at TIMESTAMP, pushed_at TIMESTAMP, git_url VARCHAR, ssh_url VARCHAR, clone_url VARCHAR, svn_url VARCHAR, homepage VARCHAR, size BIGINT, stargazers_count BIGINT, watchers_count BIGINT, "language" VARCHAR, has_issues BOOLEAN, has_projects BOOLEAN, has_downloads BOOLEAN, has_wiki BOOLEAN, has_pages BOOLEAN, has_discussions BOOLEAN, forks_count BIGINT, mirror_url JSON, archived BOOLEAN, disabled BOOLEAN, open_issues_count BIGINT, license STRUCT("key" VARCHAR, "name" VARCHAR, spdx_id VARCHAR, url VARCHAR, node_id VARCHAR), allow_forking BOOLEAN, is_template BOOLEAN, web_commit_signoff_required BOOLEAN, topics VARCHAR[], visibility VARCHAR, forks BIGINT, open_issues BIGINT, watchers BIGINT, default_branch VARCHAR, permissions STRUCT("admin" BOOLEAN, maintain BOOLEAN, push BOOLEAN, triage BOOLEAN, pull BOOLEAN), score DOUBLE)[]
]
```
- getting the number of items with `api_response.select("len(items)")`:
```text
┌────────────┐
│ len(items) │
│ int64 │
├────────────┤
│ 30 │
└────────────┘
```
The total number of repositories mentioning DuckDB is 3837, but the `items` object contains only 30. This is due to [pagination](https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api?apiVersion=2022-11-28) that is set to 30 records per page by default. We can retrieve a maximum of 100 records by using the `per_page` query parameter. Additionally, we use the `page` query parameter in order to retrieve the results for each page:
```python
(
duckdb_conn
.read_json("https://api.github.com/search/repositories?q=duckdb&per_page=100&page=2")
.select("len(items)")
)
```
```text
┌────────────┐
│ len(items) │
│ int64 │
├────────────┤
│ 100 │
└────────────┘
```
Because GitHub has a limit of returning maximum 1000 results in the search API and we retrieve the data daily, we added a filter to retrieve only the repositories which had a push in the last 7 days:
```python
api_url = "https://api.github.com/search/repositories?q=duckdb"
last_pushed_date = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
api_url = f"{api_url}+pushed:>={last_pushed_date}"
duckdb_conn.read_json(f"{api_url}&per_page=100&page=1").to_table("github_raw_data")
```
We store the first page in a table called `github_raw_data` and calculate how many pages we need to retrieve based on the total count returned by the first call. Then, with the [`insert_into`](https://duckdb.org/docs/1.5/clients/python/relational_api#insert_into) method, we append the data from each page into `github_raw_data`:
```python
for page in range(2, number_pages + 1):
logger.info(f"Fetching {page} out of {number_pages}")
(
duckdb_conn
.read_json(f"{api_url}&per_page=100&page={page}")
.insert_into("github_raw_data")
)
```
#### Saving Data to a Markdown File {#blog:2025-06-27-discovering-w-github::saving-data-to-a-markdown-file}
With the data available in a table, we can continue our data processing, by using the [`unnest` function](https://duckdb.org/docs/1.5/sql/query_syntax/unnest), which will flatten the `items` returned by the API:
```python
(
duckdb_conn.table("github_raw_data")
.select("unnest(items, recursive := true)")
)
```
By using [`recursive` unnesting](https://duckdb.org/docs/1.5/sql/query_syntax/unnest#recursive-unnest), the `STRUCT` objects within `items` will be flattened too; below is a sample of the columns derived from `items` after unnesting:
```text
['id',
'node_id',
'name',
'full_name',
'private',
'login',
'id',
...
'pull',
'score']
```
Our scope is to create a Markdown file containing a table, with the following format:
- **name**, containing the name, link, description, license and owner of the repository;
- **topics**, the list of topics of the repository;
- **stars**, the number of stars;
- **open issues**, the number of open issues;
- **forks**, the number of forks;
- **created at**, when the repository was created;
- **updated at**, when the repository was last updated.
To retrieve the `name` field we use [`concat_ws`](https://duckdb.org/docs/1.5/sql/functions/text#concat_wsseparator-string-) and [`concat`](https://duckdb.org/docs/1.5/sql/functions/text#concatvalue-) functions, in order to generate the hyperlink and text with the Markdown newline character(` `): ```python selection_query = ( duckdb_conn.table("github_raw_data") .select("unnest(items, recursive := true)") .select(""" concat_ws( '
', concat('[', name, '](', concat('https://github.com/', full_name),')'), coalesce(description, ' '), concat('**License** ', coalesce(name_1, 'unknown')), concat('**Owner** ', login) ) as repo_details """) ) ``` The above returns: ```text repo_details = [duckdb-web](https://github.com/duckdb/duckdb-web)
DuckDB website and documentation
**License** MIT License
**Owner** duckdb repo_details = [duckdb](https://github.com/duckdb/duckdb)
DuckDB is an analytical in-process SQL database management system
**License** MIT License
**Owner** duckdb ... ``` We also calculate metrics such as stars, open issues count and forks, which we sum into a field `activity_count`. A repository will end up in the list only if it has an `activity_count` greater than 3 and if it is not a fork. From above we also see that repositories owned by DuckDB are returned, therefore we filter them out too: ```python selection_query.filter(""" login != 'duckdb' and not fork and activity_count >= 3 """) ``` A Markdown table is similar to a CSV file, separated by pipe (` |`), but it must: - contain the separator at the beginning and end of each row; - between the table header and the first table row there needs to be a row with dashes (called delimiter row). In order to export the data as a Markdown file we are applying a few tricks. The first one is to select the header by adding dummy columns, at the beginning and end, containing `NULL`: ```python duckdb_conn.sql(""" select NULL, 'Name', 'Topics', 'Stars', 'Open Issues', 'Forks', 'Created At', 'Updated At', NULL """) ``` We then union the above header with the delimiter row: ```python .union( duckdb_conn.sql(""" select NULL as '', '--' as "Name", '--' as "Topics", '--' as "Stars", '--' as "Open Issues", '--' as "Forks", '--' as "Created At", '--' as "Updated At", NULL as '' """) ) ``` Returning: ```text ┌───────┬─────────┬──────────┬─────────┬───────────────┬─────────┬──────────────┬──────────────┬───────┐ │ NULL │ 'Name' │ 'Topics' │ 'Stars' │ 'Open Issues' │ 'Forks' │ 'Created At' │ 'Updated At' │ NULL │ │ int32 │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ int32 │ ├───────┼─────────┼──────────┼─────────┼───────────────┼─────────┼──────────────┼──────────────┼───────┤ │ NULL │ Name │ Topics │ Stars │ Open Issues │ Forks │ Created At │ Updated At │ NULL │ │ NULL │ -- │ -- │ -- │ -- │ -- │ -- │ -- │ NULL │ └───────┴─────────┴──────────┴─────────┴───────────────┴─────────┴──────────────┴──────────────┴───────┘ ``` And finally we union with the initial GitHub selection query and export the data to CSV, by disabling the header export: ```python ( ... .union(selection_query) ).to_csv("./exported_records.md", sep="|", header=False) ``` Below is a sample of the `exported_records.md` file, rendered as HTML: | Name | Topics | Stars | Open Issues | Forks | Created At | Updated At | | ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | ----------- | ----- | ------------------- | ------------------- | | [tailpipe](https://github.com/turbot/tailpipe)
select * from logs;
**License** GNU Affero General Public License v3.0
**Owner** turbot | [aws, azure, detections, devops, duckdb, forensics, gcp, incident-response, log-analysis, mitre-attack, open-source, parquet, siem, tailpipe, threat-detection] | 438 | 41 | 9 | 2024-04-18 02:44:35 | 2025-06-17 11:57:42 | Why a Markdown file you may wonder. Because it is rendered automatically by GitHub and there is no need to host our tiny data application somewhere else. In fact, we copy the above table to README, such that it is displayed automatically on the first page of the repository: ```bash echo '# Repositories using `duckdb`' > README.md cat exported_records.md >> README.md ``` Another way to create the README file is by using the [`string_agg` function](https://duckdb.org/docs/1.5/clients/python/relational_api#string_agg): ```python selected_data = ( selection_query .select(""" concat( '|', concat_ws( '|', repo_details, topics, stars, open_issues, forks, created_at, updated_at ) ) as line """) .string_agg( 'line', sep='|\n' ) ).fetchone()[0] with open('README.md', 'w') as readme_file: readme_file.write("# Repositories using `duckdb`\n") readme_file.write("|Name|Topics|Stars|Open Issues|Forks|Created At|Updated At|\n") readme_file.write(f"{duckdb_conn.sql("select concat(repeat('|--', 7),'|')").fetchone()[0]}\n") readme_file.write(f"{selected_data}|") ``` In the above code snippet we concatenate the columns with the pipe character and then we aggregate the records into a string, separated by `|\n` in order to add a pipe and newline at the end of each line. We then use Python to write to README the title of the page, the Markdown table header, the delimiter row (by using the [`repeat` function](https://duckdb.org/docs/1.5/sql/functions/text#repeatstring-count)) and the data itself. > The data size is very small, each commit having a README of approx. 35 KB. #### Automating with GitHub Workflow {#blog:2025-06-27-discovering-w-github::automating-with-github-workflow} With [GitHub workflows](https://docs.github.com/en/actions/writing-workflows/about-workflows) we automated the above data processing steps. We first define a [Makefile](https://medium.com/@petrica.leuca/5e987d537235?sk=60425654b72d870213b4ea29bae908a9) in our project to configure the steps needed to be executed in the workflow: ```makefile search-repos: uv run using_duckdb/search_repositories.py readme: echo '# Repositories using `duckdb`' > README.md && \ cat exported_records.md >> README.md ``` In the settings of the GitHub repository we create a [repository secret](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions) which contains the value of the API token generated above. We can then configure an environment variable in our workflow and define the processing steps: ```yaml name: Search on: workflow_dispatch: schedule: - cron: '37 5 * * *' env: READ_PUBLIC_REPO_TOKEN : ${{ secrets.READ_PUBLIC_REPO_TOKEN }} jobs: make-readme-md: runs-on: ubuntu-latest permissions: contents: write steps: - uses: actions/checkout@v4 with: ref: ${{ github.head_ref }} fetch-depth: 0 - name: Install uv uses: astral-sh/setup-uv@v6 - name: Install the project run: uv sync - name: Get search results run: make search-repos - name: Export current results run: make readme - name: Get datetime id: datetime run: echo "datetime=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT - uses: stefanzweifel/git-auto-commit-action@v5 with: commit_message: 'Update as of ${{ steps.datetime.outputs.datetime }}' file_pattern: 'README.md' ``` With [git-auto-commit-action](https://github.com/stefanzweifel/git-auto-commit-action) we delegate to GitHub the right to create a commit with the updated `README.md` file:
#### Visualizing Historical Data {#blog:2025-06-27-discovering-w-github::visualizing-historical-data}
Because we store the search results into the README file and we update it daily with Git, we can retrieve the history of the search by getting the Git difference for each commit:
```makefile
git-log:
echo '|Name|Topics|Stars|Open Issues|Forks|Created At|Updated At|' > git_log.md && \
echo '|--|--|--|--|--|--|--|' >> git_log.md && \
git log --follow -p --pretty=format:"" -- README.md | grep '^+|\[' | sed 's/^+//' >> git_log.md && \
git diff -- README.md | grep '^+|\[' | sed 's/^+//' >> git_log.md
```
In the above Makefile command we:
- create a file, `git_log.md`, which has on the first line the header of the Markdown table;
- add to `git_log.md` the delimiter row;
- for each commit, append to `git_log.md` the new lines of the README differences;
- append to `git_log.md` the new lines from the current change of README.
The `git_log.md` contains now the entire history of appended records to the README file and it can be used to display a bubble chart with the repositories mentioning DuckDB:
The above plot is generated with [Plotly](https://plotly.com/python/) by providing as data source a DuckDB Python relation:
```python
px.scatter(
duckdb_conn.read_csv(
"./git_log.md"
skiprows=2
)
.select("""
#2 as repo,
#3 as topics,
#4 as stars,
#5 as open_issues,
#6 as forks,
strftime(#7, '%B %d, %Y, %H:%m' ) as created_at,
#8 as updated_at,
if(stars + forks + open_issues = 0, 0, log(stars + forks + open_issues)) as log_activity_count,
substr(repo, position('[' in repo) + 1 , position(']' in repo) - 2) as repo_name,
count(distinct updated_at::date) over (partition by repo_name) as number_of_updates,
row_number() over (partition by repo_name order by updated_at desc) as rn
""")
.filter("rn = 1")
.order("updated_at"),
x="updated_at",
y="log_activity_count",
labels={
"updated_at": "Updated Date",
"log_activity_count": "Activity count, based on stars, open issues and forks"
},
...
```
> Using Markdown as a file format for our historical data might not be the best solution, because – among others – it does not support schema evolution.
> While we cannot do changes in the middle of the file (except column renames), we can remove the last column or add new ones at the end, due to the way DuckDB reads [malformed CSV files](#blog:2025-04-16-duckdb-csv-pollock-benchmark).
#### Conclusion {#blog:2025-06-27-discovering-w-github::conclusion}
In this post we showed how DuckDB can be used to process API requests and historical data from git commits.
Some of the repositories returned by the GitHub search API made it to [awesome-duckdb](https://github.com/davidgasquez/awesome-duckdb), which is our go-to list for curated DuckDB related projects; for example [tailpipe](https://github.com/turbot/tailpipe), an open source Security Information and Event Management for instant log insights, and [preswald](https://github.com/StructuredLabs/preswald), a Wasm packager for Python-based interactive data apps.
## DuckLake 0.2 {#blog:2025-07-04-ducklake-02}
**Publication date:** 2025-07-04
**Author:** Mark Raasveldt
**TL;DR:** We released the DuckLake v0.2 standard, which is supported by DuckDB v1.3.1's `ducklake` extension.
## Spatial Joins in DuckDB {#blog:2025-08-08-spatial-joins}
**Publication date:** 2025-08-08
**Author:** Max Gabrielsson
**TL;DR:** DuckDB v1.3.0 significantly improved the scalability of geospatial joins with a dedicated `SPATIAL_JOIN` operator.
#### Introduction {#blog:2025-08-08-spatial-joins::introduction}
Spatial joins are join operations that match rows based on the (geo-)spatial relationship between column(s). In practice they are often used to answer questions like “which of these points are within which of these polygons?”. Being able to connect datasets based on the **physical location(s)** they model is fundamental to the whole field of geospatial data science, but at a high level, also an extremely powerful way to correlate and enrich otherwise disparate data sources, anchor insights to the real world, and more often than not tell a great story with your analysis.
Ever since its inception, DuckDB's `spatial` extension has provided a `GEOMETRY` column type to represent locations, regions and shapes, along with plenty of _spatial predicate_ functions to use when `JOIN`ing. However, it wasn't until recently in [DuckDB v1.3.0](#blog:2025-05-21-announcing-duckdb-130) that spatial joins really became _scalable_ thanks to the introduction of the dedicated `SPATIAL_JOIN` query **operator**.
We managed to hastily sneak in some information about this new operator towards [the end of the v1.3.0 release notes](#blog:2025-05-21-announcing-duckdb-130::spatial-join-operator), but we figured we got enough to talk about to justify a separate blog post. So in this post, we will take a closer look at some of the challenges involved in optimizing spatial joins, and how the new `SPATIAL_JOIN` operator addresses them to achieve new levels of efficiency.
#### What Does a Spatial Join Look Like Again? {#blog:2025-08-08-spatial-joins::what-does-a-spatial-join-look-like-again}
Performing a spatial join in SQL is actually very straightforward. There is no special syntax or magic incantation required. Like you probably could've guessed, you just have to `JOIN` two tables with a `GEOMETRY` column using some _spatial predicate_. Below is a simple example that joins two tables `some_table` and `another_table` on the condition that the geometries in `some_table.geom` and `another_table.geom` “intersect”, using the `ST_Intersects` spatial predicate function:
```sql
SELECT *
FROM some_table
JOIN another_table
ON ST_Intersects(some_table.geom, another_table.geom);
```
Let's try to do something more advanced. We're going to analyze the [NYC _Citi Bike Trip_ dataset](https://citibikenyc.com/system-data), which contains around 58 million rows representing rental bike trips in New York City, including the start and end locations of each trip. We want to find neighborhoods in NYC that have the most bike trips starting in them. Therefore, we join the bike trip data with a [dataset of NYC neighborhood polygons](https://www.nyc.gov/content/planning/pages/resources/datasets/neighborhood-tabulation). We then count and group the number of trips starting in each neighborhood, and finally sort and limit the results to return the top-3 neighborhoods in which most trips originate. We've compiled the datasets and extracted the relevant columns for this example into two tables, `rides` and `hoods`, into a [218 MB DuckDB database file](https://blobs.duckdb.org/data/biketrips.duckdb).
Let's make sure the `spatial` extension is installed and loaded:
```sql
INSTALL spatial; -- Install the spatial extension
LOAD spatial; -- Load the spatial extension
```
Our query looks like this:
```sql
SELECT neighborhood, count(*) AS num_rides
FROM rides
JOIN hoods ON ST_Intersects(rides.start_geom, hoods.geom)
GROUP BY neighborhood
ORDER BY num_rides DESC
LIMIT 3;
```
```text
┌──────────────┬───────────┐
│ neighborhood │ num_rides │
│ varchar │ int64 │
├──────────────┼───────────┤
│ Midtown │ 6253835 │
│ Chelsea │ 6238693 │
│ East Village │ 4047396 │
└──────────────┴───────────┘
```
This query joins the 58,033,724 rides with the 310 neighborhood polygons and takes about **30 seconds** to run on my laptop (MacBook with an M3 Pro CPU and 36 GB RAM).
While 30 seconds might not seem that impressive at first glance (a `HASH_JOIN` over similarly sized input would finish an order of magnitude faster), I'm actually pleased DuckDB is able to execute a spatial join at this scale in a time frame that is, if not great, still acceptable for exploratory analysis (on a laptop no less!). To understand the difference in execution time, compared to, e.g., a `HASH_JOIN`, we first need to take a closer look at how DuckDB used to do spatial joins and why spatial joins are so challenging to optimize. Then, we'll dive into how the new `SPATIAL_JOIN` operator changes the game.
#### How DuckDB (Used to) Do Spatial Joins {#blog:2025-08-08-spatial-joins::how-duckdb-used-to-do-spatial-joins}
##### Spatial Predicates {#blog:2025-08-08-spatial-joins::spatial-predicates}
The first thing to understand is that a _spatial predicate_ is really just a function that evaluates some spatial relation between two geometries and returns `true` or `false`, e.g., “does `a` _contain_ `b`” or “is `a` _within distance_ `x` of `b`”. In theory you could write your own spatial predicate function, but in practice, you will probably use one of the many that come with the `spatial` extension. The nuances of the different spatial predicates are beyond the scope of this post, but here is a quick overview of the most commonly used ones, taking two geometries `a` and `b`:
| Function | Description |
| ------------------------------------------------------ | ------------------------------------------------------------ |
| `ST_Intersects(a, b)`{:.language-sql .highlight} | Whether `a` intersects `b` |
| `ST_Contains(a, b)`{:.language-sql .highlight} | Whether `a` contains `b` |
| `ST_ContainsProperly(a, b)`{:.language-sql .highlight} | Whether `a` contains `b` without `b` touching `a`'s boundary |
| `ST_Within(a, b)`{:.language-sql .highlight} | Whether `a` is within `b` |
| `ST_Overlaps(a, b)`{:.language-sql .highlight} | Whether `a` overlaps `b` |
| `ST_Touches(a, b)`{:.language-sql .highlight} | Whether `a` touches `b` |
| `ST_Equals(a, b)`{:.language-sql .highlight} | Whether `a` is equal to `b` |
| `ST_Crosses(a, b)`{:.language-sql .highlight} | Whether `a` crosses `b` |
| `ST_Covers(a, b)`{:.language-sql .highlight} | Whether `a` covers `b` |
| `ST_CoveredBy(a, b)`{:.language-sql .highlight} | Whether `a` is covered by `b` |
| `ST_DWithin(a, b, x)`{:.language-sql .highlight} | Whether `a` is within distance `x` of `b` |
> We use `ST_Intersects` in the example query above to keep it simple, even if `ST_Contains` or `ST_ContainsProperly` might be more appropriate for point-in-polygon joins.
Since all the above spatial predicates live in the `spatial` extension, DuckDB's query planner doesn't really know anything about them, except their names and that they are functions that take two `GEOMETRY` arguments and return a boolean value. This means that vanilla DuckDB on its own can't apply any special optimizations to them, and has to treat them like **any other function** in the database.
##### Optimization Challenges {#blog:2025-08-08-spatial-joins::optimization-challenges}
When DuckDB plans a join where the join condition is an arbitrary function, it normally can't use any of the advanced built-in join strategies like `HASH_JOIN` or `RANGE_JOIN` which are optimized for equality or range comparisons. Instead DuckDB has to fall back to the simplest join strategy, which is to perform a **nested loop join** (NLJ), i.e., to evaluate the join condition for every possible pair of rows in the two tables. In pseudo-code, a nested loop join would look something like this:
```python
for row_a in table_a:
for row_b in table_b:
if join_condition(row_a, row_b):
emit(row_a, row_b)
```
This is the most general way to implement a join, but it is also the least efficient. Since the _complexity_ of the join is `O(nm)`, where `n` and `m` are the number of rows in the two tables, the time it takes to perform the join grows quadratically with the size of the input.
##### Complexity Challenges {#blog:2025-08-08-spatial-joins::complexity-challenges}
Obviously, quadratic complexity quickly becomes infeasible for large joins, but DuckDB's raw execution power usually makes it bearable when joining small- to medium-sized tables.
However, what makes the nested loop join strategy impractical for spatial joins in particular, even at small- to medium-scale is that spatial predicates can be _very_ computationally expensive to evaluate in the first place. This is mostly due to the sheer complexity of the algorithms involved, but also due to the denormalized nature of geometries, in which a single geometry can contain a very large number of points. Also, besides the inherent theoretical complexity, the reality in `spatial` is that most of the spatial predicates implemented with the help of third-party libraries require a deserialization step to convert the internal binary format of the `GEOMETRY` column to an executable data structure. This usually also requires allocating memory outside of DuckDB's own memory management system, which increases pressure and lock contention on the global memory allocator, limiting the effectiveness of additional parallelism.
To illustrate how this plays out in practice, let's take a look at the query plan for the example query above, but with the `SPATIAL_JOIN` optimization disabled to force DuckDB to use a nested loop join instead:
##### Experiment {#blog:2025-08-08-spatial-joins::experiment}
```sql
LOAD spatial; -- Load the spatial extension
-- Disable the spatial join optimizer
SET disabled_optimizers = 'extension';
-- Print the new plan, using EXPLAIN to verify
-- that we are using a nested loop join
EXPLAIN SELECT neighborhood, count(*) AS num_rides
FROM rides
JOIN hoods ON ST_Intersects(rides.start_geom, hoods.geom)
GROUP BY neighborhood
ORDER BY num_rides DESC
LIMIT 3;
```
Result of EXPLAIN
```text ┌───────────────────────────┐ │ TOP_N │ │ Top: 3 │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ HASH_GROUP_BY │ │ ... │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ... │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ BLOCKWISE_NL_JOIN │ │ ──────────────────── │ │ Join Type: INNER │ │ ├──────────────┐ │ Condition: │ │ │ ST_Intersects(start_geom, │ │ │ geom) │ │ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ SEQ_SCAN ││ SEQ_SCAN │ │ ──────────────────── ││ ──────────────────── │ │ Table: rides ││ Table: hoods │ │ ... ││ ... │ │ ~58033724 Rows ││ ~310 Rows │ └───────────────────────────┘└───────────────────────────┘ ```Result of EXPLAIN
```text ┌───────────────────────────┐ │ TOP_N │ │ ──────────────────── │ │ Top: 3 │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ HASH_GROUP_BY │ │ ──────────────────── │ │ ... │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ... │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ FILTER │ │ ──────────────────── │ │ ST_Intersects(start_geom, │ -- Here we evaluate the actual │ geom) │ -- precise spatial predicate └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PIECEWISE_MERGE_JOIN │ │ ──────────────────── │ │ Join Type: INNER │ │ │ │ Conditions: │ │ ST_XMin(ST_Extent_Approx │ -- Very complex join condition, │ (start_geom)) <= ST_XMax │ -- but its all just inequality checks! │ (ST_Extent_Approx(geom)) │ │ ST_XMax(ST_Extent_Approx │ │ (start_geom)) >= ST_XMin ├──────────────┐ │ (ST_Extent_Approx(geom)) │ │ │ ST_YMin(ST_Extent_Approx │ │ │ (start_geom)) <= ST_YMax │ │ │ (ST_Extent_Approx(geom)) │ │ │ ST_YMax(ST_Extent_Approx │ │ │ (start_geom)) >= ST_YMin │ │ │ (ST_Extent_Approx(geom)) │ │ │ │ │ │ ~58033724 Rows │ │ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ SEQ_SCAN ││ SEQ_SCAN │ │ ──────────────────── ││ ──────────────────── │ │ ... ││ ... │ │ ~58033724 Rows ││ ~310 Rows │ └───────────────────────────┘└───────────────────────────┘ ```Result of EXPLAIN
```text ┌───────────────────────────┐ │ TOP_N │ │ ──────────────────── │ │ Top: 3 │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ HASH_GROUP_BY │ │ ──────────────────── │ │ ... │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ ... │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ SPATIAL_JOIN │ │ ──────────────────── │ │ Join Type: INNER │ │ │ │ Conditions: ├──────────────┐ │ ST_Intersects(start_geom, │ │ │ geom) │ │ │ │ │ │ ~58033724 Rows │ │ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ SEQ_SCAN ││ SEQ_SCAN │ │ ──────────────────── ││ ──────────────────── │ │ ... ││ ... │ │ ~58033724 Rows ││ ~310 Rows │ └───────────────────────────┘└───────────────────────────┘ ```
> In the script `reconcile_results.py`, the results between the DuckDB and `scikit-learn` preprocessing steps are reconciled, demonstrating that both implementations produce the same results.
In the above examples, we demonstrated how to perform data preprocessing with SQL expressions during training.
In practice, the same preprocessing steps must be applied at inference time, such that new data is transformed consistently with the training data.
With `scikit-learn`, this is achieved by persisting the pipeline alongside the model and applying the pipeline at inference time.
With DuckDB, the equivalent consistency is achieved by persisting the original training data (or the metrics returned by the `scaling_params` macro, computed during training).
Despite the (transformed) training data being much larger than the model artifacts, versioning the data and features calculated at training time is a common practice, which ensures model traceability and reproducibility.
> For efficient (training) data management, one can leverage solutions that provide time travel, such as [DuckLake](https://ducklake.select/docs/stable/duckdb/usage/time_travel).
#### Conclusion {#blog:2025-08-15-ml-data-preprocessing::conclusion}
In this article we showed how DuckDB offers a performant and SQL-native approach to data preprocessing for machine learning workflows. By handling tasks such as missing value imputation, categorical encoding, and feature scaling directly within the database engine, one can eliminate unnecessary data movement, during training, and reduce preprocessing latency.
## Big Data on the Move: DuckDB on the Framework Laptop 13 {#blog:2025-09-08-duckdb-on-the-framework-laptop-13}
**Publication date:** 2025-09-08
**Author:** Gábor Szárnyas
**TL;DR:** We put DuckDB through its paces on a 12-core ultrabook with 128 GB RAM, running TPC-H queries up to SF10,000.
#### Background {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::background}
When DuckDB's design was first sketched out in 2018, a key insight was that laptops had gotten powerful enough to handle most analytical workloads that data scientists can throw at them.
This made implementing a new analytical database as a single-node system that can run everywhere – including laptops – a feasible proposition: enter DuckDB.
In 2020, the release of Apple Silicon gave DuckDB-on-a-laptop setups a big performance boost. Today, modern MacBooks have CPUs with a dozen plus cores along with lots of memory and fast disks – all of which DuckDB is very eager to make use of. However, this comes at a steep price: here in the Netherlands, speccing up a MacBook Pro with 128 GB of RAM and 8 TB of disk will set you back more than 8,500 euros.
Meanwhile, there have been significant leaps in x86 land too. AMD's [Ryzen AI 300](https://en.wikipedia.org/wiki/List_of_AMD_Ryzen_processors#Strix_Point_mobile) ultrathin mobile processors, released in 2024, pack up to 12 cores and 24 threads. DuckDB is [just as happy](https://duckdb.org/why_duckdb#portable) to run on the x86_64 architecture as it is on arm64 processors, so we were eager to find out how DuckDB performs on the new AMDs.
#### Setup {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::setup}
To run this experiment, we purchased the 13" modular laptop by [Framework](https://frame.work/gb/en) and kitted it out with 128 GB of RAM and 8 TB of disk – all for [just under 3,000 euros](#blog:2025-09-08-duckdb-on-the-framework-laptop-13::cost-breakdown).
We installed [Omarchy 2.0](https://omarchy.org/), an Arch Linux-based distribution with a vibrant community and first-class support for Framework laptops. Assembling the laptop and installing Omarchy took less than an hour in total. Installing the DuckDB command line client only took seconds using the installer script:
```bash
curl https://install.duckdb.org | bash
```
We changed the theme to _[Osaka Jade](https://github.com/Justikun/omarchy-osaka-jade-theme)_ for a more Matrix-like look and got to work:
#### Experiments {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::experiments}
To see how this laptop performs, we ran a few benchmarks focusing on loading and query processing times.
##### CSV Loading {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::csv-loading}
To measure CSV loading performance, we used one of our favorite datasets: the [Dutch railway services](https://duckdb.org/docs/1.5/guides/snippets/dutch_railway_datasets). We picked the [full dataset spanning the last 80 months](https://blobs.duckdb.org/nl-railway/railway-services-80-months.zip) (between Jan 2019 and Aug 2025).
We can fetch and decompress the file as follows:
```bash
wget https://blobs.duckdb.org/nl-railway/railway-services-80-months.zip
unzip railway-services-80-months.zip
```
The resulting directory is approximately 20 GB. Let's see how quickly DuckDB can load the files:
```bash
duckdb
```
```sql
.timer on
CREATE TABLE services AS FROM 'services/services-*.csv';
-- Run Time (s): real 10.219 user 217.596664 sys 7.348692
```
It turns out that DuckDB can load 20 GB of CSV files into memory in just 10.2 seconds, at 1.96 GB/s!
##### TPC-H Benchmarks {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::tpc-h-benchmarks}
To see query performance, we used TPC-H.
We have already run TPC-H queries in all
[sorts](#blog:2021-10-29-duckdb-wasm)
[of](#blog:2024-12-06-duckdb-tpch-sf100-on-mobile)
[environments](#blog:2025-01-17-raspberryi-pi-tpch)
and were curious: how far can DuckDB scale on this laptop?
###### Data Generation {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::data-generation}
Of course, we first needed some big TPC-H datasets, and we needed to generate them locally – downloading them could take days.
Luckily, we could use the [`tpchgen-cli`](https://github.com/clflushopt/tpchgen-rs/) tool, a pure Rust implementation of the TPC-H generator that can produce large-scale datasets on the laptop in just a few hours. We generated the data as Parquet files and loaded them into DuckDB.
###### SF3,000 {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::sf3000}
We first ran all 22 TPC-H queries on the SF3,000 dataset, which corresponds to 3 TB of CSV files.
The total runtime of the queries was 47.5 minutes with a geometric mean query runtime of 86.5 seconds.
During the experiments, we noticed that the laptop bottom cover heated up [above 45 degrees Celsius](https://www.notebookcheck.net/Framework-Laptop-13-5-Ryzen-AI-9-review-Skip-the-Intel-version-for-better-performance.997363.0.html): while the keyboard was still usable, you definitely won't want to keep this machine on your lap while running data crunching workloads.
Obviously, the excess heat also results in some thermal throttling that causes the CPU to be downclocked, making the queries slower.
We are no stranger to such issues: last year, we [dipped an iPhone 16 in dry ice to improve its cooling](#blog:2024-12-06-duckdb-tpch-sf100-on-mobile::a-song-of-dry-ice-and-fire), but this seemed impractical with the laptop, so we resorted to more practical measures.
For our “cool run”, we simply inserted 5-minute sleeping periods between queries to allow the laptop to cool down. This worked very well: the total runtime of the queries (with the cooling periods excluded) went down to 30.8 minutes and the geometric mean runtime reduced to 58.2 seconds – a 32% speedup!
> Most interactive data science workloads have the query execution interspersed with the time spent on coding and analysis – which allows such a cooling period for the laptop.
###### SF3,000 on Battery {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::sf3000-on-battery}
So far, we ran our experiments with the laptop plugged into the charger. But laptops are portable devices after all, so we were curious – can we run the queries on the SF3,000 dataset while on the move?
It turns out that we can! Running all TPC-H queries on the SF3,000 dataset took 46.9 minutes (with a geomean runtime of 83.7 seconds) and it drained a fully charged battery to about 30%.
That said, if you analyze terabyte-sized datasets on your laptop, you may still want to carry a [power bank](https://www.anker.com/products/a1340-250w-power-bank) just to be on the safe side.
###### TPC-H SF10,000 {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::tpc-h-sf10000}
Finally, it was time for our ultimate challenge – can this laptop handle the SF10,000 dataset, corresponding to 10 TB of CSV files?
To find out, we:
* generated the data – _about 4 TB in Parquet files,_
* loaded the files into DuckDB – _about 2.7 TB in DuckDB's file format,_
* cleaned up the Parquet files – _knowing that the extra space will come in handy later,_
* ran the queries – _occasionally witnessing DuckDB spilling 3.6 TB on disk,_
and ultimately saw them finishing without an issue!
**The run took a total of 4.2 hours with a geometric mean query runtime of 6.6 minutes.**
Unsurprisingly, some thermal throttling also occurred here, so we repeated the experiment with cooling periods. We found that this brings down the total query runtime to 3.8 hours with a geomean runtime of 5.7 minutes (14% speedup). This means that the differences are smaller than for the SF3,000 dataset, which makes perfect sense: given the longer query runtimes, the cooling periods no longer have that much of an effect.
#### Conclusion {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::conclusion}
In short, we found that you can build a laptop for less than 3,000 euros that can load CSV files at almost 2 GB / second, run the full range of TPC-H queries on SF3,000 while on battery, and complete all queries on the SF10,000 dataset.
While ours is not an official TPC-H run, it's worth mentioning that among the current ranking of [audited SF10,000 TPC-H implementations](https://www.tpc.org/tpch/results/tpch_perf_results5.asp?resulttype=all&version=3), the cost of the cheapest setup capable of doing TPC-H SF10,000 is well over 1 million euros or 1.2 million U.S. dollars – our setup costs less than 0.3% of that and will do the job for 99% of data science workloads.
Just don't forget to pause every now and then to allow the laptop to cool down a bit.
#### Appendix {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::appendix}
##### Cost Breakdown {#blog:2025-09-08-duckdb-on-the-framework-laptop-13::cost-breakdown}
Here's the cost breakdown of the laptop using retail prices in euros (including the 21% Dutch VAT). The items were purchased in August 2025.
| Item | Cost (EUR) |
| ------------------------------------------------------------------------------------------- | -----------: |
| [Framework Laptop 13 DIY Edition w/ AMD Ryzen AI 9 HX 370 CPU](https://frame.work/laptop13) | 1,785.00 |
| Framework Laptop 13 Bezel – Translucent Green | 55.00 |
| Keyboard – US English | 109.00 |
| Power Adapter – 60W – EU | 49.00 |
| HDMI (3rd Gen) Expansion Card | 20.00 |
| USB-A Expansion Card | 10.00 |
| USB-C Expansion Card – Translucent Green (2 pcs) | 20.00 |
| [WD_BLACK SN850X NVMe SSD 8TB](https://www.amazon.nl/dp/B0D9WT512W?th=1) | 582.69 |
| [Crucial DDR5 RAM 128GB Kit](https://www.amazon.nl/dp/B0DSQMKYLN) | 327.87 |
| **Total (euros, VAT included)** | **2,974.87** |
Note that Framework currently only sells 96 GB memory kits but both the motherboard and the CPU can handle 128 GB (2 × 64 GB) kits.
## Solving Letter Scramble Puzzles with DuckDB {#blog:2025-09-11-solving-letter-scramble-puzzles}
**Publication date:** 2025-09-11
**Author:** Gábor Szárnyas
**TL;DR:** In this lighthearted post, we solve a puzzle type that's on display in Dutch trains.
The [Dutch National Railways (NS)](https://www.ns.nl/) is releasing a “letter scramble”-style puzzle every week where
they give a term whose letters can be found in a Dutch train station's name.
In order to have a match, it doesn't have to be a perfect anagram – for example, `Amsterdam` (9 letters) matches both `mastered` (8 letters) and `Dream Master` (11 letters) because all three terms contain the same letters, just with different amounts of repetitions. Let's call this a “weak anagram”.
The puzzle in the first week of September was `Clumsy Rental Red`. Let's try to find the solution using DuckDB!
#### Letter Scrambling Macro {#blog:2025-09-11-solving-letter-scramble-puzzles::letter-scrambling-macro}
First, let's create a [macro](https://duckdb.org/docs/1.5/sql/statements/create_macro) that turns a string into an ordered list of unique characters:
```sql
CREATE MACRO order_letters(s) AS
lower(s) -- convert all characters to lowercase
.regexp_replace( -- remove all non-Unicode letters
'[^\p{L}]', '', 'g'
)
.string_to_array('') -- turn the string into a list
.list_distinct() -- eliminate duplicate elements from the list
.list_sort(); -- sort the list
```
We can use this to see whether two terms are weak anagrams:
```sql
SELECT
order_letters('Amsterdam') AS letters_1,
order_letters('mastered') AS letters_2,
order_letters('Dream Master') AS letters_3,
letters_1 = letters_2 AS matches_1,
letters_1 = letters_3 AS matches_2;
```
| letters_1 | letters_2 | letters_3 | matches_1 | matches_2 |
| --------------------- | --------------------- | --------------------- | --------: | --------: |
| [a, d, e, m, r, s, t] | [a, d, e, m, r, s, t] | [a, d, e, m, r, s, t] | true | true |
Indeed, both expressions are weak anagrams of `Amsterdam`!
#### Matching against Station Names {#blog:2025-09-11-solving-letter-scramble-puzzles::matching-against-station-names}
To solve the puzzle, we need a list of train stations.
Luckily, one of our go-to datasets at DuckDB is the [Dutch railway datasets](https://duckdb.org/docs/1.5/guides/snippets/dutch_railway_datasets), including its services and train stations. We can create a table with the station names:
```sql
CREATE TABLE stations AS
FROM 'https://blobs.duckdb.org/nl-railway/stations-2023-09.csv';
```
Then, we can select the station names that are weak anagrams of the puzzle:
```sql
SELECT name_long
FROM stations
WHERE order_letters(name_long) = order_letters('Clumsy Rental Red');
```
We are not going to spoil the solution but you can reveal it below.
Click to see the solution.
[Lelystad Centrum](https://en.wikipedia.org/wiki/Lelystad_Centrum_railway_station)Click to see the end-of-life (EOL) dates for DuckDB releases.
| Version | Codename | End of community support |
|---|---|---|
| 0.* | 2024-06-03 | |
| 1.0.* | Nivis | 2024-09-09 |
| 1.1.* | Eatoni | 2025-02-05 |
| 1.2.* | Histrionicus | 2025-05-21 |
| 1.3.* | Ossivalis | 2025-09-16 |
| 1.4.* LTS | Andium | 2026-09-16 |
Click to see the code snippet showing Iceberg writes.
```sql -- Having setup an Iceberg REST catalog using -- https://github.com/duckdb/duckdb-iceberg/blob/main/scripts/start-rest-catalog.sh INSTALL iceberg; LOAD iceberg; ATTACH '' AS iceberg_datalake ( TYPE iceberg, CLIENT_ID 'admin', CLIENT_SECRET 'password', ENDPOINT 'http://127.0.0.1:8181' ); CREATE SECRET ( TYPE S3, KEY_ID 'admin', SECRET 'password', ENDPOINT '127.0.0.1:9000', URL_STYLE 'path', USE_SSL false ); USE iceberg_datalake.default; ATTACH 'duckdb.db' AS duckdb_db; CREATE TABLE duckdb_db.t AS SELECT a FROM range(4) t(a); CREATE TABLE t AS SELECT * FROM duckdb_db.t; FROM iceberg_datalake.default.t; ``` ```text ┌───────┐ │ a │ │ int64 │ ├───────┤ │ 0 │ │ 1 │ │ 2 │ │ 3 │ └───────┘ ```
While this iterator is great at sequential access, some sorting algorithms require random access. With this design, we cannot simply add an offset to a pointer to get the address of a tuple. Instead, we compute the page index and offset within the page using integer division/modulo, as the number of tuples per page is always the same (except for the last page). However, integer division/modulo is not cheap compared to the simple pointer arithmetic that can be used for contiguous blocks of memory, so we use [`fastmod`](https://github.com/lemire/fastmod) to reduce the cost. ##### Sorting Algorithm {#blog:2025-09-24-sorting-again::sorting-algorithm} With the components described so far, we are able to generate large sorted runs, that can be spilled to storage page-by-page, rather than in an all-or-nothing fashion. We use a combination of three sorting algorithms to achieve good sorting performance and high adaptivity to pre-sorted data: 1. [Vergesort](https://github.com/Morwenn/vergesort) 2. [Ska Sort](https://github.com/skarupke/ska_sort) 3. [Pattern-defeating quicksort](https://github.com/orlp/pdqsort) Vergesort detects and merges runs of _(almost) sorted_ data, which greatly reduces the effort it takes to process, _e.g._, time-series data, which is often stored in sorted order already. If Vergesort cannot detect any patterns, it falls back to Ska Sort, which performs an adaptive Most Significant Digit (MSD) [radix sort](https://en.wikipedia.org/wiki/Radix_sort) on the first 64-bit integer of the sort key. If radix partitions in the recursion become too small, or if the data is not fully sorted after the first 64-bit integer, it falls back to Pattern-defeating [quicksort](https://en.wikipedia.org/wiki/Quicksort). #### Merging {#blog:2025-09-24-sorting-again::merging} Prior to v1.4.0, DuckDB would materialize the fully-merged data. However, with a _k_-way merge, it is possible to output chunks of sorted data directly from the sorted runs, in somewhat of a streaming fashion. This means that data can be output before the full merge has been computed. We visualize this for four sorted runs:
Chunk 1 can be output to the next pipeline before all sorted runs have been merged. One of the reasons that this is useful is large `ORDER BY ... LIMIT ...` queries. If the `LIMIT` is small, DuckDB uses a min-heap, which is much faster than sorting the entire input. However, for large `LIMIT`s, the min-heap approach becomes worse than fully sorting and then applying the `LIMIT`. With a _k_-way merge, the merge can be stopped by a `LIMIT` at any point, meaning that the cost of fully merging the sorted runs is never incurred. Traditionally, the _k_-way merge is evaluated sequentially using a [tournament tree](https://en.wikipedia.org/wiki/K-way_merge_algorithm). However, with modern multi-core CPUs, this leaves a lot of performance on the table. The question is, how can we do this in parallel? ##### _K_-Way Merge Path {#blog:2025-09-24-sorting-again::_k_-way-merge-path} Various algorithms to parallelize merge sort exist, such as [Merge Path](https://arxiv.org/pdf/1406.2628.pdf), which DuckDB's sort used prior to v1.4.0, and [Bitonic Merge Sort](https://en.wikipedia.org/wiki/Bitonic_sorter). However, these algorithms parallelize a _cascading two-way merge sort_, not a _k_-way merge sort. So, while these algorithms are parallel and skew-resistant, they are unattractive for larger-than-memory sorting, as they produce much more I/O. For _k_-way merging, fewer options for parallelization exist. The work can be divided using [value-based splitting](https://pages.cs.wisc.edu/~dewitt/includes/paralleldb/parsort.pdf). However, it is easy to see that parallelism breaks down when the input distribution is extremely skewed, e.g., when half of the input has the same value, as there is no splitting value that can divide the work into evenly-sized tasks. After searching the web, the only skew-resistant parallel _k_-way merge that we could find is a [bachelor thesis from 2014](https://ae.iti.kit.edu/download/Bachelor-Thesis_Andreas_Eberle.pdf). We wanted a very fine-grained approach, so instead, we _generalized Merge Path to k sorted runs_. In the previous figure, there is a horizontal line in each sorted run that indicates how much of each sorted run went into the output chunk. The general idea of Merge Path, as explained in [our blog post on sorting four years ago](#blog:2021-08-27-external-sorting), is to compute _where these lines are_, i.e., where the sorted runs intersect. Merge Path does this efficiently for merging two sorted runs using _binary search_. We generalize this approach to _k_ sorted runs, which allows us to choose an arbitrary output chunk size, and compute where the sorted runs intersect such that when they are merged, the resulting chunk will be of the chosen size. This allows for very fine-grained skew-resistant parallelism, which is not possible when choosing specific _splitting values,_ as the size of the chunks that this produces depends on the data distribution. This is the pseudo-code for _k_-way merge path: ```python def compute_intersections(sorted_runs, chunk_size): intersections = [0 for _ in range(len(sorted_runs))] while chunk_size != 0: delta = ceil(chunk_size / len(sorted_runs)) min_idx = 0 min_val = sorted_runs[0][intersections[0] + delta] for run_idx in range(1, len(sorted_runs)): val = sorted_runs[run_idx][intersections[run_idx] + delta] if val < min_val: min_idx = run_idx min_val = val intersections[min_idx] += delta chunk_size -= delta return intersections ``` This has been greatly simplified, as this does not take into account any edge cases or going out-of-bounds on the sorted runs. The general idea is that we move up the lower bound for the intersection of one sorted run in each iteration of the `while` loop. This has a worse complexity than the binary search used in the original Merge Path, but it is also has to be called fewer times because a _k_-way merge can merge all sorted runs in a single pass, rather than in many passes. Profiling shows that this computation takes up just 1-2% of the overall execution time. Threads can compute the intersections independently, and, therefore, in parallel. Once threads have computed the intersections, they are free to merge the data between the intersections, as the data is guaranteed not to overlap with that of other threads. The merged chunks can immediately be output in parallel due to DuckDB's [_order-preserving parallelism_](https://github.com/duckdb/duckdb/pull/3700). #### Benchmark {#blog:2025-09-24-sorting-again::benchmark} So, how does the new sorting implementation perform compared to the old one? We run a few experiments on my laptop (M1 Max MacBook Pro with 10 threads and 64 GB of memory). ##### Raw Performance {#blog:2025-09-24-sorting-again::raw-performance} We first benchmark raw integer sorting performance. We have three types of inputs (pre-sorted ascending, pre-sorted descending, and randomly ordered), at three different sizes (10, 100, and 1000 million rows). We've generated the data using the following queries: ```sql CREATE TABLE ascending10m AS SELECT range AS i FROM range(10_000_000); CREATE TABLE descending10m AS SELECT range AS i FROM range(9_999_999, 0, -1); CREATE TABLE random10m AS SELECT range AS i FROM range(10_000_000) ORDER BY random(); -- and so on for 100m and 1000m ``` We took the median of 5 runs of each of these queries, for each table size: ```sql SELECT any_value(i) FROM (FROM ascending10m ORDER BY i); SELECT any_value(i) FROM (FROM descending10m ORDER BY i); SELECT any_value(i) FROM (FROM random10m ORDER BY i); -- and so on for 100m and 1000m ``` This query causes DuckDB to evaluate the entire sort, without materializing the whole table as a query result. This allows us to better isolate the performance of the sorting implementation. ###### Results {#blog:2025-09-24-sorting-again::results} | Table | Rows [Millions] | Old [s] | New [s] | Speedup vs. Old [x] | | :--------- | --------------: | ------: | --------: | ------------------: | | Ascending | 10 | 0.110 | **0.033** | 3.333 | | Ascending | 100 | 0.912 | **0.181** | 5.038 | | Ascending | 1000 | 15.302 | **1.475** | 10.374 | | Descending | 10 | 0.121 | **0.034** | 3.558 | | Descending | 100 | 0.908 | **0.207** | 4.386 | | Descending | 1000 | 15.789 | **1.712** | 9.222 | | Random | 10 | 0.120 | **0.094** | 1.276 | | Random | 100 | 1.028 | **0.587** | 1.751 | | Random | 1000 | 17.554 | **6.493** | 2.703 | This shows that the new implementation is highly adaptive to pre-sorted data: it is roughly 10x faster at ascending/descending data than the old implementation. It has much better raw sorting performance: it is more than 2× faster at sorting randomly ordered data (at 1000 million). We also plot the results on a __log-log scale__:
Here, we can see that the new implementation scales much better: the execution time of the new implementation increases less steeply with input size than the old implementation. ##### Wide Table {#blog:2025-09-24-sorting-again::wide-table} The first benchmark evaluated raw sorting performance. In this next benchmark, we sort a _wide table_, i.e., we select many columns to be sorted by the `ORDER BY` clause. We sort the `lineitem` table from TPC-H which has 15 columns, by the `l_shipdate` column, at scale factors 1 (~6 million rows), 10 (~60 million rows) and 100 (~600 million rows), generated using [DuckDB's TPC-H extension](https://duckdb.org/docs/1.5/core_extensions/tpch). We took the median execution time of 5 runs of this query for each scale factor: ```sql SELECT any_value(COLUMNS(*)) FROM (FROM lineitem ORDER BY l_shipdate); ``` ###### Results {#blog:2025-09-24-sorting-again::results} | Table | SF | Old [s] | New [s] | Speedup vs. Old [x] | | :-------------------------------------- | ---: | ------: | ---------: | ------------------: | | TPC-H SF 1 `lineitem` by `l_shipdate` | 1 | 0.328 | **0.189** | 1.735 | | TPC-H SF 10 `lineitem` by `l_shipdate` | 10 | 3.353 | **1.520** | 2.205 | | TPC-H SF 100 `lineitem` by `l_shipdate` | 100 | 273.982 | **80.919** | 3.385 | We have set the memory limit to 30 GB, so the data no longer fits in memory at scale factor 100. The new implementation is roughly 2× faster at scale factors 1 and 10, and more than 3× faster at scale factor 100. This shows that the new _k_-way merge sort reduces data movement and I/O, is much more efficient at sorting wide tables than the old cascaded 2-way merge sort. Again, we plot the results on a __log-log scale__:
And we can see that the new implementation scales much better, especially when the data no longer fits in main memory. ##### Thread Scaling {#blog:2025-09-24-sorting-again::thread-scaling} Finally, we benchmark how well the sorting implementation scales with threads. We sort the table with 100 million randomly ordered integers from before, with 1, 2, 4, and 8 threads. We use the same data and query as in the first benchmark, and take the median of five runs. ###### Results {#blog:2025-09-24-sorting-again::results} | Threads | Old [s] | New [s] | Old Speedup vs. 1 Thread [x] | New Speedup vs. 1 Thread [x] | | ------: | --------: | --------: | ---------------------------: | ---------------------------: | | 1 | **3.240** | 4.234 | **1.000** | **1.000** | | 2 | **2.121** | 2.193 | 1.527 | **1.930** | | 4 | 1.401 | **1.216** | 2.312 | **3.481** | | 8 | 0.920 | **0.654** | 3.521 | **6.474** | As we can see, the new, single-threaded sorting performance is around 30% slower than the old one. This is due to the new sorting implementation using an _in-place MSD radix sort_, rather than an _out-of-place Least Significant Digit (LSD) radix sort_. This makes the old implementation perform better specifically on this workload, at the cost of using much more memory. However, if we increase the number of threads to 2, this advantage is already gone. At 8 threads, the old implementation has a speedup of only ~3.5× over 1 thread, while this speedup is ~6.5× for the new implementation. Again, we plot the results on a __log-log scale__:
This shows that the new implementation's parallel scaling is much better than the old implementation. #### Conclusion {#blog:2025-09-24-sorting-again::conclusion} DuckDB's new sorting implementation has greatly improved performance over the old sorting implementation. It is highly adaptive to pre-sorted data, performs less I/O when sorting data that does not fit in main memory, and scales much better with additional threads. If you've upgraded to v1.4.0, you can enjoy the improved performance when using the `ORDER BY` clause. The new sorting implementation has already been integrated into the window operator, so we expect to see a performance improvements when using the `OVER` clause as well. For v1.5.0, we aim to integrate the new sorting implementation into the joins that use sorting such as the `ASOF` join. ## Announcing DuckDB 1.4.1 LTS {#blog:2025-10-07-announcing-duckdb-141} **Publication date:** 2025-10-07 **Author:** The DuckDB team **TL;DR:** Today we are releasing DuckDB 1.4.1, the first patch release of our LTS edition. In this blog post, we highlight a few important fixes and convenience improvements in DuckDB v1.4.1, the first patch release in [DuckDB's 1.4 LTS line](#blog:2025-09-16-announcing-duckdb-140). You can find the complete [release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.4.1). To install the new version, please visit the [installation page](https://duckdb.org/install). #### Iceberg Improvements {#blog:2025-10-07-announcing-duckdb-141::iceberg-improvements} The DuckDB [`iceberg` extension](https://duckdb.org/docs/1.5/core_extensions/iceberg/overview) received a number of patches: * You can now attach to an Iceberg REST Catalog and specify an access delegation mode. This fixes a bug when using catalogs that did not vend credentials. The `ATTACH` statement will now look like this: ```sql ATTACH '⟨warehouse_name⟩' AS my_datalake ( TYPE iceberg, ENDPOINT '⟨endpoint⟩', ACCESS_DELEGATION_MODE '⟨delegation_mode_option⟩', SECRET '⟨my_secret⟩' ); ``` The current `ACCESS_DELEGATION_MODE` options are `vended_credentials` (default) and `none`. * When attaching to AWS-managed REST Catalogs, the `http_timeout` setting is now respected. * Attempting to rename or replace a table within a transaction now throws a clear error message. * AWS Athena can now read Iceberg tables written by DuckDB. #### AWS Improvements {#blog:2025-10-07-announcing-duckdb-141::aws-improvements} The `aws` extension received a number of changes, which makes it easier to configure and troubleshoot. See the [`aws` documentation page](https://duckdb.org/docs/1.5/core_extensions/aws) for more details. ##### Secret Validation {#blog:2025-10-07-announcing-duckdb-141::secret-validation} Since DuckDB v1.4.0, the AWS `credential_chain` provider looks for any required credentials during `CREATE SECRET` time, failing if absent/unavailable. Since v1.4.1 this behavior can also be configured via the `VALIDATION` option as follows: ```sql CREATE OR REPLACE SECRET secret ( TYPE s3, PROVIDER credential_chain, VALIDATION 'exists' ); ``` Two validation modes are supported: * `exists` (default) requires present credentials. * `none` allows `CREATE SECRET` to succeed for `credential_chains` with no available credentials. ##### S3 Default Region {#blog:2025-10-07-announcing-duckdb-141::s3-default-region} Previously, setting the S3 region incorrectly could result in difficult-to-debug situations (` Unknown error for HTTP HEAD to ...`). DuckDB v1.4.1 [removes `us-east-1` as the default S3 region](https://github.com/duckdb/duckdb/pull/19087) and returns a 301 error code if an incorrect region is used. #### Fixes for Missing Data {#blog:2025-10-07-announcing-duckdb-141::fixes-for-missing-data} Users reported two cases where DuckDB omitted some data: * The Parquet reader had a [regression which caused it to omit some rows](https://github.com/duckdb/duckdb/issues/19131) when using predicate pushdown on certain string columns. * In certain edge cases, DuckDB’s ART index could [omit rows](https://github.com/duckdb/duckdb/issues/19190) rows non-deterministically when running on multiple threads. Note that this index is only used when you manually specify an index with [`CREATE INDEX`](https://duckdb.org/docs/1.5/sql/indexes). DuckDB v1.4.1 fixes both of these issues. #### Autoloading {#blog:2025-10-07-announcing-duckdb-141::autoloading} In DuckDB v1.4.0, the [`httpfs` extension](https://duckdb.org/docs/1.5/core_extensions/httpfs/overview) was not always autoloaded. For example, running: ```sql COPY (SELECT 42 AS answer) TO 's3://my_bucket/my_file.parquet'; ``` without loading `httpfs` manually returned the following error: ```console Cannot open file "s3://my_bucket/my_file.parquet": No such file or directory ``` With v1.4.1, autoloading works and DuckDB can write to the bucket right away. #### Docker Image {#blog:2025-10-07-announcing-duckdb-141::docker-image} We now officially distribute a [Docker image](https://hub.docker.com/r/duckdb/duckdb/), making it easy to run DuckDB in a containerized environment: ```bash docker run --rm -it -v "$(pwd):/workspace" -w /workspace duckdb/duckdb ``` ```text DuckDB v1.4.1 (Andium) b390a7c376 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. D ``` For more details including operational considerations and using the UI, read the [Docker image page](https://duckdb.org/docs/1.5/operations_manual/duckdb_docker). ## Adoption Metrics and Benchmark Results for DuckDB v1.4 LTS {#blog:2025-10-09-benchmark-results-14-lts} **Publication date:** 2025-10-09 **Author:** The DuckDB team **TL;DR:** The DuckDB landing page makes some strong claims about DuckDB's popularity. In this blog post, we show evidence for these claims. #### #1 on ClickBench {#blog:2025-10-09-benchmark-results-14-lts::1-on-clickbench} On October 9, 2025, DuckDB's in-memory variant hit #1 on the popular [ClickBench database benchmark](https://benchmark.clickhouse.com/):  This result was made possible due several [performance optimizations](#blog:2025-09-16-announcing-duckdb-140::performance-and-optimizations) in DuckDB v1.4. > **Update.** As of October 26, the rules of ClickBench changed. The new rules prevent [in-memory databases from showing “cold” (and “combined”) results](https://github.com/ClickHouse/ClickBench/pull/660). After this change, DuckDB is the #1 open-source system in hot runs, closely trailing the leader in that category, Umbra, a closed-source research prototype. #### #3 Most Admired System on Stack Overflow {#blog:2025-10-09-benchmark-results-14-lts::3-most-admired-system-on-stack-overflow} In Stack Overflow's 2024 Developer Survey, DuckDB was named among the [**top-3 most admired database systems**](https://survey.stackoverflow.co/2024/technology#2-databases). In the 2025 survey, it achieved position #4 (just 0.2% behind SQLite) but it made up for this by a significant increase in usage, which jumped from 1.4% in 2024 to 3.3% in 2025. #### 20+ Fortune-100 Companies Use DuckDB {#blog:2025-10-09-benchmark-results-14-lts::20-fortune-100-companies-use-duckdb} We estimated the number of Fortune-100 companies who use DuckDB by cross-checking self-reported affiliations in the [DuckDB issue tracker](https://github.com/duckdb/duckdb/issues) against a list of Fortune-100 companies. #### 25M+ Downloads / Month {#blog:2025-10-09-benchmark-results-14-lts::25m-downloads--month} DuckDB's Python packages has almost 25 million monthly downloads on [PyPI alone](https://pypistats.org/packages/duckdb). This is complemented with the downloads of other popular clients such as the CLI, Go, Node.js, Java, R, Rust and so on. #### TPC-H SF 100,000 {#blog:2025-10-09-benchmark-results-14-lts::tpc-h-sf-100000} DuckDB is not only fast but it is also scalable. We have recently run the queries of the [TPC-H workload](https://duckdb.org/docs/1.5/core_extensions/tpch) on the scale factor 100,000 dataset, which is equivalent to 100,000 GB of CSV files. Obviously, processing such a dataset size requires the ability to [spill to disk](https://duckdb.org/docs/1.5/guides/performance/how_to_tune_workloads#spilling-to-disk). We ran the experiment on an [`i8g.48xlarge` EC2 instance](https://aws.amazon.com/ec2/instance-types/i8g/), which has 1.5 TB of RAM and 192 CPU cores (AWS Graviton4, Arm64). This instance has 12 NVMe SSD disks, each 3750 GB in size. We created a RAID-0 array from them to have a single 45 TB partition and formatted it using [XFS](https://duckdb.org/docs/1.5/guides/performance/environment#local-disk). We generated the dataset with the [`tpchgen-cli`](https://github.com/clflushopt/tpchgen-rs/) tool, a pure Rust implementation of the TPC-H generator. We configured the generator to produce chunks of Parquet files and loaded them into DuckDB. The final DuckDB database was about 27 TB in size (as a single file!). DuckDB completed all 22 queries of the benchmark using its [larger-than-memory processing](#blog:2024-07-09-memory-management). For some queries, this required spilling _about 7 terabytes of data_ to disk. The median query runtime was 1.19 hours and the geometric mean runtime was 1.13 hours. ## Streaming Patterns with DuckDB {#blog:2025-10-13-duckdb-streaming-patterns} **Publication date:** 2025-10-13 **Author:** Guillermo Sanchez **TL;DR:** DuckDB used for streaming analytics? This post will show you some patterns in which you can use DuckDB to refresh your data at near real-time speed. The words “DuckDB” and “streaming” don't usually make it into the same sentence. Maybe this is because DuckDB has been positioned as an all powerful (but very lightweight) OLAP database. Or maybe this is because the ecosystem of streaming analytics has centered around names such as Kafka, Flink and Spark Streaming, and most recently players trying to change the game like Materialize or RisingWave. But can DuckDB be used in the context of streaming analytics? What is streaming analytics in the first place? #### Streaming Analytics Patterns {#blog:2025-10-13-duckdb-streaming-patterns::streaming-analytics-patterns} The simplest definition: streaming analytics is the _act of updating an analytical view of your data at near real-time speed as new data comes in._ For example, if three new sessions have just started in your website, the process of gathering those session events and updating the count (+3) is streaming analytics. Streaming analytics is not, in my modest opinion, just about inserting those 3 session events in a table – that would fit well within the realm of a transactional workload. Streaming analytics is also not pushing this events to a Kafka topic and sinking to another system. If you don't update the analytical view of your data, I wouldn't call it streaming analytics. Now that we have a definition, let's take a look at three common architectural patterns in streaming analytics. The names given to these patterns are of my own making, but I think they help differentiating them from one another. {: .lightmode-img } - In the **Materialized View Pattern**, it is very common to use a cloud data warehouse with support for materialized views (such as BigQuery or Snowflake). The stream of events is usually sunk to a raw table and a materialized view is created on top. This pattern is generally conceived as having a higher latency than the next two. However, there is not that much benchmarking around to conclude anything. - The **Streaming Engine Pattern** uses the more traditional ETL approach. A separate process using a streaming engine consumes the messages from the source, queries are then done on the fly and results are stored in a persisted table. Common engines are Spark Streaming, Flink, Kafka Streams or most recently Arroyo. This has traditionally come with a set of complications (e.g., dealing with watermarks, state management, increased memory load for infinite queries, etc.). - The **Streaming Database Pattern** is similar to the previous one in terms of latency but drastically simplifies the experience. Streaming databases like RisingWave or Materialize can directly read from the streaming source and update your materialized view on the fly. They aim at keeping ACID consistency and allowing clients to query data using the PostgreSQL wire protocol. “Where does DuckDB fit in all this?” – you may ask. Well, DuckDB fits well with patterns one and two. Even if DuckDB does not support materialized views ([yet](https://duckdb.org/roadmap)), we can work around this limitation and implement these patterns to still get very good results. > Interestingly, the streaming engine industry doesn't have many official benchmarks. The [Nexmark](https://github.com/nexmark/nexmark) benchmark seems to be the most common, but there are not many published results comparing engines using this benchmark. #### Materialized View Pattern: Cooking Our Own Materialized View with DuckDB {#blog:2025-10-13-duckdb-streaming-patterns::materialized-view-pattern-cooking-our-own-materialized-view-with-duckdb} We know that DuckDB is very fast at aggregating data on the fly and also performs very well in transactional workloads (for an OLAP system). So does DuckLake's lakehouse format, thanks to its [data inlining feature](https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining). In this section we are going to see both DuckDB and DuckLake in action, acting as a sink for Kafka and calculating new metric values based on deltas. > All the patterns are going to do the same thing in different ways. That is, read events from a Kafka topic and update the analytical view, which can be a persisted table or a view on top of a raw table. What happens in between is what differentiates these patterns. ##### Querying Deltas with DuckDB {#blog:2025-10-13-duckdb-streaming-patterns::querying-deltas-with-duckdb} {: .lightmode-img } The key component in this diagram is what I call “Delta Processor”. This component is basically a function that loops periodically and runs a query to aggregate new data inserted in the `raw_events` table and to update the analytical view, in this case a persisted table called `user_clicks`. This is the query that runs periodically to update `user_clicks` with the new delta: ```sql MERGE INTO user_clicks AS dest USING ( SELECT user_id, user_name, count(*) AS count_of_clicks, max(timestamp) AS updated_at FROM raw_events WHERE event_type = 'CLICK' AND (⟨LATEST_UPDATED_AT⟩ IS NULL OR timestamp > ⟨LATEST_UPDATED_AT⟩) GROUP BY user_id, user_name ) AS src ON dest.user_id = src.user_id WHEN MATCHED THEN UPDATE SET count_of_clicks = dest.count_of_clicks + src.count_of_clicks, updated_at = src.updated_at WHEN NOT MATCHED THEN INSERT (user_id, user_name, count_of_clicks, updated_at) VALUES (src.user_id, src.user_name, src.count_of_clicks, src.updated_at); ``` You can check the full pipeline in [this repository](https://github.com/guillesd/duckdb-streaming-patterns/blob/main/pipelines/pattern_1_1.py). ##### Using DuckLake's Change Data Feed {#blog:2025-10-13-duckdb-streaming-patterns::using-ducklakes-change-data-feed} {: .lightmode-img } This pattern is very similar to 1.1 but with some specifics to DuckLake: 1. We are using [DuckLake's Data Inlining](https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining) to speed up insertion without writing too many small files. 2. The Delta Processor component can take advantage of [DuckLake's Data Change Feed](https://ducklake.select/docs/stable/duckdb/advanced_features/data_change_feed) to avoid scanning unnecessary data. 3. We have an extra component, called “Inline Flusher”, that periodically flushes inlined data from the metadata catalog to parquet files of the specified file size (512 MB by default). This is a maintenance operation that will keep DuckLake performant. You can check the full pipeline in [this repository](https://github.com/guillesd/duckdb-streaming-patterns/blob/main/pipelines/pattern_1_2.py). > In order to make better use of filter pushdowns and file pruning, partitioning the data by `timestamp` is recommended #### Streaming Engine Pattern: Streaming Engines and DuckDB {#blog:2025-10-13-duckdb-streaming-patterns::streaming-engine-pattern-streaming-engines-and-duckdb} Most established streaming engines (Spark Streaming, Flink, Kafka Streams) are JVM based. They can therefore insert data in DuckDB using the JDBC protocol. This pattern is usually a bit difficult to manage. Long running streaming queries tend to consume a lot of memory and restarting interrupted streaming queries always makes me skip a beat. However, it can be a very low latency solution for very large streams of data. ##### Using Spark Streaming and Sink to DuckDB {#blog:2025-10-13-duckdb-streaming-patterns::using-spark-streaming-and-sink-to-duckdb} {: .lightmode-img } In this diagram we can see that most of the components are managed by the Spark Streaming runtime. In Spark Streaming, all of this is contained in a streaming query. When the micro batching mode is being used (like it is the case in this example) you can pass a custom function to the writer that allows you to write each batch in the way you desire. In our case, we just use a JDBC connection and overwrite the destination table (` user_clicks`). We can also see that there are no intermediate results being saved, meaning in this particular case we do not have a `raw_events` table. This is not a pattern that I love since for auditing purposes I would prefer to store the raw data to ensure that my streaming job isn't doing something funky. In this case, Spark Streaming relies on checkpoints to keep the state and make sure that data is processed just once and queries are able to restart without missing or duplicating data consumed from the Kafka topic. You can check the full pipeline in [this repository](https://github.com/guillesd/duckdb-streaming-patterns/blob/main/pipelines/pattern_2.py). #### Bonus: Using DuckDB Tributary Extension to Directory Query Kafka {#blog:2025-10-13-duckdb-streaming-patterns::bonus-using-duckdb-tributary-extension-to-directory-query-kafka} {: .lightmode-img } This setup is the most similar thing to the Streaming Databases Pattern that you can do right now with DuckDB. Powered by the [`tributary` DuckDB community extension](https://duckdb.org/community_extensions/extensions/tributary), you can create a view or a table that reads directly from a Kafka topic. To simulate materialized views, we are using views for this specific example. The following query showcases how simple this process is: ```sql CREATE VIEW IF NOT EXISTS raw_events_view AS SELECT * EXCLUDE message, decode(message)::JSON AS message FROM tributary_scan_topic(⟨TOPIC⟩, "bootstrap.servers":="localhost:9092"); ``` Currently this extension has no state management. Every time this view is queried, we would be reading the whole topic from offset 0. This is not ideal since Kafka has a limited retention policy, which means that at some point it will start flushing messages. A way around this is to materialize this messages to tables and use the offset (or a timestamp) to keep track of what has been ingested. You can check the full pipeline in [this repository](https://github.com/guillesd/duckdb-streaming-patterns/blob/main/pipelines/bonus_pattern.py). > This is an experimental extension from [Query.Farm](https://query.farm/). #### Some Thoughts {#blog:2025-10-13-duckdb-streaming-patterns::some-thoughts} Conclusions always feel very subjective, so I rather write about some of my thoughts regarding streaming patterns in general and particularly around DuckDB. **The Materialized View Pattern is usually good enough.** My hot take is that most use cases for analytics are usually covered by the Materialized View Pattern without the need of complexity that comes with other patterns. I believe that DuckDB is very well suited for this pattern because for a small OLAP, it does incredibly well at handling large amounts of streaming inserts. In [this article](https://arrow.apache.org/blog/2025/03/10/fast-streaming-inserts-in-duckdb-with-adbc/) DuckDB was pushed to the limit and was able to handle more than one million rows inserted per second. Also note that **[materialized views are on the DuckDB's long-term roadmap](https://duckdb.org/roadmap#future-work)**, so this pattern will become even simpler in the near future. If you are streaming to a lakehouse, you should know that DuckLake's [Data Inlining feature](https://ducklake.select/docs/stable/duckdb/advanced_features/data_inlining) was specifically designed to deal with high-throughput inserts while solving the small file problem. This makes DuckLake a great candidate for this pattern if you have a lakehouse-like architecture. **Streaming Engines and Streaming Databases can be hard (or expensive).** At scale, Streaming Engines can be hard to manage. It is an evolving field and some work is being done to make forever running streaming queries easier to operate. For example, [Apache Fluss](https://fluss.apache.org/) is being built with the idea to solve some of the shortcomings described in this post. However, it does add another layer of complexity to an already complex streaming architecture. Streaming databases are a very elegant solution and have the potential to be very nice to use. However, if you are looking to host the solution, this will require some expertise since these systems are considerably complex (see [RisingWave's architecture](https://docs.risingwave.com/get-started/architecture)). This pushes practitioners to buy rather than host and maintain this complex system, which can be costly. Whatever you choose for your architecture, make sure that the effort you put into it corresponds to your needs. And next time you think of streaming, make sure you also think about DuckDB. ## Uncovering Financial Crime with DuckDB and Graph Queries {#blog:2025-10-22-duckdb-graph-queries-duckpgq} **Publication date:** 2025-10-22 **Author:** Daniël ten Wolde **TL;DR:** You can process graphs in DuckDB! In this post, we show how to use DuckDB and the DuckPGQ community extension to analyze financial data for fraudulent patterns with the SQL/PGQ graph syntax that's part of SQL:2023. Following the money is harder than it looks. Sophisticated criminals hide their tracks using long, complex chains of transactions, hoping to obscure the origin of illicit funds. Unraveling these networks is a classic graph problem: you're looking for suspicious patterns and hidden paths in a vast web of accounts and transactions. For years, this kind of analysis often meant exporting data to a specialized graph database, adding complexity and overhead. But what if you could perform this powerful graph analysis directly within your daily driver database? This is where DuckDB's extensibility shines. In this blog post, we'll dive into a financial dataset and use DuckDB with a graph query extension to identify the kinds of patterns that could indicate a money laundering scheme or otherwise high-risk accounts. #### From Relational Tables to a Property Graph {#blog:2025-10-22-duckdb-graph-queries-duckpgq::from-relational-tables-to-a-property-graph} Before we can hunt for suspicious activity, we need to understand our data. We're using the [LDBC Financial Benchmark dataset](https://ldbcouncil.org/benchmarks/finbench/), which simulates a financial network. To attach to the database with the dataset, run: ```sql ATTACH 'https://blobs.duckdb.org/data/finbench.duckdb' AS finbench; USE finbench; ``` > To follow along with the examples in this post, it is recommended to use DuckDB v1.4.1. In this blog post we will use a subset of the dataset with tables for `Person`, `Account`, and the `AccountTransferAccount` table that links them.  Let's start by getting a feel for the scale of our network: ```sql SELECT (SELECT count(*) FROM Person) AS num_persons, (SELECT count(*) FROM Account) AS num_accounts, (SELECT count(*) FROM AccountTransferAccount) AS num_transfers; ``` ```text ┌─────────────┬──────────────┬───────────────┐ │ num_persons │ num_accounts │ num_transfers │ │ int64 │ int64 │ int64 │ ├─────────────┼──────────────┼───────────────┤ │ 785 │ 2055 │ 8132 │ └─────────────┴──────────────┴───────────────┘ ``` This query gives us a quick overview of the number of entities and connections we're dealing with. As the schema diagram above illustrates, these tables of accounts and transfers already form a graph, a structure made of _nodes_ or _vertexes_ (the entities), connected by _edges_ (representing relations – _hint hint_ – between the entities). To make our queries more powerful, we'll use the [Property Graph model](https://en.wikipedia.org/wiki/Property_graph). This is just a formal way of saying we can add descriptive labels, which are general types like `Account` and `Person`, as well as specific properties, like `accountId` and `nickname`. If you're thinking this sounds a lot like the relational model, you're exactly right. A `Person` table is just a collection of nodes with the label `Person`, and its columns are the properties. This natural mapping is what makes a high-performance relational database like DuckDB a perfect foundation for graph analytics. ##### Property Graphs in DuckDB {#blog:2025-10-22-duckdb-graph-queries-duckpgq::property-graphs-in-duckdb} To write our graph queries, we could use DuckDB and the SQL we are familiar with. But let us make life a little simpler for ourselves and leverage DuckDB's rich extension ecosystem. We will be using [`DuckPGQ`](https://duckdb.org/community_extensions/extensions/duckpgq), a community extension that adds support to DuckDB's parser for a new visual graph syntax. This new syntax is SQL / Property Graph Queries (SQL/PGQ), which is part of the official SQL:2023 standard. SQL/PGQ is partially inspired by the popular graph query language [Cypher](https://en.wikipedia.org/wiki/Cypher_(query_language)). > The DuckPGQ extension started out as a [research](https://duckdb.org/library/duckpgq/) [prototype](https://duckdb.org/library/duckpgq-demo/) and is now available as a community extension. Installing and loading the extension is as simple as it gets: ```sql INSTALL duckpgq FROM community; LOAD duckpgq; ``` In DuckPGQ, the first step is to create the property graph, acting as a layer on top of the tables we have created earlier. ```sql CREATE PROPERTY GRAPH finbench VERTEX TABLES ( Person, Account ) EDGE TABLES ( AccountTransferAccount SOURCE KEY (fromId) REFERENCES Account (accountId) DESTINATION KEY (toId) REFERENCES Account (accountId) LABEL Transfer, PersonOwnAccount SOURCE KEY (personId) REFERENCES Person (personId) DESTINATION KEY (accountId) REFERENCES Account (accountId) LABEL PersonOwn ); ``` During the creation of the property graph, we make a clear distinction between `VERTEX` tables and `EDGE` tables. For `VERTEX` tables, we only have to specify the name of the table. For `EDGE` tables, a little more work is required since for both the `SOURCE` and the `DESTINATION`, we need to specify the column in the edge table that forms the key for the `SOURCE` or `DESTINATION`. This is the same principle as defining a [`FOREIGN KEY` constraint](https://duckdb.org/docs/1.5/sql/constraints#foreign-keys), linking our edge table back to the node tables it connects. The `LABEL` clause gives a clean name to the relationship type. While our table is named `AccountTransferAccount`, the edges within it represent a `Transfer` relationship. This is the name we'll use in our graph queries. Now that we have created our property graph, we are ready to investigate the financial data and uncover its secrets! #### Graph Processing {#blog:2025-10-22-duckdb-graph-queries-duckpgq::graph-processing} When talking about graph processing in databases, we typically refer to these types of operations: - Pattern matching, finding a pattern in our data. - Path-finding, finding a path in our data, potentially of variable length. Let's see how we can leverage DuckDB and DuckPGQ for these two tasks. ##### Hunting for Suspicious Activities {#blog:2025-10-22-duckdb-graph-queries-duckpgq::hunting-for-suspicious-activities} As previously mentioned, SQL/PGQ introduces a visual graph syntax to formulate graph patterns more naturally. Now, let's use it to hunt for patterns that might indicate a money laundering scheme. A common technique used to hide illicit funds is called [_smurfing_](https://www.investopedia.com/terms/s/smurf.asp). The goal of smurfing is to break down a single large transfer, potentially triggering reporting requirements, into smaller transactions over time. We can search for this behavior by looking for pairs of accounts with a high number of transactions but a relatively low average amount. Let's set the threshold for the average amount at $50,000 and see if we can find any high-frequency relationships: ```sql SELECT fromName, count(amount) AS number_of_transactions, round(avg(amount), 2) AS avg_amount, toName FROM GRAPH_TABLE (finbench MATCH (a:Account)-[t:Transfer]->(a2:Account) COLUMNS (a.nickname AS fromName, t.amount, a2.nickname AS toName ) ) GROUP BY ALL HAVING avg_amount < 50_000 ORDER BY number_of_transactions DESC, avg_amount ASC LIMIT 5; ``` Running the query leads us to the following result: ```text ┌───────────────────┬────────────────────────┬────────────┬───────────────────┐ │ fromName │ number_of_transactions │ avg_amount │ toName │ │ varchar │ int64 │ double │ varchar │ ├───────────────────┼────────────────────────┼────────────┼───────────────────┤ │ Noe Trites │ 1 │ 49365.04 │ Dale Croucher │ │ Madeleine Bussing │ 1 │ 46663.56 │ Delphine Primiano │ │ Bonnie Centeno │ 1 │ 46663.56 │ Maile Boon │ │ Darci Sheedy │ 1 │ 44856.02 │ Carmella Estelle │ │ Marguerita Gurne │ 1 │ 44393.68 │ Delphine Primiano │ └───────────────────┴────────────────────────┴────────────┴───────────────────┘ ``` The query worked, but the result does not show any signs of suspicious activity with the number of transactions always being 1. To understand why, let's break down how the query was constructed. The magic happens inside the `FROM` clause. The `GRAPH_TABLE (finbench ...)` function allows us to run a graph query over the property graph we have just created and treat its output like a regular table. The `MATCH (a:Account)-[t:Transfer]->(a2:Account)` clause is the core of our pattern. It visually describes what we're looking for: a simple transfer from one account `(a:Account)` to another `(a2:Account)`. The `()` denote nodes and the `[]` denotes the connecting edge with the ASCII-style arrow `->` showing the direction of the edge. The `COLUMNS(...)` clause then acts like a `SELECT` list for our pattern, pulling out the nickname from the accounts and the amount from the transfer. The beauty of SQL/PGQ is that the result of this graph pattern match can be seamlessly returned into the standard SQL we already know. We use `GROUP BY ALL` to aggregate all transfers between the same two people, and our `HAVING avg_amount < 50_000` clause filters for the smurfing pattern we defined. We know our query is correct but also that this simple _smurfing_ pattern is not present in our dataset. This means we must investigate further, using potentially more complex patterns. This leads us to a more powerful feature of graph queries: finding structural patterns that are very difficult to express with traditional SQL `JOIN`s, such as transaction paths. ##### Finding Paths in the Transactions {#blog:2025-10-22-duckdb-graph-queries-duckpgq::finding-paths-in-the-transactions} Another classical example of possible fraudulent behavior is a cycle of transactions where the money circles back to the person who sent the first transaction in the chain. Writing a SQL query to answer this question is, using the traditional syntax, incredibly difficult. Try it yourself, after first reading this section! We will show the answer in the next section.  With SQL/PGQ writing queries that involve finding paths has become significantly easier. The diagram above illustrates the query pattern we'll use with the goal of finding a path of one or more transfers between two different accounts (` A1` and `A2`) owned by the same person (` P`). Remember that persons can own multiple account. With the following query we will try to find cycles between all the accounts that the person with id `125` owns: ```sql FROM GRAPH_TABLE(finbench MATCH p = ANY SHORTEST (p:Person)-[o1:PersonOwn]->(a1:Account) -[t:Transfer]->+ (a2:Account)<-[o2:PersonOwn]-(p:Person) WHERE p.personId = 125 AND a1.accountId <> a2.accountId COLUMNS ( path_length(p) AS path_length, a1.accountId AS start_account, a2.accountId AS end_account ) ) ORDER BY path_length; ``` The result shows that there are cycles for this person owning multiple accounts of varying lengths: ```text ┌─────────────┬─────────────────────┬─────────────────────┐ │ path_length │ start_account │ end_account │ │ int64 │ int64 │ int64 │ ├─────────────┼─────────────────────┼─────────────────────┤ │ 8 │ 4753267931712848113 │ 4794926228266025204 │ │ 8 │ 4769874955338776819 │ 4794926228266025204 │ │ 8 │ 4796615078126289138 │ 4769874955338776819 │ │ 9 │ 4753267931712848113 │ 4769874955338776819 │ │ 9 │ 4769874955338776819 │ 4753267931712848113 │ │ 9 │ 4794926228266025204 │ 4753267931712848113 │ │ 9 │ 4796615078126289138 │ 4753267931712848113 │ │ 9 │ 4796615078126289138 │ 4794926228266025204 │ │ 12 │ 4794926228266025204 │ 4769874955338776819 │ └─────────────┴─────────────────────┴─────────────────────┘ ``` Once again, the magic happens inside the `FROM` clause where we now create a `MATCH` that finds `ANY SHORTEST` path along the given pattern. The first part of the pattern finds all the accounts owned by the Person 125: `(p:Person)-[o1:PersonOwn]->(a1:Account)`. In the second part the path-finding occurs. Take special note of the `+`, which indicates that for the pattern `(a1:Account)-[t:Transfer]->+(a2:Account)` the two nodes `(a1)` and `(a2)` do not necessarily need to be connected. With the `+`, we indicate there must be one or more `Transfer`s between these two accounts, with no upper bound set. Finally, to tie it all up, we take the destination account and check whether it is owned by the same person `p`. The results confirm our suspicions. Our query found multiple non-obvious paths between the accounts owned by Person 125, with path lengths ranging from 8 to 12 transfers. Each row represents a hidden chain of transactions connecting two of the person's accounts. More interestingly, we can see clear cyclical patterns. For instance, the query found a 9-step path from account `4753267931712848113` to `4769874955338776819`, and another 9-step path flowing in the opposite direction. This suggests a sophisticated and intentional effort to move money between accounts, a strong indicator that warrants further investigation. #### Doing it the Old-Fashioned Way {#blog:2025-10-22-duckdb-graph-queries-duckpgq::doing-it-the-old-fashioned-way} Earlier, we challenged you to think about how you would find these ownership cycles using traditional SQL. As promised, here is the answer. Before we dive into the query, there are two important notes to keep in mind when comparing it to the SQL/PGQ version: 1. **Performance Safeguard:** The query requires a manual upper bound on the path length (` ps.depth < 11`) to prevent infinite recursion and potentially quadratic runtimes on dense graphs. The SQL/PGQ `->+` syntax does not require this. 2. **Path Length Difference:** You'll notice the `path_length` in this query's result is two hops shorter than the result from DuckPGQ. This is because this query only counts the `Transfer` edges, whereas the `DuckPGQ` query also includes the two `PersonOwn` edges in its path calculation. With that in mind, here is the traditional recursive CTE to find the shortest path between any two accounts owned by the same person: ```sql WITH RECURSIVE owned_accounts AS ( SELECT accountId FROM PersonOwnAccount WHERE personId = 125 ), path_search(start_node, end_node, path, depth) AS ( -- Base case: a direct transfer from one of the person's accounts SELECT fromId, toId, [fromId, toId], 1 FROM accounttransferaccount WHERE fromId IN (SELECT accountId FROM owned_accounts) UNION ALL -- Recursive step: find the next transfer in the path SELECT ps.start_node, t.toId, list_append(ps.path, t.toId), ps.depth + 1 FROM path_search ps JOIN accounttransferaccount t ON ps.end_node = t.fromId WHERE t.toId NOT IN (SELECT unnest(ps.path)) AND ps.depth < 11 ) SELECT distinct start_node, end_node, min(depth) AS path_length FROM path_search WHERE end_node IN (SELECT accountId FROM owned_accounts) AND start_node <> end_node GROUP BY ALL ORDER BY path_length; ``` As you can see, the logic requires a `WITH RECURSIVE` clause, manual path tracking in a list, and explicit cycle detection. This is exactly the kind of verbose and complex query that the visual syntax of SQL/PGQ is designed to eliminate. #### Wrapping Up {#blog:2025-10-22-duckdb-graph-queries-duckpgq::wrapping-up} We began this post with a simple goal: to see if we could use DuckDB to hunt for the complex patterns and hidden paths typical of graph analysis. After diving into the Financial Benchmark dataset, the conclusion is clear: you can. The key takeaway is the drastic improvement in usability. We saw how the visual syntax of SQL/PGQ, enabled by the DuckPGQ extension, transformed a sophisticated “ownership cycle” query from a monstrous recursive CTE into a few readable lines of code. This is exactly the kind of expressive power needed for real-world analytical tasks. For more information and complete documentation on DuckPGQ, be sure to visit its official website: [duckpgq.org](https://duckpgq.org). Just as importantly, this entire investigation was performed directly within DuckDB. By leveraging the extension ecosystem, we tapped into the power of graph queries without ever needing to export our data or manage a separate, specialized system. Everything runs on top of DuckDB's high-performance vectorized engine, right where the data lives. For powerful, analytical graph queries, DuckDB isn't just a viable alternative, it's a powerful, natural solution. The next time you think about analyzing connections in your data, remember that the tools you need are just an `INSTALL` away. ## Frozen DuckLakes for Multi-User, Serverless Data Access {#blog:2025-10-24-frozen-ducklake} **Publication date:** 2025-10-24 **Author:** Mark Harrison **TL;DR:** We show how you can build high-performance data lakes with no moving parts. ## Relational Charades: Turning Movies into Tables {#blog:2025-10-27-movies-in-databases} **Publication date:** 2025-10-27 **Author:** Hannes Mühleisen **TL;DR:** You can store and even process videos in DuckDB. In this post, we show you how. > Quote Your scientists were so preoccupied with whether they could, they didn't stop to think if they should. > > -- Dr. Ian Malcolm, Jurassic Park (1993) Here at team DuckDB, we *love* tables. Tables are a timeless elegant abstraction that precedes literature by [about a thousand years](https://www.youtube.com/watch?v=-wCzn9gKoUk). Relational tables specifically can represent *any* kind of information imaginable. But just because something *can* be done it is not always a great idea to do so. Can we build a rocket propelled by a nuclear chain reaction that irradiates the land it flies over? [Yes](https://en.wikipedia.org/wiki/Project_Pluto). Should we? Probably not. #### Disclaimer {#blog:2025-10-27-movies-in-databases::disclaimer} Array-like data like images and videos are a *textbook example* of something that [might not benefit](https://stackoverflow.com/questions/3748/storing-images-in-db-yea-or-nay) from storing them in a database. While of course any binary data can be added to tables as `BLOB`s, there is not that much added value from it. Sure, it's harder to lose the image compared to the industry standard solution, storing a file name that points to the image. But there are not that many meaningful operations that can be done on BLOBs other than store and load. Without adding some [overhyped](https://pluralistic.net/2025/09/27/econopocalypse/) AI tech, you can't even ask the database [what the picture shows](https://xkcd.com/1425/). Array data also has its own world of highly specialized file formats and compression algorithms. Just think of the ubiquitous [MPEG-4 standard](https://en.wikipedia.org/wiki/MPEG-4) to store movies. They are approximate (not exact, lossy) formats that are designed around human perception models, which is why they can avoid storing things people do not notice. They achieve impressive compression rates, with a two hour “Full” HD movie compressing to about 2 GB using MPEG-4. #### Ignoring the Disclaimer {#blog:2025-10-27-movies-in-databases::ignoring-the-disclaimer} But what would it feel like to turn a movie into a table? (Very) deep down, a movie is just a series of fast-moving pictures (“frames”), typically at something around 25 frames per second. At that speed, our monkey-brain cannot distinguish the separate images any more and is fooled into thinking that we are watching smooth movement. Side note for the younger generations, a strip of pictures was the way we [shipped around movies](https://en.wikipedia.org/wiki/Film_stock#/media/File:ButterflyDancebis.jpg) for more than 100 years. So a series of pictures it is. Each picture can be further deconstructed into a two-dimensional array (a “matrix”) of points, so-called “pixels”. Every pixel in turn consists of three numbers, one each for the intensity of red, green and blue, or [RGB](https://en.wikipedia.org/wiki/RGB_color_model) for short. Note that we're ignoring the audio tracks in this post, but in principle it would work the exact same way, just with a different kind of intensity. As an added complexity, the relational model (famously) [does not require](https://www.reddit.com/r/Database/comments/1l7tbrc/why_is_inherent_order_in_rdbms_so_neglected/) an absolute order of records. So all the various offsets have to be made explicit to not lose information. This of course greatly increases the size of our data set. We end up with a table that looks like this: | i | y | x | r | g | b | | ---: | ---: | ---: | ---: | ---: | ---: | | 0 | 0 | 0 | 4 | 5 | 1 | | 0 | 0 | 1 | 4 | 5 | 1 | | 0 | 0 | 2 | 5 | 6 | 2 | | 0 | 0 | 3 | 8 | 9 | 4 | | 0 | 0 | 4 | 9 | 10 | 5 | | 0 | 0 | 5 | 11 | 12 | 8 | | 0 | 0 | 6 | 11 | 12 | 8 | | 0 | 0 | 7 | 11 | 12 | 8 | | 0 | 0 | 8 | 9 | 10 | 5 | | 0 | 0 | 9 | 9 | 10 | 5 | We have the time offset or frame number `i`, we have `x` and `y` for the pixel position in the frame, and `r`, `g` and `b` for the color components red, green and blue. Quite involved. But now the movie is just a single table. If only just we had a conventional and guaranteed total order of rows, we could in theory skip all columns except for `r`, `g` and `b`, because with a known resolution all other columns can be inferred. This is coincidentally also how actual movie data files are stored, ignoring compression. This is another reason why maybe relational tables are not the best place for a movie to live in, but if [all you have is a hammer](https://en.wikipedia.org/wiki/Law_of_the_instrument). We could also have used some more modern features of SQL and use nested fields (a `LIST` in DuckDB), but let's keep it to a table even [System R](https://en.wikipedia.org/wiki/IBM_System_R) could have dealt with. In addition, having explicit offsets does not require nebulous conventions or additional metadata to know along with axis order the array data was serialized. #### Experiments {#blog:2025-10-27-movies-in-databases::experiments} To investigate this daft idea further (for [Science](https://www.ru.nl/personen/muhleisen-h)!), we convert the 1963 classic [“Charade”](https://en.wikipedia.org/wiki/Charade_(1963_film)), a “romantic screwball comedy mystery film” starring Audrey Hepburn and Cary Grant to a DuckDB table. This movie was picked because it is *accidentally in the Public Domain* because a screw-up in the wording of the copyright notice (no, really). Because of this, you can actually [freely download this movie](https://archive.org/details/Charade19631280x696) from the Internet Archive.  Since we're just creating a table, we will use DuckDB's native storage format. Here is the *complete* code snippet we used to convert the movie. In fact, this code should actually be generic enough to convert anything that `ffmpeg` can read to a table. Just in case you would want to try this at home on your own movies. ```python import imageio import duckdb # setup movie reading vid = imageio.get_reader("Charade-1963.mp4", "ffmpeg") dim_x = vid.get_meta_data()['size'][0] dim_y = vid.get_meta_data()['size'][1] rows_per_frame = dim_y * dim_x # setup a DuckDB database and table con = duckdb.connect() con.execute("ATTACH 'charade.duckdb' AS m (STORAGE_VERSION 'latest'); USE m;") con.execute("CREATE TABLE movie (i BIGINT, y USMALLINT, x USMALLINT, r UTINYINT, g UTINYINT, b UTINYINT)") # those offsets don't change between frames, so pre-compute them con.execute("CREATE TEMPORARY TABLE y AS SELECT unnest(list_sort(repeat(range(?), ?))) y", [dim_y, dim_x]) con.execute("CREATE TEMPORARY TABLE x AS SELECT unnest(repeat(range(?), ?)) x", [dim_x, dim_y]) # loop over each frame in the movie and insert the pixel data for i_idx, im in enumerate(vid): v = im.flatten() r = v[0:len(v):3] g = v[1:len(v):3] b = v[2:len(v):3] con.execute('''INSERT INTO movie FROM repeat(?, ?) i -- frame offset POSITIONAL JOIN y -- temp table POSITIONAL JOIN x -- temp table POSITIONAL JOIN r -- numpy scan POSITIONAL JOIN g -- numpy scan POSITIONAL JOIN b -- numpy scan ''', [i_idx, rows_per_frame]) ``` This script makes use of not just one, but (at least) *two* cool DuckDB features. First, we use so-called [replacement scans](https://duckdb.org/docs/1.5/clients/c/replacement_scans) to directly query the NumPy arrays `r`, `g`, and `b`. Note that those have not been created as tables in DuckDB nor registered in any way, but they are referenced by name in the `INSERT`. What happens here is that DuckDB inspects the Python context for the missing “tables” and finds objects with those names that it can read. The other cool feature is the [`POSITIONAL JOIN`](https://duckdb.org/docs/1.5/sql/query_syntax/from#positional-joins), which lets us stack multiple tables horizontally by position without running an actual (expensive) `JOIN`. This way, we assemble all the columns we need for a single frame in a bulk `INSERT`, which executes quite efficiently. The movie file we have has a frame rate of 25 frames per second at a (DVD-ish) resolution of 720x392 pixels. The total runtime is 01:53:02.56 seconds, which comes down to 169 563 individual frames. Because we have a row for each pixel we end up with 169 563 * 720 x 392 rows, or 47 857 461 120. 47 billion rows! Finally [Big Data](https://motherduck.com/blog/big-data-is-dead/)! When stored as a DuckDB database however, the file size is “only” around 200 GB. Totally doable on a laptop! DuckDB's [lightweight compression](#blog:2022-10-28-lightweight-compression) performs quite well here, given that in a naive binary format we would have to store at least 15 bytes per row. If we multiply that with the row count (47 billion, remember) we would end up at around 700 GB in storage for this hypothetical naive format. Of course, by turning the data into a relational table we add a bunch of previously implicit information due to the lack of ordering in relations. If we just stored the raw pixel bytes, for example as an implicitly ordered series of BMP (bitmap) files, we would end up with the same amount of bytes as the rows above times three, or 133 GB. Even *including materializing all the offsets*, the DuckDB file still manages to end up at a comparable size (200 GB). And of course, comparing the size of the table with the MPEG-4 version of the movie is not entirely fair because MPEG-4 is a *lossy* compression format. Databases can't just randomly decide to compromise on the numerical accuracy of the tables they store! To prove that the transformation is accurate, let's try to turn the table data for one random frame back into a human-consumable picture: we will retrieve the corresponding rows from DuckDB, and use some Python magic to turn them back into a PNG image file: ```python import duckdb import numpy as np import PIL.Image frame = 48000 con = duckdb.connect('charade.duckdb', read_only=True) dim_y, dim_x = con.execute("SELECT max(y) + 1 dim_y, max(x) + 1 dim_x FROM movie WHERE i=0").fetchone() res = con.execute("SELECT r, g, b FROM movie WHERE i = ? ORDER BY y, x", [frame]).fetchnumpy() v = np.zeros(dim_y * dim_x * 3, dtype=np.uint8) v[0:len(v):3] = res['r'] v[1:len(v):3] = res['g'] v[2:len(v):3] = res['b'] img = PIL.Image.fromarray(v.reshape((dim_y, dim_x, 3))) img.save(f'frame.png') ```  And voila, we can see a wonderful frame with Audrey and Cary appear. This trick can also be used to create a sequence of pictures and write them to a MPEG-4 file again using – for example – the `moviepy` library. But now that we have a table, we can have some fun with it. Let's do some basic exploration first: we start with `DESCRIBE`, which basically tells us the schema. We knew this of course. ```sql DESCRIBE movie; ``` | column_name | column_type | null | key | default | extra | | ----------- | ----------- | ---- | ---- | ------- | ----- | | i | BIGINT | YES | NULL | NULL | NULL | | y | USMALLINT | YES | NULL | NULL | NULL | | x | USMALLINT | YES | NULL | NULL | NULL | | r | UTINYINT | YES | NULL | NULL | NULL | | g | UTINYINT | YES | NULL | NULL | NULL | | b | UTINYINT | YES | NULL | NULL | NULL | No surprises there. How many rows are there? ```sql FROM movie SELECT count(*); ``` | count_star() | | -----------: | | 47857461120 | Ah yes, 47 billion. What are the numeric properties of the columns? DuckDB has this neat `SUMMARIZE` statement that computes single-pass summary statistics on a table (or arbitrary query). ```sql SUMMARIZE movie; ``` This one is admittedly a bit of a flex. DuckDB can compute elaborate summary statistics on all the 47 billion rows in ca. 20 minutes on a MacBook. Here are the results: | column_name | column_type | min | max | approx_unique | avg | std | q25 | q50 | q75 | count | null_percentage | | ----------: | ----------- | ---: | -----: | ------------: | -----------------: | -----------------: | ----: | ----: | -----: | ----------: | --------------: | | i | BIGINT | 0 | 169562 | 150076 | 84781.0 | 48948.621846957954 | 42429 | 84751 | 127137 | 47857461120 | 0.00 | | y | USMALLINT | 0 | 391 | 430 | 195.5 | 113.16028455346597 | 98 | 196 | 294 | 47857461120 | 0.00 | | x | USMALLINT | 0 | 719 | 840 | 359.5 | 207.84589644146592 | 180 | 359 | 540 | 47857461120 | 0.00 | | r | UTINYINT | 0 | 255 | 252 | 65.32575855816732 | 44.85627602555231 | 27 | 54 | 96 | 47857461120 | 0.00 | | g | UTINYINT | 0 | 249 | 249 | 56.79713844669577 | 37.03562456032193 | 28 | 44 | 77 | 47857461120 | 0.00 | | b | UTINYINT | 0 | 255 | 252 | 43.249715985643995 | 38.39218963268899 | 16 | 28 | 61 | 47857461120 | 0.00 | Since we're basically storing a lot of colors, just how many different combinations of red, green and blue are there, DuckDB? ```sql FROM (FROM movie SELECT DISTINCT r, g, b) SELECT count(*); ``` Any seasoned data engineer would rightfully caution you to run a `DISTINCT` on this many rows. There have just been too many production outages caused by overflowing aggregations. But thanks to DuckDB's [larger-than-memory aggregate hash table](#blog:2024-03-29-external-aggregation), we can confidently issue this query. We even get a nice progress bar and (since 1.4.0) a surprisingly accurate estimate of how long the query will take. | count_star() | | -----------: | | 826568 | So roughly 800 thousand different colors. Computing this took about 2 minutes in the end. But what are the frequencies of those colors? Let's compute a histogram of the 10 most-used colors! ```sql FROM movie SELECT r, g, b, count(*) AS ct GROUP BY ALL ORDER BY ct DESC LIMIT 10; ``` | r | g | b | ct | | ---: | ---: | ---: | --------: | | 17 | 20 | 15 | 106521429 | | 23 | 25 | 15 | 93004303 | | 23 | 25 | 13 | 85552738 | | 13 | 22 | 15 | 81734796 | | 22 | 24 | 13 | 76560295 | | 24 | 26 | 15 | 75376896 | | 15 | 19 | 8 | 74285763 | | 23 | 24 | 19 | 72904497 | | 22 | 24 | 12 | 69269099 | | 24 | 26 | 16 | 62230136 | The most common colors here seems to be dark shades of grey. Makes sense! Keep in mind that the MPEG-4 compression is lossy and will probably produce some odd colors as rounding artifacts. But we can also have some more fun. We have an analytical database system. How about we compute the average frame for every thousand frames and stitch the results back into a movie? It's just a big aggregation. We first create the actual averages: ```sql CREATE TABLE averages AS FROM movie SELECT i // 1000 AS idx, y, x, avg(r)::UTINYINT AS r, avg(g)::UTINYINT AS g, avg(b)::UTINYINT AS b GROUP BY ALL ORDER BY idx, y, x; ``` Then, we use Python again to turn this `averages` table into a movie: ```python # some setup omitted # fetch a bunch of frames in bulk res = con.execute("SELECT r, g, b FROM averages ORDER BY i, y, x").fetchnumpy() # split the rgb arrays by frame again r_splits = np.split(res['r'], num_frames) g_splits = np.split(res['g'], num_frames) b_splits = np.split(res['b'], num_frames) # generate pictures image_files = [] for i in range(num_frames): v = np.zeros(dim_y * dim_x * 3, dtype=np.uint8) v[0:len(v):3] = r_splits[i] v[1:len(v):3] = g_splits[i] v[2:len(v):3] = b_splits[i] image_files.append(v.reshape((dim_y, dim_x, 3), order='C')) # write movie file clip = moviepy.video.io.ImageSequenceClip.ImageSequenceClip(image_files, fps=25) clip.write_videofile('averages.mp4') ``` There is some wrangling here because we want to retrieve the whole frame dataset in bulk and not run a query for every single one. We then use NumPy to split them into frames and stitch the RGB-channels together into the three-dimensional array that the image libraries like. This does not achieve any business purpose but the results are kind of funny, here is average frame #68, with apologies to the actors:  We can also stitch all the averages together to make a somewhat twitchy average movie:Click here to see the twitchy movie generated from “Charade”:
| ', r, g, b) || IF(x = 719, ' |
Click to see the SQL code sample for Iceberg updates.
```sql ATTACH '⟨warehouse_name⟩' AS iceberg_catalog ( TYPE iceberg, ⟨other options⟩ ); CREATE TABLE iceberg_catalog.default.simple_table (col1 INTEGER, col2 VARCHAR); INSERT INTO iceberg_catalog.default.simple_table VALUES (1, 'hello'), (2, 'world'), (3, 'duckdb is great'); DELETE FROM iceberg_catalog.default.simple_table WHERE col1 = 2; UPDATE iceberg_catalog.default.simple_table SET col1 = col1 + 5 WHERE col1 = 1; ```Click here to see the entropy of a plaintext database
Click here to see the entropy of an encrypted database
Click here to see the individual query runtimes.
| Q | SF100/MacBook | SF100/MOREFINE | SF300/MacBook | SF300/MOREFINE |
|---|---|---|---|---|
| 1 | 1.247 | 7.363 | 4.528 | 26.475 |
| 2 | 0.117 | 1.058 | 0.474 | 4.101 |
| 3 | 0.697 | 8.563 | 2.759 | 32.432 |
| 4 | 0.570 | 7.348 | 2.331 | 27.185 |
| 5 | 0.631 | 8.498 | 3.217 | 34.462 |
| 6 | 0.180 | 1.236 | 1.395 | 13.225 |
| 7 | 0.620 | 7.702 | 3.119 | 37.411 |
| 8 | 0.640 | 5.593 | 3.611 | 29.914 |
| 9 | 1.906 | 30.560 | 6.670 | 99.884 |
| 10 | 0.923 | 11.755 | 4.036 | 40.412 |
| 11 | 0.102 | 1.037 | 0.709 | 4.444 |
| 12 | 0.535 | 6.422 | 2.918 | 31.501 |
| 13 | 1.847 | 21.185 | 6.394 | 74.081 |
| 14 | 0.408 | 5.616 | 3.240 | 26.613 |
| 15 | 0.252 | 2.652 | 1.906 | 17.454 |
| 16 | 0.273 | 3.108 | 0.879 | 11.480 |
| 17 | 0.805 | 5.184 | 4.655 | 28.469 |
| 18 | 1.538 | 15.492 | 7.619 | 71.845 |
| 19 | 0.779 | 9.143 | 4.379 | 39.111 |
| 20 | 0.441 | 4.993 | 3.234 | 25.967 |
| 21 | 1.996 | 23.231 | 9.503 | 96.452 |
| 22 | 0.441 | 5.036 | 1.237 | 18.709 |
Click here to see a more detailed breakdown of the benchmark results.
This figure shows the results per query, including the standard deviation error bar.  The following is the summary of the sizes of the datasets. Note that both Parquet v1 and v2 are using the default compression used by the DuckDB Parquet writer, which is Snappy. In this case, Vortex is not using any general-purpose compression but still keeps the data sizes competitive. | Table | parquet_v1 | parquet_v2 | vortex | | :---- | :---- | :---- | :---- | | customer | 1.15 | 0.99 | 1.06 | | lineitem | 21.15 | 16.02 | 18.14 | | nation | 0.00 | 0.00 | 0.00 | | orders | 6.02 | 4.54 | 5.03 | | part | 0.59 | 0.47 | 0.54 | | partsupp | 4.07 | 3.33 | 3.72 | | region | 0.00 | 0.00 | 0.00 | | supplier | 0.07 | 0.06 | 0.07 | | **total** | 33.06 | 25.40 | 28.57 |If you are interested in more details, click here.
Due to technical reasons, `httplib` is still the library we use for downloading the `httpfs` extension. When `httpfs` is loaded with the (now default) `curl` backend, subsequent extension installations go through `https://`, with the default endpoint for core extensions pointing to [`https://extensions.duckdb.org`](https://extensions.duckdb.org). All core and community extensions are cryptographically signed, so installing them through `http://` does not pose a security risk. However, some users reported issues about `http://` extension installs in environments with firewalls.See the code that creates the example databases.
```sql ATTACH 'numbers1.db'; ATTACH 'numbers2.db'; ATTACH 'animals.db'; CREATE TABLE numbers1.tbl AS FROM range(1, 3) t(i); CREATE TABLE numbers2.tbl AS FROM range(2, 6) t(i); CREATE TABLE animals.ducks AS FROM (VALUES (1, 'Labrador Duck', 1878), (2, 'Mallard', NULL), (3, 'Crested Shelduck', 1964), (4, 'Wood Duck', NULL), (5, 'Pink-headed Duck', 1949) ) t(id, name, extinct_year); CREATE TABLE animals.swans AS FROM (VALUES (1, 'Aurora', 'Mute Swan', 'White', 'European lakes and rivers'), (2, 'Midnight', 'Black Swan', 'Black', 'Australian wetlands'), (3, 'Tundra', 'Tundra Swan', 'White', 'Arctic and subarctic regions') ) t(id, name, species, color, habitat); DETACH numbers1; DETACH numbers2; DETACH animals; ```
###### DuckDB #1 {#blog:2026-05-12-quack-remote-protocol::svg-classiconuse-hrefdatabase-01usesvg-duckdb-1}
```sql
CALL quack_serve(
'quack:localhost',
token = 'super_secret'
);
CREATE TABLE hello AS
FROM VALUES ('world') v(s);
```
quack:
###### DuckDB #2 {#blog:2026-05-12-quack-remote-protocol::svg-classiconuse-hrefdatabase-01usesvg-duckdb-2}
```sql
CREATE SECRET (
TYPE quack,
TOKEN 'super_secret'
);
ATTACH 'quack:localhost' AS remote;
FROM remote.hello;
```
This should show the content of the remote table hello, `world` in DuckDB #2. Witchcraft! We can also copy data from the local instance to the remote one:
###### DuckDB #1 {#blog:2026-05-12-quack-remote-protocol::svg-classiconuse-hrefdatabase-01usesvg-duckdb-1}
```sql
-- Step two
FROM hello2;
```
quack:
###### DuckDB #2 {#blog:2026-05-12-quack-remote-protocol::svg-classiconuse-hrefdatabase-01usesvg-duckdb-2}
```sql
-- Step one
CREATE TABLE remote.hello2 AS
FROM VALUES ('world2') v(s);
```
Similarly, you should see `world2` in the output on DuckDB #1. Obviously those are the most basic examples we can think of. Tables can be much more complex, queries can be much more complex, data volumes can be quite vast (see below). There is also a way to just ship an entire verbatim query to the remote side using the `query` function, which is better for very complex queries on large datasets and offers more control over what exactly is executed remotely:
###### DuckDB #1 {#blog:2026-05-12-quack-remote-protocol::svg-classiconuse-hrefdatabase-01usesvg-duckdb-1}
```sql
-- Waiting to serve data
```
quack:
###### DuckDB #2 {#blog:2026-05-12-quack-remote-protocol::svg-classiconuse-hrefdatabase-01usesvg-duckdb-2}
```sql
FROM remote.query(
'SELECT s FROM hello'
);
```
Of course there is much more to see here. Please [consult our documentation](https://duckdb.org/docs/1.5/quack/overview) for more details.
#### Protocol Design {#blog:2026-05-12-quack-remote-protocol::protocol-design}
##### HTTP-Based {#blog:2026-05-12-quack-remote-protocol::http-based}
Quack is built straight on the venerable HTTP, the Hypertext Transfer Protocol. From its humble beginnings at CERN, HTTP has become a de-facto protocol layer on top of TCP and all the stuff below. The entire stack is optimized to transmit HTTP message streams efficiently. The protocol has surprisingly low overhead if implemented properly. Everyone and their little brother knows how to deal with HTTP in load balancing, authentication, firewalls, intrusion detection etc. It would be rather misguided not to build a database protocol on top of HTTP in 2026. HTTP also allows the [DuckDB-Wasm distribution to speak Quack natively](https://duckdb.org/docs/1.5/quack/setup/quack_wasm)! So DuckDB running in a browser can e.g., directly connect to a DuckDB instance running in an EC2 server using Quack.
##### Request-Response Pattern {#blog:2026-05-12-quack-remote-protocol::request-response-pattern}
Interactions on Quack are always driven by the client in a request-response pattern. Quack messages are for example connection requests, to authenticate with a token as seen above. See below on how authentication and authorization are handled in detail. Subsequent messages are requests to execute a query and return the first part of the response and follow-up fetch messages to retrieve large results, possibly from multiple threads in parallel.
##### Serialization {#blog:2026-05-12-quack-remote-protocol::serialization}
Requests and responses are encoded using the new MIME type application/duckdb. This encoding leverages DuckDB’s internal efficient serialization primitives for complex structures like data types and result sets. We have been using the same primitives for example in our Write-Ahead Log (WAL) files for years, meaning they are fairly well-optimized and battle-tested.
##### Encryption {#blog:2026-05-12-quack-remote-protocol::encryption}
While we want Quack to “just work” we also are wary of the security nightmares of attaching a database directly to the evil internet, as has happened before. This is why Quack will by default generate a random authentication token at server start-up, which then has to be given to the client. In addition, the Quack server will by default only bind to localhost (which can of course be overridden). Quack does not use SSL by default, because it is a bit silly to bring all that infrastructure and add dependencies just for localhost communication. We do not recommend opening up a DuckDB Quack endpoint directly to the Internet. Instead we strongly recommend that you use a common HTTP endpoint like [nginx](https://nginx.org/) if you should choose to expose Quack to the World Wide Web and have that proxy terminate SSL (e.g., with Let's Encrypt). The Quack client will assume SSL is enabled for non-local connections, this can be overridden. We provide a [guide for this in our documentation](https://duckdb.org/docs/1.5/quack/setup/reverse_proxy).
##### Round-Trips {#blog:2026-05-12-quack-remote-protocol::round-trips}
We have been careful to optimize the number of protocol round trips or request/response pairs for queries. Once connected, a query can be completely handled with a single round trip. This is a critical optimization for latency-sensitive environments. At the same time, we have seriously optimized Quack for efficient bulk response transfer. As far as we know, Quack is currently the fastest way to shove tables through a socket, and millions of rows can be transferred in a few seconds. Below are a few benchmark results.
##### Authentication and Authorization {#blog:2026-05-12-quack-remote-protocol::authentication-and-authorization}
Authentication and authorization of database queries are an endless source of joy and complexity. We are likely unable to capture everyone’s use case, certainly not in a first release. The smart thing is therefore not to try. For Quack, we have chosen an auth model that ties into DuckDB’s philosophy of extensibility. There are hundreds of DuckDB extensions out there already. Quack ships with a default Authentication method and no authorization restrictions, but both can be overridden by user-supplied code. As you have seen above, the Quack server generates a default random authentication token on startup. When a client connects, it provides an authentication string. The server side will call an authentication callback. By default, it will compare the client-supplied token with the one that was randomly generated before. But this callback can be changed through configuration! You can bring your own authentication function that for example queries an LDAP directory, reads a text file, or just rolls the dice. Up to you. Similarly, the authorization function can be changed. The default authorization function just says “yes” to everything, but you can inspect each query a client attempts to execute, correlate the query to the previously used authentication string etc. Those callbacks can even be plain SQL macros! Please see our documentation for more details.
##### Default Port {#blog:2026-05-12-quack-remote-protocol::default-port}
By default, a Quack server listens on port `9494`, the number `94` being easy to remember as the year [Netscape Navigator](https://en.wikipedia.org/wiki/Netscape_Navigator) was released.
#### Benchmarks {#blog:2026-05-12-quack-remote-protocol::benchmarks}
We have set up two benchmarks to showcase the Quack protocol. Those benchmarks were run on AWS virtual machines running Ubuntu on Arm. We picked the [m8g.2xlarge](https://instances.vantage.sh/aws/ec2/m8g.2xlarge) instance type, which has 8 vCPUs and 32 GB of RAM and, importantly, “up to 15 Gbps” network bandwidth. We recreated a real-world scenario where client and server are in the same data center, but on different machines. We made sure both instances were in the same “availability zone”. Ping time between the instances averaged around 0.280 ms.
##### Bulk Transfer {#blog:2026-05-12-quack-remote-protocol::bulk-transfer}
The first benchmark tests bulk transfer, the case where a fairly large number of rows should be transferred over the database protocol. If you’ve read the paper we linked above, you know that this is a case where traditional database protocols were struggling. We compare Quack with two systems: the widespread PostgreSQL protocol and the newer Arrow Flight SQL protocol. Arrow Flight is provided by the [GizmoSQL](https://docs.gizmosql.com/#/) server that also uses DuckDB internally. We transfer an increasing number of rows of the TPC-H lineitem table, all the way up to a whopping 60 million rows (76 GB in CSV format!) and report the median wall clock time over 5 runs. We expect the modern bulk-oriented protocols to far outclass the PostgreSQL protocol. Here are the results:
Runtimes of bulk transfer operations (lower is better)
Would you like to see the results as a table? Click here.
| Rows | DuckDB Quack | Arrow Flight | PostgreSQL |
|---|---|---|---|
| 100k | 0.07 s | 0.07 s | 0.20 s |
| 1M | 0.24 s | 0.38 s | 2.20 s |
| 10M | 0.89 s | 2.90 s | 25.64 s |
| 60M | 4.94 s | 17.40 s | 158.37 s |
Would you like to see the results as a table? Click here.
| Threads | DuckDB Quack | Arrow Flight | PostgreSQL |
|---|---|---|---|
| 1 | 1,038 tx/s | 469 tx/s | 839 tx/s |
| 2 | 1,956 tx/s | 799 tx/s | 1,094 tx/s |
| 4 | 3,504 tx/s | 1,224 tx/s | 2,180 tx/s |
| 8 | 5,434 tx/s | 1,358 tx/s | 4,320 tx/s |
###### DuckDB Server {#blog:2026-05-20-announcing-duckdb-153::svg-classiconuse-hrefdatabase-01usesvg-duckdb-server}
```sql
CALL quack_serve(
'quack:localhost',
token = 'super_secret'
);
CREATE TABLE hello AS
FROM VALUES ('world') v(s);
```
quack:
###### DuckDB Client {#blog:2026-05-20-announcing-duckdb-153::svg-classiconuse-hrefdatabase-01usesvg-duckdb-client}
```sql
CREATE SECRET (
TYPE quack,
TOKEN 'super_secret'
);
ATTACH 'quack:localhost' AS remote;
FROM remote.hello;
```
Please note that Quack is still in beta state and breaking changes may happen in the protocol, in function names, etc.
We plan to release the production-ready version of Quack together with [DuckDB v2.0](https://duckdb.org/release_calendar) in fall 2026.
##### DuckLake with Quack {#blog:2026-05-20-announcing-duckdb-153::ducklake-with-quack}
DuckLake now supports DuckDB with Quack as its catalog database ([ducklake#1151](https://github.com/duckdb/ducklake/pull/1151)).
Let the example speak for itself!
###### DuckDB Server {#blog:2026-05-20-announcing-duckdb-153::svg-classiconuse-hrefdatabase-01usesvg-duckdb-server}
```sql
CALL quack_serve(
'quack:localhost',
token => 'oogieboogie'
);
```
quack:
###### DuckDB Client {#blog:2026-05-20-announcing-duckdb-153::svg-classiconuse-hrefdatabase-01usesvg-duckdb-client}
```sql
INSTALL ducklake;
CREATE SECRET (
TYPE quack, TOKEN 'oogieboogie'
);
ATTACH 'ducklake:quack:localhost'
AS lake (DATA_PATH 'data');
USE lake;
CREATE TABLE pond (
id INT,
species VARCHAR,
weight DOUBLE
);
INSERT INTO pond VALUES
(1, 'mallard', 1.2),
(2, 'pintail', 0.9);
INSERT INTO pond VALUES
(3, 'wood duck', 0.7);
SELECT * FROM pond ORDER BY id;
```
##### AWS Extension Features {#blog:2026-05-20-announcing-duckdb-153::aws-extension-features}
The [AWS extension](https://duckdb.org/docs/1.5/core_extensions/aws) now supports the [`web_identity` chain type for IAM Roles for Service Accounts (IRSA) support](https://github.com/duckdb/duckdb-aws/pull/136).
This was made possible through a contribution by community member [Marcel Steinbach (@mst)](https://github.com/mst).
The AWS extension now also supports [IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html) for managed PostgreSQL databases running on RDS/Aurora. For more details, see the [AWS RDS IAM Authentication section](https://duckdb.org/docs/1.5/core_extensions/postgres/secrets#aws-rds-iam-authentication) in the documentation.
##### `HTTP_PROXY` Variable for the HTTPS Extension {#blog:2026-05-20-announcing-duckdb-153::http_proxy-variable-for-the-https-extension}
Setting the `HTTP_PROXY` environment variable now sets the `http_proxy` DuckDB configuration option ([duckdb#22541](https://github.com/duckdb/duckdb/pull/22541)).
This option makes sure that extensions installs are also passing through the proxy, which may come in handy in e.g. environments that use firewalls.
Note that since the [introduction of `curl` into DuckDB's network stack](#blog:2026-03-09-announcing-duckdb-150::network-stack), `curl` automatically uses `HTTP_PROXY` and `HTTPS_PROXY`, so now implicitly also DuckDB handles those parameters when the `httpfs` extension is loaded with the default `curl` backend.
##### Iceberg {#blog:2026-05-20-announcing-duckdb-153::iceberg}
The DuckDB-Iceberg extension has shipped a number of features between DuckDB v1.5.2 and v1.5.3. Most notably:
* `MERGE INTO` is now supported for Iceberg tables ([iceberg#788](https://github.com/duckdb/duckdb-iceberg/pull/788))
* The `INSERT` and `UPDATE` statements are now supported on partitioned Iceberg tables with a `truncate` or `bucket` transform ([iceberg#879](https://github.com/duckdb/duckdb-iceberg/pull/879))
* [CTAS](https://duckdb.org/docs/1.5/sql/statements/create_table#create-table--as-select-ctas) statements in DuckDB-Iceberg using [ADBC](https://duckdb.org/docs/1.5/clients/adbc) are now possible ([iceberg#974](https://github.com/duckdb/duckdb-iceberg/pull/974))
* We added the `iceberg_schema_properties`, `set_iceberg_schema_properties`, and `remove_iceberg_schema_properties` functions to allow getting, setting, and removing Iceberg schema properties ([iceberg#960](https://github.com/duckdb/duckdb-iceberg/pull/960))
* `ALTER TABLE` support has been added for Iceberg tables ([iceberg#932](https://github.com/duckdb/duckdb-iceberg/pull/932), [iceberg#928](https://github.com/duckdb/duckdb-iceberg/pull/928), [iceberg#924](https://github.com/duckdb/duckdb-iceberg/pull/924), [iceberg#912](https://github.com/duckdb/duckdb-iceberg/pull/912), [iceberg#904](https://github.com/duckdb/duckdb-iceberg/pull/904), [iceberg#853](https://github.com/duckdb/duckdb-iceberg/pull/853), [iceberg#985](https://github.com/duckdb/duckdb-iceberg/pull/985), [iceberg#981](https://github.com/duckdb/duckdb-iceberg/pull/981))
* Support for the `GEOMETRY` type has been added for Iceberg tables ([iceberg#968](https://github.com/duckdb/duckdb-iceberg/pull/968), [iceberg#902](https://github.com/duckdb/duckdb-iceberg/pull/902))
#### Development and Internals {#blog:2026-05-20-announcing-duckdb-153::development-and-internals}
##### Shipping jemalloc as a Statically Linked Library {#blog:2026-05-20-announcing-duckdb-153::shipping-jemalloc-as-a-statically-linked-library}
The [jemalloc allocator](https://duckdb.org/docs/1.5/internals/jemalloc) is now part of core DuckDB ([duckdb#22603](https://github.com/duckdb/duckdb/pull/22603)) as a static third-party library which is included and linked by default on Linux.
Previously jemalloc was a statically-linked extension – the new packaging is cleaner since other DuckDB extensions can be loaded dynamically.
##### `DISABLE_EXTENSION_LOAD` Flag {#blog:2026-05-20-announcing-duckdb-153::disable_extension_load-flag}
The `DISABLE_EXTENSION_LOAD` compile-time flag was fixed in [duckdb#22019](https://github.com/duckdb/duckdb/pull/22019).
When compiling DuckDB with this flag, loading extensions is disabled.
#### Coming Up {#blog:2026-05-20-announcing-duckdb-153::coming-up}
We have two events coming up in the next few weeks:
**DuckCon #7.** On June 24, we'll host our next user conference, [DuckCon #7](https://duckdb.org/events/2026/06/24/duckcon7/), in Amsterdam's beautiful [Royal Tropical Institute](https://www.kit.nl/about-us/).
**Ubuntu Summit Talk.** Next week, Gábor Szárnyas of DuckLabs will give a talk titled [“DuckDB: Not Quack Science”](https://duckdb.org/library/duckdb-not-quack-science/) at the [Ubuntu Summit](https://ubuntu.com/summit). Yes, his talk will include the new [Quack](#blog:2026-05-20-announcing-duckdb-153::quack-as-a-core-extension) protocol.
#### Conclusion {#blog:2026-05-20-announcing-duckdb-153::conclusion}
This post is a short summary of the changes in v1.5.3. As usual, you can find the [full release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.5.3).
## Test-Driving the Lance Lakehouse Format in DuckDB {#blog:2026-05-21-test-driving-lance}
**Publication date:** 2026-05-21
**Authors:** LanceDB team, Guillermo Sanchez
**TL;DR:** Lance is an open lakehouse format with a design geared toward AI workloads. LanceDB and DuckLabs have partnered to bring you fast vector and hybrid search directly from DuckDB SQL, without leaving your analytical workflow. In this post, we explain what Lance is, how to use it in DuckDB, and, of course, show some benchmark results.
With the [`lance` extension](https://duckdb.org/docs/1.5/core_extensions/lance), DuckDB users can query Lance datasets with the same familiar SQL interface (via the CLI or SDKs), while adding capabilities for AI and retrieval workloads. This blog post highlights how Lance is a good option for workloads that need to support storage and querying of vectors, rich table operations, and AI-oriented access patterns, while also supporting scan-friendly analytical workloads at scale. And with DuckDB, it becomes trivial to query those kinds of datasets in SQL.
> In this blog, there will be some mentions of “retrieval workloads” and “AI data patterns” or “AI datasets”. By “retrieval workloads,” we mean queries that find rows by similarity or keyword relevance, such as vector search and full-text search, rather than by exact filters or aggregations. By “AI data patterns,” we mean datasets that mix embeddings, images, or audio alongside scalar metadata.
#### What is Lance? {#blog:2026-05-21-test-driving-lance::what-is-lance}
[Lance](https://lance.org/) is an open lakehouse format designed for modern ML and AI workloads. Unlike Parquet, Lance is a file format, a table format, and a lightweight catalog spec all at once. At the table format level, Lance supports versioning, schema evolution, indexes, and transactional updates through MVCC and ACID-style semantics. In practice, this means Lance is built for datasets that change over time and need more than read-only scans.
This matters because many AI datasets are no longer just rows of scalar values. They often contain embeddings, long-form text, images, audio, metadata for filtering, and indexes used for retrieval. A format that works well for these workloads needs to do more than store and scan columns efficiently. It also needs to support search, updates, and lifecycle operations without forcing users into managing multiple different systems.
The mental model is still familiar to users coming from Parquet: columnar data in an open format, queried with standard analytical tools. Lance's fragment-based layout stores data in small columnar chunks. This design enables efficient random access without trade-offs in scan performance or memory utilization, something that has historically been difficult to achieve in columnar formats, and that the Lance team examines in [this 2025 paper on adaptive structural encodings](https://arxiv.org/abs/2504.15247).
On the data evolution side, adding columns or backfilling existing rows with new data only writes new files without touching existing ones. That makes schema changes lightweight in practice, which is useful for workflows where columns are added incrementally, such as appending derived features or embeddings to an existing dataset.
#### The Lance DuckDB Extension {#blog:2026-05-21-test-driving-lance::the-lance-duckdb-extension}
The [`lance` extension](https://duckdb.org/docs/1.5/core_extensions/lance) brings Lance into DuckDB as part of the SQL-based workflow. You can read Lance datasets directly, write to them via `COPY`, attach them as table namespaces, build indexes, and query them with regular DuckDB SQL. On top of that, the extension exposes Lance-native search functionality through SQL table functions.
This fits naturally with how DuckDB is already used: as a single, embedded SQL query engine that operates on many different data sources and file formats. With the Lance extension, DuckDB remains the familiar query engine, while Lance provides the storage, indexing, and search capabilities underneath, which is especially beneficial when your data is multimodal and includes embeddings.
##### Example Usage {#blog:2026-05-21-test-driving-lance::example-usage}
Installing and using the extension is straightforward:
```sql
INSTALL lance;
LOAD lance;
SELECT *
FROM 'path/to/dataset.lance'
LIMIT 10;
```
DuckDB can also write Lance datasets directly:
```sql
COPY (
SELECT *
FROM (
VALUES
(1::BIGINT, 'duck', [0.9, 0.7, 0.1]::FLOAT[3]),
(2::BIGINT, 'horse', [0.3, 0.1, 0.5]::FLOAT[3]),
(3::BIGINT, 'dragon', [0.5, 0.2, 0.7]::FLOAT[3])
) AS t(id, animal, vec)
) TO 'path/to/out.lance' (FORMAT lance, mode 'overwrite');
```
Once the data is in Lance, DuckDB can query it with Lance-native search operators. For example, hybrid search combines vector similarity and keyword relevance in one SQL query:
```sql
SELECT id, text, _hybrid_score, _distance, _score
FROM lance_hybrid_search(
'path/to/dataset.lance',
'vec',
[0.1, 0.2, 0.3, 0.4]::FLOAT[4],
'text',
'puppy',
k = 10,
prefilter = false,
alpha = 0.5,
oversample_factor = 4
)
ORDER BY _hybrid_score DESC;
```
The extension also exposes `lance_vector_search(...)` for vector similarity search and `lance_fts(...)` for full-text search, so users can choose the retrieval mode that fits their workload.
If you want table-style access instead of path-based access, you can attach a directory as a Lance namespace:
```sql
ATTACH 'path/to/dir' AS ns (TYPE lance);
SELECT count(*)
FROM ns.main.my_table;
```
Index creation also happens through SQL. For example, a vector index can be created directly on a Lance dataset:
```sql
CREATE INDEX vec_idx ON 'path/to/dataset.lance' (vec)
USING IVF_FLAT WITH (num_partitions = 1, metric_type = 'l2');
```
The extension surface goes well beyond read-only scans. In the current implementation, DuckDB can:
- Read Lance datasets with direct path scans
- Write and append Lance datasets with `COPY ... TO ... (FORMAT lance)`
- Run vector, full-text, and hybrid search with SQL functions
- Attach local directories or custom catalogs via REST namespaces
- Create, update, delete, merge, and alter tables in attached namespaces
- Create and manage vector, scalar, and full-text indexes
- Run maintenance operations such as compaction, cleanup, and index optimization
Note that this extension is not just a file reader, but it also gives DuckDB users a way to work with Lance as an operational table format from inside SQL.
#### Why Lance and DuckDB? {#blog:2026-05-21-test-driving-lance::why-lance-and-duckdb}
The combination of Lance and DuckDB is compelling for three reasons.
First, it gives users one SQL surface for analytics plus retrieval. The same DuckDB workflow can scan a dataset, filter it, join it with other tables, compute aggregates, and then run vector search or hybrid search over the result set. That is a good fit for AI applications where retrieval is only one step in a larger analytical pipeline.
Second, Lance is a table format for more than traditional analytics. Many AI pipelines need versioned datasets, updates, deletes, `MERGE`-style changes, index management, and schema evolution. The DuckDB extension exposes these capabilities through SQL, which means users do not need to leave the DuckDB environment just because their dataset is doing more than serving analytical reads.
Third, the workflow scales from local files to remote storage without changing the mental model. You can start with a local `lance` dataset, then [move to object storage](https://duckdb.org/docs/1.5/core_extensions/lance#query-a-lance-dataset).
The extension also supports REST namespaces, so DuckDB can connect to a remote Lance catalog (including [LanceDB Enterprise](https://docs.lancedb.com/enterprise/index)) and treat it like an attached database. That makes the local-to-remote storage progression feel incremental rather than disruptive.
To sum up, DuckDB remains the familiar SQL engine, while Lance adds storage and indexing features that are especially useful when the same dataset powers both analytics and retrieval.
#### Performance Experiment {#blog:2026-05-21-test-driving-lance::performance-experiment}
[LAION](https://laion.ai/) is an open dataset of image/caption pairs scraped from the public web, originally released to support research on models like [CLIP](https://openai.com/index/clip/), which learn a shared embedding space for images and text. The full release spans billions of pairs. For this experiment, we used the `lance-format/laion-1m` subset on Hugging Face Hub, which is easy to reproduce locally.
Each row carries a caption, a 768-dimensional CLIP image embedding, the raw image bytes, and scalar metadata like width, height, and NSFW flags. This mix of scalar, text, vector, and blob data in a single table makes it a useful workload for comparing formats, and it is structurally different from the wide-but-flat schemas like TPC-H or ClickBench that are traditionally used for analytical benchmarks.
The public Hugging Face export used by the benchmark currently materializes 69,632 rows locally, not the full million-row source dataset. The runner first downloads the public Parquet shards, then builds all local artifacts from that same baseline: an LZ4-compressed Parquet file, an indexed DuckDB database, and a Lance dataset. Generated files are reused across runs, so the initial download is the only networked step.
> The experiments were run on an Apple MacBook Pro with a 10-core M1 Max CPU and 32 GB of RAM, running DuckDB 1.5.2.
The benchmark was run using DuckDB as the query engine for the following three storage formats:
- **Parquet**: DuckDB scanning the LZ4-compressed Parquet baseline directly, with no auxiliary indexes.
- **DuckDB indexed**: the same baseline loaded into a DuckDB table, with DuckDB's `vss` (HNSW) and `fts` extensions layered on top, plus scalar indexes on filter columns. This is the typical “build it yourself in DuckDB” stack.
- **Lance native**: the same baseline written to a Lance dataset with a vector index, a full-text index, and native blob storage, queried through the DuckDB lance extension.
The workloads are aligned by task across the three paths, even though the exact SQL differs by storage/indexing backend:
- `fts`: find rows by keyword search over the caption text.
- `vector_exact`: run nearest-neighbor search over the CLIP embedding column without using an approximate vector index.
- `vector_indexed`: run nearest-neighbor search over the same embedding column using the available vector index.
- `hybrid`: combine text search and vector search into one retrieval query, returning the best-ranked matches from both signals.
- `blob_read`: fetch image bytes for selected rows, which exercises random access to large binary values rather than just scalar or vector columns.
Each workload is run five times by default, and the tables below report the average. The full scripts and SQL queries are in the `laion_1m` benchmark directory.
##### Cold Results {#blog:2026-05-21-test-driving-lance::cold-results}
The table below runs each workload cold, in a fresh DuckDB process, so it captures process startup, file open, and first-query cost. It's closest to what a one-off script or a cron job would see.
| Workload | Parquet | DuckDB indexed | Lance native |
| :--------------- | ------: | -------------: | -----------: |
| `fts` | 12 ms | 11 ms | 21 ms |
| `vector_exact` | 695 ms | 61 ms | 89 ms |
| `vector_indexed` | 761 ms | 104 ms | 12 ms |
| `hybrid` | 465 ms | 80 ms | 17 ms |
| `blob_read` | 1559 ms | 271 ms | 278 ms |
In the cold run, Lance stands out in the `vector_indexed` and `hybrid` workloads. DuckDB’s own format does well in `vector_exact` and `fts`, while the `blob_read` workload is pretty much on par. Parquet is not well optimized for vector searches or blob reads, but does well on a simple text search powered by regex.
##### Warm Results {#blog:2026-05-21-test-driving-lance::warm-results}
The warm results are from running all workloads in a single DuckDB session after a silent warmup pass, so caches, memory-mapped pages, and loaded indexes are already primed.
| Workload | Parquet | DuckDB indexed | Lance native |
| :--------------- | ------: | -------------: | -----------: |
| `fts` | 12 ms | 10 ms | 7 ms |
| `vector_exact` | 703 ms | 30 ms | 50 ms |
| `vector_indexed` | 755 ms | 2 ms | 5 ms |
| `hybrid` | 471 ms | 11 ms | 8 ms |
| `blob_read` | 1484 ms | 266 ms | 276 ms |
When caches and indexes are already warm, both DuckDB and Lance are significantly faster than using Parquet on retrieval workloads.
#### Conclusion {#blog:2026-05-21-test-driving-lance::conclusion}
Lance is a relatively new addition to the world of open lakehouse formats. It is designed for datasets that change over time, contain more than scalar values, and need to support both search and retrieval alongside traditional scan workloads. From DuckDB, the extension makes these capabilities available through SQL, while preserving the familiar embedded workflow. The benchmark results reflect, particularly in cold runs, how Lance is a good alternative to DuckDB’s own format for vector and hybrid search.
> The Lance support in DuckDB was made possible through a collaboration between [DuckLabs](https://ducklabs.com/) and [LanceDB](https://www.lancedb.com/).
## New DuckDB-Iceberg Features in v1.5.3 {#blog:2026-05-29-new-iceberg-features}
**Publication date:** 2026-05-29
**Authors:** Tom Ebergen, Thijs Bruineman
**TL;DR:** DuckDB-Iceberg now has a number of new features supporting Iceberg Tables and Iceberg REST Catalogs: `MERGE INTO`, `ALTER TABLE`, partition transforms, V3 support, and others!
Despite the work required to develop the features needed for DuckLake v1.0 and Quack, the DuckLabs team is still hard at work on the [DuckDB-Iceberg extension](https://duckdb.org/docs/1.5/core_extensions/iceberg/overview).
In this blog post, we will demonstrate some of the features that are available in [DuckDB v1.5.3](#blog:2026-05-20-announcing-duckdb-153). Many of these features were earmarked for a future release in our last Iceberg-themed blog post [“Writes in DuckDB-Iceberg”](#blog:2025-11-28-iceberg-writes-in-duckdb) – you can think of this post as “Part 2” to that blog post.
#### Getting Started {#blog:2026-05-29-new-iceberg-features::getting-started}
To experiment with the new DuckDB-Iceberg features, you will need to connect to your favorite Iceberg REST Catalog. There are many ways to do so: please have a look at the [Connecting to REST Catalogs page](https://duckdb.org/docs/1.5/core_extensions/iceberg/catalogs), which has instructions for catalogs such as [Apache Polaris](https://polaris.apache.org/) and [Lakekeeper](https://lakekeeper.io/). If you would like to connect to [Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html), please consult the [Connecting to S3 Tables page](https://duckdb.org/docs/1.5/core_extensions/iceberg/catalogs#amazon-s3-tables). In any case, your `ATTACH` command will look something like this:
```sql
ATTACH '⟨warehouse_name⟩' AS my_datalake (
TYPE iceberg,
⟨other options⟩
);
```
#### `MERGE INTO` Support {#blog:2026-05-29-new-iceberg-features::merge-into-support}
DuckDB's [`MERGE INTO`](https://duckdb.org/docs/1.5/sql/statements/merge_into) statement is the recommended way to express upserts when the target table does not have a primary key – which is the case for all [lakehouse formats](https://duckdb.org/docs/1.5/lakehouse_formats).
With v1.5.3, `MERGE INTO` is now fully supported against Iceberg tables. You can apply a change set to an Iceberg table in a single statement, deciding per row whether to insert, update or delete.
Let's take this table for example:
```sql
CREATE TABLE my_datalake.default.people (
id INTEGER,
name VARCHAR,
salary FLOAT
);
INSERT INTO my_datalake.default.people
VALUES (1, 'John', 92_000.0), (2, 'Anna', 100_000.0);
```
```text
┌───────┬─────────┬──────────┐
│ id │ name │ salary │
│ int32 │ varchar │ float │
├───────┼─────────┼──────────┤
│ 1 │ John │ 92000.0 │
│ 2 │ Anna │ 100000.0 │
└───────┴─────────┴──────────┘
```
Let's run an update on this table with two records, one increasing person 1's salary and another adding a new person with id 3.
```sql
MERGE INTO my_datalake.default.people AS target
USING (
FROM (VALUES
(1, 'John', 105_000.0),
(3, 'Sarah', 95_000.0)
) t(id, name, salary)
) AS upserts
ON (upserts.id = target.id)
WHEN MATCHED THEN UPDATE
WHEN NOT MATCHED THEN INSERT;
```
When querying the result, we get the following:
```sql
SELECT *
FROM my_datalake.default.people
ORDER BY id;
```
```text
┌───────┬─────────┬──────────┐
│ id │ name │ salary │
│ int32 │ varchar │ float │
├───────┼─────────┼──────────┤
│ 1 │ John │ 105000.0 │
│ 2 │ Anna │ 100000.0 │
│ 3 │ Sarah │ 95000.0 │
└───────┴─────────┴──────────┘
```
You can also combine matched and unmatched branches with `WHEN MATCHED THEN DELETE` to express a delete set in the same statement. As with `UPDATE` and `DELETE`, `MERGE INTO` uses merge-on-read semantics and writes positional deletes to the Iceberg table.
#### `ALTER TABLE` Support {#blog:2026-05-29-new-iceberg-features::alter-table-support}
In DuckDB v1.4's Iceberg extension, the lack of schema evolution of Iceberg tables was a [documented limitation](https://duckdb.org/docs/1.4/core_extensions/iceberg/iceberg_rest_catalogs#unsupported-operations).
In v1.5.3, the `ALTER TABLE` statement is now supported against Iceberg tables, covering the most common schema-evolution operations.
```sql
-- Create the table
CREATE TABLE my_datalake.default.simple_table AS
FROM (VALUES
(1, 'Andy'),
(2, 'Bob'),
(3, 'Claire'),
(4, 'Mr. Duck')) t(col1, col2);
-- Rename the table
ALTER TABLE my_datalake.default.simple_table
RENAME TO renamed_table;
-- Add a column
ALTER TABLE my_datalake.default.renamed_table
ADD COLUMN col3 DOUBLE;
-- Rename a column
ALTER TABLE my_datalake.default.renamed_table
RENAME COLUMN col2 TO name;
-- Drop a column
ALTER TABLE my_datalake.default.renamed_table
DROP COLUMN col3;
-- Set the format-version
ALTER TABLE my_datalake.default.renamed_table
SET ('format-version' = 3);
```
If we query the table after the schema changes, we get the following:
```sql
SELECT *
FROM my_datalake.default.renamed_table
ORDER BY col1;
```
```text
┌───────┬──────────┐
│ col1 │ name │
│ int32 │ varchar │
├───────┼──────────┤
│ 1 │ Andy │
│ 2 │ Bob │
│ 3 │ Claire │
│ 4 │ Mr. Duck │
└───────┴──────────┘
```
In the background, each `ALTER TABLE` statement updates the `current-schema-id` of the Iceberg table. The changes are visible to other Iceberg-aware engines the next time they query the `LoadTableInformation` endpoint. Iceberg schema evolution is metadata-only, so no data files are rewritten.
#### `truncate` and `bucket` Support {#blog:2026-05-29-new-iceberg-features::truncate-and-bucket-support}
The Iceberg specification defines several [partition transforms](https://iceberg.apache.org/spec/#partition-transforms) that determine how data files are laid out on disk. In v1.5.3, DuckDB-Iceberg supports creating, inserting into, and updating tables that use the `bucket` and `truncate` partition transforms.
The `bucket(N, col)` transform hashes the column's value into `N` buckets, which is useful when you want stable partitioning on a high-cardinality column. `truncate(W, col)` groups rows by the first `W` characters (or by the column's value rounded down to a multiple of `W` for numeric columns), which is useful for prefix-based partitioning.
```sql
CREATE TABLE my_datalake.default.events (
event_id BIGINT,
user_id BIGINT,
country VARCHAR,
payload VARCHAR
)
PARTITIONED BY (bucket(16, user_id), truncate(2, country));
INSERT INTO my_datalake.default.events
VALUES
(1, 1001, 'United States', 'click'),
(2, 1002, 'United Kingdom', 'view'),
(3, 1003, 'Germany', 'click'),
(4, 1004, 'Netherlands', 'view');
```
You can inspect the resulting data files to verify the partitioning:
```sql
SELECT file_path, record_count
FROM iceberg_metadata(my_datalake.default.events)
WHERE content = 'EXISTING';
```
Updates and deletes against bucket- and truncate-partitioned tables are also supported, using positional deletes under merge-on-read semantics.
#### Iceberg Schema Properties {#blog:2026-05-29-new-iceberg-features::iceberg-schema-properties}
Iceberg catalogs allow arbitrary key-value [properties](https://iceberg.apache.org/spec/#namespaces) to be attached at the schema (namespace) level. These properties are typically used to record ownership, descriptions, default storage locations, or any other metadata that applies to every table in a schema.
* `iceberg_schema_properties`
* `set_iceberg_schema_properties`
* `remove_iceberg_schema_properties`
You can use them as follows:
```sql
-- to set schema properties
CALL set_iceberg_schema_properties(my_datalake.default, {
'owner': 'analytics-team',
'description': 'Default analytics schema'
});
-- to read schema properties
SELECT * FROM iceberg_schema_properties(my_datalake.default);
```
```text
┌─────────────┬──────────────────────────┐
│ key │ value │
│ varchar │ varchar │
├─────────────┼──────────────────────────┤
│ owner │ analytics-team │
│ description │ Default analytics schema │
└─────────────┴──────────────────────────┘
```
```sql
-- to remove schema properties
CALL remove_iceberg_schema_properties(
my_datalake.default,
['description']
);
```
Schema properties are written through the Iceberg REST Catalog, so any other Iceberg-aware engine attached to the same catalog will see the updates immediately. The returned value is the number of remaining schema properties.
#### V3 Support {#blog:2026-05-29-new-iceberg-features::v3-support}
The [Iceberg v3 specification](https://iceberg.apache.org/spec/#version-3) introduces several new features that DuckDB-Iceberg now supports for both reads and writes:
* `VARIANT` and `TIMESTAMP_NS` data types
* Schema-level [default values](https://iceberg.apache.org/spec/#default-values) for columns
* Binary deletion vectors
* Row lineage tracking
The biggest change in practice is binary deletion vectors. In v2 tables, DuckDB-Iceberg writes positional deletes as Parquet files; in v3 tables, the same information is encoded as a much more compact binary deletion vector ([Puffin file](https://iceberg.apache.org/puffin-spec/)). DuckDB picks the right format automatically based on the table's `format-version`.
You can create a v3 table by setting the `format-version` table property at creation time:
```sql
CREATE TABLE my_datalake.default.v3_table
WITH ('format-version' = 3) AS
FROM (VALUES
(1, {'kind': 'click', 'x': 10}::VARIANT, TIMESTAMP_NS '2026-05-20 12:00:00.123456789'),
(2, {'kind': 'view'}::VARIANT, TIMESTAMP_NS '2026-05-20 12:00:00.987654321')
) t(id, payload, event_time);
-- Deletes against a v3 table are written as binary deletion vectors
DELETE FROM my_datalake.default.v3_table
WHERE id = 1;
SELECT * FROM my_datalake.default.v3_table;
```
```text
┌───────┬──────────────────┬───────────────────────────────┐
│ id │ payload │ event_time │
│ int32 │ variant │ timestamp_ns │
├───────┼──────────────────┼───────────────────────────────┤
│ 2 │ {"kind": "view"} │ 2026-05-20 12:00:00.987654321 │
└───────┴──────────────────┴───────────────────────────────┘
```
Looking at the metadata for the table confirms that the delete was written as a deletion vector rather than as a positional-delete Parquet file:
```sql
SELECT manifest_content, content, file_format
FROM iceberg_metadata(my_datalake.default.v3_table);
```
```text
┌──────────────────┬──────────────────┬─────────────┐
│ manifest_content │ content │ file_format │
│ varchar │ varchar │ varchar │
├──────────────────┼──────────────────┼─────────────┤
│ DATA │ EXISTING │ parquet │
│ DELETE │ POSITION_DELETES │ puffin │
└──────────────────┴──────────────────┴─────────────┘
```
> The Geography type and Unknown type are not yet supported in DuckDB-Iceberg; we are planning to add those in DuckDB v2.0.0.
#### Conclusion and Future Work {#blog:2026-05-29-new-iceberg-features::conclusion-and-future-work}
With these features, DuckDB-Iceberg has closed many of the gaps called out in the [previous blog post](#blog:2025-11-28-iceberg-writes-in-duckdb): partitioned writes, schema evolution, `MERGE INTO`, and many Iceberg v3 features are now available. There is still more to come, and as always, if you would like to see a specific feature prioritized, please reach out to us in the [DuckDB-Iceberg GitHub repository](https://github.com/duckdb/duckdb-iceberg) or [get in touch](https://ducklabs.com/contact/) with our engineers.
## Announcing DuckDB 1.4.5 LTS (Andium) {#blog:2026-06-17-announcing-duckdb-145}
**Publication date:** 2026-06-17
**Author:** The DuckDB team
**TL;DR:** Today we are releasing DuckDB 1.4.5 LTS with bugfixes and performance improvements.
> We are simultaneously releasing two DuckDB versions:
> v1.4.5 LTS (Andium) and
> v1.5.4 (Variegata).
> **This blog post is about the LTS version, v1.4.5 (Andium).**
> Unless you are using specifically using the LTS version, you are likely looking for the latest stable version,
> [v1.5.4 (Variegata)](#blog:2026-06-17-announcing-duckdb-154).
In this blog post, we highlight a few important fixes in DuckDB v1.4.5, the fifth patch release in [DuckDB's 1.4 LTS line](#blog:2025-09-16-announcing-duckdb-140).
The release ships bugfixes, performance improvements and security patches. You can find the full [release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.4.5).
To install the new version, please visit the [installation page](https://duckdb.org/install).
#### Fixes {#blog:2026-06-17-announcing-duckdb-145::fixes}
##### Correctness {#blog:2026-06-17-announcing-duckdb-145::correctness}
* [`#21482`](https://github.com/duckdb/duckdb/pull/21482) – Correctly detect overflows when decoding integers from storage
* [`#21177`](https://github.com/duckdb/duckdb/pull/21177) – Backport support for the `* NOT SIMILAR TO 'pattern'` from [`#19232`](https://github.com/duckdb/duckdb/pull/19232)
##### Crashes and Internal Errors {#blog:2026-06-17-announcing-duckdb-145::crashes-and-internal-errors}
* [`#20804`](https://github.com/duckdb/duckdb/pull/20804) – Backport various race condition fixes from [`#20803`](https://github.com/duckdb/duckdb/pull/20803)
##### Generic Bugfixes {#blog:2026-06-17-announcing-duckdb-145::generic-bugfixes}
* [`#20760`](https://github.com/duckdb/duckdb/pull/20760) – Fix escape handling in `AddEscapes` function
* [`#21316`](https://github.com/duckdb/duckdb/pull/21316) – Fix unbounded row group growth for indexed tables on repeated load and insert cycles
* [`#21577`](https://github.com/duckdb/duckdb/pull/21577) – Fix for CSV reader buffer-boundary value read
* [`#20686`](https://github.com/duckdb/duckdb/pull/20686) – Secret Manager: Ensure secrets are created exactly once
* [`#23197`](https://github.com/duckdb/duckdb/pull/23197) – Backport out-of-bounds security fixes from [`#23100`](https://github.com/duckdb/duckdb/pull/23100)
##### Performance {#blog:2026-06-17-announcing-duckdb-145::performance}
* [`#21090`](https://github.com/duckdb/duckdb/pull/21090) – AsOf Simple Joins improvements
* [`#21178`](https://github.com/duckdb/duckdb/pull/21178) – Backport zstd improvements from [`#20943`](https://github.com/duckdb/duckdb/pull/20943)
##### Miscellaneous {#blog:2026-06-17-announcing-duckdb-145::miscellaneous}
* [`#23130`](https://github.com/duckdb/duckdb/pull/23130) – Backport `GetLocalFileSystem` improvements from [`#21983`](https://github.com/duckdb/duckdb/pull/21983)
* [`#20727`](https://github.com/duckdb/duckdb/pull/20727) – Use `FileSystem` APIs in `QueryProfiler::Write`
#### Conclusion {#blog:2026-06-17-announcing-duckdb-145::conclusion}
This post was a short summary of the changes in v1.4.5. As usual, you can find the [full release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.4.5).
We would like to thank our contributors for providing detailed issue reports and patches.
Stay tuned for [future DuckDB releasese](https://duckdb.org/release_calendar), including v2.0.0 in the fall!
PS: Next week, we'll host [DuckCon #7 in Amsterdam](https://duckdb.org/events/2026/06/24/duckcon7/).
Join us [in-person](https://luma.com/dj4svsmm) or on the online stream.
## Announcing DuckDB 1.5.4 (Variegata) {#blog:2026-06-17-announcing-duckdb-154}
**Publication date:** 2026-06-17
**Author:** The DuckDB team
**TL;DR:** Today we are releasing DuckDB 1.5.4 (Variegata) with bugfixes and performance improvements.
> We are simultaneously releasing two DuckDB versions:
> v1.4.5 LTS (Andium) and
> v1.5.4 (Variegata).
> **This blog post is about the latest non-LTS version, v1.5.4 (Variegata).**
> If you are looking for the LTS version, check out the [v1.4.5 (Andium) announcement](#blog:2026-06-17-announcing-duckdb-145).
In this blog post, we highlight a few important fixes in DuckDB v1.5.4, the fourth patch release in [DuckDB's 1.5 (Variegata) line](#blog:2026-03-09-announcing-duckdb-150).
The release ships bugfixes, performance improvements and security patches. You can find the full [release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.5.4).
To install the new version, please visit the [installation page](https://duckdb.org/install).
#### Fixes {#blog:2026-06-17-announcing-duckdb-154::fixes}
This version ships a number of performance improvements and bugfixes.
##### Correctness {#blog:2026-06-17-announcing-duckdb-154::correctness}
* [#23031](https://github.com/duckdb/duckdb/pull/23031) – Fix VARIANT cast reading wrong rows under a filter
* [#23014](https://github.com/duckdb/duckdb/pull/23014) – MERGE INTO: only consider target table when binding `WHEN NOT MATCHED` and source table when binding `WHEN NOT MATCHED BY TARGET`
* [#22825](https://github.com/duckdb/duckdb/pull/22825) – Fix case-insensitive column match in INSERT ... SELECT ON CONFLICT
* [#22911](https://github.com/duckdb/duckdb/pull/22911) – Use non-deleted row count in `RowGroupReorderer`
* [#23194](https://github.com/duckdb/duckdb/pull/23194) – Fix variant shredding analysis logic discrepancy with shredded writing
* [#23234](https://github.com/duckdb/duckdb/pull/23234) – Fix problem with re-use of cached transform data for differently shredded files
* [#22844](https://github.com/duckdb/duckdb/pull/22844) – Window Self-Join Limits: don't apply the self-join optimisation more than once
##### Crashes and Internal Errors {#blog:2026-06-17-announcing-duckdb-154::crashes-and-internal-errors}
* [#21854](https://github.com/duckdb/duckdb/pull/21854) – Fix double free and memory leak in Arrow GeoArrow CRS serialization
* [#22836](https://github.com/duckdb/duckdb/pull/22836) – Fix progress bar output and crash when piping SQL
* [#23174](https://github.com/duckdb/duckdb/pull/23174) – Fix crash when storage path is not set
* [#23232](https://github.com/duckdb/duckdb/pull/23232) – Fix gzip compression write overflow
* [#23156](https://github.com/duckdb/duckdb/pull/23156) – Avoid trying to bind an expression that doesn't exist in `UNPIVOT`
* [#23189](https://github.com/duckdb/duckdb/pull/23189) – Guard against null row group reorder stats
##### Generic Bugfixes {#blog:2026-06-17-announcing-duckdb-154::generic-bugfixes}
* [#22855](https://github.com/duckdb/duckdb/pull/22855) – Fix json_keys with wildcard paths
* [#23144](https://github.com/duckdb/duckdb/pull/23144) – Fix json argument order affecting result
* [#23116](https://github.com/duckdb/duckdb/pull/23116) – Reject NULL json key
* [#23137](https://github.com/duckdb/duckdb/pull/23137) – Fix `ignore_errors` silently accepting invalid JSON
* [#22882](https://github.com/duckdb/duckdb/pull/22882) – Fix geometry stats checkpointing when no changes are detected
* [#22815](https://github.com/duckdb/duckdb/pull/22815) – Render MAP values as valid SQL in `Value::ToSQLString()`
* [#23254](https://github.com/duckdb/duckdb/pull/23254) – Fix NULL propagation for date parts of infinite dates
* [#23190](https://github.com/duckdb/duckdb/pull/23190) – Fix selection vector use in Arrow extension callbacks
##### Performance {#blog:2026-06-17-announcing-duckdb-154::performance}
* [#23253](https://github.com/duckdb/duckdb/pull/23253) – Trim the system heap in the allocator flush path on jemalloc builds
* [#23140](https://github.com/duckdb/duckdb/pull/23140) – Fix native geometry Parquet stats pruning and add `OPERATOR_ROW_GROUPS_SCANNED` to Parquet reader
##### Miscellaneous {#blog:2026-06-17-announcing-duckdb-154::miscellaneous}
* [#23246](https://github.com/duckdb/duckdb/pull/23246) – Add explicit `-dark-mode` and `-light-mode` options to the CLI, and improve terminal background color detection
* [#23100](https://github.com/duckdb/duckdb/pull/23100) – Add hardening to many DuckDB/Parquet decompression and deserializing paths
* [#22690](https://github.com/duckdb/duckdb/pull/22690) – Add `vacuum_rebuild_indexes` as an (experimental) ATTACH option
#### Conclusion {#blog:2026-06-17-announcing-duckdb-154::conclusion}
This post was a short summary of the changes in v1.5.4. As usual, you can find the [full release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.5.4).
We would like to thank our contributors for providing detailed issue reports and patches.
Stay tuned for [future DuckDB releases](https://duckdb.org/release_calendar), including v2.0.0 in the fall!
PS: Next week, we'll host [DuckCon #7 in Amsterdam](https://duckdb.org/events/2026/06/24/duckcon7/).
Join us [in-person](https://luma.com/dj4svsmm) or on the online stream.
## Announcing DuckDB 1.5.5 {#blog:2026-07-22-announcing-duckdb-155}
**Publication date:** 2026-07-22
**Author:** The DuckDB team
**TL;DR:** Today we are releasing DuckDB 1.5.5 with bugfixes and performance improvements.
In this blog post, we highlight a few important fixes in DuckDB v1.5.5, the fifth patch release in [DuckDB's 1.5 (Variegata) line](#blog:2026-03-09-announcing-duckdb-150).
The release ships bugfixes, performance improvements and security patches. You can find the full [release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.5.5).
To install the new version, please visit the [installation page](https://duckdb.org/install).
#### Fixes {#blog:2026-07-22-announcing-duckdb-155::fixes}
Here are the most important fixes from the DuckDB v1.5.5 release, organized by category:
##### Correctness {#blog:2026-07-22-announcing-duckdb-155::correctness}
* [`#23693`](https://github.com/duckdb/duckdb/pull/23693) – Fix swapped min/max for multi-row-group 128-bit `DECIMAL` in `RETURN_STATS`
##### Crashes and Internal Errors {#blog:2026-07-22-announcing-duckdb-155::crashes-and-internal-errors}
* [`#23517`](https://github.com/duckdb/duckdb/pull/23517) – Fix min/max aggregate stats when row groups are filtered, which caused a crash
* [`#23351`](https://github.com/duckdb/duckdb/pull/23351) – Fix deadlock in `TemporaryMemoryManager`
* [`#23566`](https://github.com/duckdb/duckdb/pull/23566) – Fix C API scalar bind subquery crash
* [`#23757`](https://github.com/duckdb/duckdb/pull/23757) – Fix segfault in external hash aggregate when radix bits grow after going external
* [`#23861`](https://github.com/duckdb/duckdb/pull/23861) – Fix concurrent `ALTER` and `INSERT` crash
* [`#23593`](https://github.com/duckdb/duckdb/pull/23593) – Fixed unsafe iteration when parent is `NULL` in string cast
* [`#23341`](https://github.com/duckdb/duckdb/pull/23341) – Add additional guards to `DICT_FSST` to prevent exception during compression with small block sizes
##### Generic Bugfixes {#blog:2026-07-22-announcing-duckdb-155::generic-bugfixes}
* [`#23458`](https://github.com/duckdb/duckdb/pull/23458) – Fix false RLE corruption error
* [`#23534`](https://github.com/duckdb/duckdb/pull/23534) – Fix Arrow type extension bugs
* [`#23507`](https://github.com/duckdb/duckdb/pull/23507) – Fix `ALTER TABLE ADD COLUMN IF NOT EXISTS ... DEFAULT` regression
* [`#23714`](https://github.com/duckdb/duckdb/pull/23714) – Fix `DROP COLUMN` corrupting per-column metadata block bookkeeping
* [`#23808`](https://github.com/duckdb/duckdb/pull/23808) – Fix `ALTER` dependency preservation in DependencyManager
* [`#23790`](https://github.com/duckdb/duckdb/pull/23790) – `TryLookupEntry` now uses default schema as fallback
* [`#23354`](https://github.com/duckdb/duckdb/pull/23354) – Prevent `NULL MAP` keys in `MultiFileReader` due to missing default values
* [`#23803`](https://github.com/duckdb/duckdb/pull/23803) – Fix malformed JSON when rendering via duckbox
* [`#23879`](https://github.com/duckdb/duckdb/pull/23879) – Fix reset of `empty_range` in the `TIMESTAMP` `range()` table function
* [`#23479`](https://github.com/duckdb/duckdb/pull/23479) – Fix eviction node memory leak when using external files
##### Performance {#blog:2026-07-22-announcing-duckdb-155::performance}
* [`#23483`](https://github.com/duckdb/duckdb/pull/23483) – Enable `ALP` and `ALP_RD` for storage version v1.5.0 and up with smaller block sizes
##### Miscellaneous {#blog:2026-07-22-announcing-duckdb-155::miscellaneous}
* [`#21293`](https://github.com/duckdb/duckdb/pull/21293) – Add support for `duckdb://` URI scheme in ADBC
* [`#23230`](https://github.com/duckdb/duckdb/pull/23230) – Support the ADBC Statistics API
* [`#23316`](https://github.com/duckdb/duckdb/pull/23316) – Add request body length to HTTP logs
* [`#23327`](https://github.com/duckdb/duckdb/pull/23327) – Show transport errors in HTTP log
* [`#23752`](https://github.com/duckdb/duckdb/pull/23752) – Include extension header in libduckdb archives
#### Conclusion {#blog:2026-07-22-announcing-duckdb-155::conclusion}
This post was a short summary of the changes in v1.5.5. As usual, you can find the [full release notes on GitHub](https://github.com/duckdb/duckdb/releases/tag/v1.5.5).
We would like to thank our contributors for providing detailed issue reports and patches.
Stay tuned for [future DuckDB releases](https://duckdb.org/release_calendar), including v2.0.0 in the fall!
## Asynchronous I/O in DuckDB: Work, Thread, Work {#blog:2026-07-31-asynchronous-io}
**Publication date:** 2026-07-31
**Author:** Pedro Holanda
**TL;DR:** Starting with v2.0, scheduled for fall 2026, DuckDB will support asynchronous reads of Parquet and CSV files. This can significantly speed up queries when synchronous I/O does not saturate the available bandwidth, as is typical in EC2/S3 compute-storage setups.
It doesn't matter how fast query operators are in a database system if we can't pull in the data quickly. For most of DuckDB's history, however, this problem was largely avoided by pruning data early. By pushing down filters and projections, we could ensure that we only read what we actually needed.
This worked particularly well because DuckDB primarily ran locally, with its main use case being as a quick-draw database engine for querying data directly from your machine's SSD. We could split the data into several partitions, such as row groups for Parquet files or fixed-size buffers for CSV files, and load them with low latency and high bandwidth. As a result, the main bottlenecks were elsewhere: subqueries, joins, aggregations, and so on. The actual data access path received less attention because synchronous access was perfectly suitable for this use case.
As usual, things changed. We realized that DuckDB's architecture was a great fit for querying remotely stored large-scale datasets, such as data lakes (e.g., [DuckLake](https://ducklake.select/)). Since [May this year](#blog:2026-05-12-quack-remote-protocol), we can even run DuckDB as a server using the [Quack protocol](https://duckdb.org/quack). The original expectation of data files sitting on a local SSD therefore no longer always holds.
The practical implication of these changes is that many current DuckDB setups need to transfer files from remote storage to the machine that will actually process them. For data lakes, for example, a typical setup is to store the data in blob storage, such as S3, and process it on an EC2 machine in the same region. In this setup, latency and bandwidth play a much more significant role. If we cannot issue enough concurrent requests to use the available network bandwidth, performance can suffer drastically, with threads spending a large amount of their time waiting for remote reads instead of processing data.
As an example, let's consider a simple query over a remote Parquet file. For simplicity, let's assume we only have a single thread executing.
```sql
FROM read_parquet('s3://bucket/file.parquet');
```
A Parquet scan is partitioned into row-group-based jobs, with each job containing one or more fetch tasks that issue byte-range requests. With synchronous I/O, the worker thread will be blocked, waiting for the data to arrive at the machine before performing actual work, such as decoding, aggregating, and so on. You can see a visual depiction in the figure below, where the thread is blocked from doing any work while it waits for the read to finish.
{: .lightmode-img }
*Synchronous read*{: .caption }
To address this, we have been implementing asynchronous I/O pipelines in DuckDB. They are currently implemented for Parquet and for uncompressed, seekable UTF-8 CSV files, with support for other formats, such as DuckDB's native format and JSON, still to come. In the remainder of this blog post, we will give a simple explanation of how asynchronous I/O is implemented in DuckDB and provide benchmarks for both Parquet and CSV files.
> If you would like to try asynchronous I/O now, you can do so by using [DuckDB's v2.0.0-dev preview builds](https://duckdb.org/install/preview).
> Asynchronous I/O will be used by default from the next major DuckDB version, v2.0, released [in the fall](https://duckdb.org/release_calendar).
#### Asynchronous I/O {#blog:2026-07-31-asynchronous-io::asynchronous-io}
The conceptual idea of asynchronous I/O is rather simple: we should be able to start an I/O operation without blocking the worker thread that requested it. Applied to our Parquet example, the same picture would look like the following:
{: .lightmode-img }
*Asynchronous read*{: .caption }
In this example, we have two `ASYNC` threads and one regular worker thread. The `ASYNC` threads keep _fetch tasks_ in flight while the worker thread decodes data. During the initial warm-up, the scan task parks, leaving the worker thread free to run other pipeline tasks. Once the first job is ready, fetching and decoding can overlap.
In DuckDB, we implemented something similar. We have [two separate thread pools](https://github.com/duckdb/duckdb/blob/34f2a32f08e9c454205f1ce45a4e9fa38adf4f53/src/include/duckdb/common/enums/task_scheduler_type.hpp#L15-L17):
* `REGULAR` – This pool contains our worker threads (by default: one for each available CPU thread). These are the ones that do real work, like decoding, joins, and aggregations. They prioritize regular work but can also perform I/O tasks when idle.
* `ASYNC` – A pool of threads intended for asynchronous tasks, primarily blocking I/O.
The main reason we have these two different pools is that, for remote I/O, these threads can spend almost all their time blocked, waiting for an HTTP response, for example, and hence have very little CPU utilization. Because of that, we have many more `ASYNC` workers than system threads, with the default setting being `4 * system threads` and the total being capped at 256.
It's of utmost importance to keep as many of our `ASYNC` threads busy as possible. To ensure that, we implement a read-ahead strategy instead of issuing reads on demand. This means scheduling fetch tasks ahead of what our regular worker threads currently need.
One thing we need to be attentive to is that read-ahead buys throughput by holding memory. If decoding is slow and the network is fast, prefetched data can accumulate and lead to out-of-memory issues. To mitigate this, we also implemented asynchronous memory governance. Both read-ahead and memory governance will be explained in more detail in the following sections.
##### Read-Ahead Queue {#blog:2026-07-31-asynchronous-io::read-ahead-queue}
The idea of read-ahead is also straightforward. Instead of starting a read at the exact moment a regular worker needs the data, we schedule fetch tasks for work that lies further ahead. While a regular worker decodes the current job, the ASYNC threads are already pulling in data for the next jobs. The goal is to keep enough fetch tasks in flight to hide the latency of remote storage.
The jobs are units of work that can be scheduled and processed independently, and they can be different depending on the underlying file format. For a Parquet file, a job is one row group of one file. For a CSV file, a job is a scan boundary that generally covers a fixed byte range within the file.
A Parquet job might be broken down into multiple fetch tasks depending on the query projections, filter pushdowns, physical column locations, and which nearby byte ranges can be combined. The two fetch tasks in the figure below are illustrative, as their exact grouping and sizes depend on the file and query.
For CSV files, we don't have the same granularity of information as we do for Parquet files. A job's fetch tasks load its starting buffer if it is not already in memory and, when the scan boundary reaches the end of that buffer, the following buffer as well (e.g., to handle lines that are split across two buffers).
{: .lightmode-img }
*Jobs*{: .caption }
Filling the queue requires no dedicated producer thread. Any regular worker that comes looking for scan work first tops up the queue as far as it is allowed to. The limit is either given by a user-specified number of slots or by a memory budget. If there is space, a job and its fetch tasks are created. The fetch tasks are scheduled immediately on the `ASYNC` pool, while the job is admitted to the read-ahead queue in batch order.
`ASYNC` threads execute individual fetch tasks independently of the job queue's claim order. Fetch tasks from the same job can run concurrently, although no particular assignment to `ASYNC` threads is guaranteed. All fetch tasks of a job share a countdown, and the fetch task that brings it to zero completes the job's I/O.
A worker thread claims the oldest job in the queue and checks that countdown. If I/O is done, the worker starts decoding the job. If not, it parks the scan task and is free to run other pipeline tasks. The last fetch task then unblocks the scan task, which may resume on any regular worker.
Claiming the job also immediately frees a queue slot, allowing any regular worker looking for scan work to produce a replacement job at the back of the queue. The figure below depicts this cycle:
{: .lightmode-img }
*Read-ahead cycle*{: .caption }
##### Memory Management {#blog:2026-07-31-asynchronous-io::memory-management}
Keeping more fetch tasks in flight consumes more memory. To determine a budget and avoid out-of-memory issues, we introduced the `read_ahead_depth` configuration option. It can have three types of values:
* `-1` (default): unlimited depth, bounded by memory.
* `N > 0`: at most `N` jobs ahead, with no memory budget.
* `0`: read-ahead is off, each scan task schedules I/O only for its own job.
To configure it, use the `SET` clause, e.g.:
```sql
SET read_ahead_depth = 5;
```
In the default mode, the budget is negotiated with the temporary memory manager, which is the same manager that splits memory between concurrent joins, sorts, and window operators. When there is a lot of memory pressure, for example, because an operator is using a large amount of memory, queue reservations might instantly be over budget. In practice, this means that the queue will only allow one job at a time, and the scan will behave close to a synchronous scan.
When the memory-heavy operator finishes, the memory manager has more budget to give, and the queue fills back up.
#### Benchmarks {#blog:2026-07-31-asynchronous-io::benchmarks}
Asynchronous I/O should have the largest effect when the latency of synchronous requests prevents us from using the available remote bandwidth. To measure this effect, we ran [TPC-H Query 6 at SF100](https://duckdb.org/docs/current/core_extensions/tpch), with the data sitting on S3, and compared the results against [DuckDB v1.5.5](#blog:2026-07-22-announcing-duckdb-155), our latest stable release. The SF100 dataset was written as a single file per table for both the Parquet and CSV benchmarks, with the `lineitem` table containing 600,037,902 rows.
For compute, we used an EC2 [r7i.16xlarge](http://instances.vantage.sh/aws/ec2/r7i.16xlarge) machine (64 vCPUs and 512 GB of RAM), with both the machine and the S3 bucket with the data located in the same region. We executed the query five times and report the mean execution time. The files were never cached (i.e., `SET enable_external_file_cache = false;`{:.language-sql .highlight}), meaning that every execution read the data straight from S3.
##### Parquet {#blog:2026-07-31-asynchronous-io::parquet}
The Parquet file is approximately 22 GB and has around 4,880 row groups, with each row group containing approximately 122,880 rows. With asynchronous I/O, the mean runtime drops from 8.230 seconds to 2.844 seconds, making the query almost 3× faster.
| Version | Q6 runtime |
| ----------------------------- | ---------: |
| v1.5.5 (synchronous) | 8.230 s |
| v2.0.0-dev (asynchronous I/O) | 2.844 s |
Below we also show the network throughput over the course of the query:
{: .lightmode-img }
*Network throughput*{: .caption }
In it, we run DuckDB v1.5.5 and two variations of DuckDB v2.0.0-dev. One with the read-ahead depth determined by the memory governor, and one tuned for this machine, where we cap the read-ahead at 64 in-flight jobs and adjust the I/O settings (` SET async_threads = 48; SET http_retries = 8; SET http_retry_wait_ms = 50; SET http_retry_backoff = 2`{:.language-sql .highlight}). We can see that v2.0.0-dev uses the available bandwidth much more effectively, approaching the network limit and reaching it at several points. The tuned version goes further. With fewer, hotter connections and cheap retries, the throughput variance drops to a minimum and the 25 Gbit/s network stays almost fully saturated. Its query time was 2.227 seconds, reducing the runtime of the untuned v2.0.0-dev run by 21.7% and making it about 3.7× faster than DuckDB v1.5.5. In comparison, v1.5.5 stays around 5 Gbit/s because its synchronous reads do not keep enough requests in flight to saturate the network.
One other detail worth noting is that, in all experiments, a few hundred milliseconds pass before the first bump in network traffic, followed by another few hundred milliseconds before the main data transfer begins. The first gap is the time needed to open a DuckDB connection, perform the first TLS handshake, and open the file. The bump corresponds to downloading the file footer, while the second gap comes from processing the information in the footer before executing the query. We believe this is an area we can investigate and optimize further before the v2.0 release.
> We sampled the NIC's received-byte counter every 50 ms and calculated the throughput from the change in bytes between samples. We independently confirmed that the machine can access the network at 25 Gbit/s with both a DuckDB full-file read and the [`s5cmd` tool](https://github.com/peak/s5cmd).
###### Local Disk {#blog:2026-07-31-asynchronous-io::local-disk}
Remote storage is the main target for asynchronous I/O, but cold local reads give us a useful contrast. To measure them, we ran TPC-H Query 6 over the SF100 Parquet file, this time with the file sitting on the local disk of a MacBook Pro (Apple M4 Max, 14 cores, and 36 GB of RAM). Since the benefit of asynchronous I/O on local disks comes from cold reads, we cleared the OS caches (with the macOS `purge` command) between runs, making sure every execution actually read the file from disk.
| Version | Q6 runtime |
| ----------------------------- | ---------: |
| v1.5.5 (synchronous) | 1.321 s |
| v2.0.0-dev (asynchronous I/O) | 0.883 s |
We can see that, for cold runs, asynchronous I/O is approximately 1.5× faster, reducing runtime by about 33%. The performance difference is much smaller than in the cases presented above, due to the SSD having much lower latency and much higher bandwidth than the EC2/S3 network. For hot runs, the difference is negligible, as there is no disk access happening if data is properly cached.
###### Small Files {#blog:2026-07-31-asynchronous-io::small-files}
Partitioned datasets are a particularly relevant use case here, as partitioning can easily spread the data across many small files. To see how asynchronous I/O behaves in this setup, we also performed a Parquet run using the same TPC-H SF100 dataset. Instead of using one file, we generated 976 files with five row groups each. Each file contains approximately 615,000 rows and is around 22 MB in size.
| Version | Q6 runtime |
| ----------------------------- | ---------: |
| v1.5.5 (synchronous) | 9.344 s |
| v2.0.0-dev (asynchronous I/O) | 2.945 s |
We can see that v2.0.0-dev delivers a similar performance improvement here as it does for the single-file benchmark, running around 3× faster. This shows that read-ahead can also parallelize across multiple files without becoming bottlenecked by opening files or fetching their footers.
###### Large Row Groups {#blog:2026-07-31-asynchronous-io::large-row-groups}
We also wanted to see what happens at the other extreme, when a Parquet file has only a few very large row groups. For this experiment, we generated six versions of the same TPC-H SF100 `lineitem` table as a single file, changing only the requested row group (RG) size, and ran Q6 using DuckDB v2.0.0-dev. The table below reports the runtime for each version.
| Rows / RG | RGs | Approx. RG size (MB) | Total file size (MB) | Time |
| ----------: | ----: | -------------------: | -------------------: | ------: |
| 122,880 | 4,886 | ~4 MB | ~21,600 MB | 2.74 s |
| 1,966,080 | 306 | ~70 MB | ~21,400 MB | 2.11 s |
| 9,375,593 | 64 | ~320 MB | ~20,500 MB | 2.27 s |
| 62,914,560 | 10 | ~1,500 MB | ~14,700 MB | 3.69 s |
| 150,009,476 | 4 | ~3,200 MB | ~12,800 MB | 8.01 s |
| 600,037,902 | 1 | ~12,300 MB | ~12,300 MB | 25.26 s |
At first, larger row groups decrease query times. As we increase the row-group size, request latency is amortized over much larger transfers. However, beyond a certain point, the available parallelism starts to fall. A row group is DuckDB's unit of Parquet scan parallelism, so ideally a scan should expose at least one row group per system thread. On this 64-vCPU machine, the version with 64 row groups provides exactly that and finishes in 2.27 seconds, while the fastest run comes from the version with 306 row groups, at 2.11 seconds.
However, with fewer row groups than threads, we lose parallelism and can no longer saturate the network. For Q6, the projections and the physical location of the columns result in two fetch requests per row group. Four row groups therefore expose only about eight concurrent S3 streams, raising the runtime to 8.01 seconds. For the largest configuration, the file contains a single row group, and its I/O is effectively reduced to two giant streams, pushing the runtime to 25.26 seconds. This happens even though better compression makes the file a little over half the size of the version with 4,886 row groups. In this case, the extra bandwidth required by smaller row groups is cheaper than the parallelism lost with extremely large ones.
###### Concurrent Queries {#blog:2026-07-31-asynchronous-io::concurrent-queries}
The effect becomes even clearer when several queries run at the same time. For this experiment, we ran TPC-H queries 1, 6, 9, and 18 concurrently against the same SF100 Parquet dataset on S3, using a single DuckDB instance. We picked these queries because they cover a mix of scans, aggregations, and joins, with different CPU and memory requirements. We repeated the experiment with the default memory configuration and with memory limits of 16 GB and 8 GB. The total runtime is the wall-clock time until all four queries finish. We report the average and peak CPU utilization (number of cores utilized), the peak bandwidth (bw.) and the peak resident set size (RSS).
| Version | Mem. limit | Runtime | Avg. CPU | Peak CPU | Peak bw. | Peak RSS |
| ---------- | ---------: | ------: | -------: | -------: | ----------: | -------: |
| v1.5.5 | default | 35.8 s | 5.9 | 35.7 | 10.7 Gbit/s | 14.5 GB |
| v2.0.0-dev | default | 15.6 s | 48.1 | 64.0 | 24.9 Gbit/s | 20.1 GB |
| v1.5.5 | 16 GB | 35.6 s | 6.1 | 25.7 | 17.4 Gbit/s | 14.1 GB |
| v2.0.0-dev | 16 GB | 22.7 s | 35.2 | 63.4 | 24.8 Gbit/s | 15.7 GB |
| v1.5.5 | 8 GB | 35.9 s | 6.9 | 38.8 | 16.8 Gbit/s | 10.4 GB |
| v2.0.0-dev | 8 GB | 24.2 s | 30.3 | 63.7 | 25.0 Gbit/s | 11.5 GB |
With the default memory configuration, DuckDB v1.5.5 keeps an average of only about 6 of the 64 cores busy. In other words, around 90% of the machine sits idle waiting for synchronous S3 reads. DuckDB v2.0.0-dev, on the other hand, averages 48 busy cores, reaches all 64 at its peak, and saturates the 25 Gbit/s network. As a result, all four queries finish in less than half the time.
The memory results are also interesting. As we lower the limit, the memory governor reduces the read-ahead backlog, while memory-heavy operators such as those in Q18 can spill to disk. This lowers the peak physical memory used by the DuckDB process (i.e., RSS) of DuckDB v2.0.0-dev from 20.1 GB with the default configuration to 15.7 GB with a 16 GB limit and 11.5 GB with an 8 GB limit. The additional spilling and reduced read-ahead also lower average CPU utilization and increase the runtime, but v2.0.0-dev continues to saturate the network and remains substantially faster than v1.5.5 in both cases.
> One might notice that the 8 GB result still peaks at 11.5 GB of RSS. This is because [jemalloc](https://duckdb.org/docs/1.5/internals/jemalloc) keeps recently freed pages resident for about one second so they can be reused. This memory is no longer counted by DuckDB's memory manager, and v1.5.5 shows the same allocator behavior.
##### CSV {#blog:2026-07-31-asynchronous-io::csv}
The effect is larger on CSV files. The CSV file is 80.89 GB, and asynchronous I/O reduces the mean runtime from 878 seconds to just 45 seconds, making the query almost 20× faster. CSV is row-oriented, so the scan transfers substantially more data and performs fixed-size buffer reads, making concurrent remote reads especially valuable.
| Version | Q6 runtime |
| ----------------------------- | ---------: |
| v1.5.5 (synchronous) | 877.563 s |
| v2.0.0-dev (asynchronous I/O) | 45.264 s |
As in the other experiments, we used the default memory-governed read-ahead depth, so this run was not tuned to keep the 25 Gbit/s network saturated on average.
#### Conclusion {#blog:2026-07-31-asynchronous-io::conclusion}
In this blog post, we presented the recent work on asynchronous I/O for Parquet and CSV files. Most of its benefit comes from accessing remote data, but cold local reads can also benefit, albeit less. Next, we plan to add async reads for JSON and DuckDB-native files, as these are the two other formats most relevant to DuckDB core. Formats that live in out-of-tree extensions are not on the roadmap yet. We will also investigate [`io_uring`](https://man7.org/linux/man-pages/man7/io_uring.7.html), Linux's asynchronous I/O interface, which could reduce system-call overhead and the number of threads blocked on I/O. If it proves beneficial in practice, we will integrate it into DuckDB. One important thing to notice is that any of the data lake solutions supported in DuckDB can already benefit from asynchronous I/O automatically as long as the underlying data format is Parquet (or CSV, if you are brave enough).
## Thank You for 40 000 Stars on GitHub {#blog:2026-08-05-github-40k-stars}
**Publication date:** 2026-08-05
**Author:** The DuckDB team
**TL;DR:** DuckDB just reached 40 000 stars on GitHub! Here's what happened since the last 10 000-star milestone.
The [`duckdb/duckdb` GitHub repository](https://github.com/duckdb/duckdb) has just passed 40 000 stars!
{: .lightmode-img }
We’d like to use this milestone to stop for a moment and revisit recent developments in the “Duck Stack” ecosystem since last summer (2025), when we surpassed [30 000 stars](https://duckdb.org/2025/06/06/github-30k-stars).
#### Releases {#blog:2026-08-05-github-40k-stars::releases}
* Since the last milestone, the DuckLabs team has released [DuckDB 1.4.0](https://duckdb.org/2025/09/16/announcing-duckdb-140), our first long-term support release, and [DuckDB 1.5.0](https://duckdb.org/2026/03/09/announcing-duckdb-150). Both releases are packed with many new features, performance optimizations, and bugfixes.
* The DuckLabs team also published the [DuckLake 1.0](https://ducklake.select/2026/04/13/ducklake-10/) standard, our production-ready format for SQL-as-a-lakehouse.
* Finally, we released [the Quack remote protocol](https://duckdb.org/2026/05/12/quack-remote-protocol), which lets you run DuckDB in a client-server setup with multiple concurrent writers. That’s right: DuckDB instances can now talk to each other!
#### Community Metrics {#blog:2026-08-05-github-40k-stars::community-metrics}
Besides the GitHub stars, the DuckDB community has also experienced a lot of growth in other metrics:
* Each month, `duckdb.org` receives traffic from over 8 million unique visitors – more than double last summer’s numbers. We also see over 2 PB (!) in traffic from millions of extension downloads. [Thanks again to Cloudflare](https://duckdb.foundation/#technical-sponsors) for sponsoring the project.
* DuckDB now has [50M+ monthly downloads in PyPI](https://pypistats.org/packages/duckdb), more than double the 20M we reported last time.
*(As usual, we’d like to emphasize that while we’re happy to see these metrics grow, we are not glorifying them and they are not a target per se in accordance with [Goodhart’s law](https://en.wikipedia.org/wiki/Goodhart%27s_law).)*
#### Events {#blog:2026-08-05-github-40k-stars::events}
Since 30 000 stars, we ran DuckDB meetups in [Amsterdam](https://duckdb.org/events/2025/09/17/duckdb-amsterdam-meetup-3/), [Berlin](https://duckdb.org/events/2025/06/26/duckdb-berlin-meetup/), and [London](https://duckdb.org/events/2025/09/04/duckdb-science-and-education-london-meetup/). We also hosted a [Developer Meeting](https://duckdb.org/events/2026/01/30/duckdb-developer-meeting-1/) focused on DuckDB internals and extension development.
Finally, we hosted [DuckCon #7](https://duckdb.org/events/2026/06/24/duckcon7/), our largest DuckCon so far, which we also streamed online. These events allowed us to connect with our community of engineers, builders, and scientists, and learn about the many ways they are using DuckDB.
Next up, we'll host the [DuckDB Paris meetup](https://luma.com/p9n5hkvf) with [Altertable](https://altertable.ai/) and we are setting up a
[meetup in Boston](https://duckdb.org/events/2026/09/03/duckdb-boston-meetup/).
Stay tuned!
#### DuckDB in the Wild {#blog:2026-08-05-github-40k-stars::duckdb-in-the-wild}
Here are just a few of the projects, extensions, and announcements that happened in the DuckDB community since the last 10 000-star milestone:
* **Blog:** Petrica Leuca shows you [how to use DuckDB for keyword, full-text, and semantic similarity search](https://duckdb.org/2025/06/13/text-analytics) with embeddings for lightweight text analytics.
* **Video:** GizmoEdge: [A distributed DuckDB engine for IoT](https://www.youtube.com/watch?v=xlvjN_eFJvM) (from the first DuckDB Developer Meeting) by Philip Moore.
* **Blog:** [DuckDB for streaming analytics?](https://duckdb.org/2025/10/13/duckdb-streaming-patterns) Yes, says Guillermo Sanchez, you can use DuckDB to refresh your data at near real-time speed.
* **Use Case Walkthrough:** Uncovering [Financial Crime with DuckDB and SQL/PGQ](https://duckdb.org/2025/10/22/duckdb-graph-queries-duckpgq) graph syntax that's part of SQL:2023 by Daniël ten Wolde.
* **For Fun:** DuckDB co-creator Hannes Mühleisen shows you [how to store and even process video in DuckDB](https://duckdb.org/2025/10/27/movies-in-databases).
* **Blog:** [Iceberg in the Browser](https://duckdb.org/2025/12/16/iceberg-in-the-browser): learn how to read and write tables in Iceberg catalogs without needing to manage any infrastructure – directly from your browser!
* **Presentation:** Denis Hirn from University of Tübingen introduces you to DuckPL: [A procedural language in DuckDB](https://www.youtube.com/watch?v=cjmtEBz_hSc).
* **Hardware benchmarks:** Hannes Mühleisen tests [DuckDB on a Loongson CPU](https://duckdb.org/2026/01/06/duckdb-on-loongarch-morefine) and Gábor Szárnyas put [DuckDB to work on a MacBook Neo](https://duckdb.org/2026/03/11/big-data-on-the-cheapest-macbook).
* **Video:** Barry Smart walks you through 20 years of wind data while [auditing UK energy policy without a cluster](https://www.youtube.com/watch?v=7dUEgMLuUcI) (DuckCon).
* **Blog:** Interested in [building DuckDB extensions in C#](https://duckdb.org/2026/03/20/duckdb-extensionkit-csharp)? Giorgi Dalakishvili shows you how to build DuckDB extensions in C# with DuckDB.ExtensionKit.
* **Talk:** Kian Mehrabani takes you on a tour of [how Spotify is using DuckDB](https://www.youtube.com/watch?v=-9GY1CCJG5o) as a SQL layer over user listening history in this video from DuckCon.
* **Preview:** In DuckDB v2.0, scheduled for fall 2026, DuckDB will support async reads of Parquet and CSV files. Pedro Holanda walks you through the details of [asynchronous I/O](https://duckdb.org/2026/07/31/asynchronous-io).
#### Closing Thoughts {#blog:2026-08-05-github-40k-stars::closing-thoughts}
We would like to extend a heartfelt **thank you** to the DuckDB community of users and contributors. We look forward to growing the community around the “DuckStack” and hope the GitHub stars follow. We'll report back when we reach 50 000 stars.
## A Preview of DuckDB v2.0 {#blog:2026-08-17-duckdb-20-highlights}
**Publication date:** 2026-08-17
**Authors:** Mark Raasveldt, Hannes Mühleisen
**TL;DR:** DuckDB v2.0 is coming this fall. In this post, we preview its headline features: DuckDB as a server, triggers, the VARIANT type, asynchronous I/O, a new SQL parser, a new storage format, and much more.
> **Update.** DuckDB v2.0.0 was released on October 21, 2026. Some details changed since this preview; see the [release announcement](#blog:2026-10-21-announcing-duckdb-200) and the [migration guide](https://duckdb.org/docs/2.0/operations_manual/migration).
DuckDB v2.0 will be named “Cinnamon Teal” after the [cinnamon teal](https://en.wikipedia.org/wiki/Cinnamon_teal) *(Spatula cyanoptera),* a strikingly reddish-brown duck found in the western Americas.
A major version bump is not something we do lightly, and it is not just ceremony: v2.0 ships a new SQL parser, a new default storage format, a reworked C API, and a small number of carefully chosen breaking changes. But above all, it is a feature release, built from over 10,000 commits since we released v1.5 in March. Where last year was the year of the lakehouse, this release kicks off the year of DuckDB as a server. We previewed many of these features in the [“State of the Duck” talk at DuckCon #7](https://www.youtube.com/watch?v=iFPKNQu0FtE), if you prefer to watch instead of read.
DuckDB is moving rather quickly, and we can only cover a small fraction of the changes here. Condensing all new features down to a shortlist is always a fight over what gets in, and yes, we know that what follows is technically a listicle *(Ten Things Coming to DuckDB v2.0, Number Eight Will Shock You).* We are not proud of the format, but it works, so here it is, starting with the SQL-level features and working down into the engine.
#### 1. DuckDB as a Server: Quack and `CONNECT` {#blog:2026-08-17-duckdb-20-highlights::1-duckdb-as-a-server-quack-and-connect}
DuckDB has been an in-process database since day one. But people have asked us – very persistently – for a client/server mode, and we have finally caved. The [`quack` extension](https://github.com/duckdb/duckdb-quack) implements DuckDB's native protocol for talking to other DuckDBs. It was [released as a preview](#blog:2026-05-12-quack-remote-protocol) shortly before DuckCon #7, and it is a big part of where DuckDB is headed: any DuckDB process can serve its databases over the network, and any other DuckDB can attach to it and route queries there using the new `CONNECT` statement. For example:
###### DuckDB server {#blog:2026-08-17-duckdb-20-highlights::svg-classiconuse-hrefdatabase-01usesvg-duckdb-server}
```sql
CALL quack_serve(
token = 'my_token'
);
```
quack:
###### DuckDB client {#blog:2026-08-17-duckdb-20-highlights::svg-classiconuse-hrefdatabase-01usesvg-duckdb-client}
```sql
ATTACH 'quack:server.example.com'
AS qk (TOKEN 'my_token');
CONNECT qk;
SELECT count(*) FROM events;
-- executes on the server,
-- results stream back
DISCONNECT;
```
`CONNECT` is the successor to the `remote.query($$...$$)` workaround we showed when Quack was first revealed – we looked at that syntax and said: no, this cannot be it. And `CONNECT` is not limited to Quack: it points your session at any remote database that supports it, and the new remote pushdown optimizer ([#22914](https://github.com/duckdb/duckdb/pull/22914)) ships SQL directly to PostgreSQL and MySQL instead of pulling tables over the wire:
```sql
CONNECT 'postgres://localhost/mydb';
SELECT count(*) FROM orders; -- runs on the PostgreSQL server
DISCONNECT;
```
If you have worked with analytical systems in the past, you may assume that DuckDB cannot handle transactional workloads. But DuckDB has been built as a transactional, multi-connection database with full MVCC and transaction isolation since day one. Most users just never needed that in a single-user scenario. It turns out DuckDB handles transactions well: it's fast enough to compete with general-purpose databases like PostgreSQL on quite a few workloads, and the client/server pattern finally lets that machinery shine in multi-tenant, long-running deployments.
Running DuckDB long-term also comes with new challenges, which is why v2.0 pushes on better metrics, logs, and observability (see, e.g., the metrics layer rework in [#22799](https://github.com/duckdb/duckdb/pull/22799)) that let you look at a DuckDB instance and see what it is actually doing. People even built standalone clients for the Quack protocol within weeks of the preview. We thought we were extending DuckDB to talk to other DuckDBs; the world said no, no, no, and built their own clients. Who would have thought.
#### 2. `VARIANT` Becomes a First-Class Citizen {#blog:2026-08-17-duckdb-20-highlights::2-variant-becomes-a-first-class-citizen}
The `VARIANT` type shipped in [DuckDB v1.5](#blog:2026-03-09-announcing-duckdb-150), and the way to think about it is JSON on steroids. Basically, imagine if JSON were fast. Like JSON, a `VARIANT` column can store differently-shaped data in every row. Unlike JSON, it is not a text format: DuckDB automatically detects the common structure hidden in your semi-structured data and “shreds” it, so it compresses well in storage and executes fast in queries, all without you ever declaring a schema. This makes `VARIANT` a natural fit for real-time log ingestion, where streams of JSON-ish records share structure but evolve over time.
In v2.0, this pipeline works end to end: shredded execution straight from storage ([#20912](https://github.com/duckdb/duckdb/pull/20912)), extraction pushdown into scans ([#22478](https://github.com/duckdb/duckdb/pull/22478)), shredded `VARIANT` reading *and* writing for Parquet, and a family of `variant_*` functions:
```sql
CREATE TABLE events (payload VARIANT);
INSERT INTO events
VALUES ('{"user": {"id": 42, "tags": ["a", "b"]}}'::JSON::VARIANT);
SELECT variant_type(payload), variant_keys(payload)
FROM events;
SELECT *
FROM events
WHERE variant_contains(payload, {'user': {'id': 42}}::VARIANT);
```
Longer term, likely soon after v2.0 (but don't hold us to it), we plan to back the regular `JSON` type with `VARIANT`, so existing JSON workloads get all of these benefits without changing a single query.
#### 3. Triggers {#blog:2026-08-17-duckdb-20-highlights::3-triggers}
Triggers have been a long-standing feature request, and DuckDB v2.0 delivers them in full: `BEFORE` and `AFTER` triggers, `FOR EACH ROW` and `FOR EACH STATEMENT`, transition tables via `REFERENCING OLD/NEW TABLE`, multiple triggers per event, `RETURNING` on triggered tables, and `DROP TRIGGER`.
The classic use case is audit tables: something happens in the system, and a trigger records what changed. For example:
```sql
CREATE TABLE target (id INTEGER, val INTEGER);
CREATE TABLE audit (id INTEGER, old_val INTEGER, new_val INTEGER);
CREATE TRIGGER trg_audit AFTER UPDATE ON target
REFERENCING OLD TABLE AS o NEW TABLE AS n
FOR EACH STATEMENT
INSERT INTO audit
SELECT n.id, o.val, n.val
FROM o
JOIN n ON o.id = n.id;
INSERT INTO target VALUES (1, 10), (2, 20);
UPDATE target SET val = val * 10 WHERE id <= 2;
SELECT * FROM audit;
```
| id | old_val | new_val |
| ---: | ------: | ------: |
| 1 | 10 | 100 |
| 2 | 20 | 200 |
Triggers fit naturally with long-running DuckDB services, and we are also planning to use them internally to build several upcoming features. They are fully exposed at the SQL level too, so you can build your own cool stuff with them.
#### 4. SQL Dialect Additions {#blog:2026-08-17-duckdb-20-highlights::4-sql-dialect-additions}
As always, DuckDB's SQL dialect keeps growing. A few favorites from this release cycle:
With **`NEAREST` joins** ([#24137](https://github.com/duckdb/duckdb/pull/24137)), top-k similarity search becomes a join clause, handy for vector and embedding workloads:
```sql
SELECT q.user_id, t.product_id
FROM users q
INNER JOIN products t APPROX NEAREST 2
BY SIMILARITY array_cosine_similarity(q.embedding, t.embedding);
```
**DML inside CTEs** ([#21634](https://github.com/duckdb/duckdb/pull/21634), [#21997](https://github.com/duckdb/duckdb/pull/21997), [#24217](https://github.com/duckdb/duckdb/pull/24217)) lets you use `INSERT`, `UPDATE`, `DELETE`, and `COPY` as pipeline steps:
```sql
WITH moved AS MATERIALIZED (
DELETE FROM staging RETURNING *
)
INSERT INTO archive SELECT * FROM moved;
```
**Nested schemas** ([#23492](https://github.com/duckdb/duckdb/pull/23492), [#24222](https://github.com/duckdb/duckdb/pull/24222)) allow schemas within schemas:
```sql
CREATE SCHEMA finance;
CREATE SCHEMA finance.reports;
CREATE TABLE finance.reports.q3 (revenue DECIMAL);
```
The new **variable syntax** ([#21194](https://github.com/duckdb/duckdb/pull/21194)) lets you write `$x` anywhere an expression is allowed, no more [`getvariable(...)` verbiage](https://duckdb.org/docs/1.5/sql/functions/utility#getvariablevariable_name):
```sql
SET VARIABLE threshold = 100;
SELECT * FROM orders WHERE amount > $threshold;
```
The **JSON mutation functions** `json_set`, `json_insert`, `json_replace`, and `json_remove` ([#23786](https://github.com/duckdb/duckdb/pull/23786)) finally let you modify JSON documents in place:
```sql
SELECT json_set('{"a":1}', '$.b', '2');
```
| json_set('{"a":1}', '$.b', '2') |
| ------------------------------- |
| {"a":1,"b":2} |
And **recursive CTEs with [`USING KEY`](#blog:2025-05-23-using-key) aggregation** ([#19481](https://github.com/duckdb/duckdb/pull/19481)) enable iterative algorithms in pure SQL, backed by the rewritten recursive CTE engine described below:
```sql
WITH RECURSIVE tbl(a, b) USING KEY (a, avg(b)) AS (
SELECT 1, 5
UNION ALL
SELECT a, b - 1 FROM tbl WHERE b > 0
)
TABLE tbl;
```
| a | b |
| ---: | ---: |
| 1 | 2.5 |
There is more: SQL-standard `FETCH FIRST 2 ROWS ONLY` ([#23533](https://github.com/duckdb/duckdb/pull/23533)), `OVERLAY()` ([#22456](https://github.com/duckdb/duckdb/pull/22456)), `UNNEST` in `GROUP BY` ([#23644](https://github.com/duckdb/duckdb/pull/23644)), and well-defined `MERGE` / `UPDATE ... FROM` semantics for multi-matched rows ([#24058](https://github.com/duckdb/duckdb/pull/24058)).
#### 5. Asynchronous I/O {#blog:2026-08-17-duckdb-20-highlights::5-asynchronous-io}
Interacting with object stores like S3 is central to the DuckDB experience: your data has to come from somewhere, and it often sits in object storage. DuckDB has long been able to read from object stores in parallel, but synchronous access placed a limit on how fast this could go. DuckDB v2.0 introduces asynchronous I/O throughout the engine. We described the design in detail in [a dedicated blog post](#blog:2026-07-31-asynchronous-io).
Thanks to asynchronous access, the I/O layer now scales independently from the query processing layer, which means far more parallelism for remote reads and dramatically faster queries on network storage. Parquet support came first ([#23662](https://github.com/duckdb/duckdb/pull/23662)), with CSV ([#23961](https://github.com/duckdb/duckdb/pull/23961)) and DuckDB's own file format ([#24654](https://github.com/duckdb/duckdb/pull/24654)) following, along with asynchronous Parquet writes ([#23283](https://github.com/duckdb/duckdb/pull/23283)) and new `MMAP` and `DIRECT_IO` modes ([#22988](https://github.com/duckdb/duckdb/pull/22988)). Local storage benefits a little too, but network storage is where you will see the big gains.
#### 6. Faster Queries Across the Board {#blog:2026-08-17-duckdb-20-highlights::6-faster-queries-across-the-board}
As with every release, a lot of work went into making your existing queries faster without you doing anything. To pick some highlights: partial aggregates are now pushed below joins ([#22572](https://github.com/duckdb/duckdb/pull/22572)) and redundant aggregations are reused ([#24543](https://github.com/duckdb/duckdb/pull/24543)), the recursive CTE engine has been rewritten ([#22211](https://github.com/duckdb/duckdb/pull/22211)), aggregations now spill to disk when they outgrow memory ([#24499](https://github.com/duckdb/duckdb/pull/24499)), and the Windows CLI got approximately 2.2× faster at multi-threaded result materialization ([#24036](https://github.com/duckdb/duckdb/pull/24036)).
How much faster can this get? Here is a microbenchmark you can run on a laptop: single-source reachability over a graph with one million edges, written as a plain [recursive CTE](https://duckdb.org/docs/1.5/sql/query_syntax/with).
```sql
CREATE TABLE edges AS
SELECT (range % 100_000)::INTEGER AS src,
((range * 13 + 7) % 100_000)::INTEGER AS dst
FROM range(1_000_000);
WITH RECURSIVE reachable(node) AS (
SELECT 0
UNION
SELECT dst FROM edges, reachable WHERE src = node
)
SELECT count(*) FROM reachable;
```
| Version | Run time |
| --------------------- | -------: |
| DuckDB v1.5.4 | 4.90 s |
| DuckDB v2.0 (preview) | 0.12 s |
As you can see, DuckDB v2.0 is about 40× faster (!) for the same recursive query.
> **Update.** We published a blog post on [how we are speeding up recursive CTEs in DuckDB v2.0](#blog:2026-08-25-how-duckdb-runs-recursive-ctes-faster).
Row-group pruning has been massively expanded: [min-max indexes (zone maps)](https://duckdb.org/docs/1.5/sql/indexes#min-max-index-zonemap) and [Parquet Bloom filters](#blog:2025-03-07-parquet-bloom-filters-in-duckdb) now skip data for structs, lists, decimals, UUIDs, `IN` filters, and even function predicates:
```sql
-- these now prune row groups instead of scanning them:
SELECT * FROM logs WHERE contains(message, 'ERROR');
SELECT * FROM t WHERE substr(code, 1, 3) = 'NL-';
SELECT * FROM 'data/*.parquet' WHERE id IN (1, 5, 9);
```
Query planning also becomes **partition-aware** ([#22336](https://github.com/duckdb/duckdb/pull/22336)). [Lakehouse formats](https://duckdb.org/docs/1.5/lakehouse_formats) (DuckLake, Iceberg and plain Hive-partitioned Parquet on S3) are all partitioned, and exploiting that partitioning is often the difference between scanning a dataset and skipping most of it. In v2.0, the planner and optimizer take full advantage of existing partitioning, and partitioned writes have been reworked as well ([#22225](https://github.com/duckdb/duckdb/pull/22225), [#22620](https://github.com/duckdb/duckdb/pull/22620)).
#### 7. Storage Format v2.0 {#blog:2026-08-17-duckdb-20-highlights::7-storage-format-v20}
DuckDB v2.0 bumps the default [storage format version](https://duckdb.org/docs/1.5/internals/storage) to v2.0.0 ([#22875](https://github.com/duckdb/duckdb/pull/22875)).
Column metadata is now loaded lazily ([#22333](https://github.com/duckdb/duckdb/pull/22333)), so wide tables open faster too. The `DICT_FSST` string compression method is enabled by default ([#23733](https://github.com/duckdb/duckdb/pull/23733)), deletes are stored compactly ([#24336](https://github.com/duckdb/duckdb/pull/24336)), and the storage layer performs much stronger corruption validation on read. In short: databases with big indexes and wide tables open faster and use far less memory. The new storage format also allows checkpoint vacuuming to compact tables with ART indexes by incrementally remapping entries whose row IDs change, avoiding a full index rebuild ([#23653](https://github.com/duckdb/duckdb/pull/23653)).
Later this year, ART indexes will be buffer-managed ([#21458](https://github.com/duckdb/duckdb/pull/21458), [#23605](https://github.com/duckdb/duckdb/pull/23605)). This will allow ART index buffers to be evicted under memory pressure, and remove the restriction on the ART index fitting completely within memory.
#### 8. A Brand New SQL Parser {#blog:2026-08-17-duckdb-20-highlights::8-a-brand-new-sql-parser}
DuckDB has famously always used a parser derived from PostgreSQL's. We have decided that enough is enough: v2.0 ships our own modern, extensible PEG-based parser ([#22194](https://github.com/duckdb/duckdb/pull/22194)), an idea we first explored in our 2024 post on [runtime-extensible parsers](#blog:2024-11-22-runtime-extensible-parsers). This change ties into the extension ecosystem: extensions can now hook into the grammar itself, so expect extensions that expose entirely new SQL syntax. It also brings better error messages with precise source locations.
You should not actually notice anything from the parser swap as we designed it to be compatible with the old one. If you do notice, please file an issue.
> **Update.** We published a new blog post on the [PEG parser](#blog:2026-08-20-duckdb-20-peg-parser).
#### 9. Timezones, Calendars, and Collations Without ICU {#blog:2026-08-17-duckdb-20-highlights::9-timezones-calendars-and-collations-without-icu}
Timezone-aware timestamps, calendars, and collations in DuckDB have always been powered by the ICU library. ICU is a fine library, but we only ever used a small slice of it, while still carrying it around in every DuckDB distribution. In v2.0, the ICU library is gone entirely: the `icu` extension now implements timezones, calendars, and collations itself ([#24463](https://github.com/duckdb/duckdb/pull/24463), [#24403](https://github.com/duckdb/duckdb/pull/24403)), with the timezone data built directly from the IANA database and compressed down to around 45 kB. Everything keeps working exactly as before:
```sql
SELECT '2026-08-14 12:00:00'::TIMESTAMPTZ AT TIME ZONE 'Europe/Paris';
SELECT * FROM names ORDER BY name COLLATE de;
```
Besides being much smaller and easier to keep up to date, the new implementation is also simply faster. Here's a quick microbenchmark on a MacBook that converts 25 million timestamps to a timezone and filters 5 million strings with a German collation:
| Query | v1.5.4 (ICU) | v2.0 (native) | Speedup |
| ------------------------------------------- | -----------: | ------------: | ------: |
| `ts AT TIME ZONE 'Europe/Paris'`, 25 M rows | 0.24 s | 0.11 s | 2.2× |
| Filter with `COLLATE de`, 5 M rows | 0.15 s | 0.06 s | 2.6× |
#### 10. Write Extensions Once, Host Them Yourself {#blog:2026-08-17-duckdb-20-highlights::10-write-extensions-once-host-them-yourself}
Extensions are one of the best things about DuckDB, but today, most of them, including our own, build against the [unstable C++ API](https://duckdb.org/docs/1.5/clients/cpp). Extension authors are required to rebuild and publish their extensions for every DuckDB release, even if the extension itself does not change. This is often a non-trivial operation for the extension author and a non-trivial coordination effort for the DuckDB team. Not rebuilding an extension means it can't be installed on the latest version of DuckDB.
##### Developing Extensions {#blog:2026-08-17-duckdb-20-highlights::developing-extensions}
DuckDB v2.0 will ship with a revamped C API (see [#24702](https://github.com/duckdb/duckdb/pull/24702) for part 1). The API will have a versioned [specification expressed in YAML](https://github.com/duckdb/duckdb/tree/main/api_spec) that uses an also versioned specification _schema_, and tooling for code generation ([#24135](https://github.com/duckdb/duckdb/pull/24135)). The schema lets us tag every symbol with its lifecycle and stability guarantees ([#24435](https://github.com/duckdb/duckdb/pull/24435)). A large part of the API will be marked stable and frozen, providing a stable ABI across DuckDB versions.
The API itself improves over the pre-2.0 C API in a number of ways. It provides coherent error handling and a unified set of conventions around naming and ownership. It covers a much larger feature surface, including scalar, aggregate, table, cast, and copy functions with named parameters and varargs, parsing and inspecting SQL statements, prepared statements, replacement scans, custom filesystems, direct access to vector buffers, values and types, and more. It supports streaming query result consumption. With this API we want to provide the primitives to access DuckDB's most powerful features.
You won't need to program against the C API directly, though. We also provide a C++ API on top of it. It is a thin layer that compiles into your extension and talks only to the stable C ABI, so the binary you ship stays independent of DuckDB versions. It gives you a convenient, typed, well-documented way to access DuckDB's API. We're also working on bindings that provide the same for extension writers using Rust.
So what will an extension using the C++ API look like? Here is a complete example, though the details may still change: a single file that registers a vectorized scalar function.
```cpp
#include "duckdb_cpp.hpp"
using namespace duckdb_api;
DUCKDB_CPP_EXTENSION_ENTRYPOINT(extension) {
// add_numbers(a BIGINT, b BIGINT DEFAULT 2): adds two BIGINTs, one vector at a time
ScalarFunction function;
function.SetName("add_numbers")
.SetSignature(FunctionSignature::Create()
.AddParameter("a", LogicalType::BIGINT())
.AddParameterDefault("b", LogicalType::BIGINT(), Value::Bigint(2))
.SetReturnType(LogicalType::BIGINT()))
.SetExecCallback([](ScalarFunction::ExecInput &input) {
auto chunk = input.GetInputChunk();
auto a = chunk.GetVector(0).GetView();
auto b = chunk.GetVector(1).GetView();
auto out = input.GetResultVector().GetDataMutableClick here to reveal the setup script.
```bash # make sure you have the required tools sudo apt install git g++ cmake python3-pip # clone the repository at the commit used for the experiments git clone https://github.com/duckdb/duckdb-tpch cd duckdb-tpch git reset --hard cf096cea8a0a054cefe0e695c842ca5bca6ec04d # install the required Python dependencies python3 -m venv venv source venv/bin/activate python3 -m pip install -r requirements.txt python3 -m pip install duckdb==1.6.0.dev379 # build the dbgen tool cd tpch_tools_3.0.1/dbgen/ make cd ../.. # generate the dataset, query streams and update streams export SF=1000 ./generate.sh ```
###### DuckDB server {#blog:2026-10-21-announcing-duckdb-200::svg-classiconuse-hrefdatabase-01usesvg-duckdb-server}
```sql
CALL quack_serve(
token = 'my_token'
);
CREATE TABLE events
(date DATE, name VARCHAR);
INSERT INTO events VALUES
('2027-04-07',
'DuckCon #8');
```
quack:
###### DuckDB client {#blog:2026-10-21-announcing-duckdb-200::svg-classiconuse-hrefdatabase-01usesvg-duckdb-client}
```sql
ATTACH 'quack:localhost'
AS qk (TOKEN 'my_token');
CONNECT qk;
SELECT * FROM events;
┌────────────┬────────────┐
│ date │ name │
│ date │ varchar │
├────────────┼────────────┤
│ 2027-04-07 │ DuckCon #8 │
└────────────┴────────────┘
```
For simplicity, this example runs the server and the client on the same machine. By default, the server only listens on `localhost`. To accept connections from other machines, keep the server on `localhost` and put it behind a [TLS-terminating reverse proxy](https://duckdb.org/docs/2.0/quack/setup/reverse_proxy). The [`allow_other_hostname` option](https://duckdb.org/docs/2.0/quack/overview#starting-a-server), which exposes the server directly, is only meant for testing.
The `CONNECT` statement is not limited to Quack: it points your session at [any remote database that supports it](https://duckdb.org/docs/2.0/sql/statements/connect#supported-databases), including PostgreSQL and MySQL. For these, the new remote pushdown optimizer ships queries directly to the remote server instead of pulling tables over the wire.
##### Quack in the CLI {#blog:2026-10-21-announcing-duckdb-200::quack-in-the-cli}
The CLI also has shortcuts for setting up a Quack client-server pair: the new [`-serve` and `-connect` arguments](https://duckdb.org/docs/2.0/clients/cli/arguments#serving-and-connecting-to-remote-databases) start a Quack server or connect to one directly from the command line:
###### CLI server {#blog:2026-10-21-announcing-duckdb-200::svg-classiconuse-hrefdatabase-01usesvg-cli-server}
```bash
duckdb -serve
```
```sql
CREATE TABLE t AS
SELECT 42 AS x;
```
quack:
###### CLI client {#blog:2026-10-21-announcing-duckdb-200::svg-classiconuse-hrefdatabase-01usesvg-cli-client}
```bash
duckdb -connect
```
```sql
FROM t;
┌───────┐
│ x │
│ int32 │
├───────┤
│ 42 │
└───────┘
```
#### Performance {#blog:2026-10-21-announcing-duckdb-200::performance}
##### Asynchronous I/O {#blog:2026-10-21-announcing-duckdb-200::asynchronous-io}
DuckDB 2.0 introduces [asynchronous I/O](https://duckdb.org/docs/2.0/data/io) for reading Parquet and CSV files, as well as [DuckDB's native storage format](https://duckdb.org/docs/2.0/internals/storage). The I/O layer now scales independently from the query processing layer, issuing reads in the background and keeping multiple requests in flight. This means far more parallelism for remote reads and dramatically faster queries on object storage such as [`s3://`](https://duckdb.org/docs/2.0/core_extensions/httpfs/s3api).
We described the design in detail in a [dedicated blog post](#blog:2026-07-31-asynchronous-io).
In that post, we showed the network throughput over the course of a query with and without asynchronous I/O:
{: .lightmode-img }
*Network throughput. Higher utilization and shorter runtime are better.*{: .caption }
##### The `VARIANT` Type Becomes a First-Class Citizen {#blog:2026-10-21-announcing-duckdb-200::the-variant-type-becomes-a-first-class-citizen}
The [`VARIANT` type](https://duckdb.org/docs/2.0/sql/data_types/variant) stores semi-structured, schema-less data such as JSON while keeping the performance of columnar execution.
So you can think of it as JSON on steroids: a `VARIANT` column can store differently shaped data in every row, but unlike JSON it is not a text format.
[DuckDB 1.5 already introduced `VARIANT`](#blog:2026-03-09-announcing-duckdb-150::variant-type), but DuckDB 2.0 fully integrates it. DuckDB 2.0 detects the common structure in your data and “shreds” it, so it compresses well and executes fast. The pipeline works end to end, from shredded execution straight out of storage to shredded `VARIANT` reading and writing for Parquet, backed by a family of [`variant_*` functions](https://duckdb.org/docs/2.0/sql/functions/variant). For example:
```sql
CREATE TABLE events (payload VARIANT);
INSERT INTO events
VALUES
('{"user": {"id": 42, "tags": ["a", "b"]}}'::JSON::VARIANT),
('{"event": "click", "x": 10, "y": 20}'::JSON::VARIANT);
SELECT variant_type(payload), variant_keys(payload)
FROM events;
```
```text
┌───────────────────────┬───────────────────────┐
│ variant_type(payload) │ variant_keys(payload) │
│ varchar │ varchar[] │
├───────────────────────┼───────────────────────┤
│ OBJECT │ [user] │
│ OBJECT │ [event, x, y] │
└───────────────────────┴───────────────────────┘
```
##### Agent Mode in the CLI {#blog:2026-10-21-announcing-duckdb-200::agent-mode-in-the-cli}
Coding agents such as Claude Code, Codex and Cursor increasingly drive the DuckDB CLI on their users' behalf. The 2.0 CLI detects when it is being run by an agent and switches to [agent mode](#blog:2026-10-09-agent-mode): it prints compact Markdown tables instead of padded boxes, says clearly when a result was truncated, stops runaway queries early and reports errors as JSON.
Long-running SQL queries report their expected cost prior to running, so the agent can decide whether to wait. In our experiments on TPC-H, agent mode cut the amount of DuckDB output the model had to read by 59%. You can turn it _on_ for manual testing using the [`-agent` flag](https://duckdb.org/docs/2.0/clients/cli/agent_mode#activating-agent-mode) or instruct your agent to turn it _off_ using the `-no-agent` flag.
#### Extensibility {#blog:2026-10-21-announcing-duckdb-200::extensibility}
##### A Brand New SQL Parser {#blog:2026-10-21-announcing-duckdb-200::a-brand-new-sql-parser}
DuckDB has always used a parser derived from PostgreSQL's. In 2.0, our own modern, extensible [PEG-based parser](#blog:2026-08-20-duckdb-20-peg-parser), which debuted as an opt-in feature in [DuckDB 1.5](#blog:2026-03-09-announcing-duckdb-150::peg-parser), becomes the default. Extensions can now hook into the grammar itself (an idea we first explored in our [research on runtime-extensible parsers](#blog:2024-11-22-runtime-extensible-parsers)), so expect extensions that expose entirely new SQL syntax. Error messages now come with precise source locations. For details, see the [PEG parser documentation](https://duckdb.org/docs/2.0/sql/peg_parser). The new parser is designed to be compatible with the old one, apart from a few deliberate changes listed in the [migration guide](https://duckdb.org/docs/2.0/operations_manual/migration#sql-parser). If you notice any other differences, please file an issue. In the meantime, you can switch back to the old parser using the [`legacy_parser` extension](https://duckdb.org/docs/2.0/core_extensions/legacy_parser) by running `SET enable_legacy_parser = true;`{:.language-sql .highlight}.
The old parser had served DuckDB since its first commit in 2018. Its grammar was generated with YACC/Bison, which produces an LALR(1) parser, so as DuckSQL grew, even small additions could collide with existing rules and raise `shift/reduce` or `reduce/reduce` conflicts. A PEG grammar avoids this by trying alternatives in the order they are written, so adding syntax no longer risks breaking unrelated rules. Python itself moved to a PEG-based parser in [version 3.9](https://peps.python.org/pep-0617/) for the same flexibility.
##### Write Extensions Once and Host Them Yourself {#blog:2026-10-21-announcing-duckdb-200::write-extensions-once-and-host-them-yourself}
DuckDB 2.0 ships a revamped [C API v2](https://duckdb.org/docs/2.0/clients/c/overview#c-api-v2), which is considerably more powerful than its predecessor and exposes components such as file systems and table functions to extensions. Before marking it as stable, we would like to test it in the field, so please give it a try and [send us your feedback](https://github.com/duckdb/duckdb/discussions/27525)!
You can also register and host your own trusted extension repositories, so an organization can host and sign its own extensions and have them install and load just like the built-in ones:
```sql
SET allow_extension_repositories = 'allowed';
CREATE EXTENSION REPOSITORY my_repo
WITH PREFIX 'https://extensions.example.org';
INSTALL AND LOAD my_ext FROM my_repo;
```
A repository is a name, a URL prefix and one or more RSA public keys that are trusted to sign the extensions served from it. The prefix can point at anything DuckDB can read: a local path, `https`, `s3`, you name it. At `CREATE` time, DuckDB fetches the repository's public keys and pins them into the repository definition, printing each key's SHA-256 fingerprint so you can compare it against one published out of band.
This feature is still a work in progress and its syntax may change in future releases. For more details, see the [Extension Repositories page](https://duckdb.org/docs/2.0/extensions/extension_repositories).
##### Core Extension Updates {#blog:2026-10-21-announcing-duckdb-200::core-extension-updates}
DuckDB's [core extensions](https://duckdb.org/docs/2.0/core_extensions/overview) are also shipping several new features, fixes and performance optimizations. These are too numerous to all list here, so we only highlight a few of them and will cover them in dedicated blog posts in the coming weeks:
* We released [DuckLake 1.1](https://ducklake.select/2026/10/21/ducklake-11/), including a new specification and [reference implementation](https://duckdb.org/docs/2.0/core_extensions/ducklake).
* The [Iceberg extension](https://duckdb.org/docs/2.0/core_extensions/iceberg/overview) strives to become a feature-complete implementation. To this end, it adds REST catalog [`ATTACH` options](https://duckdb.org/docs/2.0/core_extensions/iceberg/iceberg_options#attach-options) that derive a table's location from its namespace or load table metadata from the `metadata-location` file returned by the catalog. Further options control the URL prefix and the namespace separator.
* The [Delta extension](https://duckdb.org/docs/2.0/core_extensions/delta) can create Delta tables with `CREATE TABLE` (including partitioning and Delta table properties), supports time travel and improves `VARIANT` support.
#### SQL Features {#blog:2026-10-21-announcing-duckdb-200::sql-features}
##### Triggers {#blog:2026-10-21-announcing-duckdb-200::triggers}
DuckDB 2.0 delivers triggers: the [`CREATE TRIGGER` statement](https://duckdb.org/docs/2.0/sql/statements/create_trigger) with `BEFORE` and `AFTER` triggers, `FOR EACH ROW` and `FOR EACH STATEMENT`, transition tables via `REFERENCING OLD/NEW TABLE`, multiple triggers per event, `RETURNING` on triggered tables and `DROP TRIGGER`. The classic use case is audit tables, where a trigger records what changed in response to a table modification. Some combinations, such as `BEFORE ... FOR EACH ROW` and `UPDATE ... FOR EACH ROW` triggers, are not yet supported; see the [limitations](https://duckdb.org/docs/2.0/sql/statements/create_trigger#limitations).
```sql
CREATE TRIGGER trg_audit
AFTER UPDATE ON target
REFERENCING
OLD TABLE AS o
NEW TABLE AS n
FOR EACH STATEMENT
INSERT INTO audit
SELECT n.id, o.val, n.val
FROM o JOIN n ON o.id = n.id;
```
Triggers can also maintain materialized views incrementally! For example, to keep track of the average age of people per nationality in a table `age_statistics`, you can use two triggers – one for insert and another for delete operations – and store the sum and the count alongside the average.