What it is
Analytics Data Pivot (ADPivot) is an open-source tool that leverages the power of DuckDB and Pyodide to allow users to query and pivot basically any database, regardless of size, in the browser.
How it is done
That is possible thanks to DuckDB-Wasm's querying engine, that does not require loading the whole dataset into memory. Instead, DuckDB queries only the data needed for your pivot table by reading it in the storage (hard disk or SSD).
That is the reason why ADPivot is focused on Apache Parquet (.parquet) files. They make reading and writing data much faster than other data formats, like comma separated (.csv), JSON and Excel spreadsheets (.xlsx). Parquet files have also a much smaller file size, optimizing storage usage. For accessibility, though, ADPivot is able to convert Excel spreadsheets (.xlsx) into parquet files for use in the application.
Later, the raw data is passed to Pyodide, which through its port of Pandas to WebAssembly, shapes the data to the desired pivot format.
ADPivot motivation
The inspiration for the creation of ADPivot came from the hole that appeared on my capacity for data exploration on my company once it switched its Business Intelligence (BI) tool from Tableau to PowerBI. Tableau had an awesome pivot table engine that allowed for very fast and interactive data exploration.
The open-source tools that were available then to plug the hole that Tableau left weren't focused on tables, which required me to code manually in Python the pivot tables I needed. This massively slowed my capacity of data exploration.
ADPivot fixes this, by allowing users to build pivot tables with a few clicks, and later exporting them as an Excel spreadsheet for further analysis, while mantaining the MultiIndex (hierarchical index) defined in the app. Also, ADPivot even surpasses Tableau in some aspects, like allowing the querying of larger databases (even though the resulting pivot tables will take a while to load).