TidyPanel

Spend tokens on reasoning. Let code clean the tables.

Local CSV and Excel data cleaning for AI agents, with an R package and JSON command-line interface.

The local cleaning step consumes 0 AI model tokensR 4.1 or later100+ external-data validation cases using public data from around the world MIT license

Download and install the R source package

For AI Agents | Quick Start | Four Examples | Agent Guide

CSV / Excel / TSV / TXT / RDB -> local cleaning -> files + audit -> agent reasoning.

TidyPanel handles supported table preparation in local R code: 0 model calls, 0 AI inference tokens, no AI API key for the cleaning step. Your agent spends its context on the question, not on rewriting routine cleanup code.

Routine work, handled locally What your agent receives
Supported table layouts and column-name cleanup Cleaned files and complete column metadata
Common numeric formats and declared missing tokens Row/column counts, missing counts and warnings
Optional wide-to-long reshaping Local artifacts and an operation audit
A repeatable command-line workflow One JSON request and one JSON response

Full observations stay in local files. Previews are off by default. Agent instructions, tool calls and content sent to a model can still consume tokens; total savings depend on the workflow. Review results before analysis.

For AI Agents

Find the contract. Run the cleaner. Read only what you need.

After installing the public package, run these commands from the extracted source folder:

Rscript --vanilla exec/tidypanel.R describe
Rscript --vanilla exec/tidypanel.R clean --request request.json

describe returns the tool name tidypanel.clean, the request schema, exit codes and four fictional example request templates. Pick a template from tool.examples, choose a new output directory, then submit its request object. For caller data, declare missing markers from that source instead of guessing.

These are discoverable documentation entry points, not a hosted API, MCP server or automatic registration with every agent.

Validation Coverage

Public Data From Around The World. Tested Beyond Toy Examples.

100+ 5 0
External-data validation cases Data domains AI inference tokens for local cleaning

Real-world tables, not just tidy demos. Validation draws on public data from international organizations, national data portals and scientific services. It checks supported cleaning behavior and clear rejection of unsupported layouts.

Domain Tables checked
Economics and public statistics Prices, exchange rates and population series
Weather and environment Station observations and climate tables
Science and health Measurements and aggregate indicators
Transport and geography Counts, schedules and coded reference tables
Energy and industry Consumption and production observations

Counting rule: 100+ refers to external-data validation cases, not 100+ independent databases or a guarantee that every export from a source is supported. Different files, time windows and derived checks do not establish additional independent databases. The local cleaning step makes no model calls; agent messages and shared results can still consume tokens.

Four small fictional CSVs are included for a first run. Development datasets are not distributed.

Download And Install

R 4.1 or later, with a writable R library. No AI service is required.

  1. On this GitHub repository, select Code > Download ZIP.
  2. Extract the ZIP and set source_dir to the folder containing DESCRIPTION.
  3. Run the code below in R. It does not open a file picker or mirror-selection menu.
install.packages(
  c("dplyr", "stringr", "stringi", "readxl", "jsonlite"),
  repos = "https://cloud.r-project.org"
)
source_dir <- "path/to/extracted/TidyPanel"
install.packages(source_dir, repos = NULL, type = "source")

An agent can also install the source directory through Rscript, after the dependencies above are installed. R must be available on its PATH:

Rscript --vanilla -e "install.packages(commandArgs(TRUE)[1], repos = NULL, type = 'source')" "path/to/extracted/TidyPanel"
Prefer a file picker for manual installation?

After installing the dependencies, run this in an interactive R session and select DESCRIPTION in the extracted package folder:

install.packages(dirname(file.choose()), repos = NULL, type = "source")
Already have a built R source archive?
install.packages("TidyPanel_0.2.133.tar.gz", repos = NULL, type = "source")

A built source archive also includes the example guide:

vignette("TidyPanel_introduction", package = "TidyPanel")

Quick Start

Run a bundled example without preparing your own file:

library(TidyPanel)

input <- system.file(
  "extdata", "demo_03_subtotals.csv",
  package = "TidyPanel", mustWork = TRUE
)
result <- read_messy_panel(input, auto_pivot = TRUE, return_audit = TRUE)

result$data
result$audit

The total row is removed and the two yearly columns become a four-row long table. Replace input with your own file path; omit auto_pivot = TRUE to keep wide columns.

Agent Workflow

One request -> one JSON response -> complete local result files.

Call the cleaner, then reason over the result. TidyPanel performs the supported cleanup locally without invoking a model; your agent chooses the task and how much of the resulting data to read. Preview is off by default, so the tool response does not automatically echo the input table into model context.

Install the matching v0.2.133 public source package first. The launcher checks the installed edition and version before loading it. This is a local process interface, not an MCP server or automatic global tool registration.

From the extracted source folder, an agent can inspect the interface and run a job:

Rscript --vanilla exec/tidypanel.R describe
Rscript --vanilla exec/tidypanel.R clean --request request.json

Send a UTF-8 JSON object in the request file, or use clean --request - and send the same object on standard input, then close that input stream:

{
  "input": "inst/extdata/demo_03_subtotals.csv",
  "output_dir": "../tidypanel-job-001",
  "na_strings": [""],
  "auto_pivot": true
}

This request uses one of the four fictional demos, not an additional dataset. For another source, declare its own missing markers and choose its intended Excel sheet or ZIP archive_member. All paths are relative to the process working directory, including requests read from files.

Interface Contract
Input One local tabular file, a new output directory and explicit na_strings. Unknown keys, URLs and network-share paths are rejected.
Discovery describe returns the schema, limits, result fields, exit codes and four fictional example request templates.
Output Exactly one JSON response on stdout. Reader chatter goes to stderr; bounded warning messages are retained in the response.
Full results artifacts names the cleaned RDS, convenience CSV, audit CSV and complete column metadata JSON.
Preview Off by default. Opt in with preview_rows; limits are 10 rows, 20 columns and 160 characters per displayed label or value.
Failure A nonzero exit code and status: "error". Do not treat partial output directories as successful jobs.

On success, status is "ok", with row_count, column_count, artifacts, preview, audit_preview and warning fields. Start with the counts and artifact paths; read only the data needed for the question. A preview, path, warning or audit can contain sensitive caller information. Redact it or keep it local unless sharing with the selected model is authorized.

For a first job, select an entry from tool.examples in the describe response and use its request object. Each template names an installed demo file and its missing markers; the subtotal example also enables pivoting. Choose a new output_dir before running or repeating a job. describe does not read caller tables or write files. Its installed demo paths can identify your local environment, so keep the response local unless sharing it is authorized.

Installed entry point, limits and exit codes

To locate the launcher after installation, without relying on the source folder:

system.file("exec", "tidypanel.R", package = "TidyPanel", mustWork = TRUE)

Pass that returned path to Rscript --vanilla. R callers can alternatively use TidyPanel::tidypanel_cli(); it returns an exit status without terminating R. The bundled launcher exits with that status.

Exit code Meaning
0 Completed successfully; review any warnings.
2 Invalid arguments, malformed JSON or invalid request fields.
3 Missing dependencies or a nonmatching installed edition/version.
4 Cleaning failed, an archive has unsafe paths, or output is not one table.
5 Output directory unavailable or artifact writing failed.

Requests are limited to 65,536 bytes. Only options listed by describe are accepted. na_strings: [] declares no caller missing tokens. sheet selects one name or positive index, not "ALL". auto_pivot defaults to false. preview_rows defaults to 0; preview_columns defaults to 8 and preview_chars to 80. Preview cells are text, with null for missing positions; their column indices map to the complete schema in the metadata artifact.

The output directory must be new, have an existing parent and stay outside the installed package and launcher package folders. Existing files are never overwritten by the job interface. A failed write may leave a partial new directory; retain it for inspection and choose a new directory for a retry. The interface inherits the caller's filesystem permissions and is not a sandbox.

The RDS artifact retains the full cleaned R table and audit for a compatible trusted R environment. CSV is a convenience export with empty fields for missing values; it cannot preserve every type or distinguish every literal empty string from missingness. Consult column metadata before cross-language reading. The audit records supported operations and may be empty; it is not a complete cell-by-cell history. Preserve the source and review the result before analysis. This interface adds no new cleaning rules, unit inference or analysis model, and sends no files to an AI service.

Four Small Examples

CSV Try it for
Numbers Grouped amounts, negatives and leading-zero codes
Dates Calendar dates and missing readings
Subtotals Total-row removal and wide-to-long reshaping
Missing Tokens Explicit missing-marker selection without changing genuine zeros

Exactly four fictional CSVs, each with three input records. No personal records, development datasets or regression fixtures are distributed. See the example guide for all four workflows.

Cleaning notes and supported-layout boundaries

Automatic inference: infer_data_types() is a separate, heuristic operation. It can convert categorical text to logical values and turn nonmatching values into missing values when a threshold is met. A successful run or a stricter threshold is not proof that category meanings were preserved. Use recognized _code or _id names for identifier fields, retain the source, and review output types and missing counts. The Agent job interface uses read_messy_panel(); it does not apply an extra standalone inference pass.

Stable categorical labels in headerless numeric tables and paired free/total measurements are retained. For numeric text reports, provide the observation section as a plain table or data frame rather than including report metadata. Commented delimited headers retain their labels, including whitespace after the comment marker and quoted separators in field names or records. Explicit TAB records remain tab-delimited when labels contain spaces. Explicit two-column field,value tables retain every uniquely named record, including blank trailing values and long text. Hash characters inside delimited fields are treated as data rather than silently truncating the field. Explicit identifier suffixes remain distinct from financial measurement labels. Fields named exactly site preserve literal identifiers, including leading zeros. Fields named ident or ending in _ident retain text codes without filling missing positions. A qualified iso_country column retains literal NA only with at least two distinct other uppercase two-letter code observations. Coded reference catalogs require a separately prepared named observation table. Sectioned observation exports must be supplied as separately prepared named tables. Prefaced measurement exports require a separately prepared named observation table. Prefaced monthly series exports require a separately prepared named observation table. Qualified numeric total-uncertainty measurements are retained, while ordinary subtotal columns remain removable. Related measurement fields such as site_count remain eligible for numeric cleaning. Sampled averages retain their values when time and integer observation-count fields establish a measurement context. Sources with delimited metadata preambles must be supplied as plain observation tables. Headerless whitespace records with quoted labels must be converted to CSV or supplied as an explicitly named data frame; unsupported mixed separators fail clearly rather than silently combining fields. Numeric-context NaN placeholders become missing values without changing pure text labels. For scientific exports with a separate unit descriptor row, remove that row before supplying a named observation table. Customize na_strings when literal codes overlap with default missing markers. Numeric missing markers can also overlap with genuine measurements. Choose na_strings from the source's definitions; retain numeric markers that the source uses as observations by excluding them from that list. Hyphenated English month dates are converted only when every observed date is valid, independently of the time locale. Explicit date identifier fields remain text. Qualified population income measures retain their labels. Sparse country-year records remain present even when their measurements are missing. Explicitly qualified total-energy measurements are not treated as subtotal columns. Missing values in explicitly labelled calendar fields are not filled from neighboring records. Lowercase na is recognized in numeric contexts while pure text labels and recognized identifier fields remain literal. Exclude it from a custom na_strings when the marker has a different meaning in your table.

Explicit mixed-case ID suffixes remain identifier text and use _id in cleaned names. Sparse digit identifiers are not treated as prose footnotes. Type inference retains literal Trace observations unless the caller explicitly includes them in na_strings; it does not assign an invented exact measurement. Clock-bearing values and date suffixes remain text during type inference; fractional serial days are not reduced to dates. Explicit flag fields retain their literal tokens. Unit-qualified total rain, snow, and precipitation fields and their corresponding flags are retained as measurements. Slash-dated series exports require a separately prepared named observation table; unsupported metadata layouts fail clearly instead of merging labels. In both the reader and standalone inference, compact mixed digit/letter fields with leading-zero evidence stay text rather than losing letters or padding. This lexical protection does not validate code meanings or arbitrary punctuation. Existing compact-month calendar handling keeps precedence over this code guard. Scientific notation and formatted decimal amounts retain numeric conversion. Unsupported name-first reference exports require a named table with an identifier column first. Sectioned control-record reports require a separately prepared named observation table; the simplified public reader does not combine their sections.

Large-table subtotal checks retain the existing row-selection rules while avoiding repeated per-record matching.

Row detection classifies bounded chunks rather than retaining several full-table logical masks. Auxiliary header counters are also limited to the current row chunk. This reduces temporary storage without changing supported cleaning rules. The source table and cleaned output still need to fit in memory; this is not an out-of-core reader.

Contact

Tony Lu | xulunt123@gmail.com