datapitfalls Scanner or MCP: Which Fits Your Data Review?
Your report looks clean, but its date range was picked by hand and its SQL quietly drops nulls. Should datapitfalls review the finished file or sit inside your AI chat?
- Use the scanner to request an audit of an artifact; use MCP to bring the rules into an existing AI conversation.
- datapitfalls reviews data reasoning as well as chart presentation, using rules based on Ben Jones's Avoiding Data Pitfalls.
- Results distinguish observable problems from conditions that depend on data the reviewer has not seen.
The scanner calls a model; MCP provides the rules
For a report you are ready to submit for checking, the datapitfalls scanner returns an audit. For work still being developed with an AI assistant, MCP brings the review rules into that conversation. The web scanner accepts complete PDF reports; the CLI supports chart images and code files, including checks in a CI pipeline.
The scanner sends the input and relevant rules to the Claude API. Running the CLI requires an Anthropic API key. The MCP server instead supplies rules to the AI handling the conversation; that AI performs the review.
| Route | Job | Data movement |
|---|---|---|
| CLI or web scanner | Request an audit of an input | The input goes to the Claude API. |
| MCP rule tools | Retrieve checklists, rules and search results | Requests contain rule IDs, keywords or work types. |
| Optional MCP review prompt | Return a prompt containing supplied text | Text passed as an argument also reaches the connector. |
The MCP implementation exposes four tools: get_checklist, get_pitfall, list_pitfalls and search_pitfalls. A checklist request can specify SQL or a dashboard, for example. The server itself makes no Anthropic API call.
The project's privacy notice identifies an exception to ordinary rule retrieval: text supplied to the optional review_for_pitfalls prompt is sent to the connector and returned inside the prompt. It also says the web scanner does not store submitted work, although it sends that work to the API for inspection. A keyless MCP server does not imply an offline AI conversation.
A well-formatted chart can still support a bad conclusion
Recently shared on GeekNews, datapitfalls extends the review beyond axes and colors. Its catalog covers how a question is framed, data is collected and transformed, and results are analyzed and communicated.
The README illustrates the problem with a selectively chosen time window and a query that silently drops nulls. Rendering a chart correctly or successfully executing SQL does not resolve those reasoning problems. The tool accepts work such as chart images, code, prose and reports for review against its catalog.
An observed problem and an unverified condition need different follow-up
The report schema labels findings as active when evident from the artifact, or latent when they depend on unseen data. Findings include evidence, an explanation, confidence and remediation information.
This distinction affects automation. The CLI's --ci gate blocks on active warnings or errors. Latent findings do not block the pipeline. They identify conditions to investigate rather than established failures.
An incomplete model response is a separate state. The 0.10.0 release notes describe replacing empty results with an error when a response is truncated or fails to produce a usable report. A caller should distinguish a completed scan from a failed attempt before interpreting the finding count.
Match the route to the review you are already doing
MCP fits a conversation in which an AI is already helping with a query or report. The scanner fits a workflow that submits files and consumes a structured report or CI decision. That choice follows from where the review runs and how the result is consumed.
Reuse terms differ across components. The current repository notice keeps the code under MIT and places the rule taxonomy under CC BY-NC 4.0. The rights holder explicitly permits checking your own work, including at a job, while requiring a separate commercial license for selling the rules or incorporating them into a paid product or course. Previously released copies through 0.11.0 retain their MIT terms.
A useful review ends with evidence to examine: compare an active finding with the source material, and obtain the missing data needed to resolve a latent condition.