Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified .github/demos/check-fix.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ SQLClean in other configurations, on the same files:

| SQLClean command | Wall time |
|---|---:|
| `sqlclean check --select SQC` (every built-in Rule) | 1.33 s |
| `sqlclean check --select SQLCLEAN` (every built-in Rule) | 1.33 s |
| `sqlclean check` plus three custom Python Rules | 2.27 s |
| `sqlclean check` rerun with nothing changed (result cache) | 0.23 s |

Expand Down Expand Up @@ -114,7 +114,7 @@ def lowered(operand: ast.Expression) -> bool:


@rule(
code="XSQCTEAM001",
code="XSQLCLEANTEAM001",
message="Email comparisons must be case-insensitive",
remediation="Compare LOWER(email) with LOWER(...) instead.",
enabled_by_default=True,
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/linters/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ and peak memory with `/usr/bin/time`, and every lint command writes JSON finding
| Variant | Command |
|---|---|
| `sqlclean` | `sqlclean check --output-format json` (13 default Rules) |
| `sqlclean_all` | `sqlclean check --select SQC --output-format json` (all 45 Rules) |
| `sqlclean_all` | `sqlclean check --select SQLCLEAN --output-format json` (all 45 Rules) |
| `sqlclean_custom` | `sqlclean check --config custom.toml --output-format json` (default Rules and the three Python Rules in `rules/`) |
| `sqlclean_format` | `sqlclean format --check` |
| `sqruff` | `sqruff lint -f json .` |
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/linters/bench.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ trap 'rm -rf "$WORK"' EXIT

declare -A CMD
CMD[sqlclean]="$SQLCLEAN check --output-format json"
CMD[sqlclean_all]="$SQLCLEAN check --select SQC --output-format json"
CMD[sqlclean_all]="$SQLCLEAN check --select SQLCLEAN --output-format json"
# The default Rules plus the three Python Rules in rules/, copied into the work directory.
CMD[sqlclean_custom]="$SQLCLEAN check --config custom.toml --output-format json"
# sqruff's and SQLFluff's default rules include layout; SQLClean checks layout with its formatter.
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/linters/rules/layers.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@


@rule(
code="XSQCLAYER001",
code="XSQLCLEANLAYER001",
message="Mart reads a raw source",
remediation="Read the staging model instead.",
enabled_by_default=True,
Expand All @@ -26,7 +26,7 @@ def marts_read_staging(*, statement: Statement, ctx: RuleContext) -> list[Findin


@rule(
code="XSQCLAYER002",
code="XSQLCLEANLAYER002",
message="Staging model reads another model",
remediation="Staging models read only raw sources and seeds.",
enabled_by_default=True,
Expand All @@ -46,7 +46,7 @@ def staging_reads_sources(*, statement: Statement, ctx: RuleContext) -> list[Fin


@rule(
code="XSQCDET001",
code="XSQLCLEANDET001",
message="Model calls a nondeterministic function",
remediation="Pass the time or seed in as a column so reruns give the same rows.",
enabled_by_default=True,
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/linters/summarize.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def count_findings(variant: str, out: str) -> tuple[int, int]:
data = json.loads(out)
if variant.startswith("sqlclean"):
codes = [finding["code"] for finding in data["findings"]]
return len(codes), codes.count("SQC999")
return len(codes), codes.count("SQLCLEAN999")
if variant == "sqruff":
codes = [finding.get("code") for findings in data.values() for finding in findings]
return len(codes), sum(1 for code in codes if code in (None, "PRS"))
Expand Down
10 changes: 5 additions & 5 deletions crates/sqlclean-cli/skill/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: sqlclean
description: ALWAYS load this skill when linting, fixing or formatting SQL with SQLClean, editing sqlclean.toml, writing or testing custom Rules (Python @rule functions with XSQC codes), recording Rule exceptions, or reading SQLClean findings (SQC codes, SQC999 parse errors) in CI, JSON or SARIF output.
description: ALWAYS load this skill when linting, fixing or formatting SQL with SQLClean, editing sqlclean.toml, writing or testing custom Rules (Python @rule functions with XSQLCLEAN codes), recording Rule exceptions, or reading SQLClean findings (SQLCLEAN codes, SQLCLEAN999 parse errors) in CI, JSON or SARIF output.
---
<!-- generated-by: sqlclean skills -->

Expand All @@ -17,7 +17,7 @@ engine extracts from each statement.
`[rules]`). Paths in it are relative to its directory.
2. **Check.** `sqlclean check [PATHS]` exits 0 when clean, 1 with findings, 2 on errors. Use
`--output-format json` when you need to read findings programmatically.
3. **Explain.** `sqlclean rules SQC001` shows a Rule's message, fix safety and remediation.
3. **Explain.** `sqlclean rules SQLCLEAN001` shows a Rule's message, fix safety and remediation.
4. **Fix.** `sqlclean fix --diff` previews safe fixes plus formatting; `sqlclean fix` writes them.
Fixes marked unsafe can change query results, such as `= NULL` becoming `IS NULL`. Never pass
`--unsafe-fixes` without the user's say-so.
Expand All @@ -31,16 +31,16 @@ engine extracts from each statement.
`[[rules.rule_exceptions]]` entry with the Rule, path and a `reason`. Use
`[[rules.rule_ignores]]` only for whole paths a Rule does not apply to.
- An exception that no longer matches a finding is reported as stale; remove it.
- `SQC999` is a parse error. It cannot be ignored: fix the SQL or set the right `dialect`,
- `SQLCLEAN999` is a parse error. It cannot be ignored: fix the SQL or set the right `dialect`,
globally or per path under `[[dialects]]`.
- `--select` and `--ignore` take codes or prefixes such as `SQC001` or `SQC`.
- `--select` and `--ignore` take codes or prefixes such as `SQLCLEAN001` or `SQLCLEAN`.
- Results are cached in `.sqlclean/cache/`. Use `--no-cache` only when debugging a custom Rule
that imports helpers from outside its `[rules] custom` paths.

## Custom Rules

- A custom Rule is a function decorated with `@rule` from `sqlclean`, in a file under a path
listed in `[rules] custom`. Its code is `XSQC`, optional capital letters, then three digits.
listed in `[rules] custom`. Its code is `XSQLCLEAN`, optional capital letters, then three digits.
- The first keyword parameter names the subject: `statement: Statement`, `file: SqlFile` or
`repository: Repository`. The second is `ctx: RuleContext`.
- Return `ctx.finding(...)` values. Fixes are lists of `ctx.replace`, `ctx.insert_before`,
Expand Down
2 changes: 1 addition & 1 deletion crates/sqlclean-cli/skill/references/docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ SQLFluff ran once.
| Command | Wall time | CPU time | Peak memory | vs SQLClean defaults |
|---|---:|---:|---:|---:|
| `sqlclean check` (13 default Rules) | **0.81 s** | 2.5 s | 48 MiB | 1× |
| `sqlclean check --select SQC` (all 45 Rules) | **1.33 s** | 3.8 s | 140 MiB | 1.6× |
| `sqlclean check --select SQLCLEAN` (all 45 Rules) | **1.33 s** | 3.8 s | 140 MiB | 1.6× |
| `sqlclean check` with three [custom Rules](#custom-rules) | **2.27 s** | 6.9 s | 182 MiB | 2.8× |
| `sqlclean format --check` | **3.03 s** | 11.5 s | 49 MiB | 3.7× |
| `sqruff lint` | 41.0 s | 114 s | 235 MiB | 51× |
Expand Down
18 changes: 9 additions & 9 deletions crates/sqlclean-cli/skill/references/docs/cli/check.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ sqlclean check [OPTIONS] [PATHS]...
```bash
sqlclean check # every matching file under the configuration root
sqlclean check models/orders.sql reports/ # only these files and directories
sqlclean check --select SQC001,SQC004 # only these Rules
sqlclean check --select SQLCLEAN001,SQLCLEAN004 # only these Rules
sqlclean check --output-format sarif > sqlclean.sarif
```

Expand All @@ -50,28 +50,28 @@ when `include` does not match it, unless `exclude` does. See
| `--config <PATH>` | Use this configuration file instead of the nearest `sqlclean.toml` |

`--select` and `--ignore` take a comma-separated list, or the flag repeated. A value is an exact
code such as `SQC001` or a prefix such as `SQC` or `XSQCNAME`. A value that matches no Rule is an
code such as `SQLCLEAN001` or a prefix such as `SQLCLEAN` or `XSQLCLEANNAME`. A value that matches no Rule is an
error. See [Selecting Rules](../rules/overview.md#selecting-rules).

## Text output

```text
error[SQC004]: Row selection is nondeterministic
error[SQLCLEAN004]: Row selection is nondeterministic
--> models/customers.sql:6:1
|
6 | LIMIT 10;
| ^^^^^
= help: Add ORDER BY with a deterministic tie-breaker before LIMIT or OFFSET.

error[SQC001]: Comparison with NULL is never true
error[SQLCLEAN001]: Comparison with NULL is never true
--> models/orders.sql:8:22
|
8 | WHERE o.cancelled_at = NULL
| ^
= help: Use IS NULL or IS NOT NULL when testing for NULL.
= fix: available with `sqlclean fix --unsafe-fixes`

error: stale rule exception: SQC004 in models/orders.sql matches no finding; remove it from sqlclean.toml
error: stale rule exception: SQLCLEAN004 in models/orders.sql matches no finding; remove it from sqlclean.toml

Found 3 problems (1 fixable with `sqlclean fix --unsafe-fixes`).
```
Expand All @@ -92,7 +92,7 @@ and count characters, not bytes.
{
"findings": [
{
"code": "SQC004",
"code": "SQLCLEAN004",
"path": "models/customers.sql",
"line": 6,
"column": 1,
Expand All @@ -103,7 +103,7 @@ and count characters, not bytes.
"fix": null
},
{
"code": "SQC001",
"code": "SQLCLEAN001",
"path": "models/orders.sql",
"line": 8,
"column": 22,
Expand All @@ -116,7 +116,7 @@ and count characters, not bytes.
],
"stale_exceptions": [
{
"rule": "SQC004",
"rule": "SQLCLEAN004",
"path": "models/orders.sql",
"relation": null
}
Expand All @@ -140,7 +140,7 @@ text:

```json
{
"ruleId": "SQC001",
"ruleId": "SQLCLEAN001",
"level": "error",
"message": {
"text": "Comparison with NULL is never true. Use IS NULL or IS NOT NULL when testing for NULL."
Expand Down
6 changes: 3 additions & 3 deletions crates/sqlclean-cli/skill/references/docs/cli/fix.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,14 @@ Files are chosen as for [`sqlclean check`](check.md).
| `--no-cache` | Evaluate every file instead of reusing results for unchanged inputs |
| `--config <PATH>` | Use this configuration file instead of the nearest `sqlclean.toml` |

Only fixes of selected Rules apply. `sqlclean fix --select SQC005` removes unused CTEs and
Only fixes of selected Rules apply. `sqlclean fix --select SQLCLEAN005` removes unused CTEs and
formats the files, and leaves every other finding alone.

## Output

```text
Fixed 1 problem(s) and changed 1 file(s).
error[SQC001]: Comparison with NULL is never true
error[SQLCLEAN001]: Comparison with NULL is never true
--> models/open_orders.sql:7:24
|
7 | WHERE cancelled_at = NULL
Expand Down Expand Up @@ -88,7 +88,7 @@ findings:
formatter cannot format is reported as `<path>: could not format: <reason>` and keeps its fixes.

A file that does not parse gets no fixes and is reported as
[`SQC999`](../rules/overview.md#parse-errors-sqc999).
[`SQLCLEAN999`](../rules/overview.md#parse-errors-sqlclean999).

## Exit codes

Expand Down
28 changes: 14 additions & 14 deletions crates/sqlclean-cli/skill/references/docs/cli/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ sqlclean rules [OPTIONS] [CODE]

```bash
sqlclean rules # list every Rule
sqlclean rules SQC001 # show one Rule
sqlclean rules SQLCLEAN001 # show one Rule
sqlclean rules --json # every Rule as JSON
sqlclean rules --json SQC001 # one Rule as JSON
sqlclean rules --json SQLCLEAN001 # one Rule as JSON
```

## Flags
Expand All @@ -42,14 +42,14 @@ sqlclean rules --json SQC001 # one Rule as JSON
## Listing Rules

```text
SQC001 default Comparison with NULL is never true
SQC002 default Comma-separated sources create an implicit cartesian join
SQC003 default Non-cross join has no meaningful condition
SQC004 default Row selection is nondeterministic
SQLCLEAN001 default Comparison with NULL is never true
SQLCLEAN002 default Comma-separated sources create an implicit cartesian join
SQLCLEAN003 default Non-cross join has no meaningful condition
SQLCLEAN004 default Row selection is nondeterministic
...
SQC045 JSON payload is parsed more than once in one SELECT
XSQCNAME001 default Query reads a temporary table
XSQCSTYLE001 default COUNT(1) should be COUNT(*)
SQLCLEAN045 JSON payload is parsed more than once in one SELECT
XSQLCLEANNAME001 default Query reads a temporary table
XSQLCLEANSTYLE001 default COUNT(1) should be COUNT(*)
```

`default` marks the Rules that run when `[rules] select` is not set. See
Expand All @@ -59,11 +59,11 @@ XSQCSTYLE001 default COUNT(1) should be COUNT(*)
## Showing one Rule

```bash
sqlclean rules SQC001
sqlclean rules SQLCLEAN001
```

```text
SQC001: Comparison with NULL is never true
SQLCLEAN001: Comparison with NULL is never true
Subject: statement
Enabled by default: yes
Fixable: unsafe
Expand All @@ -73,7 +73,7 @@ Remediation: Use IS NULL or IS NOT NULL when testing for NULL.
A custom Rule also shows the Python module that defines it:

```text
XSQCSTYLE001: COUNT(1) should be COUNT(*)
XSQLCLEANSTYLE001: COUNT(1) should be COUNT(*)
Subject: statement
Enabled by default: yes
Defined in: sql_rules.naming
Expand All @@ -86,13 +86,13 @@ Remediation: Write COUNT(*).
## JSON output

```bash
sqlclean rules --json XSQCNAME001
sqlclean rules --json XSQLCLEANNAME001
```

```json
[
{
"code": "XSQCNAME001",
"code": "XSQLCLEANNAME001",
"message": "Query reads a temporary table",
"remediation": "Read from the published table instead.",
"enabled_by_default": true,
Expand Down
32 changes: 16 additions & 16 deletions crates/sqlclean-cli/skill/references/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,25 +31,25 @@ line_width = 100
indent = 4

[rules]
select = ["SQC"] # omit to run the Rules enabled by default
ignore = ["SQC027"]
select = ["SQLCLEAN"] # omit to run the Rules enabled by default
ignore = ["SQLCLEAN027"]
custom = ["sql_rules"] # Python files or directories that define custom Rules

[rules.thresholds]
max_literal_length = 100 # SQC044
max_ranking_order_by = 5 # SQC043
max_literal_length = 100 # SQLCLEAN044
max_ranking_order_by = 5 # SQLCLEAN043

[rules.rule_options.XSQCNAME001]
[rules.rule_options.XSQLCLEANNAME001]
prefixes = ["tmp_", "scratch_"]

[[rules.rule_exceptions]] # one reviewed finding that is allowed
rule = "SQC038"
rule = "SQLCLEAN038"
path = "models/calendar.sql"
relation = "dates"
reason = "Every pair of dates is needed."

[[rules.rule_ignores]] # Rules that do not apply to some paths
rules = ["SQC034"]
rules = ["SQLCLEAN034"]
paths = ["scratch/**"]
reason = "Exploratory queries."
```
Expand All @@ -76,8 +76,8 @@ directory is the root.
| `[rules] select` | Rules enabled by default | Rule codes or prefixes to run |
| `[rules] ignore` | `[]` | Rule codes or prefixes to skip |
| `[rules] custom` | `[]` | Python files or directories that define [custom Rules](custom-rules/overview.md) |
| `[rules.thresholds] max_literal_length` | `100` | The longest string literal `SQC044` allows |
| `[rules.thresholds] max_ranking_order_by` | `5` | The most `ORDER BY` expressions `SQC043` allows in a ranking window |
| `[rules.thresholds] max_literal_length` | `100` | The longest string literal `SQLCLEAN044` allows |
| `[rules.thresholds] max_ranking_order_by` | `5` | The most `ORDER BY` expressions `SQLCLEAN043` allows in a ranking window |
| `[rules.rule_options.<CODE>]` | none | Option values of one custom Rule; see [Options](custom-rules/fixes-and-options.md#options) |
| `[[rules.rule_exceptions]]` | none | One allowed finding; see [Exceptions](rules/overview.md#exceptions) |
| `[[rules.rule_ignores]]` | none | Rules that do not apply to some paths; see [Path ignores](rules/overview.md#path-ignores) |
Expand Down Expand Up @@ -112,15 +112,15 @@ dialect = "tsql"
```

A file that does not parse under its dialect is reported as
[`SQC999`](rules/overview.md#parse-errors-sqc999), with the dialect in the message. Without
[`SQLCLEAN999`](rules/overview.md#parse-errors-sqlclean999), with the dialect in the message. Without
the `[[dialects]]` entry above, this T-SQL file under `reports/` is parsed as `duckdb`:

```sql
SELECT [order id] FROM orders
```

```text
error[SQC999]: SQL could not be parsed as duckdb: Parse error at line 1, column 17: Expected RBracket, got Var ('id') near [SELECT [order id] FROM orders]
error[SQLCLEAN999]: SQL could not be parsed as duckdb: Parse error at line 1, column 17: Expected RBracket, got Var ('id') near [SELECT [order id] FROM orders]
```

Supported dialects, case-insensitive:
Expand Down Expand Up @@ -178,11 +178,11 @@ report any finding.
| `line_width` of `0` | `sqlclean.toml: [format] line_width must be positive` |
| `indent` of `0` or more than `8` | `sqlclean.toml: [format] indent must be between 1 and 8` |
| A threshold of `0` | `sqlclean.toml: [rules.thresholds] max_literal_length and max_ranking_order_by must be positive` |
| An exception or ignore without a reason | `sqlclean.toml: rule exception or ignore for SQC001 needs a reason` |
| An exception with `cte` but no `relation` | `sqlclean.toml: rule exception for SQC038 at models/calendar.sql sets cte without relation` |
| An exception or ignore without a reason | `sqlclean.toml: rule exception or ignore for SQLCLEAN001 needs a reason` |
| An exception with `cte` but no `relation` | `sqlclean.toml: rule exception for SQLCLEAN038 at models/calendar.sql sets cte without relation` |
| A `custom` path that does not exist | `[rules] custom path does not exist: sql_rules` |
| `rule_options` for a built-in Rule | `[rules.rule_options.SQC001]: built-in Rules take no options; set limits in [rules.thresholds]` |
| `rule_options` for a code no custom Rule has | `[rules.rule_options.XSQCNOPE001]: no custom Rule has this code` |
| A selector that matches no Rule | `Rule selector 'SQC9' matches no Rule` |
| `rule_options` for a built-in Rule | `[rules.rule_options.SQLCLEAN001]: built-in Rules take no options; set limits in [rules.thresholds]` |
| `rule_options` for a code no custom Rule has | `[rules.rule_options.XSQLCLEANNOPE001]: no custom Rule has this code` |
| A selector that matches no Rule | `Rule selector 'SQLCLEAN9' matches no Rule` |

Errors are printed as `sqlclean: error: <message>`.
Loading
Loading