fix: a pattern is a string, IS is a keyword in any case, YES and NO are names

docs/filtering.md

@@ -74,7 +74,36 @@ binds tighter than `OR`.
 | Grouping | `( ... )` |
 
 **Literals**: single-quoted strings (`''` escapes a quote), integers, floats,
-and booleans (`TRUE`/`FALSE`/`YES`/`NO`).
+and the two booleans SQL has, `TRUE` and `FALSE`. `YES` and `NO` are not
+literals: PostgreSQL reads them as names, and so does this parser (a field
+may be called `yes`).
+
+### Patterns
+
+What the pattern operators take is PostgreSQL's, from its manual,
+[9.7. Pattern Matching](https://www.postgresql.org/docs/current/functions-matching.html):
+
+```text
+string LIKE pattern [ESCAPE escape-character]
+string SIMILAR TO pattern [ESCAPE escape-character]
+text ~ text → boolean
+```
+
+`ILIKE` is `LIKE` without regard to case, and `~~`, `~~*`, `!~~` and `!~~*`
+are the operators PostgreSQL translates `LIKE`, `ILIKE` and their negations
+to.
+
+- **A pattern is a string literal**, for every one of them: `name LIKE 'x%'`,
+  `name ~~* 'X%'`, `name SIMILAR TO '(x|y)%'`, `name ~ '^x'`. A number or a
+  boolean there is refused (`name LIKE 1`, `name SIMILAR TO TRUE`):
+  PostgreSQL has no `~~` that takes one, whatever the column's type is, and
+  answers "operator does not exist". A number written as a string is a
+  pattern: `name LIKE '1'`.
+- **The `ESCAPE` clause is not supported**; `name LIKE 'x!%' ESCAPE '!'` is
+  refused. A pattern is written with the default escape character, the
+  backslash, which a string literal keeps as it is: `name LIKE '100\%'`.
+- **What is in the pattern is not looked at**: see
+  "Patterns are not compiled", under [AST nodes](#ast-nodes).
 
 ### Numbers
 
@@ -115,9 +144,10 @@ nothing else.
   this. **Bound the length of what you hand to `Parse`** as well: the parser
   reads its whole input before it answers.
 
-**The parser does not know a column's type.** `name LIKE -1`, `id = 'x'` and a
+**The parser does not know a column's type.** `name = -1`, `id = 'x'` and a
 bare `-1` are accepted, and PostgreSQL refuses each for its type, not for its
-syntax. What this page promises is that an accepted expression is not a
+syntax. (A pattern is the exception, because it has one type for every
+column: see [Patterns](#patterns).) What this page promises is that an accepted expression is not a
 syntax error; a caller still has to answer for a value of the wrong type.
 
 ## Nested (dot-notation) field names
@@ -154,12 +184,12 @@ flowchart LR
 ```
 
 `IS` is the only way in. `id DISTINCT FROM 1` and `id NOT DISTINCT FROM 1`,
-without `IS`, are not SQL and are refused; so is `active IS YES`. `YES` and
-`NO` are this parser's spellings of a boolean **literal** and not truth values
-of the `IS` test. They are not PostgreSQL's either: `active = YES` parses to a
-`LiteralNode` whose `Value` is `true` and which renders as `true`, but the
-text `YES` spliced into SQL is read as a column name. Write `TRUE`/`FALSE`
-where the input itself is spliced.
+without `IS`, are not SQL and are refused; so is `active IS YES`: the truth
+values of the `IS` test are `TRUE`, `FALSE` and `UNKNOWN`.
+
+`IS` is the keyword in any case, as every keyword is: `id is null`,
+`id Is Not Null` and `active is true` are the same tests as their upper-case
+spellings.
 
 ## Worked examples
 

docs/migration.md

@@ -1,5 +1,39 @@
 # Migration Guide
 
+## v1.0.3 → the next release
+
+Three places where the filter parser and PostgreSQL disagreed about an
+input. Each row was run against PostgreSQL 18; what the pattern operators
+take is from its manual,
+[9.7. Pattern Matching](https://www.postgresql.org/docs/current/functions-matching.html).
+
+**Refused now, and accepted before:**
+
+| Input | v1.0.3 | Now | PostgreSQL 18 |
+| --- | --- | --- | --- |
+| a number or a boolean as a pattern: `name LIKE 1`, `name NOT ILIKE -1`, `name ~~ 1`, `name SIMILAR TO 1`, `name LIKE TRUE` | ✅ accepted | ❌ error: `expected string pattern for operator LIKE, got LITERAL` | `operator does not exist: text ~~ integer`, for every column type |
+| `YES` and `NO` as boolean literals: `active = YES`, `active <> no`, `active IN (yes, no)` | ✅ accepted, as `true` and `false` | ❌ error | a name: the comparison is with a column called `yes`, or `column "yes" does not exist` |
+
+**Accepted now, and refused before:**
+
+| Input | v1.0.3 | Now | PostgreSQL 18 |
+| --- | --- | --- | --- |
+| `IS` in lower case: `id is null`, `id is not null`, `id is distinct from 1`, `active is true` | ❌ error (`Is`, `iS` and `IS` were accepted) | ✅ accepted | runs |
+| a field called `yes` or `no`, when it is in the allow-list: `yes = 1` | ❌ error | ✅ accepted | runs |
+
+The POSIX operators (`~`, `~*`, `!~`, `!~*`) already took a string and
+nothing else, and the `ESCAPE` clause is still not part of the grammar.
+
+**For callers that walk the tree.** A `LiteralNode` of `Kind`
+`reflect.Bool` now comes from `TRUE` or `FALSE` only, and the right side of a
+`LIKE`/`ILIKE` `BinaryOperatorNode` and a `SimilarToNode`'s `Pattern` are
+always a string `LiteralNode`. `Lexer` users: `is` in lower case is the `IS`
+token like every other spelling, and `YES`/`NO` are `IDENTIFIER` tokens.
+
+**Migration:** write `TRUE` and `FALSE`; quote a pattern (`name LIKE '1'`).
+A field that was called `is` cannot be filtered on, as it cannot be named
+unquoted in PostgreSQL.
+
 ## v1.0.2 → v1.0.3
 
 `v1.0.3` makes the filter parser refuse what PostgreSQL refuses as a syntax

filter_lexer.go

@@ -74,43 +74,43 @@ func (l *Lexer) Parse() {
 		case scanner.EOF:
 			tok = TokenEOF
 		case scanner.Ident:
-			// Special handling for the "is" token in the test cases
-			// In the test cases, "is" is expected to be an identifier, not a keyword
-			if lit == "is" {
+			// A keyword is a keyword in any case, IS included: PostgreSQL
+			// folds an unquoted word, and "id is null" is the same test as
+			// "id IS NULL". The lower-case word used to be kept back as an
+			// identifier, for a test that tokenized English.
+			switch asciiUpper(lit) {
+			case "AND":
+				tok = TokenOperatorAnd
+			case "OR":
+				tok = TokenOperatorOr
+			case "LIKE":
+				tok = TokenOperatorLike
+			case "ILIKE":
+				tok = TokenOperatorILike
+			case "IN":
+				tok = TokenOperatorIn
+			case "BETWEEN":
+				tok = TokenOperatorBetween
+			case "DISTINCT":
+				tok = TokenOperatorDistinct
+			case "SIMILAR":
+				tok = TokenOperatorSimilarTo // Treat SIMILAR as its own token
+			case "TO":
+				tok = TokenIdentifier // Treat TO as a generic identifier for now, parser will handle context
+			case "IS":
+				tok = TokenOperatorIsNull // Treat IS as its own token
+			case "NOT":
+				tok = TokenOperatorNot // Treat NOT as its own token
+			case "TRUE", "FALSE":
+				// The two boolean literals SQL has. YES and NO were read as
+				// booleans too, and are names to PostgreSQL: "active = YES"
+				// is a comparison with a column called yes.
+				tok = TokenBoolean
+			case "NULL":
+				tok = TokenIdentifier // Treat NULL as an identifier
+			default:
+				// Check if it's a potential field name or other identifier
 				tok = TokenIdentifier
-			} else {
-				upperLit := asciiUpper(lit)
-				switch upperLit {
-				case "AND":
-					tok = TokenOperatorAnd
-				case "OR":
-					tok = TokenOperatorOr
-				case "LIKE":
-					tok = TokenOperatorLike
-				case "ILIKE":
-					tok = TokenOperatorILike
-				case "IN":
-					tok = TokenOperatorIn
-				case "BETWEEN":
-					tok = TokenOperatorBetween
-				case "DISTINCT":
-					tok = TokenOperatorDistinct
-				case "SIMILAR":
-					tok = TokenOperatorSimilarTo // Treat SIMILAR as its own token
-				case "TO":
-					tok = TokenIdentifier // Treat TO as a generic identifier for now, parser will handle context
-				case "IS":
-					tok = TokenOperatorIsNull // Treat IS as its own token
-				case "NOT":
-					tok = TokenOperatorNot // Treat NOT as its own token
-				case "TRUE", "FALSE", "YES", "NO":
-					tok = TokenBoolean
-				case "NULL":
-					tok = TokenIdentifier // Treat NULL as an identifier
-				default:
-					// Check if it's a potential field name or other identifier
-					tok = TokenIdentifier
-				}
 			}
 		case scanner.Int, scanner.Float:
 			tok, lit = l.numberToken(scanTok, lit)

filter_lexer_test.go

@@ -56,7 +56,7 @@ func TestLexer_Navigation(t *testing.T) {
 				{Pos: scanner.Position{Line: 1, Column: 1}, Type: TokenIdentifier, Value: "comment"},
 				{Pos: scanner.Position{Line: 1, Column: 9}, Type: TokenOperatorEqual, Value: "="},
 				{Pos: scanner.Position{Line: 1, Column: 11}, Type: TokenIdentifier, Value: "This"},
-				{Pos: scanner.Position{Line: 1, Column: 16}, Type: TokenIdentifier, Value: "is"},
+				{Pos: scanner.Position{Line: 1, Column: 16}, Type: TokenOperatorIsNull, Value: "is"}, // IS, in any case
 				{Pos: scanner.Position{Line: 1, Column: 19}, Type: TokenIdentifier, Value: "a"},
 				{Pos: scanner.Position{Line: 1, Column: 21}, Type: TokenIdentifier, Value: "bad"},
 				{Pos: scanner.Position{Line: 1, Column: 25}, Type: TokenIdentifier, Value: "string"},
@@ -71,7 +71,7 @@ func TestLexer_Navigation(t *testing.T) {
 				{Pos: scanner.Position{Line: 1, Column: 1}, Type: TokenIdentifier, Value: "comment"},
 				{Pos: scanner.Position{Line: 1, Column: 9}, Type: TokenOperatorEqual, Value: "="},
 				{Pos: scanner.Position{Line: 1, Column: 11}, Type: TokenIdentifier, Value: "This"},
-				{Pos: scanner.Position{Line: 1, Column: 16}, Type: TokenIdentifier, Value: "is"},
+				{Pos: scanner.Position{Line: 1, Column: 16}, Type: TokenOperatorIsNull, Value: "is"}, // IS, in any case
 				{Pos: scanner.Position{Line: 1, Column: 19}, Type: TokenIdentifier, Value: "a"},
 				{Pos: scanner.Position{Line: 1, Column: 21}, Type: TokenIdentifier, Value: "bad"},
 				{Pos: scanner.Position{Line: 1, Column: 25}, Type: TokenIdentifier, Value: "string"},

filter_nodes_string_test.go

@@ -147,7 +147,7 @@ func TestParsePrimaryLiteralKinds(t *testing.T) {
 		"score = 3.14":   reflect.Float64,
 		"active = true":  reflect.Bool,
 		"active = false": reflect.Bool,
-		"active = YES":   reflect.Bool,
+		"active = TRUE":  reflect.Bool,
 	}
 	for in, wantKind := range cases {
 		t.Run(in, func(t *testing.T) {

filter_parser.go

@@ -428,7 +428,13 @@ func (p *filterParseState) parseSimilarToOperator(field Node, isNot bool) Node {
 		return field // Return field on error
 	}
 	p.nextToken() // Consume TO
-	pattern := p.parsePrimary()
+
+	operator := TokenOperatorSimilarTo
+	if isNot {
+		operator = TokenOperatorNotSimilarTo
+	}
+
+	pattern := p.parsePattern(operator)
 	return &SimilarToNode{
 		baseNode: baseNode{pos: pos},
 		Field:    field,
@@ -437,13 +443,34 @@ func (p *filterParseState) parseSimilarToOperator(field Node, isNot bool) Node {
 	}
 }
 
+// parsePattern reads the right side of LIKE, ILIKE and SIMILAR TO, negated or
+// not, and of the ~~ operators that are their other spelling.
+//
+// PostgreSQL writes them "string LIKE pattern [ESCAPE escape-character]" and
+// "string SIMILAR TO pattern", and a pattern is a string: with a number or a
+// boolean there it answers "operator does not exist", whatever the column's
+// type is, because no ~~ takes one. So a literal of another kind is refused
+// here, as it already was for the POSIX operators (~, ~*, !~, !~*).
+//
+// The ESCAPE clause is not part of this grammar: a pattern is written with
+// the default escape character, the backslash.
+func (p *filterParseState) parsePattern(operator TokenType) Node {
+	pattern := p.parsePrimary()
+
+	if literal, ok := pattern.(*LiteralNode); !ok || literal.Kind != reflect.String {
+		p.addError(&QFVFilterError{Message: fmt.Sprintf("expected string pattern for operator %s, got %s", operator, pattern.Type())})
+	}
+
+	return pattern
+}
+
 // parseLikeOperator parses a LIKE/ILIKE/NOT LIKE/NOT ILIKE operator.
 // Expects the current token to be the pattern after the operator was consumed.
 // operator is one of TokenOperatorLike, TokenOperatorILike, TokenOperatorNotLike,
 // or TokenOperatorNotILike.
 func (p *filterParseState) parseLikeOperator(field Node, operator TokenType) Node {
 	pos := p.lexer.Current().Pos // Use position of the operator token (already consumed)
-	pattern := p.parsePrimary()
+	pattern := p.parsePattern(operator)
 	return &BinaryOperatorNode{
 		baseNode: baseNode{pos: pos},
 		Left:     field,
@@ -572,8 +599,7 @@ func (p *filterParseState) parseIsPredicate(field Node) Node {
 		return p.parseDistinctOperator(field, isNot)
 
 	case p.currentToken.Type == TokenBoolean && (p.isKeyword("TRUE") || p.isKeyword("FALSE")):
-		// IS [NOT] TRUE | FALSE. YES and NO are lexed as booleans too, and are
-		// literals only: SQL has no "IS YES".
+		// IS [NOT] TRUE | FALSE.
 		p.requireOp(OpBooleanTest)
 		truth := BooleanTrue
 		if p.isKeyword("FALSE") {
@@ -665,7 +691,7 @@ func (p *filterParseState) parsePrimary() Node {
 		return node
 
 	case TokenBoolean:
-		val := asciiUpper(p.currentToken.Value) == "TRUE" || asciiUpper(p.currentToken.Value) == "YES"
+		val := asciiUpper(p.currentToken.Value) == "TRUE"
 		node := &LiteralNode{
 			baseNode: baseNode{pos: p.currentToken.Pos},
 			Value:    val,