Skip to content

Build page URLs 1.5x faster by escaping the query with CGI.escape - #933

Merged
ddnexus merged 1 commit into
ddnexus:devfrom
czj:perf/escape-query-with-cgi
Oct 30, 2026
Merged

ddnexus merged 1 commit into
ddnexus:devfrom
czj:perf/escape-query-with-cgi

Conversation

@czj

@czj czj commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 QueryUtils.escape goes through URI.encode_www_form_component, which dups, re-encodes and gsubs every key and value: on a page_url with a 3-param query, that is about a third of the call. This swaps it for CGI.escape, the C extension behind the CGI.escapeHTML pagy already uses. In absolute terms it saves about a microsecond per URL, so it won't show on a typical page; it adds up in helpers that build several URLs (page_url 1.5x, urls_hash 1.45x, series_nav 1.15x faster).

Both functions return the same bytes for every input except * and ~ (URI keeps * and escapes ~, CGI does the opposite), so strings containing either still go through URI and every URL stays byte-identical. Escaping a binary copy and tagging the result US-ASCII keeps URI's behaviour on invalid byte sequences and non-UTF-8 encodings as well. A differential fuzz over 1.3M inputs (random bytes in 8 encodings, symbols, integers, nil, every Unicode codepoint) found no difference on Ruby 3.3, 4.0 and 4.1; the new specs cover the edge cases plus a seeded 3,500-case version of it.

Benchmarks

Median of 9 runs (7 on 3.3), page 7 of 42 with q=ruby on rails&author=alice, master vs this branch:

Ruby 4.0.9 no JIT YJIT objects
page_url 3.68 → 2.43 µs (1.51x) 2.61 → 1.65 µs (1.59x) 37 → 32
urls_hash 6.95 → 4.85 µs (1.43x) 5.39 → 3.54 µs (1.52x) 78 → 68
series_nav 9.34 → 8.18 µs (1.14x) 6.77 → 5.74 µs (1.18x) 101 → 96
series_nav(:bootstrap) 10.37 → 9.12 µs (1.14x) 7.62 → 6.45 µs (1.18x) 116 → 111
Ruby 3.3.13 no JIT YJIT objects
page_url 3.59 → 2.32 µs (1.55x) 2.74 → 1.70 µs (1.62x) 42 → 33
urls_hash 6.79 → 4.79 µs (1.42x) 5.20 → 3.45 µs (1.51x) 87 → 69
series_nav 9.51 → 8.27 µs (1.15x) 7.57 → 6.49 µs (1.17x) 113 → 104
series_nav(:bootstrap) 10.06 → 9.13 µs (1.10x) 8.36 → 7.40 µs (1.13x) 125 → 116
# ruby [--yjit] bench.rb path/to/pagy/gem/lib
$LOAD_PATH.unshift ARGV.fetch(0)
require 'pagy'
require 'benchmark'
Pagy::Method

params  = { 'q' => 'ruby on rails', 'author' => 'alice', 'page' => '7' }.freeze
request = Pagy::Request.new(request: { base_url: 'http://www.example.com', path: '/posts', params:, cookie: nil })
pagy    = Pagy::Offset.new(count: 1000, page: 7, request:)

{ 'page_url'               => -> { pagy.page_url(3) },
  'series_nav'             => -> { pagy.series_nav },
  'series_nav(:bootstrap)' => -> { pagy.series_nav(:bootstrap) },
  'urls_hash'              => -> { pagy.urls_hash } }.each do |name, action|
  2_000.times { action.() }
  GC.start
  before  = GC.stat(:total_allocated_objects)
  seconds = Benchmark.realtime { 50_000.times { action.() } }
  puts format('%-24s %.2f µs  %d objects', name, seconds / 50_000 * 1e6, (GC.stat(:total_allocated_objects) - before) / 50_000)
end

Separately, a_lambda.rb still does require 'cgi', which prints "CGI library is removed from Ruby 4.0" under -W; switching it to require 'cgi/escape' would silence it, but on 3.3 and 3.4 apps relying on pagy to load the full CGI library (CGI::Cookie, CGI.parse) would break. Would you like that change too, here or in its own PR, or would you rather keep cgi as is?

QueryUtils.escape relied on URI.encode_www_form_component, which dups,
re-encodes and gsubs every key and value. CGI.escape is a C extension
returning the same bytes except for '*' and '~', which are still handed
to URI, so every generated URL stays byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ddnexus
ddnexus changed the base branch from master to dev October 30, 2026 11:13
@ddnexus
ddnexus merged commit 8bdfe6f into ddnexus:dev Oct 30, 2026
4 checks passed
@ddnexus

ddnexus commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Thanks, merged on dev. Yes to require 'cgi/escape' in a_lambda.rb, in its own PR please: pagy only uses CGI.escapeHTML and CGI.escape, both in cgi/escape on every supported Ruby (>= 3.3), and apps relying on pagy to load the full cgi library depend on an accident, not on pagy's API. Base it on dev.

@czj
czj deleted the perf/escape-query-with-cgi branch November 2, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants