coldtally

Methodology & independence

How we collect prices, compute cost per task, detect changes, and stay independent.

Prices

  • Every value comes from the provider's public pricing page or docs. Each one stores its source URL and the date we fetched it (raw JSON).
  • When a page doesn't state something (billing granularity, say, or isolation mechanism), we mark it unverified instead of guessing.
  • List prices for the default region and the cheapest self-serve plan that allows the workload. Regional multipliers, taxes, committed-use discounts and enterprise pricing are excluded.

Cost per typical task

  • Code-agent task: 2 vCPU / 4 GiB sandbox for 5 minutes at 10% CPU utilization (most time spent waiting on the LLM). Mirrors Vercel's own "AI code validation" example.
  • Browser-agent task: One 3-minute cloud browser session, headless where the provider prices it separately, with no proxy, CAPTCHA or LLM costs.
  • We apply each provider's billing rules as published. Wall-clock vs active CPU, minimum increments (for example Vercel's 1-minute memory minimum, or Browser Use's whole-minute rounding), Modal's 1 core = 2 vCPU, and Cloudflare's fixed instance types (we pick the smallest one that fits).
  • Results are usage cost per run. Plan fees, included credits, storage, egress, proxies, CAPTCHA and LLM tokens are listed separately and never netted in.
  • The cost model is open source (src/lib/cost.js) and is the same code the calculator runs in your browser.

Change detection

  • A weekly GitHub Action fetches every pricing page, extracts the text and price tokens, and diffs them against the last snapshot. It also checks that every price we publish still appears on the page.
  • Any difference opens a pull request with draft changelog entries marked needs-review. A human checks the source before anything is published to the changelog, the RSS feed or the newsletter.

Speed (secondary)

  • Coldtally's core product is price and change tracking. We run a small daily cold-start check (create → first command → destroy for sandboxes; create → CDP connect → navigate → release for browsers) from one pinned region. It's there to sanity-check the cost model, not to be a speed leaderboard.
  • For broad speed coverage we credit and link to ComputeSDK's open benchmarks. Their data is MIT-licensed (github.com/computesdk/benchmarks). Anything we show from it is attributed to them and labelled with their run date. Note that ComputeSDK also sells private benchmarking to providers. We report their public data as-is.
  • Providers whose terms forbid publishing benchmarks are listed with pricing only. Currently that's Blaxel.

Independence policy

  1. Rankings aren't for sale. Order is always by computed cost (or by the metric shown). Sponsorship never changes inclusion, order, data or method.
  2. No paid private tests. We don't sell private, early or "stealth" runs to the vendors we track, and we don't coordinate releases with them.
  3. No vendor-paid content. Nothing in the changelog or newsletter is written or paid for by a tracked vendor.
  4. Sponsor disclosure. Sponsored placements are always labelled. Vendors we rank can't sponsor the comparison tables or the changelog. All current sponsors are listed on the advertise page.
  5. Right of reply, not right of edit. Vendors can report errors with a source. We fix verifiable mistakes and log the fix publicly.
  6. Reproducible. Data, cost model, scraper and benchmark harness are public in the repository.