IBM Cloud VPC pricing + bare metal availability

Two views of IBM Cloud VPC inventory:

• Pricing scraper (ibm_vpc_pricing_scraper.py) — hourly rates for virtual server instances and bare metal, taken from the public catalog UI, with an optional live bare-metal capacity check on the same run.
• Availability report (report.html) — a filterable dashboard of every VPC bare metal profile, by region and zone, with whether that profile can be ordered right now. No prices on this page.

Bare metal availability report

report.html is a Carbon-styled snapshot of VPC bare metal supply. It loads data/current-bm-profiles.json on open and shows:

• A pixel world map of IBM Cloud regions. Each pin is the count of unique profiles that currently have capacity in that region (not the number of zone-rows). Pin color is relative to the busiest region. Click a pin to filter the table to that region. A region at zero still offers bare metal; none of those profiles have capacity at snapshot time.
• A table of one row per profile+zone, with family, CPU architecture, cores/sockets, memory, bandwidth, disks, and a yes/no/unknown capacity flag.
• Filters for profile, region, zone, and capacity (available now / no capacity / not checked), plus a free-text search over family, cores, and disks.

The JSON behind it is produced by ibm_vpc_profiles_report.py: for every VPC region it lists bare metal profiles (GET /v1/bare_metal_server/profiles) and live capacity (GET /v1/bare_metal_server/capacities), then joins them so each profile+zone row has capacity_available true or false. Mumbai (in-mum) is included. This path is API-only — no browser, no prices.

Open report.html from a local static server so the JSON can load (a file:// open will not fetch it):

mise run scrape-bm-profiles
python3 -m http.server 8765
# then http://127.0.0.1:8765/report.html

The same pair (report.html + data/current-bm-profiles.json) is what Vercel serves. There is no upload prompt: the page fetches that JSON on load. vercel.json maps / to the report; .vercelignore keeps the Python scrape out of the deployment. After mise run scrape-bm-profiles commits a fresh snapshot, vercel --prod (or a git-connected project) publishes it. The snapshot is only as new as the last deploy.

Why two different approaches for pricing vs capacity

Price isn't available from any public/stable API for either server type — it only exists in the catalog UI, backed by an internal GraphQL endpoint. So price is scraped with headless Chrome via Playwright, the same way for both VSI and bare metal.

Bare metal capacity (whether a profile is actually orderable right now in a given zone) is the opposite situation: it's a real, documented, authenticated REST API (cloud.ibm.com/docs/apis/vpc/latest#list-bare-metal-server-capacities), so both the report collector and the pricing script call it directly. Each item returned is just a {profile, zone} pair — its presence in the list means that combination currently has capacity (there's no numeric quantity to report, just yes/no). This needs an IBM Cloud IAM API key; see Auth below. Virtual server instances don't have an equivalent capacity-check endpoint, so this part only applies to bare metal.

Pricing scraping itself still needs no login/API key for either server type.

Setup

For the pricing scraper (Playwright + Chromium):

mise run setup

This installs playwright + requests and downloads the headless Chromium binary Playwright drives (a few hundred MB, one-time).

On Linux, Playwright's browser download needs a handful of system shared libraries. If mise run setup complains about missing dependencies, run once with sudo:

sudo python -m playwright install-deps chromium

macOS doesn't need this step.

For the availability report (API collector only):

pip install -r requirements-profiles.txt

(mise.toml pins Python 3.12; ibm_vpc_profiles.py is the shared collector, ibm_vpc_profiles_report.py is the bare-metal-only wrapper the HTML page reads.)

Auth (bare metal capacity only)

The capacity REST calls need an IBM Cloud IAM API key in the IBMCLOUD_API_KEY environment variable (override the name with --api-key-env). Wire it up with fnox rather than exporting it yourself:

fnox run -- python ibm_vpc_pricing_scraper.py --output-dir ./data
fnox run -- python ibm_vpc_profiles_report.py --output-dir ./data

(Check fnox --help / your fnox config for the exact subcommand your installed version uses to inject a secret as an env var — run/exec are common names but this varies by version; adjust the mise.toml tasks below if yours differs.)

If the env var isn't set, the pricing script logs that it's skipping the capacity section and continues on with pricing — it doesn't hard-fail. The report collector needs the key; without it there is nothing to join onto the profile list.

Running it

mise run scrape-bm-profiles          # BM profiles + live capacity → data/current-bm-profiles.json
mise run scrape-pricing              # VSI + bare metal pricing, + bare metal capacity (needs fnox/API key)
mise run scrape-pricing-no-capacity  # same, but skips the capacity REST calls (no API key needed)
mise run scrape-vsi-only             # virtual server instances only, skips bare metal entirely
mise run scrape-pricing-us           # just us-south / us-east, for a quick check
mise run scrape-pricing-debug        # visible browser window, for troubleshooting

Or call the scripts directly for finer control:

python ibm_vpc_profiles_report.py --output-dir ./data \
  --regions us-south,eu-de \     # limit which regions
  --available-only               # drop profile+zone rows with no capacity

python ibm_vpc_pricing_scraper.py --output-dir ./data \
  --skip-virtual-servers \        # bare metal only
  --skip-bare-metal-capacity \    # pricing only, no REST API calls
  --regions us-south,eu-de \      # limit which regions
  --api-key-env MY_IBM_KEY_VAR    # if you named the env var something else

ibm_vpc_profiles.py is the same collector as the report wrapper but includes virtual server instance profiles as well (still no prices).

Output schema

Availability report (ibm_vpc_profiles_report.py) writes, into --output-dir (default ./data/):

• current-bm-profiles.json — stable filename report.html loads
• ibm_vpc_bm_profiles_<timestamp>.json / .csv — the same snapshot, timestamped

Each row is one bare metal profile in one zone: region, zone, profile_name, family, cpu_architecture, cpu_core_count, cpu_socket_count, memory_gib, bandwidth_mbps, disks, capacity_available. capacity_available is True/False — whether that exact profile+zone combo showed up in the /bare_metal_server/capacities response at scrape time.

Pricing scraper writes two timestamped files to ./data/:

• ibm_vpc_pricing_<timestamp>.csv — every row, one file
• ibm_vpc_pricing_<timestamp>.json — same data, raw JSON

Every pricing-scraper row has row_type (pricing or capacity) and server_type (virtual_server_instance or bare_metal_server) so you can filter for whichever slice you need. The two row types don't share every column — e.g. only capacity rows have a zone — so the CSV writer fills in a blank cell for whatever a given row type doesn't have, rather than forcing everything into one row shape.

row_type = pricing (VSI and bare metal, from the browser scrape): profile_name, plan_id, metric_charge_units, family, vcpus, ram, bandwidth, instance_storage, accelerator, cost, geography, region. cost is the hourly rate as shown on the page (whatever the page's default Hourly/Monthly toggle is at scrape time — currently Hourly).

row_type = capacity (bare metal only, from the REST API) uses the same columns as the availability report. No price on these rows; join on profile_name + region against the pricing rows if you want price and capacity side by side.

How the pagination ("move forward") is handled

The pricing table caps at 100 rows per page. The script sets the page size to the max (100) and clicks the "Next page" button in a loop until it's disabled, so regions with 200+ profiles get fully covered without manual clicking.

Scheduling a recurring scrape

mise doesn't do cron itself, so pair it with your system scheduler, pointing at the mise task:

# crontab -e
# Run every day at 6am
0 6 * * * cd /path/to/this/project && /path/to/mise run scrape-pricing >> logs/scrape.log 2>&1

(macOS: launchd works too if you prefer that over cron — same idea, just call mise run scrape-pricing from the launchd plist.)

Troubleshooting: net::ERR_HTTP2_PROTOCOL_ERROR

If the very first page.goto() fails with this error, IBM's edge (bot-protection in front of cloud.ibm.com) reset the connection because Playwright's bundled headless Chromium has a recognizable automation fingerprint. The script already sets a normal desktop user-agent, disables HTTP/2 (falls back to HTTP/1.1), and retries navigation a couple of times with backoff, which clears this in most cases.

If it still happens, drive your real, locally-installed Chrome instead of the bundled one -- it's fingerprinted differently and is less likely to be challenged:

python -m playwright install chrome   # one-time, downloads/registers your local Chrome
python ibm_vpc_pricing_scraper.py --output-dir ./data --channel chrome --headed

(--headed here just so you can watch it work the first time; drop it once you trust it.)

Notes / things that could break this

• Selectors rely on IBM's hand-authored element ids (#geography, #region, #profile-family, etc.) rather than autogenerated React ids, which should be reasonably stable — but this is still screen-scraping a UI IBM doesn't guarantee as an API contract. If a run suddenly returns 0 rows, re-run with --headed and compare what you see against what ibm_vpc_pricing_scraper.py's find_profile_table() / parse_row() expect.
• The bare metal Pricing tab (/infrastructure/provision/bm) is assumed to reuse the same Carbon DataTable/select-id structure as the VSI one, since it's the same catalog UI component family — this part hasn't been walked through live the way the VSI page was, so if scrape_pricing_site() comes back with 0 bare metal pricing rows while VSI rows look fine, that's the first place to check with --headed (specifically whether #profile-family exists there and what options it has — select_all_profiles_family() already logs this).
• The published ibm-vpc SDK has no list_bare_metal_server_capacities helper, so the live capacity call is a direct httpx request using the same IAM token as the SDK client. Profile listing is SDK; capacity is not.
• The scripts are polite by default: 1.5s pause between regions for pricing scraping, 0.5s between regions for capacity calls, and they log progress line-by-line so a cron-redirected log file is readable later.