What is Crawler?

A program that visits websites automatically, for search engines, AI training, or abuse.

Where Crawler goes wrong in real apps

Each of these is a short scenario built from a real failure in an app made with AI. Read the answer here, or walk through it yourself in about four minutes.

Bills that explode · 4 minutes · Pro
A quiet week, a loud bill
Almost nobody visited your site this week. Your hosting bill is forty times higher than usual.
Why is my hosting bill high with few visitors?
On usage-based hosting you pay for every request, compute second and gigabyte, whether a person or a bot sent it. AI crawlers can send tens of thousands of requests that each trigger paid work. Block unwanted crawlers at your host, cache expensive pages and images, and set a hard spend limit.
Read the fix
AI inside your app · 4 minutes · Pro
The refund policy it made up
A customer wants the 60-day refund your support bot promised them. Your policy is 14 days. They have a screenshot.
How do I stop my chatbot making things up?
Give it your real documents with each question and tell it to answer only from them, saying it doesn't know and offering a human when they don't cover the question. Link the source so customers can check. Keep refunds, prices and other commitments with people or code, because customers and courts treat your bot's words as yours.
Read the fix
Launch day · 4 minutes · Pro
Invisible to Google
You published thirty blog posts over three months. Google shows only your homepage.
Why isn't Google indexing my app's pages?
If your app is a single-page app, each URL may return the same empty HTML with content added only by JavaScript, so crawlers see little to index. Server-render or pre-render pages, give each URL its own title and description, link pages with real links, and submit a sitemap.
Read the fix
Bills that explode · 4 minutes · Pro
500 signups overnight
You give every new account 50 free AI credits. You wake up to 500 new signups and no new customers.
How do I stop fake accounts abusing my free tier?
Make farming expensive and real signups easy: verify email before granting free credits, normalise addresses such as Gmail dot variants, rate-limit signups per IP address, and add a bot check. No single step stops a determined abuser; several small ones together make it not worth the effort.
Read the fix
Bills that explode · 4 minutes · Pro
The 40,000 SMS codes
Your phone sign-in sends a code by SMS. Overnight it sent 40,000 codes to numbers in countries you do not serve.
What is SMS pumping fraud?
SMS pumping is when attackers trigger thousands of verification texts to number ranges whose carriers share the fees with them, using your sign-in or invite endpoint and your SMS bill. Prevent it with rate limits per number and IP address, allowing only countries you serve, bot checks before sending, and alerts on unusual volume.
Read the fix
Getting paid · 4 minutes · Pro
200 declined payments in an hour
Your $5 tip jar had 214 payment attempts overnight. 203 were declined. Stripe has emailed you about unusual activity.
What is card testing and how do I stop it?
Card testing is when fraudsters use your payment form to check which stolen card numbers work, with many small attempts in a short time. Successful charges later become disputes with fees, and high decline rates can put your payment account at risk. Rate-limit attempts, add a bot check, require sign-in or a minimum amount, and enable fraud rules.
Read the fix

Rate limit: A cap on how often someone may do something, like 10 requests a minute, to stop abuse and runaway bills.
Server: A computer that runs your app's code out of sight. Users can't change what happens there, which is why checks belong there.
API: The part of an app that other programs talk to. Your app's pages ask its API for data, like a waiter taking orders to a kitchen.
Request: A message your browser (or any program) sends to a server asking for something: 'give me invoice 105' or 'save this profile'.
Policy: One rule in Row Level Security, such as 'people can read only their own orders'.
Secret key: A password your app uses to talk to a paid service. Anyone who sees it can use it, and spend your money.

All 62 terms in the glossary

Where Crawler goes wrong in real apps

Each of these is a short scenario built from a real failure in an app made with AI. Read the answer here, or walk through it yourself in about four minutes.

Bills that explode · 4 minutes · Pro

A quiet week, a loud bill

Almost nobody visited your site this week. Your hosting bill is forty times higher than usual.

Why is my hosting bill high with few visitors?

On usage-based hosting you pay for every request, compute second and gigabyte, whether a person or a bot sent it. AI crawlers can send tens of thousands of requests that each trigger paid work. Block unwanted crawlers at your host, cache expensive pages and images, and set a hard spend limit.

Read the fix

AI inside your app · 4 minutes · Pro

The refund policy it made up

A customer wants the 60-day refund your support bot promised them. Your policy is 14 days. They have a screenshot.

How do I stop my chatbot making things up?

Give it your real documents with each question and tell it to answer only from them, saying it doesn't know and offering a human when they don't cover the question. Link the source so customers can check. Keep refunds, prices and other commitments with people or code, because customers and courts treat your bot's words as yours.

Read the fix

Launch day · 4 minutes · Pro

Invisible to Google

You published thirty blog posts over three months. Google shows only your homepage.

Why isn't Google indexing my app's pages?

If your app is a single-page app, each URL may return the same empty HTML with content added only by JavaScript, so crawlers see little to index. Server-render or pre-render pages, give each URL its own title and description, link pages with real links, and submit a sitemap.

Read the fix

Bills that explode · 4 minutes · Pro

500 signups overnight

You give every new account 50 free AI credits. You wake up to 500 new signups and no new customers.

How do I stop fake accounts abusing my free tier?

Make farming expensive and real signups easy: verify email before granting free credits, normalise addresses such as Gmail dot variants, rate-limit signups per IP address, and add a bot check. No single step stops a determined abuser; several small ones together make it not worth the effort.

Read the fix

Bills that explode · 4 minutes · Pro

The 40,000 SMS codes

Your phone sign-in sends a code by SMS. Overnight it sent 40,000 codes to numbers in countries you do not serve.

What is SMS pumping fraud?

SMS pumping is when attackers trigger thousands of verification texts to number ranges whose carriers share the fees with them, using your sign-in or invite endpoint and your SMS bill. Prevent it with rate limits per number and IP address, allowing only countries you serve, bot checks before sending, and alerts on unusual volume.

Read the fix

Getting paid · 4 minutes · Pro

200 declined payments in an hour

Your $5 tip jar had 214 payment attempts overnight. 203 were declined. Stripe has emailed you about unusual activity.

What is card testing and how do I stop it?

Card testing is when fraudsters use your payment form to check which stolen card numbers work, with many small attempts in a short time. Successful charges later become disputes with fees, and high decline rates can put your payment account at risk. Rate-limit attempts, add a bot check, require sign-in or a minimum amount, and enable fraud rules.

Read the fix

Related terms

Rate limit

A cap on how often someone may do something, like 10 requests a minute, to stop abuse and runaway bills.

Server

A computer that runs your app's code out of sight. Users can't change what happens there, which is why checks belong there.

API

The part of an app that other programs talk to. Your app's pages ask its API for data, like a waiter taking orders to a kitchen.

Request

A message your browser (or any program) sends to a server asking for something: 'give me invoice 105' or 'save this profile'.

Policy

One rule in Row Level Security, such as 'people can read only their own orders'.

Secret key

A password your app uses to talk to a paid service. Anyone who sees it can use it, and spend your money.