Picture by Writer
# Introduction
Net crawling is the method of routinely visiting net pages, following hyperlinks, and amassing content material from an internet site in a structured method. It’s generally used to collect giant quantities of data from documentation websites, articles, data bases, and different net assets.
Crawling a complete web site after which changing that content material right into a format that an AI agent can truly use is just not so simple as it sounds. Documentation websites usually comprise nested pages, repeated navigation hyperlinks, boilerplate content material, and inconsistent web page constructions. On prime of that, the extracted content material must be cleaned, organized, and saved in a method that’s helpful for downstream AI workflows akin to retrieval, question-answering, or agent-based methods.
On this information, we’ll be taught why to make use of Olostep as an alternative of Scrapy or Selenium, arrange all the pieces wanted for the net crawling challenge, write a easy crawling script to scrape a documentation web site, and at last create a frontend utilizing Gradio in order that anybody can present a hyperlink and different arguments to crawl web site pages.
# Selecting Olostep Over Scrapy or Selenium
Scrapy is highly effective, however it’s constructed as a full scraping framework. That’s helpful while you need deep management, however it additionally means extra setup and extra engineering work.
Selenium is healthier identified for browser automation. It’s helpful for interacting with JavaScript-heavy pages, however it’s not actually designed as a documentation crawling workflow by itself.
With Olostep, the pitch is much more direct: search, crawl, scrape, and construction net knowledge by way of one utility programming interface (API), with assist for LLM-friendly outputs like Markdown, textual content, HTML, and structured JSON. Meaning you should not have to manually sew collectively items for discovery, extraction, formatting, and downstream AI use in the identical method.
For documentation websites, that can provide you a a lot quicker path from URL to usable content material since you are spending much less time constructing the crawling stack your self and extra time working with the content material you really want.
# Putting in the Packages and Setting the API Key
First, set up the Python packages used on this challenge. The official Olostep software program growth package (SDK) requires Python 3.11 or later.
pip set up olostep python-dotenv tqdm
These packages deal with the principle elements of the workflow:
- olostep connects your script to the Olostep API
- python-dotenv masses your API key from a .env file
- tqdm provides a progress bar so you may observe saved pages
Subsequent, create a free Olostep account, open the dashboard, and generate an API key from the API keys web page. Olostep’s official docs and integrations level customers to the dashboard for API key setup.
Then create a .env file in your challenge folder:
OLOSTEP_API_KEY=your_real_api_key_here
This retains your API key separate out of your Python code, which is a cleaner and safer method to handle credentials.
# Creating the Crawler Script
On this a part of the challenge, we’ll construct the Python script that crawls a documentation web site, extracts every web page in Markdown format, cleans the content material, and saves it regionally as particular person recordsdata. We’ll create the challenge folder, add a Python file, after which write the code step-by-step so it’s straightforward to comply with and take a look at.
First, create a challenge folder on your crawler. Inside that folder, create a brand new Python file named crawl_docs_with_olostep.py.
Now we’ll add the code to this file one part at a time. This makes it simpler to grasp what every a part of the script does and the way the complete crawler works collectively.
// Defining the Crawl Settings
Begin by importing the required libraries. Then outline the principle crawl settings, such because the beginning URL, crawl depth, web page restrict, embody and exclude guidelines, and the output folder the place the Markdown recordsdata might be saved. These values management how a lot of the documentation web site will get crawled and the place the outcomes are saved.
import os
import re
from pathlib import Path
from urllib.parse import urlparse
from dotenv import load_dotenv
from tqdm import tqdm
from olostep import Olostep
START_URL = “https://docs.olostep.com/”
MAX_PAGES = 10
MAX_DEPTH = 1
INCLUDE_URLS = [
“/**”
]
EXCLUDE_URLS = []
OUTPUT_DIR = Path(“olostep_docs_output”)
// Making a Helper Perform to Generate Secure File Names
Every crawled web page must be saved as its personal Markdown file. To do this, we’d like a helper operate that converts a URL right into a clear and filesystem-safe file identify. This avoids issues with slashes, symbols, and different characters that don’t work effectively in file names.
def slugify_url(url: str) -> str:
parsed = urlparse(url)
path = parsed.path.strip(“https://www.kdnuggets.com/”)
if not path:
path = “index”
filename = re.sub(r”[^a-zA-Z0-9/_-]+”, “-“, path)
filename = filename.exchange(“https://www.kdnuggets.com/”, “__”).strip(“-_”)
return f”{filename or ‘web page’}.md”
// Making a Helper Perform to Save Markdown Information
Subsequent, add helper features to course of the extracted content material earlier than saving it.
The primary operate cleans the Markdown by eradicating further interface textual content, repeated clean strains, and undesirable web page parts akin to suggestions prompts. This helps preserve the saved recordsdata centered on the precise documentation content material.
def clean_markdown(markdown: str) -> str:
textual content = markdown.exchange(“rn”, “n”).strip()
textual content = re.sub(r”[s*u200b?s*](#.*?)”, “”, textual content, flags=re.DOTALL)
strains = [line.rstrip() for line in text.splitlines()]
start_index = 0
for index in vary(len(strains) – 1):
title = strains[index].strip()
underline = strains[index + 1].strip()
if title and underline and set(underline) == {“=”}:
start_index = index
break
else:
for index, line in enumerate(strains):
if line.lstrip().startswith(“# “):
start_index = index
break
strains = strains[start_index:]
for index, line in enumerate(strains):
if line.strip() == “Was this web page useful?”:
strains = strains[:index]
break
cleaned_lines: record[str] = []
for line in strains:
stripped = line.strip()
if stripped in {“Copy web page”, “YesNo”, “⌘I”}:
proceed
if not stripped and cleaned_lines and never cleaned_lines[-1]:
proceed
cleaned_lines.append(line)
return “n”.be part of(cleaned_lines).strip()
The second operate saves the cleaned Markdown into the output folder and provides the supply URL on the prime of the file. There’s additionally a small helper operate to clear previous Markdown recordsdata earlier than saving a brand new crawl end result.
def save_markdown(output_dir: Path, url: str, markdown: str) -> None:
output_dir.mkdir(mother and father=True, exist_ok=True)
filepath = output_dir / slugify_url(url)
content material = f”””—
source_url: {url}
—
{markdown}
“””
filepath.write_text(content material, encoding=”utf-8”)
There’s additionally a small helper operate to clear previous Markdown recordsdata earlier than saving a brand new crawl end result.
def clear_output_dir(output_dir: Path) -> None:
if not output_dir.exists():
return
for filepath in output_dir.glob(“*.md”):
filepath.unlink()
// Creating the Foremost Crawler Logic
That is the principle a part of the script. It masses the API key from the .env file, creates the Olostep consumer, begins the crawl, waits for it to complete, retrieves every crawled web page as Markdown, cleans the content material, and saves it regionally.
This part ties all the pieces collectively and turns the person helper features right into a working documentation crawler.
def predominant() -> None:
load_dotenv()
api_key = os.getenv(“OLOSTEP_API_KEY”)
if not api_key:
increase RuntimeError(“Lacking OLOSTEP_API_KEY in your .env file.”)
consumer = Olostep(api_key=api_key)
crawl = consumer.crawls.create(
start_url=START_URL,
max_pages=MAX_PAGES,
max_depth=MAX_DEPTH,
include_urls=INCLUDE_URLS,
exclude_urls=EXCLUDE_URLS,
include_external=False,
include_subdomain=False,
follow_robots_txt=True,
)
print(f”Began crawl: {crawl.id}”)
crawl.wait_till_done(check_every_n_secs=5)
pages = record(crawl.pages())
clear_output_dir(OUTPUT_DIR)
for web page in tqdm(pages, desc=”Saving pages”):
attempt:
content material = web page.retrieve([“markdown”])
markdown = getattr(content material, “markdown_content”, None)
if markdown:
save_markdown(OUTPUT_DIR, web page.url, clean_markdown(markdown))
besides Exception as exc:
print(f”Didn’t retrieve {web page.url}: {exc}”)
print(f”Carried out. Information saved in: {OUTPUT_DIR.resolve()}”)
if __name__ == “__main__”:
predominant()
Be aware: The total script is accessible right here: kingabzpro/web-crawl-olostep, an online crawler and starter net app constructed with Olostep.
// Testing the Net Crawling Script
As soon as the script is full, run it out of your terminal:
python crawl_docs_with_olostep.py
Because the script runs, you will notice the crawler course of the pages and save them one after the other as Markdown recordsdata in your output folder.
After the crawl finishes, open the saved recordsdata to verify the extracted content material. You must see clear, readable Markdown variations of the documentation pages.
At that time, your documentation content material is able to use in AI workflows akin to search, retrieval, or agent-based methods.
# Creating the Olostep Net Crawling Net Software
On this a part of the challenge, we’ll construct a easy net utility on prime of the crawler script. As an alternative of modifying the Python file each time, this utility provides you a neater method to enter a documentation URL, select crawl settings, run the crawl, and preview the saved Markdown recordsdata in a single place.
The frontend code for this utility is accessible in app.py within the repository: web-crawl-olostep/app.py.
This utility does a couple of helpful issues:
- Enables you to enter a beginning URL for the crawl
- Enables you to set the utmost variety of pages to crawl
- Enables you to management crawl depth
- Enables you to add embody and exclude URL patterns
- Runs the backend crawler immediately from the interface
- Saves the crawled pages right into a folder primarily based on the URL
- Reveals all saved Markdown recordsdata in a dropdown
- Previews every Markdown file immediately inside the appliance
- Enables you to clear earlier crawl outcomes with one button
To begin the appliance, run:
After that, Gradio will begin an area net server and supply a hyperlink like this:
* Operating on native URL: http://127.0.0.1:7860
* To create a public hyperlink, set `share=True` in `launch()`.
As soon as the appliance is operating, open the native URL in your browser. In our instance, we gave the appliance the Claude Code documentation URL and requested it to crawl 50 pages with a depth of 5.
Once you click on Run Crawl, the appliance passes your settings to the backend crawler and begins the crawl. Within the terminal, you may watch the progress as pages are crawled and saved one after the other.
After the crawl finishes, the output folder will comprise the saved Markdown recordsdata. On this instance, you’d see that fifty recordsdata have been added.
The dropdown within the utility is then up to date routinely, so you may open any saved file and preview it immediately within the net interface as correctly formatted Markdown.
This makes the crawler a lot simpler to make use of. As an alternative of fixing values in code each time, you may take a look at completely different documentation websites and crawl settings by way of a easy interface. That additionally makes the challenge simpler to share with different individuals who might not need to work immediately in Python.
# Last Takeaway
Net crawling is just not solely about amassing pages from an internet site. The true problem is popping that content material into clear, structured recordsdata that an AI system can truly use. On this challenge, we used a easy Python script and a Gradio utility to make that course of a lot simpler.
Simply as importantly, the workflow is quick sufficient for actual use. In our instance, crawling 50 pages with a depth of 5 took solely round 50 seconds, which exhibits that you would be able to put together documentation knowledge shortly with out constructing a heavy pipeline.
This setup also can transcend a one-time crawl. You possibly can schedule it to run daily with cron or Process Scheduler, and even replace solely the pages which have modified. That retains your documentation contemporary whereas utilizing solely a small variety of credit.
For groups that want this sort of workflow to make enterprise sense, Olostep is constructed with that in thoughts. It’s considerably extra reasonably priced than constructing or sustaining an inner crawling resolution, and a minimum of 50% cheaper than comparable options available on the market.
As your utilization grows, the price per request continues to lower, which makes it a sensible selection for bigger documentation pipelines. That mixture of reliability, scalability, and powerful unit economics is why a number of the fastest-growing AI-native startups depend on Olostep to energy their knowledge infrastructure.
Abid Ali Awan (@1abidaliawan) is an authorized knowledge scientist skilled who loves constructing machine studying fashions. At present, he’s specializing in content material creation and writing technical blogs on machine studying and knowledge science applied sciences. Abid holds a Grasp’s diploma in expertise administration and a bachelor’s diploma in telecommunication engineering. His imaginative and prescient is to construct an AI product utilizing a graph neural community for college kids fighting psychological sickness.
