Web fetch — API reference

FetchError

Bases: Exception

Raised when a fetch fails.

FetchResult dataclass

FetchResult(
    url: str,
    status_code: int,
    content_type: str,
    charset: str,
    body: bytes,
)

convert_content

convert_content(
    body: bytes,
    content_type: str,
    charset: str,
    *,
    max_chars: int = DEFAULT_MAX_CHARS,
) -> str

Convert fetched content to a text representation suitable for an LLM.

PARAMETER	DESCRIPTION
body	Raw response bytes. TYPE: bytes
content_type	MIME type (e.g. "text/html"). TYPE: str
charset	Character encoding (e.g. "utf-8"). TYPE: str
max_chars	Maximum output length in characters. TYPE: int DEFAULT: DEFAULT_MAX_CHARS

RETURNS	DESCRIPTION
str	Converted text content, truncated if necessary.

fetch_url

fetch_url(
    url: str,
    *,
    timeout: int = DEFAULT_TIMEOUT,
    max_size: int = DEFAULT_MAX_SIZE,
    allow_private: bool = False,
) -> FetchResult

Fetch a URL and return the response content.

PARAMETER	DESCRIPTION
url	The URL to fetch. TYPE: str
timeout	Request timeout in seconds. TYPE: int DEFAULT: DEFAULT_TIMEOUT
max_size	Maximum response body in bytes. TYPE: int DEFAULT: DEFAULT_MAX_SIZE
allow_private	If True, allow fetching private/loopback IPs. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
FetchResult	FetchResult with the response data.

RAISES	DESCRIPTION
FetchError	If the fetch fails for any reason.

main

main()