Shuyan AI: web parsing, live search and document OCR APIs for LLMs

Anyone building an agent eventually hits the same wall: the model is smart enough, but it cannot see the page you need it to read. Either its knowledge stops at a training cutoff, or the HTML arrives buried under navigation and ads, or the decisive document exists only as a scanned PDF. Shuyan AI targets exactly this gap. It does not train models; it sells the pipeline that reshapes the outside world into something a model can ingest. Three questions decide whether it is worth wiring in: is the extracted output actually clean and structured, can a search result be traced back to a verifiable source, and does latency hold at the concurrency you need.

The company positions itself as data infrastructure for the large-model era and describes its technical route as dual-modal, combining vision and code. One path uses computer vision to interpret a page as rendered; the other extracts structure through code-level logic, and the two cross-check each other. That combination addresses the two classic weaknesses of scraping: pages rendered client-side that a plain HTTP fetch never sees, and brittle selectors that break the moment a layout changes. The vendor says its index spans more than 300 domains and hundreds of millions of structured records, with ad noise filtered out.

The product line splits into three layers. At the bottom sit the three data interfaces: parsing, search and OCR. Above them is a model marketplace bundling multi-provider calls, conversational agents, a deep-research mode and MCP workflow integration. Around the edges are dataset services and a BI analytics platform. Most engineering teams will only ever call two or three of the bottom-layer endpoints; the rest are adjacent capabilities from the same supplier and should be assessed on their own merits.

How the core capabilities work

Web parsing: rebuilding a rendered page as readable text

Send a target URL, receive structured Markdown or JSON. The selling point is not that it can fetch a page but that you do not have to write cleaning rules afterwards: body text, heading hierarchy and tables are preserved where possible, while ads and navigation are dropped. For retrieval pipelines this step sets the ceiling on chunk quality, and chunk quality sets the ceiling on recall. Dirty input produces chunks that are similar in embedding space and useless in practice. The vendor states support for dynamically rendered pages and complex layouts, which is precisely why a vision path adds something a pure DOM parser cannot.

Live search: a retrieval layer an agent can cite

The search endpoint is designed for model consumption and returns scored, structured content fragments rather than a bare list of links. Sources are described as covering news, encyclopedias, academic material and media platforms, refreshed continuously against a large crawl plus a self-built site archive. For research-style applications the visible difference is in reference breadth: the vendor claims its deep-research mode consults seven times as many pages as ordinary search, which means longer context and more room for cross-checking claims.

Document OCR: keeping hierarchy, not just characters

Upload a PDF or a scan and the engine extracts text, tables and formulas into Markdown and JSON that retain document structure. The hard part was never recognising individual glyphs; it is reconstructing organisation, knowing which line is a heading, which table spans three pages, which symbol in a formula is a subscript. Hierarchy matters downstream because a knowledge base chunks along headings. Lose the structure and the only remaining option is splitting on a fixed character count, which cuts sentences and tables in half.

Marketplace and MCP: model calls and data in one layer

The marketplace aggregates multiple LLM providers behind one endpoint with unified billing and response formats, so a team does not have to maintain several deployments. MCP integration injects search and parsing into AI workflows through a standard interface, which saves rewriting tool-call logic for teams already on that protocol. Together the practical benefit is configuration in one place: the data side and the model side stop being separate integrations with separate failure modes.

Getting started

1. Register on the official site and create an API key. New accounts include 500 free calls, enough to exercise all three interfaces before committing budget.
2. Validate the parsing endpoint against the pages you actually care about. Pick ten to twenty real URLs, including dynamically rendered ones, complex tables and paginated long articles, then inspect whether the returned Markdown keeps the structure you need. Do not judge on the vendor's sample page.
3. For OCR, assemble boundary samples first: two-column layouts, tables that cross page breaks, scans containing formulas. Structural mistakes concentrate in exactly this material, and finding them before launch is far cheaper than after.
4. If you already use Dify or Coze, mount the capability inside an agent workflow instead of writing a backend. For direct integration, call the REST endpoints and choose Markdown when humans will spot-check output, JSON when code will parse it.
5. For government or financial workloads the vendor offers high-concurrency private deployment, but delivery form, lead time and cost are not published; those require a commercial conversation.

Pricing and quotas

• The only published figure is 500 free calls for new accounts. That is enough for a proof of concept and nowhere near production volume.
• Billing is metered by token, and the vendor says pricing runs at roughly one third of comparable services while charging only for successful calls. Paying only on success matters more here than for plain text APIs, because parsing failures are inherently more common than generation failures.
• Per-endpoint unit prices, volume tiers, private deployment quotes and the settlement rules once the free quota is exhausted are all undisclosed. Confirm them in the console or through sales before forecasting cost.

Who it fits

• Developers building agents on Dify or Coze who need live page reading and search without maintaining crawlers and proxy pools.
• Retrieval and knowledge-base teams whose internal material is locked in PDFs and scans and must first become chunkable, hierarchy-aware text.
• Content aggregation and price-comparison products that continuously fetch and clean product and news data and are sensitive to advertising noise.
• Technical teams in finance or government that need private deployment and data residency while keeping live retrieval.

Limits and caveats

• Every performance number is vendor-supplied. Sub-second average latency, forty percent faster than the industry, error rates below 0.01 percent and one-third pricing have no independent benchmark, and the test corpus and page mix are not disclosed. Benchmark against your own target sites.
• Pricing transparency is limited. Beyond the 500 free calls, unit prices, concurrency ceilings, rate-limit policy and whether failed retries are billed are not stated in public material.
• Compliance detail is missing. Parsing and search involve third-party content, so copyright boundaries, robots handling and retention limits on fetched content need written confirmation, especially if you plan to redistribute what you retrieve.
• Multilingual quality is unstated. The vendor emphasises accuracy on complex Chinese pages, which is an advantage for Chinese sites but says nothing about performance on pages in other languages.
• The only confirmable media asset is a single official screenshot with a placeholder anchor, so this article carries no images. Check the console for interface and sample-output detail.

Frequently asked

Is Shuyan AI a model itself?

No. It is a data and aggregation layer. It does not train foundation models; it sells parsing, search and OCR endpoints plus a marketplace that routes to third-party models. You still choose the underlying model yourself.

How is this better than writing my own scraper?

Mostly maintenance cost. A homegrown scraper has to handle dynamic rendering, anti-bot measures, selector rot and body-text cleanup, and a single redesign can break everything. A hosted API moves that burden to the provider, at the price of unit cost and control: you get less say over extraction rules, which hurts on unusual pages.

Does it plug into Dify or Coze directly?

The vendor states both platforms are supported and configurable inside agent workflows. Confirm plugin naming and setup steps against the platform's own listing, since this article does not verify them.

What happens after the free quota?

Settlement rules and unit prices past the free tier are not public. Run your validation inside the 500 calls, then ask for a quote and confirm whether billing is success-only, so your cost model does not drift.