Instructor

[ReinierKoops]

Love the library.

What about using https://github.com/instructor-ai/instructor (or https://github.com/BerriAI/litellm) instead of Langchain? This should be easier to work.

[raznem]

Hi @ReinierKoops , thanks for sharing these libs!

I considered switching to something like Litellm since, essentially, we are using Langchain as a connector for different providers.
I assume you are familiar with them. So, could you provide a list of pros and cons of using them against the Langchain? (If you can add some examples, it will be 🔥)

[ReinierKoops]

@raznem

I think these provide some nice insights:

• https://www.thoughtworks.com/radar/languages-and-frameworks/langchain
• https://www.thoughtworks.com/radar/languages-and-frameworks/instructor
• https://www.thoughtworks.com/radar/languages-and-frameworks/litellm

In my experience, litellm is a bit more mature in that sense. But when u use instructor, you can basically use litellm. I feel like instructor is kinda a bit of an easier abstraction layer.

As a side note, I'm also comparing crawl4ai with Parsera:
I felt like crawl4ai is "more" mature as in it allows a lot of customization options, but then again, when I was using the same llm model the results using parsera out-of-the-box results were a lot better than crawl4ai. So maybe if your focus is easy scraping, parsera wins imho. It has good defaults. Then if it has integration with like litellm or instructor imho would make adoption potential bigger.

Here some of my example code:

# pip install crawl4ai google-generativeai pydantic==2.* asyncio

import os, json, asyncio
from datetime import datetime
from typing import List

from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CrawlerRunConfig,
    CacheMode,
    LLMConfig,
)
from crawl4ai.extraction_strategy import LLMExtractionStrategy

from pydantic import BaseModel, Field

class Comment(BaseModel):
    author: str
    timestamp: datetime
    content: str
    like: int
    dislike: int


class Article(BaseModel):
    title: str
    author: str
    timestamp: datetime
    content: str
    score_useful: int
    score_wow: int
    score_touched: int
    score_analytical: int
    score_recommend: int
    comments: List[Comment]


llm_strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(
        provider="gemini/gemini-2.5-flash-preview-04-17",
        api_token="MY_TOKEN,
    ),
    schema=Article.model_json_schema(),  # pydantic-generated JSON schema
    extraction_type="schema",
    instruction="""
You are given raw HTML from a single Naver News article page.
Extract a JSON object that matches the provided schema exactly.
""",
    input_format="html",  # feed raw HTML instead of Markdown
    extra_args={"temperature": 0.0, "max_tokens": 4096},
    verbose=True,
)
run_conf = CrawlerRunConfig(
    extraction_strategy=llm_strategy,
    cache_mode=CacheMode.DISABLED,
    locale="ko-KR",  # browser Accept-Language
    timezone_id="Asia/Seoul",  # emulated timezone
    scan_full_page=True,
    exclude_external_images=True,
)
browser_conf = BrowserConfig(headless=True, text_mode=False)
URL = "https://n.news.naver.com/article/032/0003366895?ntype=RANKING"


async def main():
    async with AsyncWebCrawler(config=browser_conf) as crawler:
        result = await crawler.arun(url=URL, config=run_conf)

        if not result.success or not result.extracted_content.strip():
            raise RuntimeError("LLM extraction failed: " + result.error_message)

        raw = json.loads(result.extracted_content)

        if isinstance(raw, list):
            raw = raw[0]  # or raise an explicit error
        article = Article(**raw, strict=False)

        print(json.dumps(article.model_dump(mode="json"), ensure_ascii=False, indent=2))


if __name__ == "__main__":
    asyncio.run(main())

versus Parsera:

# pip install "pydantic>=2" parsera langchain-google-genai python-dateutil

import os, re, json, asyncio
from datetime import datetime
from zoneinfo import ZoneInfo
from typing import List

from dateutil import parser as dtparse
from pydantic import BaseModel, Field, field_validator, ConfigDict

from parsera import Parsera 
from langchain_google_genai import ChatGoogleGenerativeAI

KST = ZoneInfo("Asia/Seoul")
SCROLLS = 10
ELEMENTS = {
    "title": "Article headline",
    "author": "Journalist name (in native order if present)",
    "timestamp": "Published date-time (ISO 8601, Asia/Seoul)",
    "content": "Full article body text with paragraphs joined by \\n",
    "score_useful": "‘도움돼요’ reaction count",
    "score_wow": "‘놀라워요’ reaction count",
    "score_touched": "‘감동받았어요’ reaction count",
    "score_analytical": "‘분석적이에요’ reaction count",
    "score_recommend": "‘추천해요’ reaction count",
    # comments is a *list* of dicts – Parsera handles repetition automatically
    "comments": {
        "author": "Commenter nickname",
        "timestamp": "Comment date-time (ISO 8601, Asia/Seoul)",
        "content": "Comment text",
        "like": "추천 수 for this comment",
        "dislike": "비추천 수 for this comment",
    },
}

os.environ["GOOGLE_API_KEY"] = "MY_KEY"

LLM = ChatGoogleGenerativeAI(
    model="gemini-2.5-flash-preview-04-17",
    temperature=0.0,
    timeout=120,
)

def make_scraper() -> Parsera:
    return Parsera(model=LLM)

def _to_int(text: str | int | None) -> int:
    """Normalise counts like '1,234', '3.5만' → int."""
    if text is None:
        return 0
    s = str(text).strip()
    if s.endswith("만"):  # 3.5만 → 35_000
        return int(float(s[:-1]) * 10_000)
    digits = re.sub(r"[^\d]", "", s)
    return int(digits or 0)


class Comment(BaseModel):
    author: str | None = None
    timestamp: datetime
    content: str
    like: int = Field(0)
    dislike: int = Field(0)

    @field_validator("timestamp", mode="before")
    @classmethod
    def _tz(cls, v):  # noqa: N805
        dt = v if isinstance(v, datetime) else dtparse.parse(v)
        return dt.astimezone(KST)

    @field_validator("like", "dislike", mode="before")
    @classmethod
    def _ints(cls, v):  # noqa: N805
        return _to_int(v)


class Article(BaseModel):
    model_config = ConfigDict(extra="ignore")

    title: str
    author: str | None = None
    timestamp: datetime
    content: str

    score_useful: int = Field(0)
    score_wow: int = Field(0)
    score_touched: int = Field(0)
    score_analytical: int = Field(0)
    score_recommend: int = Field(0)

    comments: List[Comment] = Field(default_factory=list)

    @field_validator("timestamp", mode="before")
    @classmethod
    def _tz(cls, v):  # noqa: N805
        dt = v if isinstance(v, datetime) else dtparse.parse(v)
        return dt.astimezone(KST)

    @field_validator(
        "score_useful",
        "score_wow",
        "score_touched",
        "score_analytical",
        "score_recommend",
        mode="before",
    )
    @classmethod
    def _ints(cls, v):  # noqa: N805
        return _to_int(v)

async def scrape_article(url: str, *, scrolls: int = SCROLLS) -> Article:
    """Fetch one news article and return a fully-validated Article object."""
    scraper = make_scraper()
    raw = (await scraper.arun(url=url, elements=ELEMENTS, scrolls_limit=scrolls))[0]
    return Article.model_validate(raw)


async def main() -> None:
    url = "https://n.news.naver.com/article/032/0003366895?ntype=RANKING"
    article = await scrape_article(url)
    print(article.model_dump_json(indent=2))

if __name__ == "__main__":
    asyncio.run(main())

The only thing I am missing now for Parsera, is that it has little debugging output compared to crawl4ai.

(and also this weird debug output because of langchain:

Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/grpc_aio.pyx.pxi", line 110, in grpc._cython.cygrpc.shutdown_grpc_aio
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/grpc_aio.pyx.pxi", line 114, in grpc._cython.cygrpc.shutdown_grpc_aio
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/grpc_aio.pyx.pxi", line 78, in grpc._cython.cygrpc._actual_aio_shutdown
AttributeError: 'NoneType' object has no attribute 'POLLER'
Exception ignored in: 'grpc._cython.cygrpc.AioChannel.__dealloc__'
Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/grpc_aio.pyx.pxi", line 110, in grpc._cython.cygrpc.shutdown_grpc_aio
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/grpc_aio.pyx.pxi", line 114, in grpc._cython.cygrpc.shutdown_grpc_aio
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/grpc_aio.pyx.pxi", line 78, in grpc._cython.cygrpc._actual_aio_shutdown
AttributeError: 'NoneType' object has no attribute 'POLLER'

)

Also, because of how Crawl4ai is setup, the code is a lot more "clean" due to the use of these aforementioned libraries (as you can see from the given example).

And because the aforementioned libraries (litellm, instructor) work hand-in-hand with Pydantic (which to me is really important for any structured llm output), I think it would be a big win for this library to sort of try to incorporate it.

Hope it makes sense.

[raznem]

@ReinierKoops so essentially it boils down to support of PyDantic schema, right?
LangChain has this built-in, so I think we can actually add this without such radical change as switching dependency.
This interface makes a lot of sense, thanks for noting that!