High memory usage when using HTTP/3 requests

[LuisSCorreia]

Hello all.
Recently I decided to try out curl_cffi due to its support for HTTP/3 requests. However, unlike what happens with the standard requests Python library (which uses HTTP/1.1), the memory used by my process/script keeps increasing steadily.

For context, I have a script which issues a request to a number of endpoints to test out their response times.

The following is a simplified/redacted version of my code:

#/usr/bin/python
from curl_cffi import requests as requests_http3  
from curl_cffi.const import CurlHttpVersion
import time

# just a placeholder
URL = "https://google.com"

headersHTTP = {
    'User-Agent': f'monitor/3'
}   

while True:
    response = requests_http3.get(
        url=URL,
        http_version=CurlHttpVersion.V3ONLY,
        timeout=30,
        headers=headersHTTP
    )

    response_time = round(response.elapsed * 1000, 3)
    response_code = response.status_code

    response = response.close()
    del response

    time.sleep(30)

I tried using mprof to debug this. When I plot out the memory usage when using curl_cffi, this is the result:

However, when using the standard requests (and HTTP/1.1 consequently), this is the result - which is the expected/correct behavior:

Why does this happen? And how can I mitigate it?
I appreciate any help that you can give, as this is really important for me.

[lexiforest]

Can you reproduce the memleak when using h2 or h1.1 with curl_cffi? Is it h3 only?

[LuisSCorreia]

Hello @lexiforest, apologies for the delay.
While I can't post the exact script I'm using, as it's part of proprietary code, I developed a simple example script to reproduce this issue. This is its full code:

#/usr/bin/python
from curl_cffi import requests as requests_http3  
from curl_cffi.const import CurlHttpVersion
import time

# just a placeholder - some example URLs which support HTTP/3
urls = [
    "https://www.google.com",
    "https://www.cloudflare.com",
    "https://www.facebook.com",
    "https://www.youtube.com",
    "https://www.instagram.com",
    "https://www.linkedin.com",
    "https://www.reddit.com",
    "https://www.amazon.com"
]

while True:
    for url in urls:
        print(f"Getting {url}")
        response = requests_http3.get(
            url=url,
            http_version=CurlHttpVersion.V3ONLY,
            timeout=30
        )

        response_time = round(response.elapsed * 1000, 3)
        response_code = response.status_code
        response_http_version = response.http_version

        response = response.close()
        del response

        print(f"Response time: {response_time}\nResponse code: {response_code}\nHTTP Version: {response_http_version}")

    print("Sleeping for 15 seconds")
    time.sleep(15)

As you can see below, it exhibits the same memory issue:

[lexiforest]

What about this?

from curl_cffi import Session

s = Session()

while True:
    for url in urls:
        print(f"Getting {url}")
        response = s.get(
            url=url,
            http_version=CurlHttpVersion.V3ONLY,
            timeout=30
        )

[LuisSCorreia]

I've tried that out and it shows the same behavior:

So, using a global session has the same issue.

[lexiforest]

OK, I will try to reproduce and investigate this on my machine.

[SF97]

I'm having the same issue :/ are there any advances here?