How to keep cookies on request retry?

[DoctorEvil92]

I have a workflow that is something like this: call the URL and check for something first. If it's not right, then use send_request to set something and raise an exception which triggers a retry on the request. I'm printing out cookies and after using send_request they are there, but on a retry they aren't there anymore.

I have use_session_pool=True in crawler init, I'm printing context.session.id and it shows the same number so it should be working on the same session.
So is there a way to keep cookies no matter what basically?

[vdusek]

Cookies aren't being lost, they're stored on the session, just not on the one being used for the retry.

The catch is that your request isn't pinned to a specific session. When the handler raises and the request is retried, the crawler picks a new random session from the pool. Since the default max_pool_size is 1000, the retry may run with a different session that doesn't have the cookies set by your earlier send_request call.

There are two ways to fix this, depending on whether you still want session rotation.

1. Use a session pool with a single session

This is the simplest option and keeps your current raise-to-retry flow:

crawler = HttpCrawler(
    session_pool=SessionPool(max_pool_size=1),
    # ...
)

2. Pin the request to the current session

If you still want session rotation for everything else, don't raise to trigger the retry. Instead, re-enqueue the same URL and bind it to the session that now has the cookies:

from crawlee import Request

@crawler.router.default_handler
async def handler(context):
    if context.request.user_data.get('prepared'):
        # Do the work - cookies from the setup step are on this session.
        return

    # First pass: establish cookies.
    await context.send_request('https://.../set-something')

    # Re-enqueue the same URL, pinned to the same session.
    await context.add_requests([
        Request.from_url(
            context.request.url,
            session_id=context.session.id,  # Pins the retry to this session.
            user_data={'prepared': True},
            always_enqueue=True,  # Bypasses deduplication so the same URL runs again.
        )
    ])

Cookies are scoped by domain, so make sure the send_request target sets them for the same host you're crawling. Also, keep in mind that if you're using a custom HTTP client, it needs to support attaching the session's existing cookies to outgoing send_request calls.

I'm aware that this isn't the most ergonomic or intuitive way to reuse sessions. We expect to rethink and redesign this in the upcoming Crawlee JS v4 and Crawlee Python v2, probably later this year.

[DoctorEvil92]

I was printing context.session.id and it had the value of 1 for original and retried request, so I thought it was going over the same session. It was just a single request call originally, that was then retried. So even with a big pool size it seems like it was going over the same session? I was printing context.session.cookies to see the cookies.

I would prefer to raise an exception and retry, because with adding requests it could go in an infinite loop of adding requests when failing. For this way, is it OK to set cookies into some other global variable after using send_request, raise an exception and then set cookies in pre_navigation_hook?