Skip to content

Keep a single EventManager per process (event loop) #2242

Description

@Mantisus

Make EventManager a single service per process (per event loop, in practice), owned by the global service_locator.

Motivation

Today it's possible to create several EventManager instances that emit the same events. BasicCrawler builds its own ServiceLocator and may put a crawler-specific event manager into it, while Snapshotter and RecoverableState always resolve the global service_locator. So a crawler-scoped manager runs a second SystemInfo sampling loop and emits PersistState that nobody listens to, while the components it was supposed to serve keep listening on the global one. Working around that is why BasicCrawler._run_crawler currently enters both managers.

The second problem is lifetime. An event manager only runs while at least one crawler is active, but several systems that depend on it are usable without a crawler. FileSystemRequestQueueClient keeps its ordering and in-progress state in RecoverableState, and the same holds for autosaved KeyValueStore values, RequestList and SitemapRequestLoader. Used standalone, they register a PersistState listener on a manager that is never started, so their state is only written on explicit teardown.

Implementation

Keep exactly one EventManager, reachable through a single access point in service_locator, resolved per running event loop so that a manager never leaks from one asyncio.run into the next. service_locator.set_event_manager stays as the way to install a custom manager, for tests or for the Apify SDK's platform event manager.

The manager starts itself on the first on or emit call and closes itself when its event loop shuts down, so it works regardless of whether a crawler is running. There is no context to enter and no async with in the calling code. All current on call sites are already inside async code, and so is every RecurringTask.start call site, so the lazy start always has a running loop to attach to.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions