imp(): move rate limiting to grouper by e11sy · Pull Request #576 · codex-team/hawk.workers

e11sy · 2026-07-07T00:38:36Z

This PR is part of work related to rate limiting issue, se collector part

Problem

Rate limiting logic lived in collector and was affected by sentry logs, metrics etc, that were filtered in sentry worker

it've been leading to rate limiting inconsistency
counter of events filled too fast, grouper managed to save 5-10 repetitions, however rate limits were configured to 1000+ per hour

Solution

Move collector rate-limiting logic to the grouper, where we can count actual amount of events that are being saved to db

Steps

Reproduce exactly same logic of projects cache, as we used in collector
this is used for db calls amount reducing (we fetch projects once in several minutes instead of fetching on each request)
Write redis helper lua script that would update counter on each event (increment or clear on time window expiration)
Cover with tests

codex-assistant · 2026-07-07T00:38:46Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-07-12T22:41:40Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-07-13T16:36:47Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-07-13T17:49:15Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-07-13T17:58:34Z

Thanks for adding a description — the PR is now marked as Ready for Review.

Copilot

Pull request overview

This PR moves per-project rate limiting enforcement into the grouper worker so limits are applied to events that actually get persisted, rather than being skewed by filtering/logging upstream.

Changes:

Added per-project rate limit enforcement in GrouperWorker using Redis-backed atomic counters.
Introduced an in-memory Mongo-backed cache (ProjectLimitsCache) to resolve effective limits (project → workspace → plan) with periodic refresh.
Added metrics + tests for rate limiting behavior, and documented new env vars.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
workers/limiter/src/index.ts	Adjusts quota-notification threshold lookup logic in limiter.
workers/grouper/src/index.ts	Enforces rate limiting before DB writes; refreshes cached limits periodically; avoids re-entering `handle()` on duplicate retry.
workers/grouper/src/redisHelper.ts	Adds Lua-script-based atomic rate limit counter update in Redis.
workers/grouper/src/projectLimitsCache.ts	New Mongo-backed cache resolving effective per-project rate limits.
workers/grouper/src/metrics/grouperMetrics.ts	Adds Prometheus counter for rate-limited (dropped) events.
workers/grouper/tests/index.test.ts	Adds integration-style test ensuring rate-limited events are dropped (not saved).
workers/grouper/tests/redisHelper.rateLimit.test.ts	New unit tests for RedisHelper rate-limit Lua logic.
workers/grouper/README.md	Documents rate limiting behavior and new environment variables + test command.
workers/grouper/.env.sample	Adds sample env vars for cache refresh period and Redis hash key.
.env.test	Updates Redis URL test configuration note (set by Jest redis container setup).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    if (eventsLimit === 0) {
+      return true;
+    }
+
+    const now = Math.floor(Date.now() / MS_IN_SEC);
+


+    } catch (error) {
+      this.logger.error(`Failed to update rate limit for project ${projectId}`, error);
+
+      return false;
+    }


+    this.projectLimitsRefreshInterval = setInterval(() => {
+      /* eslint-disable-next-line no-void */
+      void this.projectLimitsCache.refresh()
+        .catch((error) => {
+          this.logger.error('Failed to refresh project limits cache', error);
+        });
+    }, limitsUpdatePeriodSeconds * MS_IN_SEC);


+    const projects = await db.collection<ProjectDocument>('projects')
+      .find({}, { maxTimeMS: CONTEXT_TIMEOUT_MS })
+      .toArray();


+    const workspaces = await db.collection<WorkspaceDocument>('workspaces')
+      .aggregate<WorkspaceDocument>([
+        {
+          $lookup: {
+            from: 'plans',
+            localField: 'tariffPlanId',
+            foreignField: '_id',
+            as: 'plan',
+          },
+        },
+        { $unwind: '$plan' },
+      ], { maxTimeMS: CONTEXT_TIMEOUT_MS })
+      .toArray();


neSpecc · 2026-07-13T20:14:22Z

+      DEFAULT_PROJECTS_LIMITS_UPDATE_PERIOD_SECONDS,
+    );
+
+    this.projectLimitsRefreshInterval = setInterval(() => {


we already have 2 cache mechanics:

lib/cache/controller.ts

lim/memoize/index.ts

Let's use one of them and do not implement another one.

neSpecc · 2026-07-13T20:19:09Z

+   *
+   * @param projectId - project id
+   */
+  private async checkRateLimit(projectId: string): Promise<boolean> {


I’m not fond of the constant addition of new methods to the grouper index file. Let’s adhere to the principle of separation of concerns and create a separate ReteLimitter file that will encapsulate all the necessary logic.

neSpecc · 2026-07-13T20:19:59Z

+  eventsPeriod: number;
+}
+
+const CONTEXT_TIMEOUT_MS = 5000;


doc is missing

neSpecc · 2026-07-13T20:21:46Z

+    const workspaceMap = await this.loadWorkspacesWithPlans(db);
+
+    const projects = await db.collection<ProjectDocument>('projects')


I'm not sure we need to load all of thousands workspaces and projects and keep them in memory.
You can load plan and rate-limit settings from a particular project on demand and use "memoize" or "cacheController" for caching

moreover, we already query project data by getProjectPatterns. We can reorganize it someway to make 1 request that will return both "eventGroupingPatterns" and "rateLimitSettings" fields.

neSpecc · 2026-07-13T20:25:42Z

+  }
+
+  /**
+   * Reload all project limits from MongoDB.


Please provide more descriptive documentation. Explain the logic behind prioritizing.

imp(): move rate limiting to grouper

f383184

tests(): fix redis initialization in tests

7f670e2

chore(): eslint fix

2f14cb3

chore(): restore redis testcontainers

c7905d2

e11sy marked this pull request as ready for review July 13, 2026 17:58

Егор Коновалов added 2 commits July 13, 2026 21:01

chore(): yarn lock restoe

ea8c5e8

chore(): readme minor improvements

f53a5e2

e11sy mentioned this pull request Jul 13, 2026

imp(): mv rate limiting to grouper codex-team/hawk.collector#163

Open

neSpecc requested a review from Copilot July 13, 2026 20:01

Copilot started reviewing on behalf of neSpecc July 13, 2026 20:01 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

neSpecc requested changes Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

imp(): move rate limiting to grouper#576

imp(): move rate limiting to grouper#576
e11sy wants to merge 6 commits into
masterfrom
imp/mv-rate-limits-to-grouper

e11sy commented Jul 7, 2026 •

edited

Loading

Uh oh!

codex-assistant Bot commented Jul 7, 2026

Uh oh!

codex-assistant Bot commented Jul 12, 2026

Uh oh!

codex-assistant Bot commented Jul 13, 2026

Uh oh!

codex-assistant Bot commented Jul 13, 2026

Uh oh!

codex-assistant Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

neSpecc Jul 13, 2026

Uh oh!

neSpecc Jul 13, 2026

Uh oh!

neSpecc Jul 13, 2026

Uh oh!

neSpecc Jul 13, 2026

Uh oh!

neSpecc Jul 13, 2026

Uh oh!

neSpecc Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		const workspaceMap = await this.loadWorkspacesWithPlans(db);

		const projects = await db.collection<ProjectDocument>('projects')

Uh oh!

Conversation

e11sy commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Steps

Uh oh!

codex-assistant Bot commented Jul 7, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Jul 12, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Jul 13, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Jul 13, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

neSpecc Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

neSpecc Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

neSpecc Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

neSpecc Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

neSpecc Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

neSpecc Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

e11sy commented Jul 7, 2026 •

edited

Loading