Repository navigation
Watcher File Change Events Trigger Unbounded Concurrent Reimports, without any Debounce or Limit #297
Description
Describe the bug
Every Write event spawns a new goroutine with no debounce, no concurrency limit, no deduplication. When a file is saved in an IDE, it typically generates 2-5 write events in rapid succession
(write temp file, rename, metadata update). A cp or sed -i generates similar bursts. Each event spawns a separate HTTP upload to Microcks.
Why Critical:
- Thundering herd: 10 files changed simultaneously leads to 10+ goroutines, 10+ concurrent uploads, potentially overwhelming the Microcks server.
- Race condition on the server: Two concurrent uploads of the same artifact can create conflicting service definitions, especially for large specs that take seconds to process.
- No error aggregation: Each goroutine logs errors independently to stdout with no structured tracking.
- No backpressure: If the server is slow, goroutines accumulate indefinitely, this is a goroutine leak.
- fsnotify on directories: Some editors replace files rather than modifying in-place, which can cause the watcher to lose track of the file entirely (the inode changes).
Expected behavior
When watching files for changes, the CLI should debounce file system events (e.g., waiting ~200-500ms after the last edit before triggering an import) to ensure only a single logical save triggers one import. It should also limit concurrency or sequence the TriggerImport calls to avoid overwhelming the Microcks server. Furthermore, the watcher should correctly handle atomic file saves (which often trigger fsnotify.Rename or fsnotify.Create events) so it doesn't lose track of files when they are overwritten.
Actual behavior
Currently in pkg/watcher/watchManager.go, the Run() loop listens specifically for fsnotify.Write and immediately spawns a new, unbounded goroutine (go TriggerImport(entry)) for every single event.
Modern IDEs and tools (like sed -i or VS Code) trigger multiple rapid file system events when a file is saved (temp file write, rename, metadata update). This results in a "thundering herd" where multiple concurrent HTTP requests to upload the exact same artifact are fired at the Microcks server within milliseconds. This causes:
- Race conditions on the Microcks server resulting in potential database issues or duplicated logic.
- Goroutine leaks / absence of backpressure if the server is slow to respond.
- Loss of file tracking if the IDE performs an atomic save (replacing the file and changing its inode), because the CLI ignores non-Write operations.
How to Reproduce?
Steps to reproduce the issue:
- Start the microcks-cli watcher on a configuration tracking an API specification (e.g., using
microcks-cli test ...or your dedicated watch command). - Open the tracked API specification file in an editor like VS Code and hit Save, or simulate rapid disk writes via terminal:
for i in {1..5}; do touch my-api-spec.yaml; done - Inspect the
microcks-cliconsole output. You will see multiple[INFO] Re-importing changed file:logs simultaneously, meaning multiple concurrent HTTP uploads for the same file are executed.
2026/05/05 23:29:07 [INFO] Watcher added on /root/cncf/microcks-cli/test-spec.yaml
[INFO] microcks-watcher started...
[INFO] Re-importing changed file: /root/cncf/microcks-cli/test-spec.yaml
[WARN] Error re-importing /root/cncf/microcks-cli/test-spec.yaml: Post "/test-context/api/artifact/upload": unsupported protocol scheme ""
[INFO] Re-importing changed file: /root/cncf/microcks-cli/test-spec.yaml
[WARN] Error re-importing /root/cncf/microcks-cli/test-spec.yaml: Post "/test-context/api/artifact/upload": unsupported protocol scheme ""
[INFO] Re-importing changed file: /root/cncf/microcks-cli/test-spec.yaml
[WARN] Error re-importing /root/cncf/microcks-cli/test-spec.yaml: Post "/test-context/api/artifact/upload": unsupported protocol scheme ""
...
Microcks version or git rev
- Microcks CLI version or git rev:
masterbranch - Install method: Local build from source
Install method (docker-compose, helm chart, operator, docker-desktop extension,...)
No response
Additional information
The root cause is in pkg/watcher/watchManager.go:
case event := <-wm.fileWatcher.Events:
if event.Op&fsnotify.Write == fsnotify.Write {
// ...
} else {
wm.lock.Lock()
entry, exists := wm.watchEntries[event.Name]
wm.lock.Unlock()
if exists {
go TriggerImport(entry) // Spawned instantly per event
}
}
Activity
github-actions commentedon May 5, 2026
👋 @rootp1
Welcome to the Microcks community! 💖
Thanks and congrats 🎉 for opening your first issue here! Be sure to follow the issue template or please update it accordingly.
📢 If you're using Microcks in your organization, please add your company name to this list. 🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.
If you need to know why and how to add yourself to the list, please read the blog post "Join the Microcks Adopters list and Empower the vibrant open source Community 🙌"
Hope you have a great time there!
🌟 ~~~~~~~~~ 🌟
📢 If you like Microcks, please ⭐ star ⭐ our repo to support it!
🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.
rootp1 commentedon May 5, 2026
Hey @Harsh4902; What are your thoughts?
If this issue is helpful, I can submit a PR
yada commentedon Oct 14, 2026
Reproduced on master (f9550f5) against a real Microcks server, using
microcks import <file>:true --watch.echo >> filetriggered 2 re-imports on Linux.Cause: every
fsnotify.Writeevent runsgo TriggerImportWithContext(...)inpkg/watcher/watchManager.gowith no debounce, no per-file deduplication and no concurrency limit.Suggested fix: debounce per file (for example about 300 ms after the last event) and run one import at a time per file. This shares a code area with #432 (atomic saves), so one PR could cover both.
yada commentedon Oct 17, 2026
Update: PR #586 ("File watcher stops triggering after first save") looks like it addresses this. I built its head and ran
import --watchagainst a real server on macOS, compared with master:mv), write after the atomic save, second atomic save: 4 re-imports with the PR, versus 2 on master (both atomic saves lost).Not tested: Linux (inotify), where the watcher died after an atomic save on master. That should be checked before merging.
Disclosure: this comment was drafted with AI assistance, and the results were run locally.