[docs] GSoC26: Resource Aware Priority Task Scheduling for OpenWISP#269
[docs] GSoC26: Resource Aware Priority Task Scheduling for OpenWISP#269nemesifier merged 2 commits intomasterfrom
Conversation
WalkthroughAdds a new GSoC 2026 project idea "Resource Aware Priority Task Scheduling for OpenWISP" describing a three-component architecture (Resource Monitor, Resource Autoscaler, Priority Scheduler) for resource-aware, priority-based Celery task execution and migration to gevent-based I/O concurrency. The proposal includes rationale, problem statements, concrete goals, architecture sketches, migration guidance, success criteria, milestones, prerequisites, open questions, and duplicated content in the document. Also adds ten words to the spelling wordlist (async, autoscaler, autoscaling, benchmarking, bootsteps, filesystems, gevent, PoC, greenlets, prefetch, quickstart). Sequence Diagram(s)mermaid Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 🧹 Recent nitpick comments
📜 Recent review detailsConfiguration used: Organization UI Review profile: CHILL Plan: Pro ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
✏️ Tip: You can disable this entire section by setting Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@developer/gsoc-ideas-2026.rst`:
- Line 1207: Update the image reference string in the RST at the shown diff:
replace the misspelled filename
"../images/gsoc/ideas/2026/celery-priority-reseource-aware.png" with the
correctly spelled "../images/gsoc/ideas/2026/celery-priority-resource-aware.png"
so the image loads; ensure any other occurrences of "reseource" in the document
are corrected too.
- Around line 1497-1503: The bash code block contains RST inline code markup
(double backticks) around sysctl on the lines that set ping_group_range; remove
the backticks so the commands use plain sysctl and the filename /etc/sysctl.conf
as normal shell tokens (i.e., change ``sysctl`` to sysctl and
``/etc/sysctl``.conf to /etc/sysctl.conf) in the code-block so copy-pasting will
produce valid shell commands; verify the two affected lines reference
net.ipv4.ping_group_range and the sysctl invocation without any backticks.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
images/gsoc/ideas/2026/celery-priority-reseource-aware.pngis excluded by!**/*.png
📒 Files selected for processing (2)
developer/gsoc-ideas-2026.rstspelling_wordlist.txt
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (2)
spelling_wordlist.txt (1)
8-250: LGTM!All added words are correctly alphabetized and are valid technical terms relevant to the GSoC project on Celery, gevent, and task scheduling.
developer/gsoc-ideas-2026.rst (1)
1204-1687: Well-structured and comprehensive project documentation.The GSoC project idea is thoroughly documented with clear problem statement, goals, architecture, migration guidance, success criteria, and prerequisites. The technical content is accurate, including:
- Correct identification of gevent/Celery integration challenges
- Accurate code examples for ResourceMonitor, ResourceAutoscaler, and PriorityScheduler patterns
- Valid migration guidance from fping to icmplib
- Proper handling of Django database connection issues with gevent
The document follows OpenWISP documentation conventions with appropriate RST formatting, clear section hierarchy, and comprehensive coverage of requirements and deliverables.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
nemesifier
force-pushed
the
gsoc26-monitoring-scalability
branch
from
2d9d954 to
70ada53
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@developer/gsoc-ideas-2026.rst`:
- Around line 1630-1634: The RST numbered list under "Implementation Phase
(Coding Period)" is run together; fix it by converting the single inline
sentence starting "Priority order (must-haves first): 1. Core priority scheduler
package 2. OpenWISP Monitoring ``gevent`` migration 3. Integration & testing 4.
Documentation" into a proper reStructuredText numbered list: place a blank line
after the heading, then put each item on its own line prefixed with its number
and a period (e.g., "1. Core priority scheduler package", "2. OpenWISP
Monitoring ``gevent`` migration", etc.), ensuring consistent indentation and
spacing so the parser recognizes the list.
- Around line 1449-1456: Update the Django gevent/persistent-connections section
to clarify that Django 5.1+ native pooling (referenced as OPTIONS: {"pool":
{"min_size": 2, "max_size": 10}}) only works with PostgreSQL using psycopg3
(installable as psycopg[pool]) and does not support psycopg2, MySQL, SQLite, or
other backends; explicitly state the critical requirement to set CONN_MAX_AGE =
0 (Django raises ImproperlyConfigured if CONN_MAX_AGE > 0 or None) and note that
this pooling is not a universal fix for gevent + persistent connections across
all database drivers.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
images/gsoc/ideas/2026/celery-priority-resource-aware.pngis excluded by!**/*.png
📒 Files selected for processing (2)
developer/gsoc-ideas-2026.rstspelling_wordlist.txt
🚧 Files skipped from review as they are similar to previous changes (1)
- spelling_wordlist.txt
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (7)
developer/gsoc-ideas-2026.rst (7)
1203-1228: LGTM!The project metadata and introduction are well-structured and provide a clear overview of the project scope. The deliverables are clearly stated.
1229-1296: LGTM!The problem statement effectively describes the current limitations and the project goals are clearly articulated with realistic expectations about mixed pool usage.
1364-1425: LGTM!The pseudo-code examples effectively illustrate the proposed architecture. The three components (ResourceMonitor, ResourceAutoscaler, PriorityScheduler) demonstrate clear separation of concerns and the code is syntactically valid Python.
1497-1503: LGTM!The sysctl configuration commands are now correct without the erroneous RST inline code markup. The bash code block is valid and ready for users to copy-paste.
1509-1616: LGTM!The success criteria are measurable and realistic, with clear midterm checkpoints. The expected outcomes comprehensively cover all affected repositories and deliverables.
1636-1687: LGTM!The prerequisites clearly define required knowledge, and the open questions provide contributors with meaningful research directions that demonstrate understanding of the problem space.
1472-1492: No issues identified. The code example correctly demonstrates using icmplib with gevent. Theprivileged=Falseparameter is supported and works as documented for non-privileged ICMP operations, theping()function returns aHostobject with theavg_rttandpacket_lossattributes as shown, and the gevent integration pattern (using blocking APIs withingevent.spawn()with early monkey-patching) is the standard, recommended approach for this use case.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
developer/gsoc-ideas-2026.rst
Outdated
| - Django's persistent connections (``CONN_MAX_AGE > 0``) + ``gevent`` can | ||
| cause deadlocks. | ||
| - ``greenlets`` share the same thread, confusing Django's thread-local | ||
| connections. | ||
| - **Solution for Django 5.1+**: Use native connection pooling with | ||
| ``OPTIONS: {"pool": {"min_size": 2, "max_size": 10}}``. | ||
| - **Solution for earlier Django**: Set ``CONN_MAX_AGE=0`` or use | ||
| ``gevent`` aware connection pool. |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Django 5.1 native database connection pooling OPTIONS configuration
💡 Result:
In Django 5.1, “native” connection pooling is supported only for PostgreSQL when using psycopg (psycopg3), via DATABASES["default"]["OPTIONS"]["pool"]. (docs.djangoproject.com)
Minimal config (use psycopg pool defaults)
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
# NAME/USER/PASSWORD/HOST/PORT...
"CONN_MAX_AGE": 0, # required with pooling
"OPTIONS": {"pool": True},
}
}Django requires psycopg[pool] (or psycopg-pool) for this option, and it’s ignored if you’re using psycopg2. (docs.djangoproject.com)
Custom pool parameters (passed through to psycopg_pool.ConnectionPool)
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"CONN_MAX_AGE": 0,
"OPTIONS": {
"pool": {
"min_size": 2,
"max_size": 10,
"timeout": 10, # seconds to wait for a connection
"max_lifetime": 3600, # recycle connections after N seconds
"max_idle": 600, # close extra idle connections after N seconds
}
},
}
}These keys are ConnectionPool options (e.g., max_lifetime, max_idle, timeout, etc.). (psycopg.org)
Important constraint: don’t use persistent connections with pooling
If OPTIONS["pool"] is enabled, Django will raise ImproperlyConfigured if you set CONN_MAX_AGE to a positive value or None (pooling replaces Django’s persistent-connection behavior), so keep CONN_MAX_AGE = 0. (docs.djangoproject.com)
Citations:
- 1: https://docs.djangoproject.com/id/6.0/releases/5.1/?utm_source=openai
- 2: https://docs.djangoproject.com/en/5.1/ref/databases/?utm_source=openai
- 3: https://www.psycopg.org/psycopg3/docs/api/pool.html?utm_source=openai
- 4: https://docs.djangoproject.com/en/5.1/ref/databases/?utm_source=openai
Clarify Django 5.1+ connection pooling limitations in the documentation.
The native connection pooling feature does exist in Django 5.1+ with the syntax OPTIONS: {"pool": {"min_size": 2, "max_size": 10}}, but the documentation is incomplete:
- Works only with PostgreSQL using psycopg3 (requires
psycopg[pool]package; does not work with psycopg2, MySQL, SQLite, or other databases) - Critical requirement: Must set
CONN_MAX_AGE = 0when using pooling; Django will raiseImproperlyConfiguredifCONN_MAX_AGE > 0orNone - Not a universal solution for the gevent + persistent connections issue across all database backends
The documentation should explicitly state these constraints and the database/driver dependency to avoid misleading developers into attempting an incompatible solution.
🤖 Prompt for AI Agents
In `@developer/gsoc-ideas-2026.rst` around lines 1449 - 1456, Update the Django
gevent/persistent-connections section to clarify that Django 5.1+ native pooling
(referenced as OPTIONS: {"pool": {"min_size": 2, "max_size": 10}}) only works
with PostgreSQL using psycopg3 (installable as psycopg[pool]) and does not
support psycopg2, MySQL, SQLite, or other backends; explicitly state the
critical requirement to set CONN_MAX_AGE = 0 (Django raises ImproperlyConfigured
if CONN_MAX_AGE > 0 or None) and note that this pooling is not a universal fix
for gevent + persistent connections across all database drivers.
nemesifier
force-pushed
the
gsoc26-monitoring-scalability
branch
from
70ada53 to
fd31af6
Compare
1 participant
GSoC26: Resource Aware Priority Task Scheduling for OpenWISP