refactor: separate stats endpoints into two, rename keys for more clarity#387
Open
bolinocroustibat wants to merge 5 commits intomainfrom
Open
refactor: separate stats endpoints into two, rename keys for more clarity#387bolinocroustibat wants to merge 5 commits intomainfrom
bolinocroustibat wants to merge 5 commits intomainfrom
Conversation
a81bd73 to
6bbf5b1
Compare
54766bf to
bfe47c0
Compare
maudetes
approved these changes
Feb 6, 2026
Contributor
maudetes
left a comment
There was a problem hiding this comment.
Thank you for the refacto!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #382 and more.
Moved resource stats (
total_count,deleted_count,statuses_count) from/api/status/crawlerto/api/resources/stats. Kepttotal_eligible_countin crawler status (crawler-related and expensive)./api/status/crawlerless SQL-expensive and better balance the expensive queries between/api/status/crawlerand/api/resources/statusRenamed status endpoint keys for clarity (fixes Rename and add keys/values in
/api/status/crawlerresponse for more clarity #382):pending_count→needs_check_countfresh_count→up_to_date_check_countchecked_percentage→needs_check_percentagefresh_percentage→up_to_date_check_percentageAdded
in_progress_countandin_progress_percentagePerformance optimization: Combined 2 SQL queries into 1 (reduced from 3 to 2 queries total in
/api/status/crawler)Renamed variable:
total_resources_filtered→total_eligible_resources(and JSON keytotal_filtered_count→total_eligible_count)Add tests for this new endpoint, adapt the test for
/crawler/statsand add a new test case (outdated check)How it looks
GET /api/resources/stats:{ "total_count": 100, "deleted_count": 3, "statuses_count": { "null": 85, "BACKOFF": 2, "CRAWLING_URL": 1, "TO_ANALYSE_RESOURCE": 0, "ANALYSING_RESOURCE_HEAD": 0, "DOWNLOADING_RESOURCE": 0, "ANALYSING_DOWNLOADED_RESOURCE": 0, "TO_ANALYSE_CSV": 0, "ANALYSING_CSV": 0, "VALIDATING_CSV": 0, "INSERTING_IN_DB": 0, "CONVERTING_TO_PARQUET": 0, "TO_ANALYSE_GEOJSON": 0, "ANALYSING_GEOJSON": 0, "CONVERTING_TO_PMTILES": 0, "CONVERTING_TO_GEOJSON": 0, "TO_ANALYSE_PARQUET": 0, "ANALYSING_PARQUET": 0 }, "cors": { "external_resources_with_cors_data": 42, "external_resources_without_cors_data": 55, "external_resources_cors_coverage_percentage": 43.3, "external_resources_allow_origin_distribution": [ { "access_status": "Accessible (Specific Whitelist)", "unique_resources_count": 20, "percentage": 47.62 }, { "access_status": "Accessible (Wildcard *)", "unique_resources_count": 15, "percentage": 35.71 }, { "access_status": "Blocked (Missing Header)", "unique_resources_count": 5, "percentage": 11.9 }, { "access_status": "Blocked (Other Domain Only)", "unique_resources_count": 2, "percentage": 4.76 } ] } }GET /api/status/crawler:{ "checks": { "in_progress_count": 5, "in_progress_percentage": 4.76, "needs_check_count": 60, "needs_check_percentage": 63.16, "up_to_date_check_count": 35, "up_to_date_check_percentage": 36.84 }, "resources": { "total_eligible_count": 95 } }