[backend] replace ruleApply in background task to direct inferred creation (#11626)

[JeremyCloarec]

Proposed changes

• now send inferred creation operation to worker instead of rule apply

Related issues

• Fix Change rule activation in background tasks #11626

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case (coverage and e2e)
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

Further comments

[codecov]

Codecov Report

❌ Patch coverage is 17.52874% with 287 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.83%. Comparing base (adb3b03) to head (e1a4611).
⚠️ Report is 87 commits behind head on master.

Files with missing lines	Patch %	Lines
...latform/opencti-graphql/src/manager/taskManager.js	11.11%	48 Missing ⚠️
...ncti-graphql/src/rules/containerWithRefsBuilder.ts	12.19%	36 Missing ⚠️
...rc/rules/sighting-incident/SightingIncidentRule.ts	14.70%	29 Missing ⚠️
...latform/opencti-graphql/src/manager/ruleManager.ts	12.50%	28 Missing ⚠️
...src/rules/observed-sighting/ObserveSightingRule.ts	18.18%	27 Missing ⚠️
.../src/rules/indicate-sighted/IndicateSightedRule.ts	16.00%	21 Missing ⚠️
.../rules/sighting-indicator/SightingIndicatorRule.ts	16.00%	21 Missing ⚠️
...ules/sighting-observable/SightingObservableRule.ts	16.00%	21 Missing ⚠️
...tform/opencti-graphql/src/domain/inferredObject.ts	26.31%	14 Missing ⚠️
.../rules/observable-related/ObservableRelatedRule.ts	15.38%	11 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11628      +/-   ##
==========================================
- Coverage   30.86%   30.83%   -0.03%     
==========================================
  Files        2911     2913       +2     
  Lines      192439   192678     +239     
  Branches    39244    39244              
==========================================
+ Hits        59388    59404      +16     
- Misses     133051   133274     +223     

Flag	Coverage Δ
opencti	30.83% <17.52%> (-0.03%)	⬇️
opencti-front	2.46% <ø> (-0.01%)	⬇️
opencti-graphql	68.08% <17.52%> (-0.18%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[richard-julien]

We should validate that the json is correct for the creation

[JeremyCloarec]

Since it's only used internally for now, and since the validation isn't that easy to implement, we'll skip it until it's used externally

[xfournet]

would it possible to have these structure in the GraphQL schema ? or it's a bad idea ?

[JeremyCloarec]

Since it's only used internally for now, and since the validation isn't that easy to implement, we'll skip it until it's used externally

[xfournet]

can be simplified with
createInferredRelationCallback: createInferredRelationCallbackFunction = createInferredRelation

so then createInferredRelationCallback can be directly called
same apply for other methods/rules

but in fact it can be probably resolved at an higher level in rulesApplyHandler

[xfournet]

it look likes we must ignore some parameters in many places, it would be more readable to have an object that contains the different methods

moreover we are replicating the method signature in many places (it was already the case before this change and also apply to other methods like clean & update). It would be more efficient to reuse types, either:

• typing insert : const insert: RuleRuntime['insert'] = async (element, _createInferredEntityCallback, createInferredRelationCallback) => {
• inlining insert in the return so it can be infered by TS:

  return {
    ...def,
    insert: async (element, _createInferredEntityCallback, createInferredRelationCallback) => applyUpsert(element, createInferredRelationCallback),
    update: async (element) => applyUpsert(element),
    clean: async (element: StoreObject, deletedDependencies: Array<string>) => {
      await deleteInferredRuleElement(def.id, element, deletedDependencies);
    },
  };

[xfournet]

To make sure I understand correctly:

In the new approach, bundles are split using RULE_APPLY_MAX_BUNDLE_SIZE. The same logic should also be applied to standardOperationCallback, which currently doesn’t split and creates a single (potentially very large) bundle.
Adding queues introduces an extra step and increases code complexity.

Is this additional complexity intended to take advantage of the worker queues’ consumption regulation?

[JeremyCloarec]

standardOperationCallback bundles are already split (at least in query tasks) during the elList with MAX_TASK_ELEMENTS (it can be found in the opts in taskQuery).

As for your question, just to make sure I understand it correctly also: you're asking specifically about the new RULE_APPLY_MAX_BUNDLE_SIZE, and not the overall design of sending inferred objects in the bundle right?
The RULE_APPLY_MAX_BUNDLE_SIZE was used to prevent very large bundle from being sent to the worker, since a single ruleApply call can lead to a very large amount of inferred data. Only sending a bundle at the end of the ruleApply could lead to a bundle having >100k objects, which in theory could be handled by the worker, but it would put unnecessary load on the splitting of the bundle

[xfournet]

It should be removed since it has already been done

[Copilot]

Pull request overview

This PR refactors the rule application system to send inferred object creation operations directly to workers instead of applying rules in the background task manager. The change enables background tasks to generate inferred entity and relation creation instructions that workers can process asynchronously.

Key changes:

• Rule insert methods now accept callback functions for creating inferred entities/relations
• Background task manager generates inferred creation bundles instead of applying rules directly
• New GraphQL mutations and Python client methods handle inferred object creation

Reviewed changes

Copilot reviewed 20 out of 22 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
SightingObservableRule.ts	Updated insert method signature to accept inferred creation callbacks
SightingIndicatorRule.ts	Updated insert method signature to accept inferred creation callbacks
SightingIncidentRule.ts	Modified to use callbacks and generate standard IDs for inferred entities
relationWithRelationBuilder.ts	Updated insert method to use callback pattern
relationToRelationBuilder.ts	Updated insert method to use callback pattern
ObserveSightingRule.ts	Updated insert method to use callback pattern
ObservableRelatedRule.ts	Updated insert method to use callback pattern
LocalizationOfTargetsRule.ts	Updated insert method to use callback pattern
IndicateSightedRule.ts	Updated insert method to use callback pattern
containerWithRefsBuilder.ts	Updated to use callbacks and refactored bundle element building
inferredObject.ts	New domain file with inferred creation methods
inferredObjectResolvers	New resolver for inferred object mutations
taskManager.js	Added ruleApplyCallback to generate inferred creation bundles
ruleManager.ts	Updated to accept callback functions throughout rule processing
opencti.graphql	Added inferredRelationAdd and inferredEntityAdd mutations
relay.schema.graphql	Added inferredRelationAdd and inferredEntityAdd mutations
opencti_stix2.py	Added handling for inferred_entity and inferred_rel operations
opencti_api_inferred.py	New Python API client for inferred object creation
opencti_api_client.py	Integrated OpenCTIApiInferred into client