Skip to content

feat: add interceptor to bypass serialization for heavy resp#2524

Open
minottic wants to merge 1 commit intomasterfrom
fast_pub
Open

feat: add interceptor to bypass serialization for heavy resp#2524
minottic wants to merge 1 commit intomasterfrom
fast_pub

Conversation

@minottic
Copy link
Member

@minottic minottic commented Feb 6, 2026

Description

The thumbnail in the publication data can be quite big (we have cases of >8MB). This causes, in certain envs (e.g. a pod in k8s) the backend to be killed with OOM. The cause seems to be due to nestjs after serialization logics. This allow to bypass the additional interceptors from nestjs and avoid the memory peak.

If this feature is not desired by other facilities, I will be happy to make it conditional or configurable.

@minottic minottic requested a review from a team as a code owner February 6, 2026 15:32
@Junjiequan
Copy link
Member

This looks quite hacky solution to me.
In which field of publishedData do you store the images? I'm not very familiar with the publishedData flow

@minottic
Copy link
Member Author

it's stored in the thumbnail, which, as long as it's under the 16MB limit, it's fine for mongo. What I could do is making the fastResponse thingy parametric and test if the provided field has length > 5 MB otheriwse keep the nestjs serialisers

@HayenNico
Copy link
Member

The thumbnail field for publishedData is deprecated for APIv4 and only kept for backwards compatibility with APIv3 - the intended way to do this is to ingest an attachment separately (attachments can point to publishedData, proposals and samples besides datasets).
My guess would be the memory spike comes from running the publishedData.metadata including the encoded image through validation after

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants