Skip to content

Handle arrays of images when building prompts#1247

Open
cpinn wants to merge 17 commits intomainfrom
caitlin/modify-nunjucks-parsing-to-match-custom-mustache-behavior
Open

Handle arrays of images when building prompts#1247
cpinn wants to merge 17 commits intomainfrom
caitlin/modify-nunjucks-parsing-to-match-custom-mustache-behavior

Conversation

@cpinn
Copy link
Contributor

@cpinn cpinn commented Jan 8, 2026

Users would like to be able to reference an array when handling images.

This PR introduces the following changes:

  • Variables that contain arrays of images or files will now be automatically expanded when used.

    • If a user has referenced exactly an array of attachment and everything in the variable array is a type of attachment.
  • Nunjucks is updated to mirror what was done for Mustache html parsing, escaping is now disabled in the Nunjucks environment.

  • Mustache JSON-stringifies objects by default. For Nunjucks, we preserve its native behavior and do not JSON-stringify by default. Instead, a tojson filter has been added so that users familiar with Nunjucks or Jinja can explicitly convert data to JSON using standard renderer formats.

Note: I went back and forth on whether to keep Mustache’s default behavior of automatically calling JSON.stringify on objects. Ultimately, I chose not to replicate this in Nunjucks. Since Nunjucks is a different templating language than Mustache, this behavioral difference made more sense to me.

@cpinn cpinn force-pushed the caitlin/modify-nunjucks-parsing-to-match-custom-mustache-behavior branch from 97a2bde to 74e0bc1 Compare January 8, 2026 22:37
@cpinn cpinn closed this Jan 8, 2026
@cpinn cpinn reopened this Jan 8, 2026
throwOnUndefined,
});

// Intercept renderString to wrap objects with custom toString
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nunjucks doesn't really support the type of custom escaping we were doing for mustache. This implements some of the custom parsing we were doing just to make sure that [Object object] doesn't get printed out during the toString method. Kind of hacky but it works to match the behavior.

@cpinn cpinn force-pushed the caitlin/modify-nunjucks-parsing-to-match-custom-mustache-behavior branch from 77b2ee2 to d6a79e6 Compare January 8, 2026 23:49
@cpinn cpinn requested a review from ibolmo January 12, 2026 22:24
return new nunjucks.Environment(null, {
autoescape: true,
const env = new nunjucks.Environment(null, {
autoescape: false,
Copy link
Collaborator

@ibolmo ibolmo Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd double check our docs to make sure we call out that we don't escape by default in both template renderers. perhaps how to escape, for those that want to

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't call this out at all, never did which is why I was quite surprised by it.

{ templateFormat: "nunjucks" },
);
// Arrays output as comma-separated (Nunjucks default behavior)
expect(resultWithAccess).toBe("User: Bob, Age: 25, Items: 1,2,3");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i expected the array to be ["1", "2", "3"] 🤔 (json stringify)

Copy link
Contributor Author

@cpinn cpinn Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't decided if I want to overwrite this or not because this is just the behavior of nunjucks.

if you have an array it doesn't get output as [object object] it just gets outputs the array as such, this is regular rendering. Its a different behavior than mustache above which would print out the array.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like mustache does do as I'd expected. 🤔

@cpinn cpinn requested a review from ibolmo January 14, 2026 00:23
@cpinn cpinn changed the title modify nunjucks parsing to stop parsing to string as [Object object] Handle arrays of images when building prompts Jan 14, 2026
@cpinn cpinn force-pushed the caitlin/modify-nunjucks-parsing-to-match-custom-mustache-behavior branch from 22803a9 to 5df5f53 Compare January 14, 2026 05:25
@cpinn cpinn marked this pull request as draft January 30, 2026 20:56
@cpinn cpinn force-pushed the caitlin/modify-nunjucks-parsing-to-match-custom-mustache-behavior branch from 5bc4989 to 2c9a086 Compare January 30, 2026 22:46
if (!Array.isArray(value)) return null;

// Check if array contains attachments or image URLs (supports mixed arrays)
const allValid = value.every(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flow is that the backend will attempt to turn all the references into URLs. Now if something goes wrong and one of the attachments isn't turned into a URL this would still allow expansion by ensuring the other values are attachments.

A different flow I experimented with was passing the attachment map from the backend for verification. I did not like that mixing of concerns though so I did not think it was a good idea to introduce that logic here.

@cpinn cpinn marked this pull request as ready for review February 3, 2026 00:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants