Skip to content

Adds support for image pasting from clipboard, and multiple file attachments #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sdrinf wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Adds support for image pasting from clipboard, and multiple file attachments #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions src/renderer/src/components/footer/footer.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ interface MessageInputProps {
onKeyDown: (e: React.KeyboardEvent<HTMLTextAreaElement>) => void
onCompositionStart: () => void
onCompositionEnd: () => void
onPaste: (e: React.ClipboardEvent<HTMLTextAreaElement>) => void
attachmentsCount: number
onClearAttachments: () => void
}

// Reusable components
Expand Down Expand Up @@ -81,28 +84,36 @@ const MessageInput = memo(({
onKeyDown,
onCompositionStart,
onCompositionEnd,
onPaste,
attachmentsCount,
onClearAttachments,
}: MessageInputProps) => {
const { t } = useTranslation();

return (
<InputGroup flex={1}>
<Box position="relative" width="100%">
<IconButton
aria-label="Attach file"
aria-label="Clear attachments"
variant="ghost"
{...footerStyles.footer.attachButton}
onClick={onClearAttachments}
>
{attachmentsCount > 0 && (<>
{attachmentsCount}
</>)}
<BsPaperclip size="24" />
</IconButton>
Comment on lines 96 to 106
Copy link

@gemini-code-assist gemini-code-assist bot Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using the paperclip icon (BsPaperclip) for an action that clears attachments is counter-intuitive, as this icon is universally understood to mean "attach". This can lead to a confusing user experience, even though the aria-label is correct. To improve clarity, consider changing the icon when there are attachments to visually indicate the "clear" action. For example, you could show a close icon (IoClose from react-icons/io5, which would need to be imported) next to the attachment count.

Suggested change
<IconButton
aria-label="Attach file"
aria-label="Clear attachments"
variant="ghost"
{...footerStyles.footer.attachButton}
onClick={onClearAttachments}
>
{attachmentsCount > 0 && (<>
{attachmentsCount}
</>)}
<BsPaperclip size="24" />
</IconButton>
<IconButton
aria-label="Clear attachments"
variant="ghost"
{...footerStyles.footer.attachButton}
onClick={onClearAttachments}
disabled={attachmentsCount === 0}
>
{attachmentsCount > 0 ? (
<HStack spacing={1}>
<Box as="span">{attachmentsCount}</Box>
<IoClose size="20" />
</HStack>
) : (
<BsPaperclip size="24" />
)}
</IconButton>

<Textarea
value={value}
onChange={onChange}
onKeyDown={onKeyDown}
onPaste={onPaste}
onCompositionStart={onCompositionStart}
onCompositionEnd={onCompositionEnd}
placeholder={t('footer.typeYourMessage')}
{...footerStyles.footer.input}
/>
/>
</Box>
</InputGroup>
);
Expand All @@ -118,6 +129,9 @@ function Footer({ isCollapsed = false, onToggle }: FooterProps): JSX.Element {
handleKeyPress,
handleCompositionStart,
handleCompositionEnd,
handlePaste,
attachmentsCount,
clearAttachments,
handleInterrupt,
handleMicToggle,
micOn,
Expand Down Expand Up @@ -146,6 +160,9 @@ function Footer({ isCollapsed = false, onToggle }: FooterProps): JSX.Element {
onKeyDown={handleKeyPress}
onCompositionStart={handleCompositionStart}
onCompositionEnd={handleCompositionEnd}
onPaste={handlePaste}
attachmentsCount={attachmentsCount}
onClearAttachments={clearAttachments}
/>
</HStack>
</Box>
Expand Down
6 changes: 6 additions & 0 deletions src/renderer/src/hooks/footer/use-footer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ export const useFooter = () => {
handleKeyPress: handleKey,
handleCompositionStart,
handleCompositionEnd,
handlePaste,
attachmentsCount,
clearAttachments,
} = useTextInput();

const { interrupt } = useInterrupt();
Expand Down Expand Up @@ -49,6 +52,9 @@ export const useFooter = () => {
handleKeyPress,
handleCompositionStart,
handleCompositionEnd,
handlePaste,
attachmentsCount,
clearAttachments,
handleInterrupt,
handleMicToggle,
micOn,
Expand Down
47 changes: 44 additions & 3 deletions src/renderer/src/hooks/footer/use-text-input.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,15 @@ import { useAiState } from '@/context/ai-state-context';
import { useInterrupt } from '@/components/canvas/live2d';
import { useChatHistory } from '@/context/chat-history-context';
import { useVAD } from '@/context/vad-context';
import { useMediaCapture } from '@/hooks/utils/use-media-capture';
import { ImageData, useMediaCapture } from '@/hooks/utils/use-media-capture';

export function useTextInput() {
const [inputText, setInputText] = useState('');
const [isComposing, setIsComposing] = useState(false);
const [attachedImages, setAttachedImages] = useState<ImageData[]>([]);
const clearAttachments = () => setAttachedImages([]);


const wsContext = useWebSocket();
const { aiState } = useAiState();
const { interrupt } = useInterrupt();
Expand All @@ -20,13 +24,46 @@ export function useTextInput() {
setInputText(e.target.value);
};

const handlePaste = (e: React.ClipboardEvent<HTMLInputElement | HTMLTextAreaElement>) => {
const { items } = e.clipboardData;
let foundImage = false;

for (let i = 0; i < items.length; i += 1) {
const item = items[i];
if (item.type.startsWith('image/')) {
const file = item.getAsFile();
if (file) {
foundImage = true;
const reader = new FileReader();
reader.onload = () => {
setAttachedImages(prev => [
...prev,
{
source: 'clipboard',
data: reader.result as string,
mime_type: file.type,
},
]);
};
reader.readAsDataURL(file);
}
}
}
// Prevent the raw image from being inserted as text
if (foundImage) e.preventDefault();
};
Comment on lines +27 to +54
Copy link

@gemini-code-assist gemini-code-assist bot Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The handlePaste function can be refactored to be more declarative and easier to read. Instead of using a for loop and a boolean flag (foundImage), you can convert e.clipboardData.items to an array and use modern JavaScript array methods like filter and forEach. This makes the intent of the code clearer and improves maintainability.

  const handlePaste = (e: React.ClipboardEvent<HTMLInputElement | HTMLTextAreaElement>) => {
    const imageItems = Array.from(e.clipboardData.items).filter((item) =>
      item.type.startsWith('image/'),
    );

    if (imageItems.length > 0) {
      e.preventDefault();
      imageItems.forEach((item) => {
        const file = item.getAsFile();
        if (file) {
          const reader = new FileReader();
          reader.onload = () => {
            setAttachedImages((prev) => [
              ...prev,
              {
                source: 'clipboard',
                data: reader.result as string,
                mime_type: file.type,
              },
            ]);
          };
          reader.readAsDataURL(file);
        }
      });
    }
  };


const handleSend = async () => {
if (!inputText.trim() || !wsContext) return;
if (!inputText.trim() && attachedImages.length === 0) return;
if (!wsContext) return;

if (aiState === 'thinking-speaking') {
interrupt();
}

const images = await captureAllMedia();
const captured = await captureAllMedia();
const images = [...attachedImages, ...captured];


appendHumanMessage(inputText.trim());
wsContext.sendMessage({
Expand All @@ -37,6 +74,7 @@ export function useTextInput() {

if (autoStopMic) stopMic();
setInputText('');
setAttachedImages([]);
};

const handleKeyPress = (e: React.KeyboardEvent<HTMLInputElement>) => {
Expand All @@ -58,5 +96,8 @@ export function useTextInput() {
handleKeyPress,
handleCompositionStart,
handleCompositionEnd,
handlePaste,
attachmentsCount: attachedImages.length,
clearAttachments,
};
}
4 changes: 2 additions & 2 deletions src/renderer/src/hooks/utils/use-media-capture.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ declare class ImageCapture {
grabFrame(): Promise<ImageBitmap>;
}

interface ImageData {
source: 'camera' | 'screen';
export interface ImageData {
source: 'camera' | 'screen' | 'clipboard';
data: string;
mime_type: string;
}
Expand Down