Regarding the issue of model response prompt words

[2450292765]

Hi, @Md-Ashraful-Pramanik @rizwan09
As I am very interested in this work, I am currently replicating this experiment. I have encountered two problems and hope to receive your guidance.
1. When I reproduced the experiment using the codesim method, the model sometimes output some prompt words, resulting in a relatively low accuracy rate. The model I used was the locally deployed Qwen2.5-7B-Instruct, and I employed vllm to accelerate reasoning. Below is a screenshot of Result.jsonl

2. **Whether or not vllm is used, there will always be a large amount of repetitive information in codesim. Is this normal?**

[Md-Ashraful-Pramanik]

Thank you for reaching out. I have a question regarding your experiment. Do you use quantization?

…

On Sat, Oct 18, 2025, 10:56 AM Zhiquan Jia ***@***.***> wrote: *2450292765* created an issue (kagnlp/CodeGenerator#3) <#3> Hi, @Md-Ashraful-Pramanik <https://github.com/Md-Ashraful-Pramanik> @rizwan09 <https://github.com/rizwan09> As I am very interested in this work, I am currently replicating this experiment. I have encountered two problems and hope to receive your guidance. *1. When I reproduced the experiment using the codesim method, the model sometimes output some prompt words, resulting in a relatively low accuracy rate. The model I used was the locally deployed Qwen2.5-7B-Instruct, and I employed vllm to accelerate reasoning. Below is a screenshot of Result.jsonl* image.png (view on web) <https://github.com/user-attachments/assets/b33c1d7a-f9c5-4fe2-a9d2-d884b80036ea> 2. **Whether or not vllm is used, there will always be a large amount of repetitive information in codesim. Is this normal?** image.png (view on web) <https://github.com/user-attachments/assets/f7a5877e-4369-4b5e-bf4d-a99e740dfb5f> — Reply to this email directly, view it on GitHub <#3>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANO3RTUYBZJXVU7RPKHCHSL3YHB73AVCNFSM6AAAAACJRFB37KVHI2DSMVQWIX3LMV43ASLTON2WKOZTGUZDQMBYGY2TGNA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[2450292765]

Thank you for reaching out.
I have a question regarding your experiment. Do you use quantization?
感谢您的联系。
关于您的实验，我有一个问题。您使用量化了吗？
…

Thank you very much for your reply. I didn't use quantification. I have solved this problem. The reason for this problem is that I didn't use 'tokenizer.apply_chat_template' at the beginning. Also, for the 'Direct' method, is its pass@1 a bit low？

[Md-Ashraful-Pramanik]

Great. In our paper we didn't experiment with Qwen so i have no idea about the ***@***.*** using Qwen.

…

On Thu, Oct 23, 2025, 7:53 PM Zhiquan Jia ***@***.***> wrote: *2450292765* left a comment (kagnlp/CodeGenerator#3) <#3 (comment)> Thank you for reaching out. I have a question regarding your experiment. Do you use quantization? 感谢您的联系。 关于您的实验，我有一个问题。您使用量化了吗？ … <#m_6475068254802628752_> Thank you very much for your reply. I didn't use quantification. I have solved this problem. The reason for this problem is that I didn't use 'tokenizer.apply_chat_template' at the beginning. Also, for the 'Direct' method, is its ***@***.*** a bit low？ — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANO3RTQLO5DMZGFMKOABVLL3ZDMUHAVCNFSM6AAAAACJRFB37KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMZXGE2TIOBRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>