Skip to content

Commit 300997f

Browse files
committed
add codecontests results
1 parent 0595010 commit 300997f

File tree

1 file changed

+56
-4
lines changed

1 file changed

+56
-4
lines changed

index.html

Lines changed: 56 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -267,10 +267,62 @@ <h3>How It Works:</h3>
267267

268268
</section>
269269

270-
<!-- <section>
271-
<h2 id="video">Video</h2>
272-
<iframe height="528" src="#" title="Supplemental video" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
273-
</section> -->
270+
<section>
271+
<h2 id="results">Results</h2>
272+
<p>We evaluate Librarian against state-of-the-art code agents on MiniCode across all three domains. The results demonstrate Librarian's effectiveness at creating compressed, correct refactorings.</p>
273+
274+
<figure class="table-figure">
275+
<table class="table-styled">
276+
<thead>
277+
<tr>
278+
<th><strong>Agent</strong></th>
279+
<th><strong>Tokens</strong></th>
280+
<th><strong>CC</strong></th>
281+
<th><strong>Pass %</strong></th>
282+
<th><strong>MDL</strong></th>
283+
<th><strong>MDL %</strong></th>
284+
</tr>
285+
</thead>
286+
<tbody>
287+
<tr>
288+
<td>original</td>
289+
<td>12946</td>
290+
<td>238</td>
291+
<td>82.0</td>
292+
<td>13346.09</td>
293+
<td>100.0</td>
294+
</tr>
295+
<tr>
296+
<td>sonnet 3.7</td>
297+
<td>18157</td>
298+
<td>280</td>
299+
<td>93.9</td>
300+
<td>14357.77</td>
301+
<td>107.4</td>
302+
</tr>
303+
<tr>
304+
<td>sonnet 4</td>
305+
<td>12756</td>
306+
<td>224</td>
307+
<td>84.4</td>
308+
<td>10292.62</td>
309+
<td>77.1</td>
310+
</tr>
311+
<tr>
312+
<td>codex-mini</td>
313+
<td>13178</td>
314+
<td>238</td>
315+
<td>82.0</td>
316+
<td>11608.58</td>
317+
<td>86.8</td>
318+
</tr>
319+
</tbody>
320+
</table>
321+
<figcaption style="text-align: center;">Table 2: Results on the MiniCode CodeContests split</figcaption>
322+
</figure>
323+
324+
Check out the paper for more!
325+
</section>
274326

275327
<section class="section" id="citation">
276328
<div class="container is-max-desktop content">

0 commit comments

Comments
 (0)