Large Language Models (LLMs) have shown remarkable aptitude in code
generation but still struggle on challenging programming tasks. Self-repair —
in which the model debugs and fixes mistakes in its own code — has recently
become a popular way to boost performance in these settings. However, only very
limited studies on how and when self-repair works effectively exist in the
literature, and one might wonder to what extent a model is really capable of
providing accurate feedback on why the code is wrong when that code was
generated by the same model. In this paper, we analyze GPT-3.5 and GPT-4’s
ability to perform self-repair on APPS, a challenging dataset consisting of
diverse coding challenges. To do so, we first establish a new evaluation
strategy dubbed pass@t that measures the pass rate of the tasks against the
total number of tokens sampled from the model, enabling a fair comparison to
purely sampling-based approaches. With this evaluation strategy, we find that
the effectiveness of self-repair is only seen in GPT-4. We also observe that
self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback
on the programs generated by GPT-3.5 and using expert human programmers to give
feedback on the programs generated by GPT-4, we unlock significant performance
gains.