Denis Neumüller

Context
We are convinced that automatically detecting algorithm implementations in a code base can be helpful to gain knowledge about, which concerns are present in the code base, how they are solved and which components are involved. This knowledge can then support the tasks of software comprehension, software architecture recovery and software maintenance.
Examples of algorithms that could be interesting to detect and some of the insights their detection provides:

• Quicksort -> The application sorts data structure x.
• A* -> The application does path search.
• Raft (Consensus Algorithm) -> The application is a distributed, fault-tolerant system.

Research Question
Large Language Models (LLMs) achieve impressive results on code-related tasks such as code clone detection, code summarization and code generation. 
Our first experiments using LLMs for algorithm detection indicate promising performance, with F1 scores of about 77%.
However, our evaluation has so far focused only on smaller algorithms such as Binary Search, Bubble Sort, Matrix Transposition etc.
We are now interested in evaluating LLMs on more complex algorithms implemented in real code bases.
Therefore, the main research question of this thesis is:

• How do LLMs perform in recognizing complex algorithms in source code bases?

Tasks

• Create a dataset of more complex algorithms (e.g. Levenshtein distance, Raft), preferably from real open-source projects, using code from GitHub, Maven, or other web resources.
• Adapt the current LLM evaluation code written in Python to support different evaluation strategies, such as file-based and API-based evaluations.
• Evaluate different LLMs (e.g., Deepseek, Mixtral, LLaMA3, ChatGPT) using the dataset and the implemented evaluation strategies.
• Furthermore, we are interested in explaining the predictions of LLMs, (especially for failure cases).

No prior knowledge of machine learning (ML) is required for this thesis. However, you should be open to familiarizing yourself with a new subject area.

Related Work

• Publications regarding the (automated) curation of datasets:
  • https://bdqnghi.github.io/files/SANER_2019_bilateral_dependency.pdf
  • https://arxiv.org/pdf/1710.06159
  • https://link.springer.com/article/10.1007/s10664-017-9512-6
  • https://link.springer.com/article/10.1007/s10664-020-09825-8
• Publications regarding the application of LLMs to (other) code related tasks:
  • https://ieeexplore.ieee.org/document/10576984
  • https://arxiv.org/abs/2308.01191
• Examples of LLM API interaction:
  • https://dl.acm.org/doi/pdf/10.1145/3650212.3680384
  • https://arxiv.org/abs/2408.02232
• Publications regarding explanations:
  • https://dl.acm.org/doi/10.1145/3524610.3527914
  • https://arxiv.org/abs/2407.19200
  • https://arxiv.org/abs/2401.12874

Contact
If you are interested and/or have any questions, feel free to contact me any time.
We can discuss the topic further and try to adapt it to your personal preferences.

Denis Neumüller