Revolutionizing Higher Ed Rankings

The USNews college and graduate school rankings have always been criticized with its flawed methodology. Despite that, universities still strive hard to chase even the slightest improvement in that ranking, showing its weight and importance to prospective students and parents. In the computer science community, frustrated by the flawed USNews ranking, Dr. Emery Berger from University of Massachusetts Amherst created the CSRankings (https://csrankings.org/) in 2017, which has quickly became the gold standard for applicants to computer science graduate schools all over the world. 

CSRankings is a big step forward in comparison with USNews because instead of relying on questionnaires to department chairs and deans who might be several years removed from first-line research as the USNews ranking, CSRanking directly counts the number of publications in top computer science conferences by each CS faculty. However, over the years, the importance of CSRankings has grown, which has impacted hiring practices in many universities to hire people who publishes a significant amount of papers. This has in turn encouraged graduate students who want to be faculties to value quantity over quality, and we are now seeing fresh PhD graduates hired as incoming assistant professors with already more than 70 publications in top conferences within only a 5-year career. Such games to improve CSRankings encourages quantity over quality, which not only adds a significant amount of reviewer load and mediocre quality publications, but also is toxic to the research culture in general of the entire computer science discipline.

Objectives


Develop a new mechanism of ranking authors by the usage of Large Language Models (LLMs) and connect it with the website developed during last year’s capstone project.

During last year’s capstone project, we have already 1) developed a website and 2) collected names of researchers to be displayed on the website. However, the original idea of a survey-based ranking is not easy to materialize because of the low response rate to the survey. Hence, a new idea is to utilize large language models (LLMs) to measure research impact. The general idea would be to:

1) Input papers into large language models (e.g. Llama 3 as a free LLM) and locate the references that each paper finds to be important or lavish high praises to, then, we can collect the author lists and affiliations from those important papers and assign scores to them. 

2) Afterwards, we will proceed to remove geographical and cultural bias, and 

3) Create a fair and equitable ranking formula to be displayed on the website of the rankings.

Motivations


I believe it is time to create a new ranking system that will upend this toxic trend and restore sanity in computer science research, redirect the focus from quantity to quality. This new ranking should not merely count number of papers but focus on the research impact of each paper. How do we measure research impact? Last year we planned to do a survey, but we had low response rate. So this year, we will try to utilize Large Language Models (LLMs) to work on it.

Qualifications


Minimum Qualifications:
  • Python programming and debugging experience.

Preferred Qualifications:
  • Experience using large language model APIs.
  • Experience building websites


Details


Project Partner:

Fuxin Li

NDA/IPA:

No Agreement Required

Number Groups:

1

Project Status:

Accepting Applicants

Website:
https://github.com/Torin-Perkins/revolutionizing-higher-ed-rankings
Keywords:
WebPythonMachine Learning (ML)Consultancy
Card Image Capstone