Fair Ranking: A Critical Review, Challenges, and Future Directions

Members of the MD4SG Discrimination and Equality in Algorithmic Decision-Making Working Group Gourab Kumar Patro, Lorenzo Porcaro, Laura Mitchell, Qiuyue Zhang, Meike Zehlike, and Nikhil Garg wrote a position paper on the current state of fair rankings (Fair Ranking: A Critical Review, Challenges, and Future Directions) that has been accepted for presentation at ACM Conference on Fairness, Accountability, and Transparency (FAccT) 2022!

The paper summarizes the position of fairness in retrieval systems. More specifically, current fair ranking (or recommendation) system mechanisms often fail to recognize several real-world nuances like delayed impacts, uncertainties in outcomes, and ecosystem behavior. Thus, one must design fairness interventions using an impact-oriented approach with a holistic and long-term view of the ranking system in mind. With regard to these issues, algorithmic impact assessment could be of help. Various applied modeling techniques and simulation frameworks can be used for impact-oriented studies of fairness in ranking systems. However, data bottlenecks and legal hurdles might challenge the efforts toward a holistic view of ranking systems fairness.

How did you meet? #

Our first meeting was held virtually in early December 2020, as part of the Ranking and Recommendation Systems project, a subgroup of the MD4SG Bias, Discrimination, and Fairness Working Group. We are a group of ten people—some from academia (PhD students, postdocs, and an assistant professor) and others from public and private institutions—spread all over the world.

How did you come up with the idea? #

We spent the first two to three months meeting once every two weeks and discussing what each of us was interested in exploring. Some of us were excited to do more experimental research with recommender systems and rankings, while others were more interested in reviewing the fair ranking literature. The difference in interest was partly due to a difference in our backgrounds. However, we all agreed that one of the major limitations of the current approaches in modeling fairness is its static representation and lack of a long-term, impact-oriented perspective. In the end, we chose to take that position and start our literature review around the topic to make the project as inclusive as possible, trying to take advantage of the interdisciplinarity of our group.

How did you collaborate? #

Based on the initial few discussions on fairness and ranking, we started splitting the literature review into main areas and assigning one part to each group member. Then, during our meetings, we discussed the main results found. We did this back-and-forth until we started to delineate a meaningful overall structure. At some point, we also started to look at deadlines for conferences and workshops in the area of fairness in rankings. Several times we failed to prepare something submittable (often going back to review more literature) before finally submitting our work to FAccT 2022.

What were some obstacles you faced? #

Collaborating with people with a different social, cultural, linguistic, and academic background requires a lot of effort to make everyone feel comfortable and part of the project. Small differences, for instance in the vocabulary used for describing specific issues, can also occur between people coming from different backgrounds. In addition, collaborating during a global pandemic negatively influenced how we experienced this period of our lives. Apart from that, finding suitable time slots for meetings was also a challenge, since the group members were from different continents. However, we are quite happy that we overcame these obstacles, executed the project successfully, and published a very nice position paper with a lot of insights for the future of the fair ranking domain.

Any other messages for the general MD4SG (now EAAMO) audience? #

During the project execution, we realized that things that were obvious for some of us were not for others, due to our different backgrounds. As part of a working group with a strong interdisciplinary culture, it was important to take little steps to make everyone comfortable in discussing our views on collective decisions.

Since MD4SG collaborations often last for a long time—during which some might finish their PhD, others might change jobs, or postdocs might become professors—members should try to come out of their disciplinary comfort zones and preferably look for long-term cross-disciplinary collaboration goals. It is always important to ensure that everyone is doing their best based on what they know and that collaborating in a MD4SG working group is first and foremost an opportunity for personal and collective growth. While our team has not been able to meet in person, with COVID-19 restrictions being phased out, we hope to see future project groups having physical meetups at events like conferences and workshops.