According to Zeynep Tufecki’s recent keynote at NeurIPS (which I summarize here), the problem is that society is built on assumptions that certain things will be hard (or “load bearing frictions”), i.e., that only humans can generate outputs with certain properties. LLMs break our ability to conclude there is proof of effort, or of authenticity or sincerity. Gatekeeping is a necessary function, and when the old mechanisms stop working, other measures will step in, like relying on the prestige of the candidate’s institution or their connections to decide who to hire, or what papers to cite or publish. When those things are no longer hard, some mechanism must step in in its place, and it may not be ideal. The point being if you break something important, you don’t necessarily get something better unless you build something better.