Monday, May 27th, 2024 | 11:00 - 12:00 a.m.
Part 1: "Adapter Fairness": "Current natural language processing (NLP) research tends to focus on only one or, less frequently, two dimensions -- e.g., performance, privacy, fairness, or efficiency -- at a time, which may lead to suboptimal conclusions and often overlooking the broader goal of achieving trustworthy NLP. Work on adapter modules focuses on improving performance and efficiency, with no investigation of unintended consequences on other aspects such as fairness. To address this gap, we conduct experiments on three text classification datasets by either (1) finetuning all parameters or (2) using adapter modules."
Part 2: "Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget”: "Compared to standard language model (LM) pretraining (i.e., from scratch), Knowledge Distillation (KD) entails an additional forward pass through a teacher model that is typically substantially larger than the target student model. As such, KD in LM pretraining materially slows down throughput of pretraining instances vis-a-vis pretraining from scratch. Scaling laws of LM pretraining suggest that smaller models can close the gap to larger counterparts if trained on more data (i.e., processing more tokens)—and under a fixed computation budget, smaller models are able be process more data than larger models. We thus hypothesize that KD might, in fact, be suboptimal to pretraining from scratch for obtaining smaller LMs, when appropriately accounting for the compute budget.”
Part 3: Most likely, Duc will also discuss the ideas we have for his research stay with us (~Cross-cultural Hate Speech). Feedback is highly welcome!
Short Bio
I'm a PhD student at JGU Mainz, advised by Katharina von der Wense. My research focuses on analyzing and developing techniques that balance efficiency and fairness in NLP models. While numerous approaches have been developed to enhance the resource efficiency,
their impact on model fairness remains largely unclear. Prior to this, I completed my bachelor's degree in "Mathematics in Business and Economics", and subsequently pursued a master's degree in "Data Science" with a strong emphasis on NLP. Following the completion
of my master's degree, I transitioned into the industry, where I worked as a data scientist in the autonomous driving field.
Institution
- Professorship of Data Science
