Towards Large Language Model Organization: A Case Study on Abstractive Summarization

In this work we propose ”LLM organization”, an organizational structure-based LLM workflow for improving the performance of standard abstractive summarization techniques and mitigate unfaithful summary generation. We formulated the organizational structure-based LLM workflow as a directed acyclic graph (DAG), where each node corresponds to an LLM and each edge to a communication protocol. Our workflow is benchmarked on 5 datasets from various domains, using 7 evaluation metrics. The results indicate that LLM organization could mitigate unfaithfulness and increase the overall performance of abstractive summarization methods.