Technical Report: Auxiliary Tuning and its Application to Conditional\n Text Generation

We introduce a simple and efficient method, called Auxiliary Tuning, for\nadapting a pre-trained Language Model to a novel task; we demonstrate this\napproach on the task of conditional text generation. Our approach supplements\nthe original pre-trained model with an auxiliary model that shifts the output\ndistribution according to the target task. The auxiliary model is trained by\nadding its logits to the pre-trained model logits and maximizing the likelihood\nof the target task output. Our method imposes no constraints on the auxiliary\narchitecture. In particular, the auxiliary model can ingest additional input\nrelevant to the target task, independently from the pre-trained model's input.\nFurthermore, mixing the models at the logits level provides a natural\nprobabilistic interpretation of the method. Our method achieved similar results\nto training from scratch for several different tasks, while using significantly\nfewer resources for training; we share a specific example of text generation\nconditioned on keywords.\n