Effectively copying the knowledge in LLMs’ embeddings with small models

In recent years, we have witnessed the booming of large language models (LLMs), e.g., GPT models, which lead to better performance on various downstream applications. However, due to the massive cost of constructing LLMs, ordinary users can only access related services via application programming interface (API) queries with high expenses and long-time waiting. Therefore, effective copying methods to capture the specific embedding ability of LLMs are urgently required. To that end, in this paper, we propose a novel contrastive copying approach for copying the GPT model via a basic open-source small model. Specifically, we first utilize only a subset of data relevant to the target domain to obtain the GPT embeddings and then apply the contrastive copying method to copy the knowledge. In detail, our method could constrain the small model to generate similar embeddings with target LLMs, while at the same time, different from the other sample embeddings. Extensive experiments have validated that our solution could generate an adequate small model to achieve similar performance with the GPT models compared with other competitive methods.