Testing Mulitple Local LLM with LM Studio

Testing multiple local LLMs can be a cumbersome process for developers and researchers. Manually assessing the performance of different models often involves tedious tasks such as tracking response times, readability scores, and collating results from various sources. This workflow addresses these frustrations by automating the testing process, allowing users to focus on interpreting results rather than getting bogged down in the data collection. By eliminating manual entry and data aggregation, it saves valuable time and reduces the potential for human error, enhancing productivity in model evaluation.

This n8n workflow utilizes a series of integrated nodes to facilitate the testing of local LLMs. It begins with the 'chatTrigger' node, which initiates the workflow based on user input. Next, the 'lmChatOpenAi' node processes the input through various LLMs to generate responses. The 'httpRequest' node is employed to fetch necessary data, while 'dateTime' nodes are used to timestamp each interaction. Responses are analyzed for readability scores, utilizing the 'code' node for score calculations. Finally, results are recorded in 'googleSheets' for easy access and analysis, with 'set' and 'splitOut' nodes used to organize the output.

This workflow is particularly beneficial for data scientists, machine learning engineers, and academic researchers who frequently work with language models. For instance, a data scientist could use it to compare the effectiveness of different LLMs in natural language processing tasks. Similarly, an academic researcher might assess the readability of model outputs for educational content creation. Teams focused on AI development and testing will find this automation invaluable for enhancing their evaluation processes.

To get started with this n8n workflow, simply import the template into your n8n instance using FlowEngine. You can customize nodes to fit your specific requirements, such as adding more LLMs or changing output formats. Once configured, you can deploy the workflow directly within n8n and begin testing multiple models efficiently.