this post was submitted on 03 Jul 2023
1 points (100.0% liked)

Machine Learning - Learning/Language Models

0 readers
1 users here now

Discussion of models, thier use, setup and options.

Please include models used with your outputs, workflows optional.

Model Catalog

We follow Lemmy’s code of conduct.

Communities

Useful links

founded 1 year ago
MODERATORS
[email protected]
1
GitHub - wgryc/phasellm: Large language model evaluation and workflow framework from Phase AI. (github.com)
submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]
0 comments fedilink hide all child comments
 

Docs: https://phasellm.com/docs/phasellm/eval.html

This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

Features:

  • 200+ tasks implemented. See the task-table for a complete list.
  • Support for models loaded via transformers (including quantization via AutoGPTQ), - GPT-NeoX, and Megatron-DeepSpeed, with a flexible tokenization-agnostic interface.
  • Support for commercial APIs including OpenAI, goose.ai, and TextSynth.
  • Support for evaluation on adapters (e.g. LoRa) supported in HuggingFace's PEFT library.
  • Evaluating with publicly available prompts ensures reproducibility and comparability between papers.
  • Task versioning to ensure reproducibility when tasks are updated.
no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here