guanaco-65B is my favorite. It’s pretty hard to go back to 33B models after you’ve tried a 65B.

It’s slow and requires a lot of resources to run though. Also, not like there are a lot of 65B model choices.

With a quantized GGML version you can just run on it on CPU if you have 64GB RAM. It is fairly slow though, I get about 800ms/token on a 5900X. Basically you start it generating something and come back in 30minutes or so. Can’t really carry on a conversation.

That’s the impression I got from playing with it. I don’t really use LLMs for anything practical, so I haven’t done anything too serious with it. Here’s are a couple examples of having it write fiction: https://gist.github.com/KerfuffleV2/4ead8be7204c4b0911c3f3183e8a320c

I also tried with plain old llama-65B: https://gist.github.com/KerfuffleV2/46689e097d8b8a6b3a5d6ffc39ce7acd

You can see it makes some weird mistakes (although the writing style itself is quite good).

If you want to give me a prompt, I can feed it to guanaco-65B and show you the result.

I was pretty impressed by guanaco-65B, especially how it was able to remain coherent even way past the context limit (with llama.cpp’s context wrapping thing). You can see the second story is definitely longer than 2,048 tokens.

It’s a different type of model. llama.cpp only supports LLaMA models while GGML (the machine learning library llama.cpp is based on) has examples of various models with different architectures. WizardCoder, MPT, Bloom, probably very soon Falcon. Also some separate projects use GGML to support other models (including some of the ones I listed). For example the Rust “llm” project can support LLaMA models, MPT, BLOOM.

It looks like a frontend that just bundles a bunch of stuff together. Oobabooga’s webui thing is similar: you can run stuff with llama.cpp, GPTQ, etc. What models and features are supported is going to depend on how the frontend manages that stuff. There are also forks of llama.cpp like koboldc++ which may support different models/features/formats (I know koboldc++ supports some older GGML file formats that llama.cpp broke compatibility with).

I don’t know if it does or doesn’t, I was just saying those two projects seemed similar: presenting a frontend for running inference on models while the user doesn’t necessarily have to know/care what backend is used.

If you use DuChinese, I also made a userscript that adds a number of helpful features.

If you’re technically inclined, you can look in the beta-0.6 branch - that version adds even more stuff like a pinyin section and the ability to view the vocabulary list in long form.

---

For the impatient (includes screenshot): https://github.com/KerfuffleV2/duchinese-helper

Feedback welcome!

Note: Currently I’ve only tested with with the ViolentMonkey userscript browser extension (available for both Chrome and Firefox). I’m willing to work with anyone that has problems running it with a different userscript manager.

Abbreviated version of the README from the repo above follows. See link for the full version.

---

Disclaimer

Note: This is a third-party extension and not supported by the official DuChinese site: https://duchinese.net/

Unofficial extensions like this can cause unexpected behavior. If you have any issues using duchinese.net, please make sure you’ve completely disabled this userscript and restarted your browser before contacting their support.

Synopsis

This extension adds a number of features to DuChinese, a site that provides learning content for people studying Mandarin Chinese.

It is released under a permissive open source license (MIT) and the full source code is available for anyone to look at/use.

Note: The extension provides an alternative interface for viewing/interacting with the lesson content. It does not replace any existing elements.

Features

Fonts

You can configure font size and weight. The extension will provide a number of options for font families as well.

Themes

It’s possible configure the color scheme the lesson content/interface is provided in.

Annotations

There are two main types of annotation available: text based (for example, to provide the pinyin for a character) and color-based (for example, to show the tone of the syllable).

Visibility

Annotations (tone colors included, but configurable independently) can have their visibility based on:

1. Whether it’s the first time the word has been seen in a lesson.
2. HSK level.
3. Whether the containing sentence is hovered.

Text

Characters can be annotated with pinyin, IPA, zhuyin (bopomofo), HSK level, tone number, traditional or simplified hanzi (to aid in learning the alternative set of characters).

The annotation position can also be set to appear before, after, above or below the word or even oriented vertically (useful for zhuyin).

Tone Color

Characters can be colored based on the tone of the syllable.

Sandhi

You can enable displaying a mark below characters where a tone change occurs or is expected to occur. For example, multiple tone 3 syllables in a row.

Audio

The extension adds a normal audio element, this allows adjusting the volume or seeking through the content directly. It is also possible to download the audio using this. The extension also adds the ability to set a default volume level and playback speed.

As with the normal site interface, it is possible to click within the lesson text to seek to that part of the audio and while audio is playing the current syllable/word/sentence will be tracked.

Vocabulary

The extension adds a collapsible vocabulary section which will show a list of words in the lesson, organized by HSK level. If the word doesn’t appear in HSK, it will appear as “Other” or “Name”. There are also some statistics available such as average HSK level. It is possible to click on a word to hear it spoken (from the location it appeared in the lesson).

Misc

You can copy the text of the lesson to clipboard.

Installation

Please see: https://github.com/KerfuffleV2/duchinese-helper#installation

Did You Have Permission

Please see: https://github.com/KerfuffleV2/duchinese-helper#did-you-have-permission-to-make-this

Assuming ALPHV/BlackCAT follows through this will make him liability for profits.

I mean, just blackmailing people is enough to get someone in big legal trouble. So I’m going to guess he’s 1) pretty anonymous and 2) living in a country that won’t cooperate with the US or extradite their citizens.

Sweet. So now everyone has free access to ChatGPT 4?