I’ve never used oobabooga but if you use llama.cpp directly you can specify the number of layers that you want to run on the GPU with the -ngl flag, followed by the number.

So, as an example, a command (on linux) from the directory you have the binary, to run its server would look something like: ./llama-server -m "/path/to/model.gguf" -ngl 10

Another important flag that could interest you is -c for the context size.

This will put 10 layers of the model on the GPU, the rest will be on RAM for the CPU.

I would be surprised if you can’t just connect to the llama.cpp server or just set text-generation-webui to do the same with some setting.

At worst you can consider using ollama, which is a llama.cpp wrapper.

But probably you would want to invest the time to understand how to use llama.cpp directly and put a UI in front of it, Sillytavern is a good one for many usecases, OpenWebUI can be another but - in my experience - it tends to have more half baked features and the development jumps around a lot.

As a more general answer, no, the safetensor format doesn’t directly support quantization, as far as I know

AFAIK it is still a tuning of llama 3[.1], the new Base models will come with the release of 4 and the “Training Data” section of both the model cards is basically a copy paste.

Honestly I didn’t even consider the fact they would not be giving Base models anymore before reading this post and, even now, I don’t think this is the case. I went to search the announcements posts to see if there was something that could make me think about it being a possibility, but nothing came out.

It is true that they released Base models with 3.2, but there they had added a new projection layer on top of that, so the starting point was actually different. And 3.1 did supersede 3…

So I went and checked the 3.3 hardware section and compare it with the 3 one, the 3.1 one and the 3.2 one.

So yeah, I’m pretty sure the base of 3.3 is just 3.1 and they just renamed the model in the card and added the functional differences. The instruct and base versions of the models have the same numbers in the HW section, I’ll link them at the end just because.

All these words to say: I’ve no real proof, but I will be quite surprised if they will not release the Base version of 4.

• 3
  • 70B
  • 70B Instruct
• 3.1
  • 70B
  • 70B Instruct
• 3.2
  • 90B Vision
  • 90B Vision Instruct
• 3.3
  • 70B Instruct

Small note: I did delete my previous post because I had messed up the links, so I had to recheck them, whoops

“Sailor moon eating a crow, while wearing a crow, while riding a crow, and is also a crow.”

That’s a murder for sure

The only text-to-audio model I can think of at the moment is Stable Audio Open, which AFAIK is rather underwhelming for your use-case, if it can even handle stuff more complex than basic sounds - and no lyrics.
It is even under the “new” membership licensing of SAI.

I remember reading about a more recent one, but I currently can’t find it, and I don’t think that that one too could handle lyrics.

I suppose the Music industry is a lot harder to fight, so not a lot of people want to entangle themself with it.

The more images I see and the more impressive Flux seems.

People could already make it run on 16GB cards, let’s hope it can get squeezed down even more.

I wish they used them all, especially XDG_CACHE_HOME which can become pretty big pretty fast.

disable this system security feature temporarily,

This should be - if I’m not mistaken - possible using the pip env var I posted about earlier, like this:

PIP_BREAK_SYSTEM_PACKAGES=1 sudo apt install howdy

Or exporting it for the current shell, before running the installation

export PIP_BREAK_SYSTEM_PACKAGES=1

But I personally highly discourage it, because - AFAIK - if it even works it will mess up the deps in your system.

I’m no python expert but reading around it seems your only real solution is using a virtual environment, through pipx or venv as you already had found out, or using the

--break-system-packages

* Allow pip to modify an EXTERNALLY-MANAGED Python installation

  (environment variable: `PIP_BREAK_SYSTEM_PACKAGES`)

pip flag which, as the name suggest, should be avoided.

EDIT: After rereading I got your problem better and I was trying to read the source for Howdy to see how to do it, so far no luck.

It’s an error with a dependency written in Rust, the workaround is to use an older toolchain (1.72), it is fixed in the newer code of tokenizers, but probably it is not updated in AUTOMATIC1111 yet: you should check their bug tracker

To have more info you can read this issue: Link

Considering you are not using the Flatpak anymore it is, indeed, strange. The only reasons I can think of are: your network manager is using the wrong network interface to route your traffic ( if you go on an ip checking site like for example ipinfo do you see yours or the VPN’s IP?) or that you have WebRTC enabled and the broadcaster is getting your real ip through that.

For the first case it can get pretty complicated, but it is probably an error during the installation of the VPN app or you set up multiple network managers and it gets confused on which one to configure. You should also enable the Advanced Kill Switch in the configuration.

For the second case you could try adding something like the Disable WebRTC add-on for firefox and check if it works. Remember to enable it for Private Windows too.

The last thing I can think of is that you allowed the broadcaster to get your real geolocation (in firefox it should be a small icon on the left of the address bar), or you are leaking some kind of information somewhere: there are a bunch of site that check for ip leak, but I don’t know if that goes too deep for you.
If you want to check anyway the first two results from DDG are browserleaks and ipleak. Mullvad offered one too but it is currently down.

EDIT: If you enable the Advanced Kill Switch, and the app is working correctly, internet will not work while you are not connected to a VPN server or until you disable the switch again, so pay attention to that.

Probably when you installed the second linux you overwrote the boot loader instead of adding a new UEFI entry point.

But I’ve never had a Mac, so take this with a pinch of salt, and honestly considering things can change based on what, in which order, and how you installed things… it could be something else.

Unfortunately, in general, people tend to just read, vote, and then forget about it without checking back: that’s why I always try to add some source or ways to verify what I post.

And - in this specific case - probably some people just like LTT, I assume.

On the contrary, in my opinion if they are clearly labelled as a joke, they are a great way for people who don’t understand them to ask why and, in the process, being a little more informed on what not to do and what it’s dangerous.

Especially because there’s really no risk of emulation in this case.

Linus is an investor in the framework company: Source on the framework forums that links to the video on youtube

Currently they have direct shipping for some EU’s members (Germany, France, Ireland, Netherlands, Austria, Italy, Spain, Belgium) Source

And I think the shipment costs are currently already in the price for the laptop Source

But I’ve not bought one so I don’t have direct experience

Because, as pointed in the page, Servo is being developed as a(n embeddable) Rendering Engine, not as a full blown end user Browser.
Its alternatives are not Chrome, Safari or Firefox, but Webkit, Blink and Gecko

There’s an example GUI called Servoshell, but it is more of a testing ground and example on how to embed the engine in an app than a serious alternative to anything currently in the market.

Already this kind of work is difficult and daunting. Adding to it a full GUI would make it completely impossible for the current size and financial backing Servo has.

Big words aside it just means that Servo wants to be only one of the parts that compose a real browser: the one that takes HTML, Javascript, WASM and translates them into the things you see on your monitor. All the user facing functionality are left to the devs of the app that embed it.