Open(ish) hardware GPU cluster for local LLM work?

General
1

Hey all!

Is anyone aware of any work being done vis a vis an open or mostly-open hardware stack being used for local LLM work? I know there’s the whole GPU problem, but I’ve got a couple of researchers that are doing highly sensitive image analysis work that needs to be air-gapped and having more trusted hardware stack for the work would be useful to them. If anyone knows of any projects in this space, LMK!

Thanks!

2 Likes
2

Maybe this is at least adjacent to what you’re asking.

1 Like
3

Some of my colleagues work with medical images locally; we have a few PCs with a couple of 4090s.

Another group is working with open-source LLMs for edge solutions without connectivity.

If you need more information, you can reach me at jecarrique@sinc.unl.edu.ar.

This is a problem that interests me because I’m starting to work with patient data for a personal health solution for the prevention and improved treatment of type 1 diabetes.

Best regards.