Feed aggregator

Ask HN: Affordable hardware for running local large language models?

Hacker News - 4 hours 3 min ago

A while back there was a post about running stable diffusion on a Raspberry Pi Zero 2 [1] which was slow but incredibly impressive! And that sparked my curiosity with this question, what is considered affordable hardware for running large language models locally today? I'm aware that there's a lot of work underway to make inference cheap to run at the edge but I'm curious to understand the landscape at present that anyone could purchase. I seen people running models on flagship smartphones but those are more expensive then a mac mini with worse performance.

By affordable I mean no greater then the cost of a current gen base model mac mini ($599) but ideally around the price of a raspberry pi 5 ($79) which when searching for budget PC gets mentioned[2]. Both devices have the same amount of ram in my case (8gb) but different performance observed given the importance of memory bandwidth. I mention these two because I've had experience running llama 3 via ollama on both with success although of slower speeds compared to a full workstation with a commodity GPU i.e. RTX 4090 which starts at ($1599). I'm interested in learning about what other devices are out there that people consider cheap and use for LLMs locally.

[1]: https://news.ycombinator.com/item?id=38646969

[2]: https://www.pcmag.com/picks/the-best-budget-desktop-computers

Comments URL: https://news.ycombinator.com/item?id=40262206

Points: 1

# Comments: 0

Categories: Hacker News

Force Push Gin

Hacker News - 4 hours 9 min ago

Article URL: https://fridaydeployment.co/

Comments URL: https://news.ycombinator.com/item?id=40262182

Points: 1

# Comments: 0

Categories: Hacker News

I realized I am a bad programmer

Hacker News - Sat, 05/04/2024 - 11:59pm

These days I realized I am a bad programmer. I'm sure this is not imposter syndrome.

I graduated in CS at 26, working since age of 24 only in web development, mostly front-end, but more recently full stack.

I sort of coasted through various jobs in the last 13 years.

Managed to became tech lead 5 years ago, but went back to just development due to burnout.

I have never applied any algorithm or particular data structure and I have never worked really really hard, beyond "just" coding.

Now I feel I have wasted all my best years and left with some coding skills that will become obsolete in a few years (if I 'm lucky).

I don't have any satisfaction from my job and I did not develop other soft skills that might help me...

I'm not exactly sure on what to do now... Study DSA? Transition to management? Construction?

Comments URL: https://news.ycombinator.com/item?id=40262152

Points: 1

# Comments: 2

Categories: Hacker News

Single Points of Failure

Hacker News - Sat, 05/04/2024 - 11:52pm
Categories: Hacker News

Adobe Project Neo

Hacker News - Sat, 05/04/2024 - 11:35pm

Article URL: https://projectneo.adobe.com/

Comments URL: https://news.ycombinator.com/item?id=40262078

Points: 1

# Comments: 0

Categories: Hacker News

Pages