Thoughts on Mac Studio M3 Ultra with 256gb for open claw and running models locally

I know a lot of people say to just pay for API usage and those models are better, and I plan to keep doing that for all of my actual job work.

But for building out my own personal open claw to start running things on the side, I really like the idea of not feeding all of my personal data right back to them to train on. So I would prefer to run locally.

Currently I have my gaming desktop with a 4090 that I can run some models very quickly on, but I would like to run a Mac with unified memory so I can run some other models, and not care too much if they have lower tokens per second since it will just be background agentic work.

So my question is: M3 ultra with 256gb of unified memory good? I know the price tag is kinda insane, but I feel like anything else with that much memory accessible by a GPU is going to be insanely priced. And with the RAM and everything shortages...I'm thinking the price right now will be looking like a steal in a few years?

Alternatively, is 96gb of unified memory enough with an M3 Ultra? Both happen to be in stock near me still, and the 256gb is double the price....but is that much memory worth the investment and growing room for the years to come?

Or just everyone flame me for being crazy if I am being crazy. lol.