Rendered at 16:34:04 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
felix089 34 minutes ago [-]
Quick update, we doubled the models tested based on community contributions, and a new model took the #1 spot. also well over 1000 games have been played against decision model ghosts.
https://opper.ai/jevman-benchmark/#leaderboard
hung 14 hours ago [-]
Shouldn't it be running at every tick of the game rather than just the junctions? Pac-Man should be able to change direction at any point.
felix089 45 minutes ago [-]
Ideally yes but that would not work with the model latency. But it is already partly covered when a ghost enters pacmans corridor he is asked again. Here's how the decisions work in detail https://github.com/opper-ai/jevman-benchmark/blob/main/READM...
hyperhello 11 hours ago [-]
It may not matter if the ghosts' next move is predictable over that maximum distance.
theowaway213456 3 hours ago [-]
That requires much more thinking (because of calculation) though doesn't it? Which decision models are not optimized for.
felix089 6 hours ago [-]
[flagged]
NichoPaolucci 15 hours ago [-]
This is neat. At least I'm still better than the models at pac-man.
Also love the idea of a shared pool for users to try things out. I was considering more of a crowdfunded approach for one of my toy projects, something like... Giving it $10 in credits to begin and somehow allowing users to feed a buck in if they wanted.
felix089 6 hours ago [-]
Great to hear, if it's about inference credits we actually built a thing called an AI wallet where users can just easily pay for their own inference on apps. if that's interesting lmk, id be happy to support this https://opper.ai/ai-wallet
Jev is useful to run locally overnight: it can classify the results while you sleep, ready for you to review in the morning.
fancyswimtime 10 hours ago [-]
Yes, try with toxic/toxichat and CFPB
soltanov 12 hours ago [-]
Running cloud API inference for Pac-Man is peak modern software engineering. A 2 MB local ONNX model on a CPU core beats network round-trip jitter every time.
ozozozd 16 hours ago [-]
Super cool!
I wish the controls were a little easier on mobile.
felix089 8 hours ago [-]
Thank you and yea agreed fixing it now, try again in an hour from now :)
ismailmaj 6 hours ago [-]
it would be nice to have a regular LLM for reference
Also love the idea of a shared pool for users to try things out. I was considering more of a crowdfunded approach for one of my toy projects, something like... Giving it $10 in credits to begin and somehow allowing users to feed a buck in if they wanted.
I trained some to do some interesting things, including playing doom: https://github.com/nicobrenner/jeffy
I’ll try training one for this benchmark, seems like fun
I wish the controls were a little easier on mobile.
What even is this reality.