Rendered at 06:17:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
zmmmmm 1 days ago [-]
Seems like a nice win even if it just does the example they have:
void matmul(Matrix2DFloat A, Matrix2DFloat B, Matrix2DFloat C, int size) {
for (@Parallel int i=0; i<size; i++) {
for (@Parallel int j=0; j<size; j++) {
float sum = 0f;
for (int k=0; k<size; k++)
sum += A.get(i,k)*B.get(k,j);
C.set(i, j, sum);
}
}
}
But the elephant in the room is the Vector API. Is that going to end up like logging - a standardised API nobody uses because it came 10 years late to the party and isn't as good as what was already established?
And why confuse everybody by calling it a VM when it ships as a JAR file that runs in an existing VM?
pjmlp 1 days ago [-]
I actually use the logging API, it is good enough, and maybe I am old enough not to like to cowboy install dependencies if I can avoid it.
russli1993 1 days ago [-]
what about tiling? seems better to call into existing kernels, fused megakernels than rewrite kernel in java.
mikepapadim 20 hours ago [-]
You can do both. It supports nvidia's cuTile programming model through Tilecontext API or you can write megakernels in Java and compile then to CUDA or even add native libraries in the mix like cudnn. JITllm is a good example how all these can work together
zmmmmm 1 days ago [-]
i might be overly optimistic, but I'm assuming it'll do the tiling by itself with the above. As long as there's not an algorithmic reason to do it a specific way, in theory you could just optimise it for the target hardware.
mikepapadim 20 hours ago [-]
For now it supports directly cuTile through what they call TileContext API. Is a 1-1 mapping of the cuTile C++ implementation.
... which is a single JAR that does SIMD accelerated inferencing of local models, dependency free. Everything is written in Java from the tokenizer up. It can also use Metal on macOS. Small models can be surprisingly powerful these days and all you need is a fast CPU. It works not only for LLMs but also image models, video models and speech.
Remarkably, it is significantly faster than llama.cpp on some tasks like prefill (decode is memory bandwidth constrained so language hardly makes a difference). At least as long as you run on GraalVM (JVM or native image, both work). So you can actually beat C++ performance for AI in Java.
It's impressive stuff.
vips7L 1 days ago [-]
Why GraalVM specifically? Is it something special with the Graal JIT?
And why confuse everybody by calling it a VM when it ships as a JAR file that runs in an existing VM?
You can use it with any JDK21+ and Nvidia or Apple hardware and their Nvidia libs or MLX libs.
My impression was that people mainly use python.
Pros and cons?
https://github.com/Quotation/LongestCocoa
https://qxotic.ai/
... which is a single JAR that does SIMD accelerated inferencing of local models, dependency free. Everything is written in Java from the tokenizer up. It can also use Metal on macOS. Small models can be surprisingly powerful these days and all you need is a fast CPU. It works not only for LLMs but also image models, video models and speech.
Remarkably, it is significantly faster than llama.cpp on some tasks like prefill (decode is memory bandwidth constrained so language hardly makes a difference). At least as long as you run on GraalVM (JVM or native image, both work). So you can actually beat C++ performance for AI in Java.
It's impressive stuff.