Rendered at 02:23:59 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
snitty 8 hours ago [-]
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
Aurornis 6 hours ago [-]
Jev's launch was the best AI era marketing campaign I've seen so far. It was executed and timed so perfectly that Jev became synonymous with decision models overnight.
I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.
The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.
sublinear 6 hours ago [-]
I think you have a very narrow and specific experience with people claiming to be "devs".
I don't know of any devs in real life who would loudly and publicly get excited about anything unless it was their own completed project to show off.
The tools are a footnote. It's precisely because it's so hard to get a project that far along that they would be familiar with many more tools than they would ever speak about. I suppose some people have a low bar. I wouldn't really call them "devs", especially now in this "AI era".
Aurornis 5 hours ago [-]
> I don't know of any devs in real life
I think you're coming at this from a very narrow and specific experience with a few people you know.
Jev really went from the announcement to being discussed all over overnight. We're discussing how popular things are in general, not among your narrow and specific group of people you work with.
asveikau 4 hours ago [-]
I agree with what you said above, but not this. Jev was discussed "all over" by people chasing hype. I think that is actually a narrow set of people.
sublinear 4 hours ago [-]
"No you!" :-)
I think it's clear we're at the tail end of this cycle. I'm very glad to see it.
seanhunter 2 hours ago [-]
I think there are a lot of practical applications where something that works few-shot and doesn't require training is quite appealing if the cost of inference is low. That's the sweet spot for Jev and similar models. If you have time and an eval set to train it, clearly a "real" special-purpose classifier should have much better performance (and for most applications cheaper cost/faster inference), but Jev and friends are way better than using a full-fat transformer LLM for a classification task.
As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.
dinobones 6 hours ago [-]
There are a lot of people working in software that have limited knowledge of information theory or machine learning.
Unfortunately LLMs are magic to them.
When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.
lemonish97 6 hours ago [-]
it's also the unfortunate practice of trying to use the new shiny thing everywhere and anywhere to signal that you're keeping with the times. It's funny how i've seen newer devs brush off things as 'traditional ml'
cogman10 47 minutes ago [-]
Which is nuts because one thing LLMs are actually pretty good up is generating "traditional ml" harnesses. You just have to describe the problem you are trying to solve and ask the LLM for algorithm suggestions.
It's also nuts because "traditional ml" generally requires a fraction of the resources of an LLM. If you need to auto label, tag, categorize, it's never been easier to setup that sort of system in any project.
RGS1811 7 hours ago [-]
> Just use the LLM to make a thing that's actually good at what you need it to do.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
Twirrim 6 hours ago [-]
The frustrating part about an LLM is at a fundamental level it's designed to do the wrong thing, it's designed to provide an answer for things it can't do, e.g. one argument I've been in recently at work is someone using it to summarise numerical data. They've a raw data spreadsheet they're putting into ChatGPT and asking it to produce a report in a particular style. So the LLM is doing its job, predicting the tokens to emit, and not actually doing mathematical analysis of the underlying data.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
DougBTX 6 hours ago [-]
For numerical work, there’s no reason to assume the computation is happening in the LLM. All frontier models could handily write a Python script to do linear regression then report the result, so the quality of the responses here depends entirely on the prompt + harness.
hdjrudni 6 hours ago [-]
+1. I noticed ChatGPT now pulls out a calculator or writes a script instead of hallucinating numbers. Initially it was terrible, but recently it seems pretty good... that said, if you're doing serious work with this you should probably double check. And if you're doing repeated work, obviously get it to write a script or formula instead of doing 1-offs.
ashdksnndck 3 hours ago [-]
It makes sense to use an LLM to write the queries when exploring data, but if you are building a dashboard that’s going to be used repeatedly, you should save the queries so they can be validated and improved over time. The LLM shouldn’t be rewriting a new query every time you open the dashboard.
Twirrim 3 hours ago [-]
I've seen so many "truthy" results coming back that suggest it's not even remotely as smart as you might hope.
I'm mostly starting to have some success telling TPMs and engineers to ask chatgpt to give them appropriate code that generates more accurate answers.
4 hours ago [-]
charrondev 6 hours ago [-]
Definitely. I’ve found Cursor + Grok to be the best budget models for this type of tool use, although I’ve yet to try out the latest Anthropic ones.
You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.
The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.
girvo 5 hours ago [-]
They’re very much trained to do this, in fact. Even Step 3.5 Flash, a medium sized open model from way earlier this year is trained to use python to calculate answers on the fly :)
Foobar8568 5 hours ago [-]
Either chatgpt or claude are building scripts to do those and has been for a few months at least now.
trollbridge 7 hours ago [-]
This is exactly what we’ve been doing.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)
harpiaharpyja 6 hours ago [-]
(Presumably you mean the guy affected by the decision?)
jorgen123 6 hours ago [-]
I think it means the person making the decision does not have to get up early anymore in order to make a decision. The people receiving the decision still receive the decision, and a bit earlier.
trollbridge 5 hours ago [-]
A complaint from the crews was not getting the go/no-go at the same time consistently. This fixes that problem.
howunfortunate 8 hours ago [-]
I do agree with you, but I do think you're also making the classic "curse of knowledge" mistake.
You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.
flatline 7 hours ago [-]
The tooling around LLMs is still nascent and, in my opinion, quite poor. You may be able to shotgun easily-falsifiable mathematical proofs with raw compute, but get into the messy world of real data, uncertainty quantification, and a host of other things that go into building real world models, and I think that an LLM plus tooling dedicated to the task plus a human in the loop will open up capabilities not available to an LLM in a chat or development centric software environment.
m11a 8 hours ago [-]
I don't believe this is true with current frontier coding agents. Or at least, it's becoming increasingly less true, such that I think it will effectively be nullified in ~6 months.
misterderpie 7 hours ago [-]
I'm gonna cite myself here:
«We came full circle on the problem that also exists with Google Search, and why so many people don't get good results out of it.» [0]
It is not that access to all information was hidden to anyone before, it was harder to find it, sure. Now it can be found easily and aggregated, however, it is not as simple as saying "Do X" and believe that X is correct. Recall the infamous Swing & Product Management meme? [1] The problem is the absence of information you provide, the LLM will have to make a guess of what you mean, much like the human. Likelihoods of this guess being correct depends on the domain and background.
So yes, it'll likely build whatever you ask it (I do think we're there already), but if that matches what you intended to actually get is vastly different. You'll be settling with "Good enough" though and declare it working.
Others already pointed out that 6 months seem to be the constant of distance, irrespective of the date.
LLMs are the perfect tool for "finding the right words". They are literally translation machines.
JackSlateur 8 hours ago [-]
Just ask the LLM for these words, obviously !
yesco 8 hours ago [-]
It's funny you mention this, because I've been thinking lately that all the focus on code quality benchmarks for LLMs seems to miss something. In my experience, LLMs aren't actually all that good at prompting other LLMs, and that's exactly the skill that matters for proper subagent usage.
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
_matthew_ 2 hours ago [-]
Quick tip here: Where possible try to trick the model into thinking whatever task it is writing will / could be read and performed by one of your coworkers rather than another agent. It will quickly fall into line and stop writing prompts meant for dumb historic low intelligence LLMs.
dev0id 7 hours ago [-]
The "agent factories" available to employees at the labs and hyperscalers clearly prove that agent delegation does work. Today, maybe that capacity lies with Fable class models, as you pointed out, but there is no doubt with opus 5.5 that this capability will be available to cheaper models as they get better.
alienbaby 8 hours ago [-]
That's what I've been doing in areas I lack knowledge tbh. Instead of a half assed attempt at describing what I need, I ask an llm to take my base idea and write it up using industry standard jargon as if I were commissioning a team to do the work. Has been working pretty well so far!
im3w1l 8 hours ago [-]
Okay so I just asked it "What question should I be asking you but I don't because I'm not smart enough to think of it?" Response was kind of interesting I have to say.
jonatron 8 hours ago [-]
It gave me a very helpful answer, my project will be improving just because I asked.
Natsu 8 hours ago [-]
I did this just now, ironically before I even saw this discussion. I ended up questioning the AI about humanism.
staticautomatic 8 hours ago [-]
So what? The average joe shouldn't be trying to diagnose people with cancer.
JPLeRouzic 8 hours ago [-]
Healthcare is more than having medical knowledge.
Ten years ago, in France, two doctors (one in a professional setting and the other during a private consultation) asked me about a wound I had on my shoulder; obviously I couldn't identify what it was, and they offered no suggestions, so I supposed it was not something important.
It took a visit to a cardiologist—for an entirely different reason—for him to suggest that I rapidly consult a dermatologist.
It was a carcinoma, and surgery removed it.
If ChatGPT had existed back then, this carcinoma would have been removed several years earlier.
trollbridge 7 hours ago [-]
This sounds like a competency problem on the part of the GPs. They’re supposed to make these calls.
devrwoody 7 hours ago [-]
Yet here we are
gchamonlive 8 hours ago [-]
Maybe if chatgpt existed back then you'd have had your carcinoma extracted along with 10 other benign anomalies.
Automation doesn't make up for carelessness.
kaikai 7 hours ago [-]
Skin cancer is easily treated with excisions if it’s caught early. Better to have ten small false positives than a single miss.
gchamonlive 7 hours ago [-]
Even if addressing each false positive would carrying low but real risk of death or complications?
bell-cot 56 minutes ago [-]
Is your "low but real" risk comparable to the risk of driving a motorcycle, of riding in a car, or of using the bathroom?
Reasonable treatments for small, non-aggressive skin cancers (or suspected cancers) are extremely low risk. And reasonable practitioners will often suggest that you merely keep an eye on an abnormal spot - because it doesn't look worth the bother of any sort of treatment for now.
teiferer 6 hours ago [-]
Not sure why this is getting downvoted. In its specific case that's quite obviously not a good example because removing benign spots isn't harmful, but overlooking malignant ones is, so there is no downside.
But if we think a little further and generalize then there is great risk (and by extension, harm) to delegate diagnosis and treatment (of health issues in general, including potentially or supposedly fatal ones) to ChatGPT and similar tech. That we'd end up with a net positive is far from obvious.
red75prime 4 hours ago [-]
Obviously, error rate of ChatGPT should be measured by comparing to human diagnoses and when it plateaus, we should investigate a possibility that the number is driven mostly by human errors.
gchamonlive 6 hours ago [-]
I still maintain my point that it isn't harmless to blindly address benign anomalies. Every medical procedure carries risk.
senordevnyc 2 hours ago [-]
Your point doesn’t really make a lot of sense. Even with ChatGPT existing and making 90% bad calls, a doctor isn’t likely to just “blindly” be cutting out anomalies.
gchamonlive 2 hours ago [-]
As opposed to what OP said?
senordevnyc 33 minutes ago [-]
Your original comment seemed to imply that if ChatGPT had existed, maybe they’d have had ten benign things removed. But that’s not how it works.
Conversely, ChatGPT can help avoid false negatives.
bunderbunder 5 hours ago [-]
Prompted LLM classifiers are good for when training data is expensive and/or you don’t expect a high inference volume. It’s basically choosing lower model development costs in exchange for higher inference costs, which can be good for TCO, time to market, or both. Heck, I’ve worked on some applications that weren’t even feasible before LLM-prompted classification because of the cost to acquire enough data to train a supervised model.
Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.
softwaredoug 8 hours ago [-]
This is like saying we don’t need Dropbox because we have FTP servers.
People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.
atherton94027 8 hours ago [-]
I don't know, a lot of the things we used to take for granted as "hard" just aren't anymore. Writing a parser for a compiler? It used to be an endeavor that took several days, now an LLM will do it in 20 mins.
Same for annoying training jobs, with sufficient data you can probably let an LLM loose on it in autotrain mode and it will come back with a decent classifier.
catlifeonmars 5 hours ago [-]
> Writing a parser for a compiler? It used to be an endeavor that took several days, now an LLM will do it in 20 mins.
Heh this used to be an interview question I gave. You’d be surprised how far people would get in 30 minutes.
softwaredoug 7 hours ago [-]
It’s not the software development that’s hard. It’s the data science and domain knowledge to build a good training set and a meaningful objective.
trollbridge 7 hours ago [-]
Training classifiers is shockingly easy nowadays with LLMs. This is probably the biggest single piece of bespoke software I ship that I didn’t used to.
We have in the works a classifier for if a prospective customer will be long term profitable or not. This changes everything.
suoloordi 6 hours ago [-]
> We have in the works a classifier for if a prospective customer will be long term profitable or not
I'm curious, why do you use a classifier for long-term profitability? Wouldn't predicting expected lifetime profit with a forecasting model be more meaningful than just predicting whether a customer will be profitable or not?
trollbridge 5 hours ago [-]
The purpose is to sort which customer to prioritise calling back versus putting at the bottom of the pile.
Eventually we hope to identify ones who are going to be lifetime duds.
Y_Y 7 hours ago [-]
We don't need Dropbox, it's a heap of shit.
With enough marketing you can sell anything. Also a decent way to make a product is to do the hard part for someone else.
I've been training classifiers for ten years, it used to be hard. Now it's hard to do really well but it's trivial to do well enough.
nico 5 hours ago [-]
> decent way to make a product is to do the hard part for someone else.
100% this is a very good insight
kaydub 1 hours ago [-]
The LLM is like the swiss army knife in these situations. It can KINDA do it all, but not very well in a lot of cases. Better to have specialized tools in most cases. And you're right, the LLM is pretty good for figuring out or building out the right tool for the job.
johnshahbazian 4 hours ago [-]
People like me who are excited about Jev-style models aren't going to use them to diagnose cancer. We are going to use them to make fairly straightforward decisions very quickly (300 ms or less) and very cheaply ($0.042 per M input tokens - output tokens aren't billed). Even the lower end frontier models will take much longer and be much more expensive, so these decision models are perfect for certain things.
antonvs 2 hours ago [-]
But as the GP said, you’d likely be better off if you used an LLM to help you train a custom model to do what you’re trying to do.
If you really evaluate the best options, it’s very likely that a Jev-style decision model won’t give the best results.
johnshahbazian 5 minutes ago [-]
If you know the exact domain you're working in, then a trained model could potentially do better, but its not a guarantee. For example, I fine-tuned Muse Glimmer on a specific domain and then tested it against Mercury Decide - which I've found does better than Jev - and they were even (and no fine-tuning was done on Mercury). But Mercury Decide was was faster and cheaper. The P95 for latency was about 5.6x lower. Decision models are very, very fast.
cowlevel 54 minutes ago [-]
It's because you can "program" it in casual English. No code, no training runs.
andrewmutz 7 hours ago [-]
Training a model requires labels training data. Getting up and running with an LLM just requires a prompt.
I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.
nico 5 hours ago [-]
I just ran an experiment, I took the banking77 dataset, took the labels out, and got an LLM (Opus 5) to label them, then trained a logistic regression classifier (Jeffy http://github.com/nicobrenner/jeffy) on the resulting set
Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)
But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification
miltava 7 hours ago [-]
That’s true. It could help with the cold start problem for training models and maybe replace heuristics/rules in some cases.
magicalhippo 6 hours ago [-]
What we've been trying Jev for is interpreting things like short product descriptions provided by customers, and determine which of the natural language categories fits best.
The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).
What kind of dedicated classifier would be better? Genuinely curious.
anukin 6 hours ago [-]
A classifier requires a ton of labeled data to be processed for it to be good. Jev and likes are zero shot classifiers where none of the data is required.
Imo both are useful for different tasks. I am not going to use jev based models for compliance or any tasks where being wrong can bankrupt my business. For something trivial like sentiment analysis it’s a god sent.
servbot 5 hours ago [-]
I've found that while this technically true, if you're building a classifier of any importance, you're going to need to monitor it and grade the efficacy of any changes, which will require labeled data anyways.
This kind of approach works for one-shot "let's just see if it works" type stuff, but anything that I would stand behind needs at least some level of labeled data and curated dataset.
jrflo 5 hours ago [-]
Can’t you use existing open weights sentiment analysis models that are far more lightweight though?
nico 7 hours ago [-]
> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture
Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/
At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow
Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well
kristjansson 5 hours ago [-]
There's a great bit of shortform video content floating around that shows a stammering younger guy in a welding mask:
> Do you have two pieces of metal that need to become one?
> But if it became two pieces again it wouldn't be a big safety concern?
> If that's you call Bob's OK welding!
That's Jev, and ~all zero-shot classification-by-LLM. It's universally OK. And that's great, for the tranche of problems where OK is good enough. But anywhere OK is not good enough, one would be advised to use or build something somewhat more robust.
Karrot_Kream 5 hours ago [-]
Classifiers can require a lot of examples to train correctly. You can take a decision model's outputs and apply platt scaling to arrive at a good decision boundary a lot faster.
Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.
alex_duf 8 hours ago [-]
There's a significant engineering cost to having a custom classifier.
An off-the-shelf API has way less friction, if the results are good enough why bother?
dehrmann 7 hours ago [-]
For this particular case, it's probably worth it because a dermatologist visit is $200. If I were the NHS or Kaiser, I'd partner with other integrated insurance and service providers to collect as much training data as possible and fund classifier development.
sungam 5 hours ago [-]
This is the plan!
alberth 4 hours ago [-]
I think there is a parallel between engineering systems (specialized) and commodity cloud hardware (general purpose).
While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.
melenaboija 6 hours ago [-]
Use LLMs to implement, process data, test and iterate statistical and NN models.
postepowanieadm 8 hours ago [-]
Once you have a hammer everything is a nail.
heisenbit 7 hours ago [-]
True for modern hammers while older had weaker decision models limiting them to finger nails.
Y_Y 7 hours ago [-]
Even the hammer?
postepowanieadm 7 hours ago [-]
Mc Hammer?
jcims 6 hours ago [-]
>LLMs drive the cost of making those models down to near zero.
The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.
If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.
mbreese 6 hours ago [-]
It is often better to use a pretrained model. But the choice here is using a frontier LLM to classify or a purpose built classification algorithm.
I know which I’d expect to perform better.
dnautics 5 hours ago [-]
> The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models
Have you considered putting an LLM on the task of completing the bureaucratic gauntlet
jcims 5 minutes ago [-]
Looks like you might be getting downvoted but yes!
alex_suzuki 8 hours ago [-]
> Just use the LLM to make a thing that's actually good at what you need it to do.
I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?
Y_Y 7 hours ago [-]
Not if you take advantage of open-weight models and make an adaptor or distill its knowledge into the thing you want.
OtherShrezzing 7 hours ago [-]
> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture
There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.
garrickvanburen 3 hours ago [-]
> Just use the LLM to make a thing that's actually good at what you need it to do.
Yes, this.
EagnaIonat 8 hours ago [-]
> why people are excited about Jev-style decision models.
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
nico 5 hours ago [-]
> and not using in anger
Is this a typo? are you saying that this is so new people haven't struggled with Jev enough yet? very interesting
At around 4 classes already LLM-based classifiers (Jev/Laya/GLiNER et al) start falling behind
iforgotmypasswo 3 hours ago [-]
Yes, of course, with time, attention, higher maintenance costs, and slower time to respond to requirements changes you can build a specialized system that performs better than an off the shelf general purpose solution. But you cost money. And you can quit and go work somewhere else, stranding the project. And the data might be garbage you can’t train on without substantial effort. And training takes time, which has opportunity costs that are often substantial. And the requirements may change tomorrow with no notice, invalidating all of your work.
I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.
I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.
I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:
“It is difficult to get a man to understand something when his salary depends on his not understanding it.”
sungam 8 hours ago [-]
I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.
dnautics 5 hours ago [-]
A transformer trained only on cancer images could probably do better but the old model is trained on a well curated data set to do one thing. A decision model can decide on a much broader universe of things, only a sliver of which is cancer (and probably there's a smaller cancer dataset too)
whatthejesus 7 hours ago [-]
Don’t those decision models mainly route to other models? My understanding is that they wouldn’t handle a request but forward it along with some kind of semantic capability.
foolswisdom 8 hours ago [-]
For some types of applications, Jev is also useful as a stepping stone, you use it for the cases it's most confident so you already get gains from using it, and then collect all the usage to train your specialized model.
nostrebored 8 hours ago [-]
You can train gradient boosted trees on jev style features, it is a quick “good enough” approach for a lot of stuff
nostrebored 8 hours ago [-]
They make very good classifiers and are easy to fine tune for specific use cases.
8 hours ago [-]
flamedoge 4 hours ago [-]
I think the contention is CNN vs Transformer
weitendorf 8 hours ago [-]
it's precisely because you can tune them to do that off a strong base
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
zer00eyz 6 hours ago [-]
> For any application at scale it'd be way more accurate to have a dedicated classifier
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
Routine8502 8 hours ago [-]
I mean, it's the pareto principle, right? 80% of the applications require 20% of the intelligence. If I can drop in something without needing to create high quality training data that's good enough, that's great! Skin cancer probably isn't one of those applications.
holoduke 6 hours ago [-]
The only real usecase I can see for jev style like models are processing long lists of items. i.e server logs or financial transactions or sensor data in robotics.
dieortin 6 hours ago [-]
Why?
yieldcrv 7 hours ago [-]
Internet discovers If statements
subnanosecond response times
model fits on CPU
tancop 7 hours ago [-]
[dead]
tasuki 7 hours ago [-]
> I compared to a 10-year old CNN and to my surprise found that this was superior.
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
jrflo 5 hours ago [-]
A CNN is also definitely not an “outdated architecture” when it comes to image classification
koutakun 2 hours ago [-]
They have sort of been replaced by transformers for a lot of use cases (and where dataset size allows).
sungam 6 hours ago [-]
I think the reason I am surprised is that they are exceptionally good at interpreting images generally and there are tens of thousands of labeled skin cancer images on the public internet so I had assumed the general image interpretation abilities would transfer
ang_cire 2 hours ago [-]
Having actually worked with teams doing medical vision models for cancer detection, I can tell you that the datasets we used were not public, and they came from studies and private hospital partnerships that LLM companies don't have access to.
jrflo 5 hours ago [-]
The specialized medical diagnosis models are trained on that same data though, and also use neural networks. The more noteworthy story will be when they overtake specialized models.
gradascent 2 hours ago [-]
Not only do they train on that data but they train ONLY on that data and don’t have to generalize to other domains. And the architectures themselves, including hyperparameters, loss function, etc are chosen and tuned specifically for the quirks of this domain. So this result doesn’t surprise me either.
ip26 4 hours ago [-]
such as helping you create a network that's good at playing go
This is really what’s happening right now… LLMs are excellent computer users. Which makes one of the frontiers “amazing things computers always could have done, but we lacked the programmers to do”.
If basically anyone can train a high quality CNN for just about any purpose now, what new opportunities appear that computers can help with?
Legend2440 8 hours ago [-]
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
EagnaIonat 8 hours ago [-]
> This is just benchmaxxing.
I believe the normal term is "gaming a benchmark test".
Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.
The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.
Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.
Aurornis 6 hours ago [-]
Training a task-specific small model is just good developer practice.
That's just good developer practice, not "benchmaxxing".
Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.
If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.
YetAnotherNick 4 hours ago [-]
It is benchmaxxing because the test distribution in these kind of image dataset generally matches the training much more than real world image. It used to be the biggest issue in real AI in most companies that the test data and real data is either different or risk getting drifted apart.
So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.
yunnpp 4 hours ago [-]
TIL: benchmaxxing, aka overfitting?
lkois 6 hours ago [-]
What would be an unbenchmaxxed model here? A general image classifier like Inception v3, without the fine-tuning?
The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is.
A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.
__rito__ 7 hours ago [-]
This is standard practice.
I developed many models from scratch deployed in various fields.
When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.
This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.
cowlevel 52 minutes ago [-]
This is the same way LLMs are trained. Are they benchmaxxing and useless out of domain?
sungam 8 hours ago [-]
This is a good point and was an issue for skin cancer AI in the early days but has largely been solved now - newer models use ensembles of vision transformers and much greater number of training images and have been proven accurate in clinical trials. See for example https://skin-analytics.com/ (I am not linked in any way)
the__alchemist 8 hours ago [-]
Could you explain why this is benchmaxxing?
caughtinthought 5 hours ago [-]
Being good at the intended application is not benchmaxxing.
MitPitt 8 hours ago [-]
And a calculator outperforms LLMs at multiplication and division...
bayindirh 8 hours ago [-]
In a world where masses believe that LLMs are "everything boxes", some reality check is always good.
jacquesm 7 hours ago [-]
The problem is that they are 'everything boxes'. They're the first application that can actually lay claim to being a universal computing aid. What then happens is that people will start using their newfound hammer and before you know it everything becomes a nail, even if for a particular problem there is a better solution you can't beat 'nothing to invest, nothing to learn, get an answer in 10 seconds flat' from a convenience perspective. If I look at the things people use LLMs for I wonder why they take such an unsuitable tool, and then I marvel at the fact that they actually get more-or-less to where they would have gotten with the proper tools, and most likely in less time.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
ehsankia 8 hours ago [-]
LLMs should be for general/high level work. For example, an LLM with the right harness could fetch and use that CNN, or train it's own CNN or other model to use for this task.
The harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
bayindirh 8 hours ago [-]
We don't disagree by a great margin. I'm not an anti-LLM person. I'm anti-hype in this case.
grumbel 8 hours ago [-]
LLMs are pretty good at writing calculators.
bayindirh 8 hours ago [-]
I'd rather use a verified one. Scientific calculations on a ALU is tricky. There are many traps and hardware limitations to consider.
jszymborski 8 hours ago [-]
You say this, but the reaction I get from colleagues is increasingly "why bother training new models when you can just ask an LLM?"
Doesn't matter the domain or the task, the answer from some researchers is "LLM" always.
tourist2d 7 hours ago [-]
[dead]
asgraham 8 hours ago [-]
Most people need to be reminded of this. Repeatedly.
sungam 8 hours ago [-]
This does make sense, however in view of the recent maths results and since there are tens of thousands of training images for skin cancer available on the public internet I was expecting the LLMs to perform a lot better than they did
trollbridge 7 hours ago [-]
Yeah so that’s why we invented tool calls, and an LLM exercises judgment when it should simply place a tool call to do simple math.
everfree 8 hours ago [-]
And if you give the LLM a simple calculator MCP, then the calculator's outperformance disappears.
bayindirh 8 hours ago [-]
Nope. It'll be more efficient and faster.
everfree 8 hours ago [-]
That's just semantics around the word "performance".
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
bayindirh 8 hours ago [-]
Actually not. Watt per task is a metric bothers more than LLM operators and users. We care about it a lot, and working a lot to measure and improve it, and we're not doing it in context of AI or LLMs.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
everfree 8 hours ago [-]
Generally when I want to use a calculator, I use an app on my phone or computer. The watts per task is abysmal compared to a desk calculator.
The app is less efficient only in an abstract, impractical way.
bayindirh 8 hours ago [-]
> The watts per task is abysmal compared to a desk calculator.
It depends on the calculator and task at hand. Consider graphing, integration or any programmable needs, and they might converge. ALU of a processor is pretty efficient at what it does.
> The app is less efficient only in an abstract, impractical way.
Again, depends, as I said above. You tend to brush efficiency away because you only mentally measure time to completion, only, which is understandable.
guelo 8 hours ago [-]
Imagine performing literally trillions of matrix multiplications in order to figure out the answer to 1 + 2.
everfree 8 hours ago [-]
Your computer/phone just performed millions of them in order to display the contents of my comment to you.
cubefox 8 hours ago [-]
Pretty much what our brain does.
wrigby 8 hours ago [-]
Sure, but what about performance per joule?
everfree 8 hours ago [-]
If performance per joule is important in this loosely-defined hypothetical scenario, then it's important.
If it's not, then it's not.
devin 20 minutes ago [-]
It's going to be funny/sad to watch people who have deeply misapplied LLM tech discover "oh gee, this is pretty expensive. I guess I can just use ML?"
jonex 8 hours ago [-]
As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
trollbridge 7 hours ago [-]
In other words… more dimensions for a bigger set of vectors?
whiplash451 7 hours ago [-]
Not just more dimensions, but off-the-shelf multimodal encoders
lukehandcool 8 hours ago [-]
I found SkinGPT-4 when I was looking into this article. From the nature paper "among the 150 cases, a significant percentage of SkinGPT-4’s diagnoses (80.63%) were evaluated as correct or relevant by board-certified dermatologists." Which does outperform the CNN in this article.
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
gauku 2 hours ago [-]
Very sincerely curious to hear why you expect this problem to benefit from such an architectural change?
sungam 8 hours ago [-]
The CNN architecture used here is definitely outdated. I used it as a baseline to compare to the LLMs and because the model is small enough to run using WebGPU in a phone browser for demonstration. Modern approaches use ensembles of vision transformers and have proven accurate in clinical trials
i2talics 2 hours ago [-]
OK, so everyone in the replies is saying "yeah duh", but a few years ago people used to defend the negative externalities of LLM's by pointing to medical applications and saying "but think of how useful they would be for doing a cancer diagnosis"
vikramkr 1 hours ago [-]
I think at least a few of the replies including mine are saying this is a pretty meaningless over hyped demonstration because the cnn was fine tuned on the same dataset it was tested on and the actual way the eval was structured and run is nonsensical. It's basically just slop research designed to create a fun attention grabbing headline (ironically one that trashes llms).
akst 24 minutes ago [-]
I mean not too surprising, it's a specialised model that trained on relevant quantitive/qualitative inputs.
LLMs are very general and flexible, by all means they can do well on non-text based specialised problems, but it also does well because their harnesses know how to defer to the right tools a lot of the time, and produces good answers from inspecting the output. Like if you ask Codex/Claude Code to transcribe a videos, it's going to create a script that uses whisper to extract the transcript not manually process the binary. If you ask it for demographic information it'll pull census data and write a script which produces the summary stats and report it back to you (I imagine much of the recent math stuff relied on this).
Sometimes it'll pull a number from its training data but the strength of an LLM is its ability to autonomously seek out the right answer even when it doesn't know the answer initially.
You could definitely use train an LLM to utilise these efficient specialised models, by it may be more cost effective to ask the model to build a special purpose application with a new UI over the top of the efficient specialised model.
chilmers 8 hours ago [-]
Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.
Neywiny 15 minutes ago [-]
I think this is true of any trained anything. It's the fish climbing a tree scenario. The teens on Ellen getting laughed at for not knowing how to fold a paper map.
There's infinite front end code to train on. Virtually infinite python and shell scripts too. As I've worked my way away from such domains, the LLM outputs have gotten worse. It just makes sense.
boelboel 7 hours ago [-]
"average human who's read a few books on the subject and is good at bullshitting" seems like a large portion of white collar workers that make bank.
Creamsicle47 7 hours ago [-]
The most sane comment I've seen in this thread
arfelious 8 hours ago [-]
LLMs shouldn't be treated as if they were purpose-built for specific cases just because they generally output a plausible-looking answer, and this is not something that was discovered only recently
sdenton4 23 minutes ago [-]
CNNs are going the way of radiologists...
infiniteupside 5 hours ago [-]
I developed the current #1 ranking leaderboard model and seems like a good time to stop lurking and chime in... It's no surprise that a dramatically simpler model fit to the narrow ~11 label prediction task outperforms an LLM where the same task is a drop in the ocean with regards to its training corpus.
What is maybe more interesting is that, despite the availability of open weight medical image-language embedding models like Google's MedSigLIP AND the opportunity to adapt them, they don't seem to break into the Top-10. My impression is that the joint representation is valuable down-stream of the diagnostic classification task for interpretation, communication, and management of the risk, but the language corpus for dermatology images are full of 'human engineered' descriptive features (think Bitter Lesson), and we should be looking past clinical semantics for more informative visual features if we want to push the limits of performance and generalizability. Ultimately, this seems necessary: people are immediately and persistently aware of the presence and extent of their skin conditions, the skin is the most readily accessible to imaging by patients themselves, smartphones have remarkable macroscopic photo capability, and there is huge demand for dermatology care.
sungam 4 hours ago [-]
I agree - personally I do not find the descriptive terms and language used to describe features on dermoscopy helpful to me in making a diagnosis, however I think they can be helpful when teaching other humans how to learn the technique. They may not provide additional benefit for machine learning classifiers which can deduce these principles directly from the images themselves.
lifeisloving 4 hours ago [-]
I love a good leaderboard, where can I participate?
biotechbio 3 hours ago [-]
I think an important comparison is the LLM with the supervised training data in context. I wonder how the LLM would perform if you provided all, or a representative subset of, the data used to fine tune the CNN in the prompt and then asked it to classify a new case.
This strategy is how LLM-like models have achieved leading performance on tabular prediction tasks (TabPFN).
everfree 8 hours ago [-]
My dog outperforms the world's top researchers in an unassisted 100 meter sprint.
MatejKafka 7 hours ago [-]
If your goal is sprinting 100m, using the dog seems like a better idea then, no matter how good the researchers are in other areas.
everfree 7 hours ago [-]
I think you’re onto something.
7 hours ago [-]
nomilk 2 hours ago [-]
Seems obvious given how each works. Choose the right tool for the job, don't blame the tool.
podgietaru 8 hours ago [-]
This doesn't seem that surprising to me? One is tasked with a simple very narrow task. The other has a corpus including billions of other images and pieces of information. It'd seem to me that instinctively one would outperform the other, regardless of age.
4994944995 8 hours ago [-]
Great luckily you are not in charge of a pension fund, thag wkuld be horrible
armcat 6 hours ago [-]
The same holds true for most classification tasks - if you have sufficient samples and high quality labels, training an xgboost or a deep learning model with a classification head is the way to go. You can use a frontier LLM on high-effort to help you with classification reasoning and explainability.
tommica 5 hours ago [-]
But how well will the CNN be able to write me a fanfiction of Dobby being in Lord of the Rings?
Bluestein 4 hours ago [-]
This entire LLM hoo-ha is leading to a "revival" and introspection on other techniques that are being revisited. Which is wonderful ...
fastball 8 hours ago [-]
This isn't really surprising. LLMs are intended to be generally intelligent. They are spiky on the things frontier labs tend to benchmark against (software/math) but are not intended or expected to be great at everything. The general intelligence unlocks new workflows, it does not perfect existing ones.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
vikramkr 2 hours ago [-]
This is not a compelling analysis.
First, the CNN is not 10 years old. The base is but it was fine tuned for this project on the same dataset the test images were derived from:
> The test: The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration [ISIC] in 2025
> The CNN is deliberately not state of the art. I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU
The test images themselves were filtered out at least, but that's not really enough. The CNN has a home field advantage because the test images are going to be in distribution with the training images.
Second, Looking at the code, for frontier models probably you should be giving them image tools and using them in an agent loop and not trying to one shot results (e.g. https://platform.claude.com/docs/en/build-with-claude/prompt...). It is also not meaningful to ask an LLM for a numeric confidence score.
Third, all 200 of the the test images were biopsied by the clinician originally. So when you ask the model "You are assisting a dermatologist" and "Decide whether this lesion can be safely discharged, meaning it needs no treatment, no biopsy and no follow-up" why would you penalize the model for making the same call the physician did even if it was benign in retrospect?
Are you evaluating the model on if it can replace the need for a biopsy itself (in which case the ground truth is whether it in retrospect was harmful or not) or in if it can match physician judgement in determining when a biopsy is needed (in which case the ground truth is if they physician ordered a biopsy)? You're asking the model to do the latter and grading it on the former.
Overall yeah probably specialized models make a ton of sense in a bunch of areas still but idk this does not seem like a good demonstration of that
jonatron 8 hours ago [-]
It's cool that there's an in-browser skin classifier...but it says "It is not a state-of-the-art model". The obvious question is, is there a free in browser state-of-the-art model?
5 hours ago [-]
sungam 8 hours ago [-]
Would be difficult to fit in a browser as state of the art requires multiple vision transformers. Also it would not be sensible to provide access to models without human oversight as at this point in time all models have the potential to miss skin cancer.
lynndotpy 8 hours ago [-]
This isn't surprising but it's good to have reporting to point to. The only way a general purpose LLM like these will match the performance of a bespoke classifier is if it has access to such a model.
callamdelaney 8 hours ago [-]
Yeah obviously. Xgboost outperforms llms in algorithmic trading, even applying the token prediction method to markets xgboost is effectively better.
Llms have their applications. Other models still do.
natsucks 8 hours ago [-]
yeah why is this news. breaking: we still need models besides llms.
jacobgorm 6 hours ago [-]
The model architecture might be ten old, but it seems it was trained recently, so the title is a bit misleading.
glennfiedler 4 hours ago [-]
Give to Caesar what is Caesar’s, and to God what is God’s.
crorella 4 hours ago [-]
The key is to use llms to create CNN models
talon8635 5 hours ago [-]
LLMs are not 10 years old and far far more than just a CN. By the time hey hit the 10 year mark I’d be stunned if they weren’t better, given the pace of change in math and AI. Once Lon’s get specifically trained on these and other targeted tasks, they will develope
Kuyawa 8 hours ago [-]
I'd like to see it compared to chinese models like DeepSeek, Kimi, GLM, etc just for the sake of research
a19486 2 hours ago [-]
Duh?
amelius 6 hours ago [-]
Read this as: AI beaten by a 10yo.
elzbardico 1 hours ago [-]
I am not sure I would expect a transformer model to excel on this kind of task.
thomasjeff1 7 hours ago [-]
An example of "Dumb it down"
bryan0 2 hours ago [-]
In other news, 30-year-old chess engine outperforms frontier llms in chess
Kalanos 7 hours ago [-]
this should not be surprising. alphafold is supervised learning.
c0rruptbytes 6 hours ago [-]
did you ask the LLM to make a CNN instead haha, might’ve won
4 hours ago [-]
einpoklum 5 hours ago [-]
I asked ChatGPT what it thinks about this situation and it told me LLMs can diagnose skin cancer, so there.
simianwords 7 hours ago [-]
Couldn't you just train a general purpose LLM on the output of this CNN?
Anyone wanna take a bet that the general purpose model will beat these specific models in a year or two?
j45 8 hours ago [-]
It seems one is more specialized for the purpose and compare it against a general purpose LLM instead of a specalized/focused LLM?
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison
here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
spwa4 8 hours ago [-]
Which CNN? A fine-tuned version of Inception v3, fine-tuned on 68,011 dermatology images
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
malfist 8 hours ago [-]
You know AI companies are selling the dream that you don't even need that just-tested-at-iq-130 6 year old. They're selling the dream that an sub-average iq 6 year old can do it.
So yes, this is an important finding that LLMs are not better than experience.
ChrisArchitect 8 hours ago [-]
fyi CNN: convolutional neural network
starfallg 7 hours ago [-]
Yeah, I too was confused for a second.
6 hours ago [-]
CoolThings 5 hours ago [-]
[dead]
napierzaza 5 hours ago [-]
[dead]
scottchiefbaker 8 hours ago [-]
[dead]
tourist2d 7 hours ago [-]
[dead]
howunfortunate 8 hours ago [-]
I'm sorry but as an MLE this is not a remotely useful article, and does not belong on the front page.
It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.
tgv 8 hours ago [-]
It's good to be reminded that "frontier models" aren't particularly good at most tasks. They happen to be good at programming, since there's a lot of training data available, and the task has parallels with report writing and story telling. For skin cancer? Probably very little. Also, images of skin cancer probably don't translate that well to a Large Language Model's internal representation.
everfree 8 hours ago [-]
It's very strange to me that articles like this get upvoted, because they are coming from this assumption that LLMs are better at everything and that it's notable and newsworthy when there's something that they aren't actually better at.
I don't think we should need to be reminded that frontier models aren't particularly good at most tasks. Just like we shouldn't need to be reminded that a hammer is not particularly good for driving a screw, or writing an essay.
It's an admission of magical thinking about a tool that's made out of the same material of every other digital tool since the invention of computing.
dgellow 8 hours ago [-]
There is no such thing as “belong to the front page”. Flag if you think it doesn’t have its place. Otherwise, it’s up to the crowd to decide what they want upvoted. HN front page reflects upvoter interest, nothing else
howunfortunate 8 hours ago [-]
The reason I say this is that I suspect many people are reflexively upvoting the title without reading the article. If they read the article they would realize the evidence is flimsy and not really worth reading.
I will not flag it because the post is fine, it's the voters who I believe are acting poorly.
Just use the LLM to make a thing that's actually good at what you need it to do.
I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.
The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.
I don't know of any devs in real life who would loudly and publicly get excited about anything unless it was their own completed project to show off.
The tools are a footnote. It's precisely because it's so hard to get a project that far along that they would be familiar with many more tools than they would ever speak about. I suppose some people have a low bar. I wouldn't really call them "devs", especially now in this "AI era".
I think you're coming at this from a very narrow and specific experience with a few people you know.
Jev really went from the announcement to being discussed all over overnight. We're discussing how popular things are in general, not among your narrow and specific group of people you work with.
I think it's clear we're at the tail end of this cycle. I'm very glad to see it.
As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.
Unfortunately LLMs are magic to them.
When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.
It's also nuts because "traditional ml" generally requires a fraction of the resources of an LLM. If you need to auto label, tag, categorize, it's never been easier to setup that sort of system in any project.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
I'm mostly starting to have some success telling TPMs and engineers to ask chatgpt to give them appropriate code that generates more accurate answers.
You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.
The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)
You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.
«We came full circle on the problem that also exists with Google Search, and why so many people don't get good results out of it.» [0] It is not that access to all information was hidden to anyone before, it was harder to find it, sure. Now it can be found easily and aggregated, however, it is not as simple as saying "Do X" and believe that X is correct. Recall the infamous Swing & Product Management meme? [1] The problem is the absence of information you provide, the LLM will have to make a guess of what you mean, much like the human. Likelihoods of this guess being correct depends on the domain and background. So yes, it'll likely build whatever you ask it (I do think we're there already), but if that matches what you intended to actually get is vastly different. You'll be settling with "Good enough" though and declare it working.
Others already pointed out that 6 months seem to be the constant of distance, irrespective of the date.
[0] - https://news.ycombinator.com/item?id=50063427#50063744
[1] - https://www.productftw.com/productftw-2-the-tire-swing-carto...
https://waymo.com/blog/2025/10/creating-an-all-weather-drive...
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
Ten years ago, in France, two doctors (one in a professional setting and the other during a private consultation) asked me about a wound I had on my shoulder; obviously I couldn't identify what it was, and they offered no suggestions, so I supposed it was not something important.
It took a visit to a cardiologist—for an entirely different reason—for him to suggest that I rapidly consult a dermatologist.
It was a carcinoma, and surgery removed it. If ChatGPT had existed back then, this carcinoma would have been removed several years earlier.
Automation doesn't make up for carelessness.
Reasonable treatments for small, non-aggressive skin cancers (or suspected cancers) are extremely low risk. And reasonable practitioners will often suggest that you merely keep an eye on an abnormal spot - because it doesn't look worth the bother of any sort of treatment for now.
But if we think a little further and generalize then there is great risk (and by extension, harm) to delegate diagnosis and treatment (of health issues in general, including potentially or supposedly fatal ones) to ChatGPT and similar tech. That we'd end up with a net positive is far from obvious.
Conversely, ChatGPT can help avoid false negatives.
Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.
People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.
Same for annoying training jobs, with sufficient data you can probably let an LLM loose on it in autotrain mode and it will come back with a decent classifier.
Heh this used to be an interview question I gave. You’d be surprised how far people would get in 30 minutes.
We have in the works a classifier for if a prospective customer will be long term profitable or not. This changes everything.
I'm curious, why do you use a classifier for long-term profitability? Wouldn't predicting expected lifetime profit with a forecasting model be more meaningful than just predicting whether a customer will be profitable or not?
Eventually we hope to identify ones who are going to be lifetime duds.
With enough marketing you can sell anything. Also a decent way to make a product is to do the hard part for someone else.
I've been training classifiers for ten years, it used to be hard. Now it's hard to do really well but it's trivial to do well enough.
100% this is a very good insight
If you really evaluate the best options, it’s very likely that a Jev-style decision model won’t give the best results.
I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.
Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)
But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification
The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).
What kind of dedicated classifier would be better? Genuinely curious.
This kind of approach works for one-shot "let's just see if it works" type stuff, but anything that I would stand behind needs at least some level of labeled data and curated dataset.
Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/
At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow
Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well
Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.
An off-the-shelf API has way less friction, if the results are good enough why bother?
While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.
The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.
If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.
I know which I’d expect to perform better.
Have you considered putting an LLM on the task of completing the bureaucratic gauntlet
I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?
There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.
Yes, this.
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
Is this a typo? are you saying that this is so new people haven't struggled with Jev enough yet? very interesting
> it starts to break down over 20 classifications
Yes, I recently ran a benchmark on exactly that: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
At around 4 classes already LLM-based classifiers (Jev/Laya/GLiNER et al) start falling behind
I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.
I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.
I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:
“It is difficult to get a man to understand something when his salary depends on his not understanding it.”
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
subnanosecond response times
model fits on CPU
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
This is really what’s happening right now… LLMs are excellent computer users. Which makes one of the frontiers “amazing things computers always could have done, but we lacked the programmers to do”.
If basically anyone can train a high quality CNN for just about any purpose now, what new opportunities appear that computers can help with?
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
I believe the normal term is "gaming a benchmark test".
Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.
The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.
Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.
That's just good developer practice, not "benchmaxxing".
Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.
If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.
So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.
The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is. A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.
I developed many models from scratch deployed in various fields.
When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.
This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
The harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
Doesn't matter the domain or the task, the answer from some researchers is "LLM" always.
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
The app is less efficient only in an abstract, impractical way.
It depends on the calculator and task at hand. Consider graphing, integration or any programmable needs, and they might converge. ALU of a processor is pretty efficient at what it does.
> The app is less efficient only in an abstract, impractical way.
Again, depends, as I said above. You tend to brush efficiency away because you only mentally measure time to completion, only, which is understandable.
If it's not, then it's not.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
LLMs are very general and flexible, by all means they can do well on non-text based specialised problems, but it also does well because their harnesses know how to defer to the right tools a lot of the time, and produces good answers from inspecting the output. Like if you ask Codex/Claude Code to transcribe a videos, it's going to create a script that uses whisper to extract the transcript not manually process the binary. If you ask it for demographic information it'll pull census data and write a script which produces the summary stats and report it back to you (I imagine much of the recent math stuff relied on this).
Sometimes it'll pull a number from its training data but the strength of an LLM is its ability to autonomously seek out the right answer even when it doesn't know the answer initially.
You could definitely use train an LLM to utilise these efficient specialised models, by it may be more cost effective to ask the model to build a special purpose application with a new UI over the top of the efficient specialised model.
There's infinite front end code to train on. Virtually infinite python and shell scripts too. As I've worked my way away from such domains, the LLM outputs have gotten worse. It just makes sense.
What is maybe more interesting is that, despite the availability of open weight medical image-language embedding models like Google's MedSigLIP AND the opportunity to adapt them, they don't seem to break into the Top-10. My impression is that the joint representation is valuable down-stream of the diagnostic classification task for interpretation, communication, and management of the risk, but the language corpus for dermatology images are full of 'human engineered' descriptive features (think Bitter Lesson), and we should be looking past clinical semantics for more informative visual features if we want to push the limits of performance and generalizability. Ultimately, this seems necessary: people are immediately and persistently aware of the presence and extent of their skin conditions, the skin is the most readily accessible to imaging by patients themselves, smartphones have remarkable macroscopic photo capability, and there is huge demand for dermatology care.
This strategy is how LLM-like models have achieved leading performance on tabular prediction tasks (TabPFN).
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
First, the CNN is not 10 years old. The base is but it was fine tuned for this project on the same dataset the test images were derived from:
> The test: The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration [ISIC] in 2025
> The CNN is deliberately not state of the art. I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU
The test images themselves were filtered out at least, but that's not really enough. The CNN has a home field advantage because the test images are going to be in distribution with the training images.
Second, Looking at the code, for frontier models probably you should be giving them image tools and using them in an agent loop and not trying to one shot results (e.g. https://platform.claude.com/docs/en/build-with-claude/prompt...). It is also not meaningful to ask an LLM for a numeric confidence score.
Third, all 200 of the the test images were biopsied by the clinician originally. So when you ask the model "You are assisting a dermatologist" and "Decide whether this lesion can be safely discharged, meaning it needs no treatment, no biopsy and no follow-up" why would you penalize the model for making the same call the physician did even if it was benign in retrospect?
Are you evaluating the model on if it can replace the need for a biopsy itself (in which case the ground truth is whether it in retrospect was harmful or not) or in if it can match physician judgement in determining when a biopsy is needed (in which case the ground truth is if they physician ordered a biopsy)? You're asking the model to do the latter and grading it on the former.
Overall yeah probably specialized models make a ton of sense in a bunch of areas still but idk this does not seem like a good demonstration of that
Llms have their applications. Other models still do.
Anyone wanna take a bet that the general purpose model will beat these specific models in a year or two?
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
So yes, this is an important finding that LLMs are not better than experience.
It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.
I don't think we should need to be reminded that frontier models aren't particularly good at most tasks. Just like we shouldn't need to be reminded that a hammer is not particularly good for driving a screw, or writing an essay.
It's an admission of magical thinking about a tool that's made out of the same material of every other digital tool since the invention of computing.
I will not flag it because the post is fine, it's the voters who I believe are acting poorly.