I’ve been trying out ChatGPT for many things. It’s useful for drafting paragraphs of copy that I have to edit heavily, summarising content from other places that’s too verbose (“Explain this in simpler terms”), or giving me ideas.
But one thing ChatGPT definitely doesn’t do well is give accurate answers. It doesn’t give me factual answers to questions about motorcycles. I haven’t tested it in every domain (as I’m not an expert in many, and only know a few things in a few domains), but you definitely can’t rely on ChatGPT to give you accurate answers about motorcycle maintenance or motorcycle statistics, and below there are many examples why.
Why — I don’t know. I know that OpenAI draws from publicly available information on the web. I suppose a lot of that information is wrong. Just go to any forum and try to wade through the pools of misinformation. Yes, a few people on them actually know things, but AI can’t figure out who those are yet (and it’s not even always clear to the casual visitor).
I make mistakes and typos too, and constantly go back and edit old documents. Maybe ChatGPT has read my out-of-date, not very well-edited articles!
Anyway, I thought I’d just show a few examples of things ChatGPT got wrong. Loosely, I’m categorising this into basic facts about motorcycles, as well as details about motorcycle maintenance.
Are you obsessed with motorcycles?
Well, I am. That’s why I created this site — as an outlet. I love learning and sharing what others might find useful. If you like what you read here, and you’re a fraction as obsessed as I am, you might like to know when I’ve published more. (Check the latest for an idea of what you’ll see.)
ChatGPT failing at basic motorcycle facts
Firstly, I asked ChatGPT a few basic facts about motorcycles to see how it fared. Not well.
Here’s me asking it something about the Multistrada V4.
I’m actually surprised about this one in multiple ways.
Firstly, it got the basic fact wrong, when a cursory google will reveal that one of the cool things about the Multistrada V4 is that it does not have a Desmodromic valve timing system.
Secondly, it calls it a “Desmodromic Valve System” and then abbreviates that as DVT. DVT is actually Ducati’s acronym for Desmodromic Variable Timing, an equivalent of variable valve timing, and is a feature of other Ducati engines, like the Multistrada 1200 DVT.
Thirdly, it sounds coherent to the point where you want to believe it and I had to go triple check that I was right. I am! Here’s Ducati talking about it.
Secondly, I gave it a complicated question about the BMW HP2 Enduro (one of the BMW HP2 line).
This is quite a hard question, because the HP2 Sport has twin overhead cams, but the other HP2 bikes do not. Even the manuals put it oddly, saying they have two camshafts… but not specifying that it’s one camshaft per cylinder. Anyway, ChatGPT got it wrong again.
Finally, I gave ChatGPT a softball one about the BMW R nineT.
Fail again. And yet again, it sounds quite convincing. The 2021 model year was a new generation of the BMW R nineT (see my BMW R nineT model guide). But it’s still air/oil-cooled. (It might mean oil cooling as liquid cooling, but that wouldn’t mesh with the previous gens, which are also oil-cooled.)
I want to clarify that it did get some answers to other questions right. But when it gets this much wrong, I wouldn’t trust a thing it says.
A ChatGPT-Produced Motorcycle Maintenance Schedule
I gave ChatGPT the prompt to create a maintenance schedule for a Ducati Monster 900. Here’s what it produced.
I did this based on a suggestion from a reader who said that they used it for the same purpose.
But this service schedule is so riddled with basic mistakes that the whole thing is rendered unreliable. If an analyst or writer produced this for me — even as a draft — I wouldn’t employ them further.
Among the errors, it asks me to check the clutch cable as well as change the clutch fluid (pick a team!), to check / change the coolant (good luck finding coolant), and to change the oil every 600 miles, which is ridiculous.
On top of that, it didn’t ask me to sync the carburettors.
This maintenance schedule has so many errors that it isn’t worth even using as a template. I’d much rather go find the original one.
To be fair, I next tried a much more common motorcycle, the Yamaha MT-07. It was still totally wrong on a number of fronts.
So ChatGPT can’t produce a maintenance schedule for one of the most popular motorcycles in many markets.
I thought I’d get tricky and ask a leading question, too:
Yes, I did give it a leading question. But any two-bit mechanic or motorcycle owner would say “What chain? The R 1200 GS has a shaft drive”. Instead, I got a coherent-sounding answer about a chain that doesn’t exist.
I asked ChatGPT about a US-spec maintenance schedule for a common motorcycle, the Yamaha MT-07.
I had previously asked it about the service interval, and it gave me 10000 km, which I know to be the European spec. But when I asked it about the US spec, it gave me a weird answer of 7500 miles or 12 months, which is flat wrong — it should be 4000 miles or 6 months. In Europe it’s 6000 miles / 10000 km or 12 months. This is the case in all manuals for the FZ-07 and MT-07.
Previously it got other details wrong too when I asked it about region-specific valve service intervals.
So — What should you do?
Luckily, there are loads of online resources that have accurate information, made by humans (or at least heavily edited by humans) for humans.
And if all else fails, you can refer to official literature like user manuals or service manuals. I don’t always do that as they’re not very practical (they’re expensive and huge, and the content isn’t always well laid-out). But I do refer to them sometimes.
Sometimes I employ analysts to comb over articles and data and summarise them for me. But I always triple check their work. More importantly, I build a culture of integrity where correctness comes first.
Conclusion — AI is Not There Yet, but Don’t rule it Out
Artificial Intelligence is becoming more useful. It does become smarter all the time at understanding nuance in data it reads.
The only reason I’m writing this article is because I have been generally impressed with the quality of the content that AI produces, and so have been pushing it to see how far it goes.
But I think that as someone who values facts, a service I can add for now is to say: Find a better source than AI-produced content related to facts, then layer on your own judgment.
One obvious improvement for ChatGPT for people to be able to trust it would be to link to verifiable sources.
Where I think AI and machine learning will go in the future of motorcycling is a huge topic to be explored later. But some things I’ve been thinking about are
- Predictive maintenance: Intelligently estimating mean times to failure of components based on usage
- Purchasing and pricing: Helping people make purchasing decisions based on important aspects of any bike plus price, plus helping people sell items for sale
- Trip planning: Taking a series of inputs and suggesting a route
- Crash prevention: Assessing risk based on conditions (temperature, speed, traffic levels) and giving guidance to a rider
- Custom training: Assessing how you ride and giving guidance on things including braking, acceleration, cornering, and choosing a line
These are just some ideas to be explored later. Even though AI / ML is still in its early stages, I’m confident it’ll advance more rapidly than we expect!
