Apple Intelligence 2.0: 10 ways Apple can get its AI back on the right track

When Apple Intelligence launched last year, Apple promised us that its AI system would revolutionize how we use our devices. Unfortunately, this past year has been much more middling than miraculous for Apple’s AI, with the platform failing to impress in the ways we were hoping.

That means there are a lot of ways Apple Intelligence could be improved in the future. From improved automation to more powerful editing tools and beyond, Apple’s AI foundation opens up a world of possibilities for bringing new smart tools to our Apple devices. Here are 10 of the top features we’d love to see in the next version of Apple Intelligence.

Apple Intelligence 2.0: Fast improvements

Image generation

Apple’s Image Playground app is a fun way to quickly create quirky images, but it’s still very limited and rudimentary, especially when you compare it to rival AI generators such as Stable Diffusion and GPT-4o Image Generation.

For starters, Apple needs to fix those persistent bugbears that seem to afflict almost all AI generators, things like freakishly inaccurate hands and feet. But the improvements need to go beyond that if Image Playground is to gain traction. We need far more control and flexibility over the images it whips up. That means being able to import new generation models and styles, the ability to work with different aspect ratios, adjust the relative strength of each prompt, and more. 

It’s doubtful Apple will give us the ability to make phot-realistic images, but there is still a lot Image Playground can do to enhance its generated art. For example, the recent OpenAI GPT-4o model update allowed users to draw images in the style of a Studio Ghibli Japanese animation film. It was a viral sensation when it launched and exactly the kind of thing Image Playground needs to get some buzz.

Photo editing

Speaking of images, Apple has brought a few AI image editing tools to our devices, yet these can also be vastly improved. In the first version of Apple Intelligence, we can remove distractions in photos using Apple Intelligence’s Clean Up tool, for example, but having the option to add elements to your pictures would be a fun extra. This could extend to refining existing parts of your images, such as replacing one type of tree with another in a landscape.

We’d also like to get the power to remove something from one image and add it to another. Or to summon Siri and use natural language to make edits. Imagine how much hassle you could save by simply saying, “Hey Siri, change this sky to a thunderous rainstorm.”

Foundry

Apple Intelligence 2.0: Foundational upgrades

Natural language search

One of the key ideas behind artificial intelligence is that it will simplify your day-to-day life. So why isn’t Apple Intelligence very good at doing that? One key way it could improve is by adding more natural language capabilities.

This could be really effective if it was added to things like Spotlight search or the Messages app. We should be able to open Spotlight or Siri and ask, “Show me photos I took between August 3 and 5.” Right now, that doesn’t work. Spotlight brings up images, sure, but only those that contain the text you typed. With a little sprinkling of Apple Intelligence, those searches could be much more useful.

Foundry

Apple LLM

One of the more glaring examples of Apple intelligence’s ineptitude is Siri’s integration with ChatGPT, which has the unfortunate effect of directly comparing Apple’s AI engine with a far more advanced system. The comparison is not exactly flattering to Apple’s infant AI efforts.

What we really need is for Apple to launch its own LLM to compete with the likes of ChatGPT. For one thing, this would inject far more power into Apple Intelligence’s engines and allow it to stand toe-to-toe with the industry’s best platforms. For another, it would better preserve your privacy, since you could stay in the Apple intelligence sphere without venturing off into less-secure cloud-based alternatives. With an Apple LLM behind the wheel, there would be no need to switch to other AI tools, which would keep your workflow smooth and uninterrupted.

Rumors have suggested that this kind of Apple-backed LLM is in the works, but we don’t know when it might arrive. When it eventually does drop, it could be the jump-start Apple Intelligence needs to keep pace with its rivals.

Shortcuts

Apple’s Shortcuts app is one of the best tools available on your iPhone, yet I’d bet most people have never even opened it. Part of the problem is that Apple only covers the most basic of tasks, so you’re on your own if you want to run more complex workflows.

This is a place where Apple Intelligence could step in to fill the gap. If AI was integrated into Shortcuts so that you could simply tell it what you want to do, Shortcuts would instantly go from an underused utility to one of the best iPhone features ever. As one of the main tenets of Apple intelligence, Shortcuts could suggest simple time-saving shortcuts throughout the day like Siri suggestions, and turn your iPhone into a true digital assistant.

Foundry

Health and fitness

Health and fitness tracking has been one of Apple’s priorities in recent years, with the Apple Watch leading the way. There’s a lot of health data stashed on our devices, and there are plenty of ways that Apple’s AI could help improve health and fitness without compromising privacy. There are already rumors that Apple is working on an AI “coach” to interpret the data collected by your devices and give you a more complete picture of your health.

Such a platform could manifest itself in optimized workouts with Apple Intelligence crunching your fitness numbers and then building a custom workout plan based on your needs. Or specialized HIIT workouts, Pilates, weight training, or whatever it thinks you need to take your fitness to the next level.

Apple

Apple Intelligence 2.0: New features

Videos

Apple Intelligence is already being used to edit images on our iPhones, but what if it could apply its skills to videos as well? The iPhone is already one of the top video phones around and AI could take it to the next level.

Perhaps that would mean creating a list of suggested edits for any project you feed into it. Or the power to remove distractions from videos in the same way you can currently do with images. Perhaps even automated color grading and audio mixing could be added.

Of course, Apple will be very careful about treading on video editing – after all, it probably doesn’t want to imply it’s coming after the creative industries after the Crush! ad fiasco – but a few handy tools for amateur editors to play with on their iPhones would be a welcome addition.

Notes

One of the ways that Apple tried to demonstrate Apple Intelligence’s power at WWDC 2024 was through transcribing audio in the Notes app. Unfortunately, this is neither a groundbreaking feature nor one that’s even on par with rival AI platforms. It might work well for simple voice memos, but anything with more than one speaker quickly becomes a jumbled mess.

There’s a quick way Apple can fix this by teaching Apple Intelligence how to automatically label speakers—after all, services such as Otter and the tiny independent MacWhisper app have been doing it for years.

Apple

Maps 

Over the years, Apple Maps has improved so much that many iPhone users now prefer it over Google Maps. But instead of Apple running a victory lap, Apple can use AI to enhance Maps even further.

At the moment, you can search Apple Maps for things like restaurants and shops near you. But I want more than that – I want to be able to ask Apple Maps for “the best restaurants near me” and only see the highest-rated ones. Or search for eateries that serve meals with specific ingredients (this is currently very hit-and-miss). Or find out where the best viewpoints are in my city (again, this is unreliable). Apple Maps struggles with that right now, but perhaps Apple Intelligence would help it get there.

Apple Intelligence can also help with directions. Siri is already pretty good at getting us from Point A to Point B, but as with much of Siri’s current features, it’s still disappointingly limited to places and addresses. Apple Intelligence could step in here to make it more powerful. For example, you could ask Siri to “get me home using the most eco-friendly route,” or “create a circular walking route in the countryside near me.” Weaving more AI into the mix might make these requests more achievable for Apple’s virtual assistant.

Apple

Home

Apple is heavily rumored to be getting in on the smart home act with a new HomePad hub device reportedly launching later this year and early next, and Apple Intelligence will almost certainly take on a central role in how you interact with it. But

You might one day use Apple Intelligence to analyze the connected devices in your smart home to work out how you could save money on electricity bills or reduce your carbon footprint. And with rumors swirling that Apple is working on a series of smart home hubs, it’s possible that these capabilities will appear on the horizon sooner rather than later.