There are four types of problems that can be perfectly solved by algorithms. If it were the year 1960, we might say this:

• Problems algorithms are good for solving: Like sorting a big list of data

• Problems too difficult for algorithms to solve: Like planning a route from A to B. (See later)

• Problems we can't even approach with algorithms: Like telling if we have a photo of a bird, or making a computer opponent who's good at chess.

But then, we had some of the following cool algorithmic advancements:

• Being okay with algorithms being slow sometimes: We can use hashmaps and randomized methods and heuristics. This lets us plan a route from A to B across the world and have it usually be fast.

• Being okay with algorithms sometimes being wrong: Anything that can use a bloom filter will be wrong sometimes, but will be a lot faster. This lets us plan a route from A to B across the world, and sometimes it's suboptimal and that's okay.

But all the algorithms we had were about coming up with clever constructions built around the existing datastructures and algorithms we had. Faster computer speeds, analog tools, parallel computing, etc. resulted in some clever advances.

"Heuristic methods" usually means making some assumption about the underlying distribution of the dataset, or using a statistic about the dataset, or approximating the data (e.g. you can get the mean of a list before you sort it in O(n) time, which can be used to make for faster sorting, even if you're still sorting in O(n log n) time.)

"Machine learning" takes this a step forward, throwing all the knowledge we can gather about the data we're working on. We can apply the full brunt of statistics and stochastic modeling we have at a problem.

With machine learning, this leads to a place where we solve problems entirely differently. Traditional algorithm work assumes no prior knowledge about the distribution of the input data, while machine learning explicitly and only works with prior knowledge about the distribution of input data.

So, the questions you ask are, "What data will we gather?" "What answers do we want?" "How can we construct a model that maps the input domain to the output range, with parameters we can fit to the data?"

You can go wild with this, but you have to hand-craft the approach. You can, for example:

• Train a uppercase <--> lowercase model, and use that to generate "uppest case" and "lowest case" fonts

• Videogame enemies which "learn how to learn", like in MGS5 and Alien: Isolation Shadow of Mordor, or so I'm told.

• Real scientific applications, like factor analysis to help create better medical tests, or manifold learning to identify things we don't have a name for.

• Anywhere with problems needing nonconvex optimization to solve.

• Plus all the ones you probably already heard about, like "Can machine learning help us detect earth quakes or identify tumors?"

• Lots of bad things too. Google "dual use technology" to feel sad.

Machine learning was a real fascinating area with vast applications.

"Deep learning" was machine learning with deep neural networks, and that too was a fascinating area with vast applications.

"Artificial intelligence" once meant systems of known facts and relationships ("symbolic AI").

And now "machine learning" means "AI" which just means "something with an LLM built into it."

It's amazing how vast this field was in 2022, and how it's been pigeonholed into a few small areas of research. There's basically four areas of research nowadays:

• We built and trained a better txt2txt (ChatGPT) / txt2img (StableDiffusion) / img2txt model, or

• We built and trained one of those models for a niche use case (like generating small emojis instead of large images), or

• We deployed one of those models on consumer harderware, or

• We put an LLM into something which already worked and now it works better on a contrived metric.

It makes me want to pull my hair out. When I left the field in 2022, I thought there was a lot of "sniffing their own farts" and that the forces driving the research weren't aligned with what was actually useful for the world. I didn't realize how much worse it would get.

Manifold learning is a type of poetry I think, and yet nobody gives a shit about that anymore :( Except for "we forced our latent space to map to transform coordinates so you can explore your generated image in 3D" or whatever. Which, cool I guess, but don't you want to know the true shape of things? Don't you want to try? Don't you want to apply the same concept to concepts which don't have literal shapes? Don't they teach you kids about t-SNE and UMAP anymore?