Other Ways to Learn

You can use machine learning for some extremely sophisticated applications, such as creating or modifying text and images. However, these activities often consist of an intelligent combination of the three main techniques.

For example, Generative Adversarial Networks (GANs) work in a way that means that an artificial neural network A should be able to distinguish a real image of a dog from a false image or an image without a dog. We sometimes feed this network with stock images of dogs, and sometimes with images generated by another artificial neural network (let's call it B). If A makes a mistake, we find out about it and can use the mistake in the way we did before in the example of supervised learning with the dog, cat, mouse and elephant. On the other hand, if A is able to detect the false images from B, this failure is fed into network B as a negative used to optimize B. If this works, both networks will increasingly become better at their respective tasks.

Language or texts can be generated on the basis of neural networks with a memory. Currently, these are often transformer networks, although long short-term memory (LSTM) networks often do a good job, too. As supervised techniques, both variants basically only predict the probability with which sentence part, letter, or word is likely to follow next. Learning on the basis of very large bodies of language helps to develop a model that can continue texts or articles that have been started.

It would typically also be possible to implement something like this based on keywords. If you read somewhere that an artificial intelligence is now writing newspaper articles, there is actually still no intelligence involved. The point is simply that there are so many typical structures for, say, sports reports that it is easy for a model to generate a typical text from a few hard facts – who scored a goal and when, plus a couple of special features. That says more about the sports section of the newspaper than it does about AI.

This is not to say that a trained model cannot create something new. Thanks to the current copyright amendment from 2021, many people will be familiar with the term pastiche. Machine learning systems are very good at recombining things. If they receive positive feedback about what people appreciate, they can also create attractive images. If it is only a matter of imitating the style, a Neural Style Transfer (Figure 4), for example, provides attractive results more easily and stably than a GAN.

Figure 4: An example of Neural Style Transfer: The style from Wave off Kanagawa (center) is transferred to the content on the left. The result (on the right) depends on the extent to which the optimization prioritizes style and content fidelity.

But this does not change the fact that these are essentially mathematical functions with inputs and outputs that improve thanks to optimization algorithms. The fact that optimization algorithms seem to perform more and more tasks that we previously thought would require a very intelligent human might suggest that we need to clarify our terms. For example, intelligence is not exactly a clearly defined concept. Judged by the simplest method – intelligence being what an intelligence test measures – machine models are likely to be intelligent soon.

For example, a large database of typical questions from intelligence tests was presented at the International Joint Conference on Artificial Intelligence in 2019. It is probably only a matter of time before the results, which were mediocre at the time, improve. If and when this happens, the criticism levied at this simplistic definition of intelligence is likely to intensify, and the term will probably have to be developed further. If science is lucky, we will discover a better definition for intelligence that can be formulated more scientifically.

In machine learning, there has been a decades-long competition between more accurate, faster, and farther (more data). Admittedly, these aspects are still important and will remain so. However, many current questions, especially in Europe, revolve around other topics, such as Trustworthy AI, determination of uncertainty, Green AI, and others. Many aspects and discussions relating to the concept of the term Trustworthy AI are bundled in the 2021 "Ethics Guidelines for Trustworthy AI," published by the EU. Among other things, the topics of privacy and data governance, transparency and diversity, non-discrimination and fairness can be identified as specifically European impulses. Many things are happening centering around these issues in Europe right now.

People keep questioning whether Europe can keep up with competitors like China and the United States when it comes to AI – what they actually means in most cases is machine learning. The answer is probably that it depends on the topic. The term Green AI in turn comes from an article of the same name [1], the key phrase of which is: "The term Green AI refers to AI research that yields novel results while taking into account the computational cost, encouraging a reduction in resources spent." So far, little attention has been paid to resource consumption. If a few thousandth parts more accuracy can be achieved when distinguishing between different classes, this can often be worth publishing. This approach largely leaves out the costs of training and operation.

Dreams and Reality

In the end, even software based on machine learning algorithms is just software, which means it needs to be tested and maintained. And it would be great if it were open source, but that is easier to write than do.

When it comes to testing, it is important to realize that this is software that cannot easily be validated by classical testing approaches. The software itself is usually deterministic – you don't have to worry about a model classifying the same image as a dog in some cases and as a cat in others. But there are many other pitfalls. For example, in a real experiment [2], all wolves were identified as such. However, the attempt to apply the model to some images from a German zoo was depressing. There were many images of wolves in the training and test data, but all of them in the snow. The system had learned that "snow" was an essential factor for "wolf" – no snow, no wolf (Figure 5). Such failures cannot be determined by applying classical quality control.

Figure 5: This figure from the wolf identification experiment [2] shows what caused the bad decision.

The other point is: What would open source look like for software based on machine learning? All the widely-used machine learning software packages, like scikit-learn, TensorFlow/Keras, and PyTorch are released under open source licenses, but these are just the libraries that you use to build the software.

As an example, consider a piece of software that contains an artificial neural network. The weightings are not the kind of source text that is humanly readable and understandable but were extracted from a process. If you want to repeat the process, you need the libraries (no problem), information on the structure of the network (often supplied), the exact optimization parameters (this is where things start to dry up), and the dataset used for training. The last item on the list is usually off limits – you do not get to see the data.

This means that there are narrow limits as to what else can be done with a network like this if it is shared as part of open source software. It can be used for the intended purpose and with exactly the quality that was delivered. To improve it, you can usually only use this training status in the scope of transfer learning. Transfer learning means that a network has been trained for many image classes, for example.

One typical example is the ImageNet database, which contains many animals, people, and so on. A network of this kind, including the weighting values, can be taken and adapted to a new task for which data is available. For example, if you want to find South American birds only, you will not find all the species in ImageNet, but the network has already learned many useful filters. You can then get rid of the part of the network you can't use, and train the assembled network starting with the pre-trained status on your own collection of images of South American birds. This means that you can get by with fewer images and presumably achieve higher accuracy with a lower-energy overhead in training. This is good and also quite often possible.

But the actual idea behind the demand for open source software is that you can change the software and pass on the changed (and thus usually improved) form. And this goal is no longer fully met. If not all data is available, the possibilities for further development are limited – not to mention the problems posed by the numerous software patents in the machine learning environment.

Conclusions

Machine learning has been around for a long time, but its growing importance alone should be an incentive to look further behind the scenes. In addition to the power of machine learning, it is also genuinely fun.

For example, if you have a large collection of LaTeX documents, why not use them to train an LSTM or Transformer network? You can even build a feedback loop through the compiler so that the system learns to write error-free LaTeX code. The content will certainly be a bit weird, but the exercise is entertaining.

Then, once you understand what it's all about, as an open source enthusiast, you might wonder what this technology means for your own movement. The conventional "copyleft" protection of the GPL was only meant to apply to software distribution and doesn't require the modifier to release modifications for software that runs as a service. (You might recall the ASP loophole that the AGPL was supposed to close.) Personally, I believe that the widespread use of machine learning software should lead us to rethink how we want to push forward with open source development and bring everything together with the concept of open data.