Real-time image scaling techniques through the use of AI algorithms, accelerated through dedicated hardware, have gained importance in recent times. We explain how these resolution upload algorithms work, which performance benefits bring to our PCs. What will the new games of the future be like thanks to this resolution increase technology?
The increase in resolution in both video games and movies has consequences in terms of the power required to generate an image, since the more resolution pixels we want to achieve as an output resolution, the more power we will need. The idea behind using algorithms to generate higher resolution images from lower resolution ones are aimed at saving computational power and time.
Causes for using AI to generate images with increased resolution
Image scaling algorithms or super resolution algorithms are those that convert an image buffer in “X” resolution to a higher resolution.
Image scaling algorithms not based on AI
To understand the need for artificial intelligence as a tool to increase the resolution of images, it is necessary to know in advance the limitation of classical algorithms not based on AI.
But how do we go about scaling an image in terms of resolution? When we increase the resolution we find that most of the new pixels with respect to the previous resolution do not have a defined color value so we have to fill them in some way.
The easiest way to do the resolution increase is to copy the color value of the adjacent pixels, but the actual result is not an image at a higher resolution but an image at the same resolution using a higher number of dots per pixel. This algorithm is called Nearest Neighbor (a) (Close Neighbor)
Other techniques are based on interpolation algorithms, these use the color value of the adjacent pixels to know the value of the intermediate pixels using a greater or lesser number of samples to know the value of said pixels. The most famous are bilinear interpolation (b), also used for filtering textures in games, and bicubic interpolation (c).
None of these scaling algorithms reach an optimal quality level and they never produce an image as sharp as generating the image directly at higher resolution (d).
Artificial Intelligence to the rescue
When the Artificial Intelligence algorithms scale an image, what they do is imagine or invent the intermediate pixels that we do not know, for this it is necessary to train the artificial intelligence with a series of example images as long as the artificial intelligence learns a pattern from the which to reconstruct the images at a higher resolution from others with a lower resolution.
For this, it is necessary to feed the neural network during the training stage with hundreds and even thousands of images so that it finds a fixed pattern that it can then apply when generating the missing pixels, this is called AI training .
Once artificial intelligence has found the pattern, what it does is create a generative algorithm that it will always apply to create the images at higher resolution and that will execute the corresponding specialized AI hardware that we are using in the system, we call this inference of the AI.
Neural Networks for higher image resolution
There are many types of neural networks, but those that are usually used to generate images at higher resolution are the so-called generative adversary networks, in English Generative Adversial Network or GAN.
The simplest form of this type of network is as an example at the beginning of this subsection. GANs work with two neural networks that are called Generator and Discriminator.
The Generator is in charge of generating samples that are initially nothing more than noise, while the second, which we call Discriminator, evaluates them by comparing them with samples of real images. This evaluation only has two answers that are true or false. If the evaluation is positive then the image is used as feedback so that the generative network learns what the pattern is to create new images that increasingly resemble what we want it to learn to generate.
For example, if we feed the Discriminator with photos of cats at the end, the Generator will end up learning that the images it has to generate are photos of cats.
Using AI, an increase in resolution is achieved in video games
When we play a video game it is not the same as with a movie where all the frames are stored beforehand and therefore the Discriminator completely lacks sample images as an example.
We can train a GAN to make it learn to scale images from old films to a higher resolution, taking as an example for its training images from other newer and higher resolution films since we are talking about frames taken from real life.
But in the case of a video game not only is each frame unique and is not repeated again, so there is no previous frame at a higher resolution to compare, but also the fact that each game has its own visual style causes that the algorithms learned to generate an image buffer at a higher resolution of a game will not work for other games if they do not share a similar artistic style.
This forces the generative adversary network that we have to use to train the AI and thereby generate the inference algorithm to be able to generate higher resolution images, to be something different.
To train the AI to learn to raise the resolution of a video game, what is done is to use a very powerful PC with the ability to render images at very high resolution.
This very powerful PC can be something like a server with several GPUs working in unison in order to render the image at very high resolution. For example, NVIDIA uses its Saturn V supercomputer to generate the higher resolution image buffer.
On the other hand, we have a GPU rendering the same scene at a lower resolution, this lower resolution scene is used as an input sample for the Generator, which from this image will try to generate an image at a higher resolution that the discriminator itself will have to evaluate at from which the image with a higher resolution comparing both.
As expected, the result is a higher performance, an increase in resolution due to image scaling and therefore greater clarity in the gamer’s view.
