Different Kinds of Visual Transformers (TODO)
Shift Transformer (Swin Transformer)
The Swin Transformer (“Shifted windows”) took inspiration from standard CNNs:
Instead of performing self-attention over the entire sequence of tokens, one for each patch, it performs “shifted window based” self-attention, which means only performing attention over square-shaped blocks of patches. One block of patches is analogous to the receptive field of one convolution.
After every few attention blocks, there is a “merge layer”, which merges neighboring 2x2 tokens into a single token. This is analogous to pooling (by 2x2 convolution kernels, with stride 2). Merging means concatenation followed by multiplication with a matrix.
It is improved by Swin Transformer V2, which modifies upon the ViT by a different attention mechanism:
- LayerNorm immediately after each attention and feedforward layer (“res-post-norm”);
- Scaled cosine attention to replace the original dot product attention;
- log-spaced continuous relative position bias, which allows transfer learning across different window resolutions.
Timesformer
The TimeSformer was designed for video understanding tasks, and it applied a factorized self-attention, similar to the factorized convolution kernels found in the Inception CNN architecture.
Schematically, it divides a video into frames, and each frame into a square grid of patches (same as ViT). Let each patch coordinate be denoted by 𝑥, 𝑦, 𝑧, denoting horizontal, vertical, and time.
A space attention layer is a self-attention layer where each query patch
A time attention layer is where the requirement is
The TimeSformer also considered other attention layer designs, such as the “height attention layer” where the requirement is
However, they found empirically that the best design interleaves one space attention layer and one time attention layer.
VIT-VQGAN
In ViT-VQGAN there are two ViT encoders and a discriminator:
- One encodes 8x8 patches of an image into a list of vectors, one for each patch. The vectors can only come from a discrete set of “codebook”, as in vector quantization.
- Another encodes the quantized vectors back to image patches.
The training objective attempts to make the reconstruction image (the output image) faithful to the input image.
The discriminator (usually a convolutional network, but other networks are allowed) attempts to decide if an image is an original real image, or a reconstructed image by the ViT.
The idea is essentially the same as vector quantized variational autoencoder (VQVAE) plus generative adversarial network (GAN).
After such a ViT-VQGAN is trained, it can be used to code an arbitrary image into a list of symbols, and code an arbitrary list of symbols into an image.
The list of symbols can be used to train into a standard autoregressive transformer (like GPT), for autoregressively generating an image.
Further, one can take a list of caption-image pairs, convert the images into strings of symbols, and train a standard GPT-style transformer. Then at test time, one can just give an image caption, and have it autoregressively generate the image.
Video Vision Transformer (VIVIT)
It tokenizes the video in one of two ways: uniform frame sampling i.e., the same tokenization procedure adopted in TimeSformer, and tubelet embedding, which extracts tokens from small non-overlapping spatio-temporal boxes that an entire input video. Each spatio-temporal box is flattened and encoded through a linear embedding and concateanted into a long token sequence representing the whole video.
The resulting sequence is finally processed by a standard transformer encoder, as in ViT. However, ViViT offers some less computationally demanding variants, which factorise the processing ot ehs patial and temporale dimensions of the video a tiether architecture or
First, patheces are extract from video frame according to the patching echanism adopted by the speciffic mode. A D-dimensional embedding is then computed for each p, and positional embessing is addedd to encode patch position with the video. The seqeunce of patches is then input to a stack composed of L transformer encoder layers. Finally a prediction head produces the output according to the task being tackled.
MTV
The multiveiw transformer (MTV) is another spatio-temporal transformer model for video processing. MTV based on ViViT, introduces a multiview archiotecture to process video by considering multiple temporal scales. MTV tokenizes video using the aforementioned tubelet embedding. Multi-scale views of the input video are constructed by tokenizing the video multiple times, and tubelets with different temporal sizes are used at each time. Views with smaller tubelets caputre fine-grained video details, while larger tubelets capture coarser aspectrs of the video.
Uniformer
The unified transformer (UniFormer) unifies under one transformer architecture the spatio-temporal locality awareness (typical of 3D convolutions) and he skill of captuing long-term dependencies, typicals of transformers. UniFormer introduces a specific relation aggregator block with two variants: a local variant capturing …
both blocks work with multiple heads, each, with its own parameters, computing an output according to the block type. Outputs of all heads are concateanted and linearly projected back to the input space. These relation aggregator blocks are organized in such a way that local dependencies are grasped in shallow layers of the network, while global one are modelled by deeper leaders. In fact uniformer is composed of four relation aggregator blocks: