Natural language processing builds on several foundational techniques. Tokenization splits raw text into smaller units called tokens—words, subwords, or characters. Modern systems typically use subword schemes such as Byte-Pair Encoding (BPE), WordPiece, or SentencePiece, which balance vocabulary size with the ability to handle rare words and morphology. Stemming crudely chops word endings to a rough root (sometimes producing non-words), while lemmatization uses vocabulary and morphology to return proper dictionary forms and is slower but more accurate. Word embeddings are dense, low-dimensional vector representations of words where semantically similar words are nearby. Word2Vec (CBOW and Skip-gram) and GloVe are classic methods; Word2Vec famously produces analogies like king − man + woman ≈ queen, while GloVe factorizes global word co-occurrence matrices. Modern transformer-based models learn contextual embeddings that depend on surrounding text.
Higher-level NLP tasks layer on these foundations. Part-of-speech (POS) tagging assigns a grammatical category to each token, reaching 97%+ accuracy on English with neural sequence models. Named Entity Recognition (NER) locates and classifies entities such as people, organizations, locations, and dates; modern systems use BiLSTM-CRF or transformer-based models. Named entity disambiguation links a mention to a specific entity in a knowledge base—for example, distinguishing "Apple" the company from "apple" the fruit. Dependency parsing analyzes grammatical structure by linking words into typed head–dependent relations. Sentiment analysis classifies the polarity of text (positive, negative, neutral) using lexicon methods, classical ML on TF-IDF features, or fine-tuned transformers, and is widely used for brand monitoring and review analysis.
Contrastive learning has emerged as a powerful paradigm for representation learning. The idea is to pull similar pairs—such as an image and its caption, or two augmentations of the same image—together in embedding space while pushing dissimilar pairs apart. Losses such as InfoNCE, triplet loss, and supervised contrastive loss train models to produce embeddings that cluster semantically related items. CLIP trains an image encoder and a text encoder jointly with a contrastive InfoNCE loss on a massive image-text dataset, aligning the two modalities in a shared space and enabling zero-shot image classification from natural-language prompts. CLIP powers text-to-image systems like DALL·E.
Generative models learn to produce new samples resembling the training distribution. Generative Adversarial Networks (GANs) train two networks in opposition: a generator that produces fake samples from noise and a discriminator that distinguishes real from fake; through adversarial training, the generator produces increasingly realistic outputs and powers deepfakes and high-resolution image synthesis such as StyleGAN. Variational Autoencoders (VAEs) learn a probabilistic latent space by training an encoder to map inputs to a distribution and a decoder to reconstruct from sampled latent codes, optimized via the evidence lower bound; VAEs produce blurry but diverse samples. Diffusion models learn to reverse a gradual noising process—during training, real images are progressively corrupted with noise and the model learns to denoise them, while at inference, samples are generated by denoising from pure noise. Diffusion models such as Stable Diffusion and DALL·E 2 currently lead in image synthesis quality. The Boltzmann machine and its restricted variant (RBM) are older energy-based generative models foundational to deep belief nets. For evaluating text generation, BLEU measures n-gram precision between generated and reference text and is standard for machine translation, while ROUGE measures n-gram recall and is standard for summarization; both have known weaknesses for paraphrases. Discriminative models like logistic regression and SVMs learn P(y|x), while generative models like Naive Bayes, GANs, VAEs, and language models learn P(x, y) or P(x|y) and can sample new data.