Real Mode: A compatibility mode of the x86 processors, in which they operate almost identically to the original 16bit 8086/8088 CPU. In Real Mode, memory accesses are performed by adding together the contents of a 16bit segment register right-shifted by 4, to a 16bit offset, to form 20bit linear addresses. When the processor initializes from reset, it starts in this 16bit compatibility mode, and the software must take extra steps to switch the processor to the full 32bit protected mode, or 64bit long mode if it wishes to access more than 1MB of memory.
Protected Mode: The native mode of operation of 32bit x86 processors, in which it can address up to 4GB of memory is called protected mode. Protected mode is entered by setting up a GDT, flipping the PE (protection enable) bit in cr0 (control register 0), and loading all relevant segment registers (at least cs, ds and ss) with appropriate segment selectors.
Unreal Mode: Unreal mode is not an actual documented mode of operation of the x86 processor, but rather a side-effect of the fact that the processor keeps a descriptor cache for each segment register, instead of constantly looking up descriptors in the GDT for each memory access. Unreal mode works by entering protected mode temporarily, to set up 4GB segments in the GDT and to load segment registers with those selectors, before switching back to real mode. This makes the processor cache the 4GB segment limits in the descriptor cache, and after switching back to real mode, those 4GB segment limits are still in effect. The result is that even though address calculations are performed by combining an offset with a 16bit segment register, the offset is no longer constrained to a 16bit value, instead it can be a full 32bit value, allowing access to the full 4GB address space.
Virtual 8086 (or v86) Mode: A special mode of operation of the x86 processor, designed to allow running 16bit 8086 programs under the control of a 32bit protected mode operating system. A context switch to a 16bit task with the VM bit set in the eflags register is used to trigger entering v86 mode. In this mode privileged operations are trapped triggering an exception which can be handled by the 32bit operating system, to emulate whatever the 8086 program was trying to do, without allowing it to take over and/or crash the system. Use cases of v86 include the MS-DOS program execution under Windows 9x, the implementation of int86 under DOS4G/W and DPMI hosts, as well as running MS-DOS programs with a 32bit memory manager enabled, such as EMM386 or QEMM.
GDT (Global Descriptor Table): A table containing mainly segment descriptors, which define the start address, size, and protection attributes for each memory segment. Before switching the processor into protected mode, the GDT must be populated, and its address loaded into the gdtr register using the lgdt instruction. At minimum it's necessary to define a supervisor (priviledge level 0) code segment, and a supervisor data segment. The table index is called a selector, and can be loaded in a segment register, to access memory of that segment.
IDT (Interrupt Descriptor Table): A table containing interrupt descriptors, which define the address of the interrupt service routine, the corresponding code segment selector, and protection attributes for each interrupt. The IDT is populated, and its address loaded into the idtr register, using the lidt instruction. Each entry in the IDT takes 8 bytes, so when an interrupt is raised, to locate the appropriate descriptor, the processor starts from the base of the IDT, and adds the interrupt number times 8.
IVT (Interrupt Vector Table): A table starting at address 0, which contains the segment:offset address of interrupt service routines, on the 8086. Later processors use the original IVT format to vector interrupts while running in real mode, but the table can be located anywhere in memory, with the help of the idtr register.
A20 Line: For compatibility with the 8086/80888 processor, PCs start with the 21st address line disabled, to avoid generating addresses above 1MB, and instead wrap around to the start of the memory when that would occur. This was done because some early PC software depended on this memory wrap-around of earlier processors, and failed to operate correctly without it. Enabling the A20 line is a crucial step when switching to protected mode, otherwise every odd megabyte of memory will be inaccessible, and mapped to the previous even megabyte instead, with catastrophic consequences.
VBE (VESA BIOS Extensions): In the early 90s, PC graphics cards manufacturers started extending their chips beyond the capabilities of the VGA, providing higher resolutions and color depths, and did so initially in completely different and incompatible ways. To enable software interoperability without having to support each and every SVGA chip out there explicitly, the VESA industry consortium defined a set of extensions to the standard video BIOS for handling higher resolutions, and later linear video memory access (VBE 2.0), and protected mode interfaces (VBE 3.0).