Generally, the mechanisms for invoking the operating system are called interrupts. They are the gateways to the operating system. Modern computers will go into a special mode called system mode when they handle an interrupt. System mode provides the foundation for operating system security.
In most modern processors, the hardware deals with an interrupt using a simple mechanism that works much like a jump instruction except that the target address is fixed by the hardware and the processor switches into system mode. Operating systems install interrupt handling code at this address when they are booted up.
The three types of interrupts are
Software interrupts are triggered by executing programs (processes) to
request operating system services.
On many machines this is done with a single syscall
instruction.
A code stored in a specified register indicates what kind of service.
Data stored in other registers can provide parameters for the operation.
The system calls that are available depends on the operating system. They make up part of its application binary interface (ABI). The rest of the ABI consists of the machine's instruction set.
High-level languages usually provide a higher-level operating system
interface called the application program interface (API).
This consists of language library subprograms that set up registers for
system calls, then execute a syscall instruction.
Many high-level languages use the C library functions to do this.
External interrupts are signals from external devices that need attention from the operating system. For example, after a software interrupt has started a disk read operation, the operating system typically puts the requesting process on hold until the read operation completes. Since completion takes several milliseconds, the operating system will start a different process. The operation completes the disk read after the disk has triggered an interrupt that signals that the data is ready.
Exceptions indicate problems detected by the processor that it cannot handle by itself. Some indicate errors such as divide by zero. Others may or may not indicate errors - only the operating system can know for sure.
For example, with virtual memory, a flag bit in a page table entry indicates that a page is not in memory. The processor will trigger an exception if an attempt is made to access the affected page. The operating system then determines if the access is legal or not. If it is legal, the operating system will start up a disk read to bring the page in from disk swap space.
In system mode, the computer can do things that it cannot do in the normal mode, which is called user mode. It may be able to execute special instructions, or it may be able to access parts of memory that are inaccessible in user mode. System mode is the foundation of operating system security.
The code that is installed at the target address for interrupts is called an interrupt handler. The first thing that it has to do is save the state of the currently executing process. Then it calls a subprogram to deal with the specific type of interrupt. When that subprogram returns, the interrupt handler restores the state of the process that was executing when the interrupt occurred.
The interrupt handler determines which subprogram to call by looking at an interrupt code. For software interrupts, this code is passed through a register by the software. For external interrupts, it is passed to the processor by the interrupting device through data lines in the system bus. For exceptions, the code is generated internally by the processor.
In early processors, there were often attempts to implement parts of the operating system in hardware. These attempts severely limited the capability of running different operating systems on the same processor. Though it seemed that doing things in hardware would improve performance, the performance benefits were either minimal or nonexistent. The flexibility offered by the simple scheme described above is far more important.