## Run this Tcl script to generate the sqlite.html file.#set rcsid {$Id: arch.tcl,v 1.16 2004/10/10 17:24:54 drh Exp $}source common.tclheader {Architecture of SQLite}puts {<h2>The Architecture Of SQLite</h2><h3>Introduction</h3><table align="right" border="1" cellpadding="15" cellspacing="1"><tr><th>Block Diagram Of SQLite</th></tr><tr><td><img src="arch2.gif"></td></tr></table><p>This document describes the architecture of the SQLite library.The information here is useful to those who want to understand ormodify the inner workings of SQLite.</p><p>A block diagram showing the main components of SQLiteand how they interrelate is shown at the right.  The text thatfollows will provide a quick overview of each of these components.</p><p>This document describes SQLite version 3.0.  Version 2.8 andearlier are similar but the details differ.</p><h3>Interface</h3><p>Much of the public interface to the SQLite library is implemented byfunctions found in the <b>main.c</b>, <b>legacy.c</b>, and<b>vdbeapi.c</b> source filesthough some routines arescattered about in other files where they can have access to data structures with file scope.  The<b>sqlite3_get_table()</b> routine is implemented in <b>table.c</b>.<b>sqlite3_mprintf()</b> is found in <b>printf.c</b>.<b>sqlite3_complete()</b> is in <b>tokenize.c</b>.The Tcl interface is implemented by <b>tclsqlite.c</b>.  Moreinformation on the C interface to SQLite is<a href="capi3ref.html">available separately</a>.<p><p>To avoid name collisions with other software, all externalsymbols in the SQLite library begin with the prefix <b>sqlite3</b>.Those symbols that are intended for external use (in other words,those symbols which form the API for SQLite) beginwith <b>sqlite3_</b>.</p><h3>Tokenizer</h3><p>When a string containing SQL statements is to be executed, theinterface passes that string to the tokenizer.  The job of the tokenizeris to break the original string up into tokens and pass those tokensone by one to the parser.  The tokenizer is hand-coded in C in the file <b>tokenize.c</b>.<p>Note that in this design, the tokenizer calls the parser.  Peoplewho are familiar with YACC and BISON may be used to doing things theother way around -- having the parser call the tokenizer.  The authorof SQLite has done it both ways and finds things generally work out nicer forthe tokenizer to call the parser.  YACC has it backwards.</p><h3>Parser</h3><p>The parser is the piece that assigns meaning to tokens based ontheir context.  The parser for SQLite is generated using the<a href="http://www.hwaci.com/sw/lemon/">Lemon</a> LALR(1) parsergenerator.  Lemon does the same job as YACC/BISON, but it usesa different input syntax which is less error-prone.Lemon also generates a parser which is reentrant and thread-safe.And lemon defines the concept of a non-terminal destructor sothat it does not leak memory when syntax errors are encountered.The source file that drives Lemon is found in <b>parse.y</b>.</p><p>Becauselemon is a program not normally found on development machines, thecomplete source code to lemon (just one C file) is included in theSQLite distribution in the "tool" subdirectory.  Documentation onlemon is found in the "doc" subdirectory of the distribution.</p><h3>Code Generator</h3><p>After the parser assembles tokens into complete SQL statements,it calls the code generator to produce virtual machine code thatwill do the work that the SQL statements request.  There are manyfiles in the code generator:<b>attach.c</b>,<b>auth.c</b>,<b>build.c</b>,<b>delete.c</b>,<b>expr.c</b>,<b>insert.c</b>,<b>pragma.c</b>,<b>select.c</b>,<b>trigger.c</b>,<b>update.c</b>,<b>vacuum.c</b>and <b>where.c</b>.In these files is where most of the serious magic happens.<b>expr.c</b> handles code generation for expressions.<b>where.c</b> handles code generation for WHERE clauses onSELECT, UPDATE and DELETE statements.  The files <b>attach.c</b>,<b>delete.c</b>, <b>insert.c</b>, <b>select.c</b>, <b>trigger.c</b><b>update.c</b>, and <b>vacuum.c</b> handle the code generationfor SQL statements with the same names.  (Each of these files calls routinesin <b>expr.c</b> and <b>where.c</b> as necessary.)  All otherSQL statements are coded out of <b>build.c</b>.The <b>auth.c</b> file implements the functionality of<b>sqlite3_set_authorizer()</b>.</p><h3>Virtual Machine</h3><p>The program generated by the code generator is executed bythe virtual machine.  Additional information about the virtualmachine is <a href="opcode.html">available separately</a>.To summarize, the virtual machine implements an abstract computingengine specifically designed to manipulate database files.  Themachine has a stack which is used for intermediate storage.Each instruction contains an opcode andup to three additional operands.</p><p>The virtual machine itself is entirely contained in a singlesource file <b>vdbe.c</b>.  The virtual machine also hasits own header files: <b>vdbe.h</b> that defines an interfacebetween the virtual machine and the rest of the SQLite library and<b>vdbeInt.h</b> which defines structure private the virtual machine.The <b>vdbeaux.c</b> file contains utilities used by the virtualmachine and interface modules used by the rest of the library toconstruct VM programs.  The <b>vdbeapi.c</b> file contains externalinterfaces to the virtual machine such as the <b>sqlite3_bind_...</b> family of functions.  Individual values(strings, integer, floating point numbers, and BLOBs) are storedin an internal object named "Mem" which is implemented by<b>vdbemem.c</b>.</p><p>SQLite implements SQL functions using callbacks to C-language routines.Even the built-in SQL functions are implemented this way.  Most ofthe built-in SQL functions (ex: <b>coalesce()</b>, <b>count()</b>,<b>substr()</b>, and so forth) can be found in <b>func.c</b>.Date and time conversion functions are found in <b>date.c</b>.</p><h3>B-Tree</h3><p>An SQLite database is maintained on disk using a B-tree implementationfound in the <b>btree.c</b> source file.  A separate B-tree is used foreach table and index in the database.  All B-trees are stored in thesame disk file.  Details of the file format are recorded in a largecomment at the beginning of <b>btree.c</b>.</p><p>The interface to the B-tree subsystem is defined by the header file<b>btree.h</b>.</p><h3>Page Cache</h3><p>The B-tree module requests information from the disk in fixed-sizechunks.  The default chunk size is 1024 bytes but can vary between 512and 65536 bytes.The page cache is responsible for reading, writing, andcaching these chunks.The page cache also provides the rollback and atomic commit abstractionand takes care of locking of the database file.  TheB-tree driver requests particular pages from the page cache and notifiesthe page cache when it wants to modify pages or commit or rollbackchanges and the page cache handles all the messy details of making surethe requests are handled quickly, safely, and efficiently.</p><p>The code to implement the page cache is contained in the single Csource file <b>pager.c</b>.  The interface to the page cache subsystemis defined by the header file <b>pager.h</b>.</p><h3>OS Interface</h3><p>In order to provide portability between POSIX and Win32 operating systems,SQLite uses an abstraction layer to interface with the operating system.The interface to the OS abstraction layer is defined in<b>os.h</b>.  Each supported operating system has its own implementation:<b>os_unix.c</b> for Unix, <b>os_win.c</b> for windows, and so forth.Each of these operating-specific implements typically has its ownheader file: <b>os_unix.h</b>, <b>os_win.h</b>, etc.</p><h3>Utilities</h3><p>Memory allocation and caseless string comparison routines are locatedin <b>util.c</b>.Symbol tables used by the parser are maintained by hash tables foundin <b>hash.c</b>.  The <b>utf.c</b> source file contains Unicodeconversion subroutines.SQLite has its own private implementation of <b>printf()</b> (withsome extensions) in <b>printf.c</b> and its own random number generatorin <b>random.c</b>.</p><h3>Test Code</h3><p>If you count regression test scripts,more than half the total code base of SQLite is devoted to testing.There are many <b>assert()</b> statements in the main code files.In additional, the source files <b>test1.c</b> through <b>test5.c</b>together with <b>md5.c</b> implement extensions used for testingpurposes only.  The <b>os_test.c</b> backend interface is used tosimulate power failures to verify the crash-recovery mechanism inthe pager.</p>}footer $rcsid