?? metakit and tcl web server.mht
字號:
From: =?gb2312?B?08kgV2luZG93cyBJbnRlcm5ldCBFeHBsb3JlciA3ILGjtOY=?=
Subject: MetaKit and Tcl Web Server
Date: Fri, 29 Feb 2008 00:31:50 +0800
MIME-Version: 1.0
Content-Type: multipart/related;
type="text/html";
boundary="----=_NextPart_000_0074_01C87A6A.797B39A0"
X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6000.16545
這是 MIME 格式的多方郵件。
------=_NextPart_000_0074_01C87A6A.797B39A0
Content-Type: text/html;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Location: http://wiki.tcl.tk/3072
=EF=BB=BF<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" =
"http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd">
<HTML lang=3Den><HEAD><TITLE>MetaKit and Tcl Web Server</TITLE>
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8">
<STYLE type=3Dtext/css media=3Dall>@import url( /wikit.css );
</STYLE>
<LINK title=3DRSS href=3D"/rss.xml" type=3Dapplication/rss+xml =
rel=3Dalternate><!--[if lte IE 6]>=0A=
<style type=3D'text/css' media=3D'all'>@import 'ie6.css';</style>=0A=
<![endif]--><!--[if gte IE 7]>
<STYLE type=3Dtext/css media=3Dall>@import url( ie7.css );
</STYLE>
<![endif]-->
<SCRIPT type=3Dtext/javascript>=0A=
function init() {=0A=
// quit if this function has already been called=0A=
if (arguments.callee.done) return;=0A=
=0A=
// flag this function so we don't do the same thing twice=0A=
arguments.callee.done =3D true;=0A=
=0A=
try {=0A=
checkTOC();=0A=
} catch (err) {=0A=
/* nothing */=0A=
}=0A=
};=0A=
=0A=
/* for Mozilla */=0A=
if (document.addEventListener) {=0A=
document.addEventListener("DOMContentLoaded", init, false);=0A=
}=0A=
=0A=
/* for Internet Explorer */=0A=
/*@cc_on @*/=0A=
/*@if (@_win32)=0A=
document.write("<script defer src=3Die_onload.js><\/script>");=0A=
/*@end @*/=0A=
=0A=
/* for other browsers */=0A=
window.onload =3D init;=0A=
</SCRIPT>
<META content=3D"MSHTML 6.00.6000.16609" name=3DGENERATOR></HEAD>
<BODY>
<DIV class=3Dcontainer>
<DIV class=3Dheader>
<DIV class=3Dlogo><A class=3Dlogo =
href=3D"http://wiki.tcl.tk/">wiki.tcl.tk</A> </DIV>
<DIV class=3Dtitle id=3Dtitle><A title=3D"click to see reference to this =
page"=20
href=3D"http://wiki.tcl.tk/_ref/3072">MetaKit and Tcl Web =
Server</A></DIV>
<DIV class=3Dupdated id=3Dupdated>Updated 2003-11-13 02:55:13 <A =
class=3Ddelta=20
href=3D"http://wiki.tcl.tk/_diff/3072#diff0">=E2=96=B2</A></DIV></DIV>
<DIV id=3Dwrapper>
<DIV id=3Dcontent>
<P>Working on making use of <A =
href=3D"http://wiki.tcl.tk/20">MetaKit</A> within=20
the Tcl Web Server, also known as <A=20
href=3D"http://wiki.tcl.tk/2085">TclHttpd</A>...</P>
<HR>
<P><A href=3D"http://wiki.tcl.tk/6132">Jeff Smith</A> wrote - I am =
heading down=20
this path also. I would like to share my idea and hopefully get some =
feedback as=20
to whether this is feasible.</P>
<P>I hope to create a Database so my Squid proxy/cache server can =
authenticate=20
users before they go to the Internet. I will use <A=20
href=3D"http://wiki.tcl.tk/2085">TclHttpd</A> wrapped as a <A=20
href=3D"http://wiki.tcl.tk/3661">starkit</A> installed on the Squid box =
to provide=20
the server front end to the Database. The Squid client authentication =
program=20
will be a Starkit using the http package to delivery the "username" and=20
"password" to TclHttpd for authentication. Also I want my users to be =
able to=20
change their own passwords, so that should be possible via a form on the =
TclHttpd server. Also an "Administrator" form to add more users. I don't =
know if=20
this can all happen at the same time, I am reading the MetaKit =
documentation at=20
the moment and noticed there is an issue about concurrency!</P>
<P>Down the track as I become more familiar with Metakit, parse(is that =
the=20
correct term) the Squid access.log file with Tcl into individual user db =
files=20
for reporting or billing etc. and have them accessible via TclHttpd. =
Users=20
should be able to access there own reports on TclHttpd using the same=20
authentication scheme as above.</P>
<P>Finally if this all works, share this with the Squid Community, =
promote Tcl=20
and demonstrate what I think is a very powerfull combination, TclHttpd =
and=20
Tclkit.</P>
<HR>
<P><A href=3D"http://wiki.tcl.tk/3567">PS</A> A simple trick to safely =
update the=20
squid user database is to have two copies, one for reading only, the =
other for=20
updating. Assuming you are using unix and squid re-opens the database =
every time=20
(it is changed), simply [file copy -force workingcopy tmpcopy] and [file =
move=20
-force tmpcopy readingcopy] everytime you update your working copy and =
the=20
changes are guaranteed to be atomic for the squid process. Assuming your =
user=20
database is small, this adds almost no overhead at all.</P>
<P><A href=3D"http://wiki.tcl.tk/6146">NB</A> I have uploaded a small =
package, <A=20
href=3D"http://wiki.tcl.tk/6137">formkit</A> demonstrating the use of =
Metakit and=20
Tclhttpd for webforms</P>
<HR>
<P><A href=3D"http://wiki.tcl.tk/6132">Jeff Smith</A> Thanks for the =
feedback!=20
Greatly appreciated.</P>
<HR>
<P><A href=3D"http://wiki.tcl.tk/6132">Jeff Smith</A> OK I have made a =
start. Here=20
is a <A href=3D"http://wiki.tcl.tk/8657">Username/Password Database for=20
Tclhttpd</A> using Metakit.</P>
<HR>
<P>2003-07-04 <A href=3D"http://wiki.tcl.tk/6132">Jeff Smith</A> I have =
made=20
further progress check out [<A=20
href=3D"http://www.freewebs.com/headsup/userpassdb.htm" =
rel=3Dnofollow>1</A>] for a=20
Starkit. I think the best advise I can give anyone who is thinking of =
using=20
Metakit with Tclhttpd, is to use the Session module in Tclhttpd for =
session=20
management and record locking of the Metakit database when updating. =
Another=20
example can be found at [<A =
href=3D"http://www.freewebs.com/headsup/formkitse.htm"=20
rel=3Dnofollow>2</A>]</P>
<HR>
<P>2003-11-12 <A href=3D"http://wiki.tcl.tk/6132">Jeff Smith</A> Well I =
have been=20
in production now for 2 months. 1500 Squid users authenticating daily to =
my=20
Metakit database with a TclHttpd front end. It works great! The reason =
behind=20
this little project was my Squid users originally authenticated against =
a=20
database which ran on an old Firewall. The Firewall was at its end of =
life and=20
the new Firewall did not allow authentication from our Squid machines. =
Luckily=20
the passwords on the old Firewall used unix crypt so I used crypt in =
pure tcl=20
from the wiki so the passwords could remain the same. One night I =
extracted the=20
usernames, passwords and other info from the database on the old =
Firewall and=20
imported it to the Metakit database. I then made the changes to Squid so =
it=20
would authenticate against the TclHttpd Metakit database. Next day my =
Squid=20
users didn't even realise there had been a change!</P>
<P>OK, now I am feeling adventurous I want to move on to the next stage =
but as a=20
novice I am uncertain how to approach this.</P>
<P>1. I want to process the Squid access.log file and store information =
about=20
sites visited, bytes downloaded etc... by individual users in a Metakit =
database=20
for reporting and billing. At present the daily Squid access.log file is =
approx.=20
40MB around 1.2GB per month. My thoughts are to process the access.log =
file on a=20
daily basis and store individual user data in their own Metakit datafile =
(average 27KB daily). I think this maybe the best approach??</P>
<P>2. a) When I process the Squid access.log file should I first open =
1500=20
Metakit datafiles and then store the data in its respective user =
datafile as I=20
process the log (Is this a ridiculous suggestion?).</P>
<P>b) Or as I process each line of the log, open the respective user =
datafile,=20
write the information, then close the datafile.</P>
<P>c) Or as I process each line of the log, open the respective user =
datafile,=20
write the information, if the next log file line is for a different =
user, open=20
another user datafile and so on. Limit this to say a maximum of 10 =
Metakit user=20
datafiles open at any given time, closing the one that has been open the =
longest=20
without data being stored in it.</P>
<P>Any suggestions??</P>
<P>d) Process in memory and then update Metakit datafiles.</P>
<P>e) add the lines to a single metakit view, then iterate over it by =
user.</P>
<HR>
<P><A href=3D"http://wiki.tcl.tk/1620">jcw</A> - Juggling 1500 open =
files would be=20
awkward and may hit OS limits. One file per day would not be hard of =
course, but=20
it leaves the info scattered in the wrong way. One datafile would hit MK =
limits=20
(32-bit address space limits datafiles to be under 2 Gb, but practical =
limit is=20
considerably lower). Some thoughts:</P>
<UL>
<LI>think carefully about size and storage: there is a <I>lot</I> of=20
redundancy in logfiles, if you bring this down 5..10-fold, it'd all =
become a=20
lot faster and easy to maintain in a single monthly DB - this is the =
most=20
flexible / DB-like approach=20
<LI>single days can easily be converted to MK db - again, there should =
be=20
several ways to reduce (store dates and IP's as ints, create second =
view with=20
paths accessed and store index into it in the actual log view, perhaps =
store=20
some fields in separate file if not used often ...)=20
<LI>consider updating user db's on-demand: when accessed, go through =
any=20
not-yet processed daily db's and append new info, once=20
<LI>if single-db remains too large to update efficiently, consider =
using <A=20
href=3D"http://wiki.tcl.tk/9489">oomk</A>, which has a lot of =
relational/set=20
operations - what you can do is store in "blocked" format (which =
scales way=20
better), and what you can also do is store in a few datafiles, and =
join or=20
concat them during use (MK view operators work across datafiles)=20
<LI>other tricks to consider are: batching (go through parts of the=20
collection, open/close, then next part), and scanning (go through all =
data,=20
process users 0..99, then again for 100..199, etc)</LI></UL>
<P>IOW, 2a probably hits OS limits, 2b will be too slow (massive =
commits), 2c=20
could work (I'd try 100, not 10). If you want a Tcl-style approach, you =
could=20
consider the following:</P>
<UL>
<LI>append each log line to Tcl array, one item per user=20
<LI>once an entry is say 100 lines long, open the matching MK file,=20
convert/store/save, and clear array entry=20
<LI>repeat, then save all the remaining ones</LI></UL>
<P>In the worst case, this buffers 150,000 lines in memory as all data =
gets=20
processed.</P>
<P>This last approach probably leads to the best trade-offs in such =
"data=20
pivoting" tasks, assuming per-user r/o access is the most frequent =
activity.</P>
<P>2003-11-13 <A href=3D"http://wiki.tcl.tk/6132">Jeff Smith</A> Thanks =
for taking=20
the time to respond, your efforts are greatly appreciated! :)</P>
<HR>
<P><A href=3D"http://wiki.tcl.tk/3226">Category Internet</A> | <A=20
href=3D"http://wiki.tcl.tk/3114">Category Database</A></P></DIV></DIV>
<DIV id=3Dmenu_area>
<DIV id=3Dwiki_menu>
<UL id=3Dmenu>
<LI><A href=3D"http://wiki.tcl.tk/">Home</A>
<LI><A href=3D"http://wiki.tcl.tk/4">Recent changes</A>
<LI><A href=3D"http://wiki.tcl.tk/3">Help</A>
<LI><A href=3D"http://wiki.tcl.tk/_edit/3072">Edit</A>
<LI><A href=3D"http://wiki.tcl.tk/_history/3072">History</A>
<LI><A =
href=3D"http://wiki.tcl.tk/_ref/3072">References</A></LI></UL></DIV>
<FORM id=3Dsearchform action=3D/_search method=3Dget><INPUT =
type=3Dhidden=20
name=3D_charset_> <INPUT id=3Dsearchtxt onblur=3DsetSearch(); =
onfocus=3DclearSearch();=20
value=3D"Search in titles" name=3DS> </FORM>
<FORM id=3Dgsearchform action=3D/_gsearch method=3Dget><INPUT =
type=3Dhidden=20
name=3D_charset_> <INPUT id=3Dgoogletxt onblur=3DsetGoogle(); =
onfocus=3DclearGoogle();=20
value=3D"Search in pages" name=3DS> </FORM>
<DIV class=3Dnavigation>
<DIV id=3Dpage_toc></DIV></DIV>
<DIV class=3Dextra>
<DIV id=3Dwiki_toc>
<DIV class=3Dtoc1>Getting started=20
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/299">What is =
Tcl?</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/487">What is =
Tk?</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20786">Getting=20
Tcl/Tk</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20788">Getting=20
help</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20789">Learning=20
Tcl</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/20790">The =
Tcl Dev=20
Xchange</A></DIV>
<DIV class=3Dtoc3><A class=3Dtoc href=3D"http://wiki.tcl.tk/20791">About =
the=20
Wiki</A></DIV></DIV>
<DIV class=3Dtoc1>Community=20
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/590">Advocacy</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/828">Conferences</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/20792">Chat, =
news,=20
lists</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20794">History</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/17182">Humor</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/10681">People</A></DIV>
<DIV class=3Dtoc3><A class=3Dtoc href=3D"http://wiki.tcl.tk/20810">Tcl=20
websites</A></DIV></DIV>
<DIV class=3Dtoc1>Reference=20
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/1887">Companies</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20788">Getting=20
help</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20795">Online=20
books</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20796">Online=20
tutorials</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/3109">Manual =
pages</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/20797">Tcl=20
roadmap</A></DIV>
<DIV class=3Dtoc3><A class=3Dtoc=20
href=3D"http://wiki.tcl.tk/49">Acronyms</A></DIV></DIV>
<DIV class=3Dtoc1>Tcl software=20
<DIV class=3Dtoc2><A class=3Dtoc=20
href=3D"http://wiki.tcl.tk/20798">Applications</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc href=3D"http://wiki.tcl.tk/20799">The =
Tcl=20
core</A></DIV>
<DIV class=3Dtoc2><A class=3Dtoc =
href=3D"http://wiki.tcl.tk/20800">Development=20
tools</A></DIV>
?? 快捷鍵說明
復(fù)制代碼
Ctrl + C
搜索代碼
Ctrl + F
全屏模式
F11
切換主題
Ctrl + Shift + D
顯示快捷鍵
?
增大字號
Ctrl + =
減小字號
Ctrl + -