doc/build/loncapasqldatabase.html - view

File: [LON-CAPA] / doc / build / Attic / loncapasqldatabase.html
Revision 1.16: download - view: text, annotated - select for diffs
Thu Mar 15 13:43:00 2001 UTC (23 years, 2 months ago) by harris41
Branches: MAIN
CVS tags: stable_2002_spring, stable_2001_fall, HEAD

updating the correct commands to create the database structure
and insert a test case -Scott

<HTML> <HEAD> <TITLE>LON-CAPA SQL Database Documentation</TITLE> </HEAD> <BODY> <H1>LON-CAPA SQL Database Documentation</H1> <P> Scott Harrison </P> <P> Last updated: 02/15/2001 </P> <P> This file describes issues associated with LON-CAPA and a SQL database. </P> <H2>Latest HOWTO</H2> <P> <UL> <LI>Important notes <LI>Current status of documentation</LI> <LI>Current status of implementation</LI> <LI>Purpose within LON-CAPA</LI> <LI>Dependencies</LI> <LI>Installation</LI> <LI>Installation from source</LI> <LI>Configuration (automated)</LI> <LI>Manual configuration</LI> <LI>Testing</LI> <LI>Example sections of code relevant to LON-CAPA</LI> </UL> </P> <H2>Important notes</H2> <P> The current database is implemented assuming a non-adjustable architecture involving these data fields (specific to each version of a resource). <UL> <LI>title</LI> <LI>author</LI> <LI>subject</LI> <LI>notes</LI> <LI>abstract</LI> <LI>mime</LI> <LI>language</LI> <LI>creationdate</LI> <LI>lastrevisiondate</LI> <LI>owner</LI> <LI>copyright</LI> </UL> </P> <P> Security occurs as a function of the user 'www', and the permissions on the files in the /usr/local/mysql directory. (These files and directories should be 700, 600, 500, 400, etc; not allow access to anyone but user 'www'.) </P> <P> These commands create the loncapameta database. <PRE> mysql> CREATE DATABASE IF NOT EXISTS loncapa; mysql> USE loncapa; mysql> CREATE TABLE IF NOT EXISTS metadata (title TEXT, author TEXT, subject TEXT, url TEXT, keywords TEXT, version TEXT, notes TEXT, abstract TEXT, mime TEXT, language TEXT, creationdate DATETIME, lastrevisiondate DATETIME, owner TEXT, copyright TEXT, FULLTEXT idx_title (title), FULLTEXT idx_author (author), FULLTEXT idx_subject (subject), FULLTEXT idx_url (url), FULLTEXT idx_keywords (keywords), FULLTEXT idx_version (version), FULLTEXT idx_notes (notes), FULLTEXT idx_abstract (abstract), FULLTEXT idx_mime (mime), FULLTEXT idx_language (language), FULLTEXT idx_owner (owner), FULLTEXT idx_copyright (copyright)) TYPE=MYISAM; mysql> INSERT INTO metadata VALUES ('The Structure of Scientific Revolutions','Thomas S. Kuhn','scientific philosophy','/res/msu/shh1/poobah2.html','aphorisms, theories, paradigm, revolution','current','still developing','This famous book stands out in contrast to the more rigid theories of Popper.','html','seniso','1999-03-03 12:34:56','1999-03-03 3:12:00','shh1@msu.edu','default'); mysql> SELECT * FROM metadata WHERE title REGEXP "1"; </PRE> </P> <P> Current time values for things. <PRE> fenchurch.lite.msu.edu Mem: 46812K av, 45632K used, 1180K free, 14756K shrd, 4292K buff Swap: 1148608K av, 11260K used, 1137348K free 13244K cached Red Hat Linux release 6.2 (Zoot) Kernel 2.2.16-3 on an i586 </PRE> <BR>Q: How big are data records in test database? A: on average, 1000 bytes each, medline records from PubMed. <BR>Q: How big is the biggest field? A: on average, 838 bytes each <BR>Q: How much time to insert 5284 medline records into database? A: 600 seconds <BR>Q: What about when using "speed-technique" on page 130? A: 689 seconds (weird, eh?) <BR>Q: What about REGEXP searching? A: about 1-2 seconds for small fields; 10 to 20 seconds for REGEXP search on "abstract" field <BR>Q: What about FULLTEXT indexing? A: about 6 seconds for abstract field. </P> <P> An important quote from the manual: <BLOCKQUOTE> In MySQL Version 3.23.23 or later, you can also create special FULLTEXT indexes. They are used for full-text search. Only the MyISAM table type supports FULLTEXT indexes. They can be created only from VARCHAR and TEXT columns. Indexing always happens over the entire column and partial indexing is not supported. See section 25.2 MySQL Full-text Search for details. </BLOCKQUOTE> </P> <P> I plan on using a MyISAM table type with 11 metadata fields of column type=TEXT. </P> <P> It might be worthwhile to look at /usr/local/mysql/manual.html. It is quite in depth. </P> <H2>Current status of documentation</H2> <P> I am going to begin documentation by inserting what notes I have into this file. I will be subsequently rearranging them and editing them based on the tests that I conduct. I am trying to make sure that documentation, installation, and run-time issues are all consistent and correct. The current status of everything is that it works and has been minimally tested, but things need to be cleaned up and checked again! </P> <H2>Current status of implementation</H2> <P> Need to <UL> <LI>Installation: Fix binary file listings for user permissions and ownership. <LI>Installation: Make sure sql server starts, and if database does not exist, then create. (/etc/rc.d). <LI>Processes: Make sure loncron initiates lonsql on library machines. <LI>Read in metadata from right place periodically. <LI>Implement tested perl module handler. </UL> <P> Right now, a lot of "feasibility" work has been done. Recipes for manual installation and configuration have been gathered. Network connectivity of lond->lonsql->lond->lonc type tests have been performed. A binary installation has been compiled in an RPM (LON-CAPA-mysql, with perl components a part of LON-CAPA-systemperl). The most lacking test in terms of feasibility has been looking at benchmarks to analyze the load at which the SQL database can efficiently allow many users to make simultaneous requests of the metadata database. </P> <P> Documentation has been pieced together over time. But, as mentioned in the previous section, it needs an overhaul. </P> <P> The binary installation has some quirks associated with it. Some of the user permissions are wrong, although this is benign. Also, other options of binary installation (such as using binary RPMs put together by others) were dismissed given the difficulty of getting differing combinations of these external RPMs to work together. </P> <P> Most configuration questions have been initially worked out to the point of getting this SQL software component working, however there may be more optimal approaches than currently exist. </P> <H2>Purpose within LON-CAPA</H2> <P> LON-CAPA is meant to distribute A LOT of educational content to A LOT of people. It is ineffective to directly rely on contents within the ext2 filesystem to be speedily scanned for on-the-fly searches of content descriptions. (Simply put, it takes a cumbersome amount of time to open, read, analyze, and close thousands of files.) </P> <P> The solution is to hash-index various data fields that are descriptive of the educational resources on a LON-CAPA server machine. Descriptive data fields are referred to as "metadata". The question then arises as to how this metadata is handled in terms of the rest of the LON-CAPA network without burdening client and daemon processes. I now answer this question in the format of Problem and Solution below. </P> <P> <PRE> PROBLEM SITUATION: If Server A wants data from Server B, Server A uses a lonc process to send a database command to a Server B lond process. lonc= loncapa client process A-lonc= a lonc process on Server A lond= loncapa daemon process database command A-lonc --------TCP/IP----------------> B-lond The problem emerges that A-lonc and B-lond are kept waiting for the MySQL server to "do its stuff", or in other words, perform the conceivably sophisticated, data-intensive, time-sucking database transaction. By tying up a lonc and lond process, this significantly cripples the capabilities of LON-CAPA servers. While commercial databases have a variety of features that ATTEMPT to deal with this, freeware databases are still experimenting and exploring with different schemes with varying degrees of performance stability. THE SOLUTION: A separate daemon process was created that B-lond works with to handle database requests. This daemon process is called "lonsql". So, database command A-lonc ---------TCP/IP-----------------> B-lond =====> B-lonsql <---------------------------------/ | "ok, I'll get back to you..." | | / A-lond <------------------------------- B-lonc <====== "Guess what? I have the result!" Of course, depending on success or failure, the messages may vary, but the principle remains the same where a separate pool of children processes (lonsql's) handle the MySQL database manipulations. </PRE> </P> <H2>Dependencies</H2> <P> I believe (but am not 100% confident) that the following RPMs are necessary (in addition to the current ones in rpm_list.txt) to run MySQL. Basically I discovered these dependencies while trying to do external RPM based installs. I assume, and sometimes found, that these dependencies apply to tarball-based distributions too. (So to play it on the safe side, I am going to include these RPMs as part of the core, minimal RPM set.) <UL> <LI>egcs-1.1.2-30</LI> <LI>cpp-1.1.2-30</LI> <LI>glibc-devel-2.1.3-15</LI> <LI>zlib-devel-1.1.3-6</LI> </UL> </P> <H2>Installation</H2> <P> Installation of the LON-CAPA SQL database normally occurs by default when using the LON-CAPA installation CD (see http://install.lon-capa.org). It is installed as the LON-CAPA-mysql RPM. This RPM encodes for the MySQL engine. Related perl interfaces (Perl::DBI, Perl::Msql-Mysql) are encoded in the LON-CAPA-systemperl RPM. </P> <P> The three components of a MySQL installation for the LON-CAPA system are further described immediately below. <TABLE BORDER="0"> <TR><TD COLSPAN="2"><STRONG>Perl::DBI module</STRONG>- the API "front-end"...</TD></TR> <TR><TD WIDTH="10%"></TD><TD>database interface module for organizing generic database commands which are independent of specific database implementation (such as MySQL, mSQL, Postgres, etc). </TD></TR> <TR><TD COLSPAN="2"><STRONG>Perl::MySQL module</STRONG>- the API "mid-section"...</TD></TR> <TR><TD WIDTH="10%"></TD><TD>the module to directly interface with the actual MySQL database engine</TD></TR> <TR><TD COLSPAN="2"><STRONG>MySQL database engine</STRONG>- the "back-end"...</TD></TR> <TR><TD WIDTH="10%"></TD><TD>the binary installation (compiled either from source or pre-compiled file listings) which provides the actual MySQL functionality on the system</TD></TR> </TABLE> </P> <H2>Installation from source</H2> <P> Note: the mysql site recommends that Linux users install by using the MySQL RPMs (MySQL-client, MySQL, MySQL-shared, etc). While these RPMs work, I was unsuccessful at integrating this RPM-installed database with perl modules from www.cpan.org. Hence, I <STRONG>strongly</STRONG> recommend that, when installing from "source", MySQL and the perl components be in fact installed from their tarballs (.tar.gz, .tgz). (Perl components, when installed from RPMs, also wound up in incorrect locations on the disk.) Do not coordinate a source install with externally made RPMs! It is, of course, okay to use LON-CAPA RPMs such as LON-CAPA-systemperl and LON-CAPA-mysql since we, in fact, made these RPMs correctly :). <UL> <LI>http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2215.tar.gz <BR>This tarball Released 20th August 2000 <LI>http://www.mysql.com/Downloads/MySQL-3.23/mysql-3.23.33-pc-linux-gnu-i686.tar.gz <BR>This tarball Last changed 2000-11-11 <BR>This is actually a binary tarball (as opposed to source code that is subsequently compiled). <LI>http://www.cpan.org/authors/id/TIMB/DBI-1.14.tar.gz <BR>This tarball Released 14th June 2000 </UL> </P> <P>So, here is exactly how I installed MySQL-3.23. (Note that all files wind up in /usr/local/mysql-3.23.33-pc-linux-gnu-i686 except for a link from /usr/local/mysql to /usr/local/mysql-3.23.33-pc-linux-gnu-i686 and some files involved in system process handling (/etc/rc.d/*/*mysql). <PRE> (As user=root) cd /usr/local/; tar xzvf mysql-3.23.33-pc-linux-gnu-i686.tar.gz ln -s /usr/local/mysql-3.23.33-pc-linux-gnu-i686 mysql cd /usr/local/mysql chown -R www /usr/local/mysql/. chgrp -R users /usr/local/mysql/. chmod -R g-w,g-r,g-x /usr/local/mysql/. (probably also want chmod -R a-w,a-r,a-x /usr/local/mysql/.) Alter <TT>safe_mysqld</TT> and <TT>support-files/mysql.server</TT> to use 'localhosts' instead of `bin/hostname`. Also, to use user 'www' instead of 'mysql'. (These changes could be done with /etc/my.cnf, but I think this approach makes sure the database NEVER gets screwed up due to somebody forgetting to install /etc/my.cnf). Change this line in mysql.server from pid_file=$datadir/`@HOSTNAME@`.pid to pid_file=$datadir/localhost.pid Change this line in safe_mysqld from user=root to user=www Change this line in safe_mysqld from pid_file=$DATADIR/`/bin/hostname`.pid to pid_file=$DATADIR/localhost.pid Change this line in safe_mysqld from test -z "$err_log" && err_log=$DATADIR/`/bin/hostname`.err to test -z "$err_log" && err_log=$DATADIR/localhost.err cp -p support-files/mysql.server /etc/rc.d/init.d/mysql chmod 755 /etc/rc.d/init.d/mysql /sbin/chkconfig --add mysql /etc/rc.d/init.d/mysql start </PRE> </P> <P>This is how I installed the Msql-Mysql-modules perl modules. <PRE> [root@fenchurch Msql-Mysql-modules-1.2215]# perl Makefile.PL Which drivers do you want to install? 1) MySQL only 2) mSQL only (either of mSQL 1 or mSQL 2) 3) MySQL and mSQL (either of mSQL 1 or mSQL 2) 4) mSQL 1 and mSQL 2 5) MySQL, mSQL 1 and mSQL 2 Enter the appropriate number: [3] 1 Do you want to install the MysqlPerl emulation? You might keep your old Mysql module (to be distinguished from DBD::mysql!) if you are concerned about compatibility to existing applications! [y] n Where is your MySQL installed? Please tell me the directory that contains the subdir 'include'. [/usr/local/mysql] Which database should I use for testing the MySQL drivers? [test] On which host is database test running (hostname, ip address or host:port) [localhost] User name for connecting to database test? [undef] Password for connecting to database test? [undef] [root@fenchurch Msql-Mysql-modules-1.2215]# make [root@fenchurch Msql-Mysql-modules-1.2215]# make test make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql' make[1]: Leaving directory `/home/user/Msql-Mysql-modules-1.2215/mysql' make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql' PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/00base............ok t/10dsnlist.........ok t/20createdrop......ok t/30insertfetch.....ok t/40bindparam.......ok t/40blobs...........ok t/40listfields......ok t/40nulls...........ok t/40numrows.........ok t/50chopblanks......ok t/50commit..........ok t/60leaks...........skipping test on this platform t/ak-dbd............ok t/akmisc............ok t/dbdadmin..........ok t/mysql.............ok t/mysql2............ok All tests successful, 1 test skipped. Files=17, Tests=732, 40 wallclock secs (15.38 cusr + 1.30 csys = 16.68 CPU) [root@fenchurch Msql-Mysql-modules-1.2215]# make install These files are installed. /usr/bin/dbimon /usr/lib/perl5/man/man3/Bundle::DBD::mysql.3 /usr/lib/perl5/man/man3/DBD::mysql.3 /usr/lib/perl5/man/man3/Mysql.3 /usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBD/mysql.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/mysql.pm /usr/lib/perl5/site_perl/5.005/i386-linux/Mysql.pm /usr/lib/perl5/site_perl/5.005/i386-linux/Mysql/Statement.pm /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.bs /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.so /usr/man/man1/dbimon.1 /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist </PRE> </P> <P> This is how I installed the DBI perl modules. <PRE>$dbh [root@fenchurch DBI-1.14]# perl Makefile.PL *** Note: The optional PlRPC-modules (RPC::PlServer etc) are not installed. If you want to use the DBD::Proxy driver and DBI::ProxyServer modules, then you'll need to install the RPC::PlServer, RPC::PlClient, Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you. You can install them any time after installing the DBI. You do *not* need these modules for typical DBI usage. Optional modules are available from any CPAN mirror, in particular http://www.perl.com/CPAN/modules/by-module http://www.perl.org/CPAN/modules/by-module ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module Checking if your kit is complete... Looks good Writing Makefile for DBI Remember to actually *read* the README file! Use 'make' to build the software (dmake or nmake on Windows). Then 'make test' to execute self tests. Then 'make install' to install the DBI and then delete this working directory before unpacking and building any DBD::* drivers. [root@fenchurch DBI-1.14]# make [root@fenchurch DBI-1.14]# make test PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/basics............ok t/dbidrv............ok t/examp.............ok t/meta..............ok t/proxy.............skipping test on this platform t/shell.............ok t/subclass..........ok All tests successful, 1 test skipped. Files=7, Tests=179, 7 wallclock secs ( 6.46 cusr + 0.49 csys = 6.95 CPU) PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 test.pl test.pl DBI test application $Revision: 1.16 $ Using /home/user/DBI-1.14/blib Switch: DBI 1.14 by Tim Bunce, 1.14 Available Drivers: ADO, ExampleP, Multiplex, Proxy, mysql dbi:ExampleP:: testing 5 sets of 20 connections: Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Disconnecting... Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Disconnecting... Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Disconnecting... Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Disconnecting... Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Disconnecting... Made 100 connections in 0 wallclock secs ( 0.22 usr + 0.03 sys = 0.25 CPU) Testing handle creation speed... 5000 NullP statement handles cycled in 6.6 cpu+sys seconds (762 per sec) test.pl done [root@fenchurch DBI-1.14]# make install These files are installed. /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist /usr/bin/dbiproxy /usr/bin/dbish /usr/lib/perl5/man/man3/Bundle::DBI.3 /usr/lib/perl5/man/man3/DBD::ADO.3 /usr/lib/perl5/man/man3/DBD::Multiplex.3 /usr/lib/perl5/man/man3/DBD::Proxy.3 /usr/lib/perl5/man/man3/DBI.3 /usr/lib/perl5/man/man3/DBI::DBD.3 /usr/lib/perl5/man/man3/DBI::FAQ.3 /usr/lib/perl5/man/man3/DBI::Format.3 /usr/lib/perl5/man/man3/DBI::ProxyServer.3 /usr/lib/perl5/man/man3/DBI::Shell.3 /usr/lib/perl5/man/man3/DBI::W32ODBC.3 /usr/lib/perl5/man/man3/Win32::DBIODBC.3 /usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBI.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ADO.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ExampleP.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Multiplex.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/NullP.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Proxy.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Sponge.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/DBD.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/FAQ.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Format.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/ProxyServer.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Shell.pm /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/W32ODBC.pm /usr/lib/perl5/site_perl/5.005/i386-linux/Win32/DBIODBC.pm /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.bs /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.so /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBIXS.h /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/Driver.xst /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbd_xsh.h /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbi_sql.h /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbipport.h /usr/man/man1/dbiproxy.1 /usr/man/man1/dbish.1 </PRE> </P> <FONT COLOR="green"> old notes in green <P> The following set of tarballs was found to work together properly on a LON-CAPA RedHat 6.2 system: <UL> <LI>DBI-1.13.tar.gz <LI>Msql-Mysql-modules-1.2209.tar.gz <LI>mysql-3.22.32.tar.gz </UL> </P> <P> Installation was simply a matter of following the instructions and typing the several "make" commands for each </P> </FONT> <H2>Configuration (automated)</H2> <P> Not yet developed. This will be part of an interface present on LON-CAPA systems that can be launched by entering the command <TT>/usr/sbin/loncapaconfig</TT>. </P> <H2>Manual configuration</H2> <P> This is not complete. </P> <P> <STRONG>Starting the mysql daemon</STRONG>: Login on the Linux system as user 'www'. Enter the command <TT>/usr/local/bin/safe_mysqld &</TT> </P> <P> <STRONG>Set a password for 'root'</STRONG>: <TT>/usr/local/bin/mysqladmin -u root password 'new-password'</TT> </P> <P> <STRONG>Adding a user</STRONG>: Start the mysql daemon. Login to the mysql system as root (<TT>mysql -u root -p mysql</TT>) and enter the right password (for instance 'newmysql'). Add the user www <PRE> INSERT INTO user (Host, User, Password) VALUES ('localhost','www',password('newmysql')); </PRE> </P> <P> <STRONG>Granting privileges to user 'www'</STRONG>: <PRE> GRANT ALL PRIVILEGES ON *.* TO www@localhost; FLUSH PRIVILEGES; </PRE> </P> <P> <STRONG>Set the SQL server to start upon system startup</STRONG>: Copy support-files/mysql.server to the right place on the system (/etc/rc.d/...). </P> <P> <STRONG>The Perl API</STRONG> <PRE> $dbh = DBI->connect( "DBI:mysql:loncapa", "www", "SOMEPASSWORD", { RaiseError =>0,PrintError=>0}); There is an obvious need to CONNECT to the database, and in order to do this, there must be: a RUNNING mysql daemon; a DATABASE named "loncapa"; a USER named "www"; and an ABILITY for LON-CAPA on one machine to access SQL database on another machine; So, here are some notes on implementing these configurations. ** RUNNING mysql daemon (safe_mysqld method) The recommended way to run the MySQL daemon is as a non-root user (probably www)... so, 1) login as user www on the linux machine 2) start the mysql daemon as /usr/local/bin/safe_mysqld & safe_mysqld only works if the local installation of MySQL is set to the right directory permissions which I found to be: chown www:users /usr/local/var/mysql chown www:users /usr/local/lib/mysql chown -R www:users /usr/local/mysql chown www:users /usr/local/include/mysql chown www:users /usr/local/var ** DATABASE named "loncapa" As user www, run this command mysql -u root -p mysql enter the password as SOMEPASSWORD This allows you to manually enter MySQL commands. The MySQL command to generate the loncapa DATABASE is: CREATE DATABASE 'loncapa'; ** USER named "www" As user www, run this command mysql -u root -p mysql enter the password as SOMEPASSWORD To add the user www to the MySQL server, and grant all privileges on *.* to www@localhost identified by 'SOMEPASSWORD' with grant option; INSERT INTO user (Host, User, Password) VALUES ('localhost','www',password('SOMEPASSWORD')); GRANT ALL PRIVILEGES ON *.* TO www@localhost; FLUSH PRIVILEGES; ** ABILITY for LON-CAPA machines to communicate with SQL databases on other LON-CAPA machines An up-to-date lond and lonsql. </PRE> </P> <H2>Testing</H2> <P> To test the backend MySQL database, a number of commands should be run after installation. <UL> <LI><TT>cd /usr/local/mysql/sql-bench; ./run-all-tests --small-test</TT></LI> <BR>without the --small-test flag, this test can take more than 10 hours! <LI><TT>cd /usr/local/mysql; bin/mysqladmin version</TT></LI> <LI><TT>cd /usr/local/mysql; bin/mysqladmin variables</TT></LI> <LI><TT>cd /usr/local/mysql; bin/mysqlshow</TT></LI> <LI><TT>cd /usr/local/mysql; bin/mysqlshow mysql</TT></LI> <LI><TT>cd /usr/local/mysql; bin/mysql -e "select host,db,user from db" mysql</TT></LI> <LI><TT>cd /usr/local/mysql/mysql-test; ./test-run-all</TT></LI> </UL> <P> These are sections of perl code which helps test the LON-CAPA network. <PRE> <STRONG>** TEST the database connection with my current tester.pl code which mimics what command will eventually be sent through lonc.</STRONG> $reply=reply( "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID); </PRE> </P> <H2>Example sections of code relevant to LON-CAPA</H2> <P> Here are excerpts of code which implement the above handling: </P> <P> <PRE> <STRONG>**LONSQL A subroutine from "lonsql" which establishes a child process for handling database interactions.</STRONG> sub make_new_child { my $pid; my $sigset; # block signal for fork $sigset = POSIX::SigSet->new(SIGINT); sigprocmask(SIG_BLOCK, $sigset) or die "Can't block SIGINT for fork: $!\n"; die "fork: $!" unless defined ($pid = fork); if ($pid) { # Parent records the child's birth and returns. sigprocmask(SIG_UNBLOCK, $sigset) or die "Can't unblock SIGINT for fork: $!\n"; $children{$pid} = 1; $children++; return; } else { # Child can *not* return from this subroutine. $SIG{INT} = 'DEFAULT'; # make SIGINT kill us as it did before # unblock signals sigprocmask(SIG_UNBLOCK, $sigset) or die "Can't unblock SIGINT for fork: $!\n"; #open database handle # making dbh global to avoid garbage collector unless ( $dbh = DBI->connect("DBI:mysql:loncapa","www","SOMEPASSWORD",{ RaiseError =>0,PrintError=>0}) ) { my $st=120+int(rand(240)); &logthis("<font color=blue>WARNING: Couldn't connect to database ($st secs): $@</font>"); print "database handle error\n"; sleep($st); exit; }; # make sure that a database disconnection occurs with ending kill signals $SIG{TERM}=$SIG{INT}=$SIG{QUIT}=$SIG{__DIE__}=\&DISCONNECT; # handle connections until we've reached $MAX_CLIENTS_PER_CHILD for ($i=0; $i < $MAX_CLIENTS_PER_CHILD; $i++) { $client = $server->accept() or last; # do something with the connection $run = $run+1; my $userinput = <$client>; chomp($userinput); my ($conserver,$querytmp)=split(/&/,$userinput); my $query=unescape($querytmp); #send query id which is pid_unixdatetime_runningcounter $queryid = $thisserver; $queryid .="_".($$)."_"; $queryid .= time."_"; $queryid .= $run; print $client "$queryid\n"; #prepare and execute the query my $sth = $dbh->prepare($query); my $result; unless ($sth->execute()) { &logthis("<font color=blue>WARNING: Could not retrieve from database: $@</font>"); $result=""; } else { my $r1=$sth->fetchall_arrayref; my @r2; map {my $a=$_; my @b=map {escape($_)} @$a; push @r2,join(",", @b)} (@$r1); $result=join("&",@r2) . "\n"; } &reply("queryreply:$queryid:$result",$conserver); } # tidy up gracefully and finish #close the database handle $dbh->disconnect or &logthis("<font color=blue>WARNING: Couldn't disconnect from database $DBI::errstr ($st secs): $@</font>"); # this exit is VERY important, otherwise the child will become # a producer of more and more children, forking yourself into # process death. exit; } } </P> <P> <STRONG>** LOND enabling of MySQL requests</STRONG> <BR /> This code is part of every lond child process in the way that it parses command request syntax sent to it from lonc processes. Based on the diagram above, querysend corresponds to B-lonc sending the result of the query. queryreply corresponds to B-lond indicating that it has received the request and will start the database transaction (it returns "ok" to A-lonc ($client)). <PRE> # ------------------------------------------------------------------- querysend } elsif ($userinput =~ /^querysend/) { my ($cmd,$query)=split(/:/,$userinput); $query=~s/\n*$//g; print $client sqlreply("$hostid{$clientip}\&$query")."\n"; # ------------------------------------------------------------------ queryreply } elsif ($userinput =~ /^queryreply/) { my ($cmd,$id,$reply)=split(/:/,$userinput); my $store; my $execdir=$perlvar{'lonDaemons'}; if ($store=IO::File->new(">$execdir/tmp/$id")) { print $store $reply; close $store; print $client "ok\n"; } else { print $client "error:$!\n"; } </PRE> </P> </BODY> </HTML>