loncom/interface/loncoursedata.pm - diff

Return to loncoursedata.pm CVS log

Up to [LON-CAPA] / loncom / interface

Diff for /loncom/interface/loncoursedata.pm between versions 1.155 and 1.201.2.4

-version 1.155, 2006/02/05 19:14:58
+version 1.201.2.4, 2021/03/04 23:55:05
  Line 25
  # http://www.lon-capa.org/
  #
  ###
  =pod
  =head1 NAME
- loncoursedata
+ Apache::loncoursedata
  =head1 SYNOPSIS
- Line 38  Set of functions that download and proce
+ Line 37  Set of functions that download and proce
  =head1 PACKAGES USED
-  Apache::Constants qw(:common :http)
+   Apache::lonnet
-  Apache::lonnet()
+   Apache::longroup
-  Apache::lonhtmlcommon
+   Time::HiRes
-  HTML::TokeParser
+   Apache::lonmysql
-  GDBM_File
+   LONCAPA
+   Digest::MD5
  =cut
  package Apache::loncoursedata;
  use strict;
- use Apache::Constants qw(:common :http);
  use Apache::lonnet;
- use Apache::lonhtmlcommon;
+ use Apache::longroup();
- use Time::HiRes;
+ use Time::HiRes();
- use Apache::lonmysql;
+ use Apache::lonmysql();
- use HTML::TokeParser;
+ use LONCAPA;
- use GDBM_File;
+ use Digest::MD5();
  =pod
- =head1 DOWNLOAD INFORMATION
+ =head2 make_into_hash
- This section contains all the functions that get data from other servers
+ Turn a colon separated string into a hash and return a reference
- and/or itself.
+ to it.  Numbering from 0 even elements are keys and odd elements
+ are values e.g. a:b:c:d creates a hash like
+   a => b, c =>d
  =cut
- sub LoadDiscussion {
-     my ($courseID)=@_;
-     my %Discuss=();
-     my %contrib=&Apache::lonnet::dump(
-                 $courseID,
-                 $env{'course.'.$courseID.'.domain'},
-                 $env{'course.'.$courseID.'.num'});
-     #my %contrib=&DownloadCourseInformation($name, $courseID, 0);
-     foreach my $temp(keys %contrib) {
- 	if ($temp=~/^version/) {
- 	    my $ver=$contrib{$temp};
- 	    my ($dummy,$prb)=split(':',$temp);
- 	    for (my $idx=1; $idx<=$ver; $idx++ ) {
- 		my $name=$contrib{"$idx:$prb:sendername"};
- 		$Discuss{"$name:$prb"}=$idx;
- 	    }
- 	}
-     }
-     return \%Discuss;
- }
- ################################################
- ################################################
- =pod
- =item &make_into_hash($values);
- Returns a reference to a hash as described by $values.  $values is
- assumed to be the result of
-     join(':',map {&Apache::lonnet::escape($_)} %orighash);
- This is a helper function for get_current_state.
- =cut
- ################################################
- ################################################
  sub make_into_hash {
      my $values = shift;
-     my %tmp = map { &Apache::lonnet::unescape($_); }
+     my %tmp = map { &unescape($_); } split(':',$values);
-                                            split(':',$values);
      return \%tmp;
  }
- ################################################
- ################################################
- =pod
- =head1 LOCAL DATA CACHING SUBROUTINES
- The local caching is done using MySQL.  There is no fall-back implementation
- if MySQL is not running.
- The programmers interface is to call &get_current_state() or some other
- primary interface subroutine (described below).  The internals of this
- storage system are documented here.
- There are six tables used to store student performance data (the results of
- a dumpcurrent).  Each of these tables is created in MySQL with a name of
- $courseid_*****, where ***** is 'symb', 'part', or whatever is appropriate
- for the table.  The tables and their purposes are described below.
- Some notes before we get started.
- Each table must have a PRIMARY KEY, which is a column or set of columns which
- will serve to uniquely identify a row of data.  NULL is not allowed!
- INDEXes work best on integer data.
- JOIN is used to combine data from many tables into one output.
- lonmysql.pm is used for some of the interface, specifically the table creation
- calls.  The inserts are done in bulk by directly calling the database handler.
- The SELECT ... JOIN statement used to retrieve the data does not have an
- interface in lonmysql.pm and I shudder at the thought of writing one.
- =head3 Table Descriptions
- =over 4
- =item Tables used to store meta information
- The following tables hold data required to keep track of the current status
- of a students data in the tables or to look up the students data in the tables.
- =over 4
- =item $symb_table
- The symb_table has two columns.  The first is a 'symb_id' and the second
- is the text name for the 'symb' (limited to 64k).  The 'symb_id' is generated
- automatically by MySQL so inserts should be done on this table with an
- empty first element.  This table has its PRIMARY KEY on the 'symb_id'.
- =item $part_table
- The part_table has two columns.  The first is a 'part_id' and the second
- is the text name for the 'part' (limited to 100 characters).  The 'part_id' is
- generated automatically by MySQL so inserts should be done on this table with
- an empty first element.  This table has its PRIMARY KEY on the 'part' (100
- characters) and a KEY on 'part_id'.
- =item $student_table
- The student_table has 7 columns.  The first is a 'student_id' assigned by
- MySQL.  The second is 'student' which is username:domain.  The third through
- fifth are 'section', 'status' (enrollment status), and 'classification'
- (to be used in the future).  The sixth and seventh ('updatetime' and
- 'fullupdatetime') contain the time of last update and full update of student
- data.  This table has its PRIMARY KEY on the 'student_id' column and is indexed
- on 'student', 'section', and 'status'.
- =back
- =item Tables used to store current status data
- The following tables store data only about the students current status on
- a problem, meaning only the data related to the last attempt on a problem.
- =over 4
- =item $performance_table
- The performance_table has 9 columns.  The first three are 'symb_id',
- 'student_id', and 'part_id'.  These comprise the PRIMARY KEY for this table
- and are directly related to the $symb_table, $student_table, and $part_table
- described above.  MySQL does better indexing on numeric items than text,
- so we use these three "index tables".  The remaining columns are
- 'solved', 'tries', 'awarded', 'award', 'awarddetail', and 'timestamp'.
- These are either the MySQL type TINYTEXT or various integers ('tries' and
- 'timestamp').  This table has KEYs of 'student_id' and 'symb_id'.
- For use of this table, see the functions described below.
- =item $parameters_table
- The parameters_table holds the data that does not fit neatly into the
- performance_table.  The parameters table has four columns: 'symb_id',
- 'student_id', 'parameter', and 'value'.  'symb_id', 'student_id', and
- 'parameter' comprise the PRIMARY KEY for this table.  'parameter' is
- limited to 255 characters.  'value' is limited to 64k characters.
- =back
- =item Tables used for storing historic data
- The following tables are used to store almost all of the transactions a student
- has made on a homework problem.  See loncapa/docs/homework/datastorage for
- specific information about each of the parameters stored.
- =over 4
- =item $fulldump_response_table
- The response table holds data (documented in loncapa/docs/homework/datastorage)
- associated with a particular response id which is stored when a student
- attempts a problem.  The following are the columns of the table, in order:
- 'symb_id','part_id','response_id','student_id','transaction','tries',
- 'awarddetail', 'response_specific' (data particular to the response
- type), 'response_specific_value', and 'submission (the text of the students
- submission).  The primary key is based on the first five columns listed above.
- =item $fulldump_part_table
- The part table holds data (documented in loncapa/docs/homework/datastorage)
- associated with a particular part id which is stored when a student attempts
- a problem.  The following are the columns of the table, in order:
- 'symb_id','part_id','student_id','transaction','tries','award','awarded',
- and 'previous'.  The primary key is based on the first five columns listed
- above.
- =item $fulldump_timestamp_table
- The timestamp table holds the timestamps of the transactions which are
- stored in $fulldump_response_table and $fulldump_part_table.  This data is
- about both the response and part data.  Columns: 'symb_id','student_id',
- 'transaction', and 'timestamp'.
- The primary key is based on the first 3 columns.
- =item $weight_table
- The weight table holds the weight for the problems used in the class.
- Whereas the weight of a problem can vary by section and student the data
- here is applied to the class as a whole.
- Columns: 'symb_id','part_id','response_id','weight'.
- =back
- =back
- =head3 Important Subroutines
- Here is a brief overview of the subroutines which are likely to be of
- interest:
- =over 4
- =item &get_current_state(): programmers interface.
- =item &init_dbs(): table creation
- =item &update_student_data(): data storage calls
- =item &get_student_data_from_performance_cache(): data retrieval
- =back
- =head3 Main Documentation
- =over 4
- =cut
- ################################################
- ################################################
- ################################################
- ################################################
  { # Begin scope of table identifiers
  my $current_course ='';
  my $symb_table;
  my $part_table;
  my $student_table;
+ my $groupnames_table;
+ my $students_groups_table;
  my $performance_table;
  my $parameters_table;
  my $fulldump_response_table;
- Line 303  my $fulldump_timestamp_table;
+ Line 90  my $fulldump_timestamp_table;
  my $weight_table;
  my @Tables;
- ################################################
- ################################################
- =pod
- =item &init_dbs()
- Input: course id
- Output: 0 on success, positive integer on error
- This routine issues the calls to lonmysql to create the tables used to
- store student data.
- =cut
- ################################################
- ################################################
  sub init_dbs {
      my ($courseid,$drop) = @_;
      &setup_table_names($courseid);
- Line 377  sub init_dbs {
+ Line 149  sub init_dbs {
                      { name => 'section',
                        type => 'VARCHAR(100) BINARY',
                        restrictions => 'NOT NULL'},
-                     { name => 'status',
+                     { name => 'start',
-                       type => 'VARCHAR(15) BINARY',
+                       type => 'INT',
+                       restrictions => 'NOT NULL'},
+                     { name => 'end',
+                       type => 'INT',
                        restrictions => 'NOT NULL'},
                      { name => 'classification',
                        type => 'VARCHAR(100) BINARY', },
- Line 390  sub init_dbs {
+ Line 165  sub init_dbs {
          'PRIMARY KEY' => ['student_id'],
          'KEY' => [{ columns => ['student (100)',
                                  'section (100)',
-                                 'status (15)',]},],
+                                 'start',
+ 				'end']},],
+     };
+     #
+     my $groupnames_table_def = {
+         id => $groupnames_table,
+         permanent => 'no',
+         columns => [{ name => 'group_id',
+                       type => 'MEDIUMINT UNSIGNED',
+                       restrictions => 'NOT NULL',
+                       auto_inc => 'yes', },
+                     { name => 'groupname',
+                       type => 'VARCHAR(100) BINARY',
+                       restrictions => 'NOT NULL UNIQUE'},
+                    ],
+         'PRIMARY KEY' => ['group_id'],
+         'KEY' => [{ columns => ['groupname (100)',]},],
+     };
+     #
+     my $students_groups_table_def = {
+         id => $students_groups_table,
+         permanent => 'no',
+         columns => [{ name => 'student_id',
+                       type => 'MEDIUMINT UNSIGNED',
+                       restrictions => 'NOT NULL', },
+                     { name => 'group_id',
+                       type => 'MEDIUMINT UNSIGNED',
+                       restrictions => 'NOT NULL', },
+                    ],
+         'PRIMARY KEY' => ['student_id','group_id'],
+         'KEY' => [{ columns => ['student_id'] },
+                   { columns => ['group_id'] },],
      };
      #
      my $performance_table_def = {
- Line 491  sub init_dbs {
+ Line 297  sub init_dbs {
                        type => 'TINYTEXT' },
                      { name => 'response_specific_value',
                        type => 'TINYTEXT' },
+                     { name => 'response_specific_2',
+                       type => 'TINYTEXT' },
+                     { name => 'response_specific_value_2',
+                       type => 'TINYTEXT' },
                      { name => 'submission',
                        type => 'TEXT'},
                      ],
- Line 622  sub init_dbs {
+ Line 432  sub init_dbs {
                                   &Apache::lonmysql::get_error());
          return 10;
      }
+     $tableid = &Apache::lonmysql::create_table($groupnames_table_def);
+     if (! defined($tableid)) {
+         &Apache::lonnet::logthis("error creating groupnames_table: ".
+                                  &Apache::lonmysql::get_error());
+         return 11;
+     }
+     $tableid = &Apache::lonmysql::create_table($students_groups_table_def);
+     if (! defined($tableid)) {
+         &Apache::lonnet::logthis("error creating student_groups_table: ".
+                                  &Apache::lonmysql::get_error());
+         return 12;
+     }
      return 0;
  }
- ################################################
- ################################################
  =pod
- =item &delete_caches()
+ =head2 delete_caches
- This routine drops all the tables associated with a course from the
+ Drops all of the tables in the local mysql cache associated with the
- MySQL database.
+ specified course id.
- Input: course id (optional, determined by environment if omitted)
+ TODO:  The drops should be pushed into lonmysql to further isolate
+ mysql code from other modules.
- Returns: nothing
  =cut
- ################################################
- ################################################
  sub delete_caches {
      my $courseid = shift;
      $courseid = $env{'request.course.id'} if (! defined($courseid));
- Line 660  sub delete_caches {
+ Line 475  sub delete_caches {
      return;
  }
- ################################################
- ################################################
- =pod
- =item &get_part_id()
- Get the MySQL id of a problem part string.
- Input: $part
- Output: undef on error, integer $part_id on success.
- =item &get_part()
- Get the string describing a part from the MySQL id of the problem part.
- Input: $part_id
- Output: undef on error, $part string on success.
- =cut
- ################################################
- ################################################
  my $have_read_part_table = 0;
  my %ids_by_part;
- Line 726  sub get_part {
+ Line 516  sub get_part {
      return undef; # error
  }
- ################################################
- ################################################
- =pod
- =item &get_symb_id()
- Get the MySQL id of a symb.
- Input: $symb
- Output: undef on error, integer $symb_id on success.
- =item &get_symb()
- Get the symb associated with a MySQL symb_id.
- Input: $symb_id
- Output: undef on error, $symb on success.
- =cut
- ################################################
- ################################################
  my $have_read_symb_table = 0;
  my %ids_by_symb;
- Line 791  sub get_symb {
+ Line 556  sub get_symb {
      return undef; # error
  }
- ################################################
- ################################################
- =pod
- =item &get_student_id()
- Get the MySQL id of a student.
- Input: $sname, $dom
- Output: undef on error, integer $student_id on success.
- =item &get_student()
- Get student username:domain associated with the MySQL student_id.
- Input: $student_id
- Output: undef on error, string $student (username:domain) on success.
- =cut
- ################################################
- ################################################
  my $have_read_student_table = 0;
  my %ids_by_student;
  my %students_by_id;
- Line 868  sub populate_student_table {
+ Line 607  sub populate_student_table {
      &init_dbs($courseid,0);
      my $dbh = &Apache::lonmysql::get_dbh();
      my $request = 'INSERT IGNORE INTO '.$student_table.
-         "(student,section,status) VALUES ";
+         "(student,section,start,end) VALUES ";
      my $cdom = $env{'course.'.$courseid.'.domain'};
      my $cnum = $env{'course.'.$courseid.'.num'};
      my $classlist = &get_classlist($cdom,$cnum);
      my $student_count=0;
      while (my ($student,$data) = each %$classlist) {
-         my ($section,$status) = ($data->[&CL_SECTION()],
+         my ($section,$start,$end) = ($data->[&CL_SECTION()],
-                                  $data->[&CL_STATUS()]);
+ 				     $data->[&CL_START()],
+ 				     $data->[&CL_END()]);
          if ($section eq '' || $section =~ /^\s*$/) {
              $section = 'none';
          }
-         $request .= "('".$student."','".$section."','".$status."'),";
+ 	if (!defined($start)) { $start = 0; }
+ 	if (!defined($end))   { $end   = 0; }
+         $request .= "('".$student."','".$section."','".$start."','".$end."'),";
          $student_count++;
      }
      return if ($student_count == 0);
- Line 887  sub populate_student_table {
+ Line 629  sub populate_student_table {
      $dbh->do($request);
      if ($dbh->err()) {
          &Apache::lonnet::logthis("error ".$dbh->errstr().
-                                  " occured executing \n".
+                                  " occurred executing \n".
                                   $request);
      }
      return;
  }
- ################################################
+ my $have_read_groupnames_table = 0;
- ################################################
+ my %ids_by_groupname;
- =pod
+ sub get_group_id {
+     my ($groupname) = @_;
+     if (! $have_read_groupnames_table) {
+         my @Result = &Apache::lonmysql::get_rows($groupnames_table);
+         foreach (@Result) {
+             $ids_by_groupname{$_->[1]}=$_->[0];
+         }
+         $have_read_groupnames_table = 1;
+     }
+     if (! exists($ids_by_groupname{$groupname})) {
+         &populate_groupnames_table();
+         undef(%ids_by_groupname);
+         my @Result = &Apache::lonmysql::get_rows($groupnames_table);
+         foreach (@Result) {
+             $ids_by_groupname{$_->[1]}=$_->[0];
+         }
+     }
+     if (exists($ids_by_groupname{$groupname})) {
+         return $ids_by_groupname{$groupname};
+     }
+     return undef; # error
+ }
- =item &clear_internal_caches()
+ sub populate_groupnames_table {
+     my ($courseid) = @_;
+     if (! defined($courseid)) {
+         $courseid = $env{'request.course.id'};
+     }
+     &setup_table_names($courseid);
+     &init_dbs($courseid,0);
+     my $dbh = &Apache::lonmysql::get_dbh();
+     my $cdom = $env{'course.'.$courseid.'.domain'};
+     my $cnum = $env{'course.'.$courseid.'.num'};
+     my %curr_groups = &Apache::longroup::coursegroups($cdom,$cnum);
+     return if (!%curr_groups);
+     my $request = 'INSERT IGNORE INTO '.$groupnames_table.
+                   '(groupname) VALUES ';
+     foreach my $groupname (sort(keys(%curr_groups)),'none') {
+         $request .= "('".$groupname."'),";
+     }
+     chop($request);
+     $dbh->do($request);
+     if ($dbh->err()) {
+         &Apache::lonnet::logthis("error ".$dbh->errstr().
+                                  " occurred executing \n".
+                                  $request);
+     }
+     return;
+ }
- Causes the internal caches used in get_student_id, get_student,
+ my $have_read_studentsgroups_table = 0;
- get_symb_id, get_symb, get_part_id, and get_part to be undef'd.
+ my %groupids_by_studentid;
- Needs to be called before the first operation with the MySQL database
+ sub get_students_groupids {
- for a given Apache request.
+     my ($student_id) = @_;
+     if (! $have_read_studentsgroups_table) {
+         my @Result = &Apache::lonmysql::get_rows($students_groups_table);
+         foreach (@Result) {
+             push(@{$groupids_by_studentid{$_->[0]}},$_->[1]);
+         }
+         $have_read_studentsgroups_table = 1;
+     }
+     if (! exists($groupids_by_studentid{$student_id})) {
+         &populate_students_groups_table();
+         undef(%groupids_by_studentid);
+         my @Result = &Apache::lonmysql::get_rows($students_groups_table);
+         foreach (@Result) {
+             push(@{$groupids_by_studentid{$_->[0]}},$_->[1]);
+         }
+     }
+     if (exists($groupids_by_studentid{$student_id})) {
+         if (ref($groupids_by_studentid{$student_id}) eq 'ARRAY') {
+             return @{$groupids_by_studentid{$student_id}};
+         }
+     }
+     return undef; # error
+ }
- =cut
- ################################################
+ sub populate_students_groups_table {
- ################################################
+     my ($courseid) = @_;
+     if (! defined($courseid)) {
+         $courseid = $env{'request.course.id'};
+     }
+     #
+     &setup_table_names($courseid);
+     &init_dbs($courseid,0);
+     my $dbh = &Apache::lonmysql::get_dbh();
+     my $request = 'INSERT IGNORE INTO '.$students_groups_table.
+         "(student_id,group_id) VALUES ";
+     my $cdom = $env{'course.'.$courseid.'.domain'};
+     my $cnum = $env{'course.'.$courseid.'.num'};
+     my ($classlist,$keylist) = &get_classlist($cdom,$cnum);
+     my ($classgroups,$studentgroups) = &get_group_memberships($classlist,
+                                                               $keylist,
+                                                               $cdom,$cnum);
+     my $record_count = 0;
+     foreach my $student (sort(keys(%{$classgroups}))) {
+         my $student_id = &get_student_id(split(':',$student));
+         my @studentsgroups = &get_students_groups($student,'Active',$classgroups);
+         if (@studentsgroups < 1) {
+             @studentsgroups = ('none');
+         }
+         foreach my $groupname (@studentsgroups) {
+             my $group_id = &get_group_id($groupname);
+             $request .= "('".$student_id."','".$group_id."'),";
+             $record_count++;
+         }
+     }
+     return if ($record_count == 0);
+     chop($request);
+     $dbh->do($request);
+     if ($dbh->err()) {
+         &Apache::lonnet::logthis("error ".$dbh->errstr().
+                                  " occurred executing \n".
+                                  $request);
+     }
+     return;
+ }
  sub clear_internal_caches {
      $have_read_part_table = 0;
      undef(%ids_by_part);
- Line 920  sub clear_internal_caches {
+ Line 768  sub clear_internal_caches {
      $have_read_student_table = 0;
      undef(%ids_by_student);
      undef(%students_by_id);
+     $have_read_groupnames_table = 0;
+     undef(%ids_by_groupname);
  }
- ################################################
+ sub symb_is_for_task {
- ################################################
+     my ($symb) = @_;
+     return ($symb =~ /\.task$/);
- =pod
+ }
- =item &update_full_student_data($sname,$sdom,$courseid)
- Does a lonnet::dump on a student to populate the courses tables.
- Input: $sname, $sdom, $courseid
- Output: $returnstatus
- $returnstatus is a string describing any errors that occured.  'okay' is the
- default.
- This subroutine loads a students data using lonnet::dump and inserts
- it into the MySQL database.  The inserts are done on three tables,
- $fulldump_response_table, $fulldump_part_table, and $fulldump_timestamp_table.
- The INSERT calls are made directly by this subroutine, not through lonmysql
- because we do a 'bulk'insert which takes advantage of MySQLs non-SQL
- compliant INSERT command to insert multiple rows at a time.
- If anything has gone wrong during this process, $returnstatus is updated with
- a description of the error.
- Once the "fulldump" tables are updated, the tables used for chart and
- spreadsheet (which hold only the current state of the student on their
- homework, not historical data) are updated.  If all updates have occured
- successfully, $student_table is updated to reflect the time of the update.
- Notice we do not insert the data and immediately query it.  This means it
- is possible for there to be data returned this first time that is not
- available the second time.  CYA.
- =cut
- ################################################
- ################################################
  sub update_full_student_data {
      my ($sname,$sdom,$courseid) = @_;
      #
- Line 973  sub update_full_student_data {
+ Line 791  sub update_full_student_data {
      #
      # Download students data
      my $time_of_retrieval = time;
-     my @tmp = &Apache::lonnet::dump($courseid,$sdom,$sname);
+     my @tmp = &Apache::lonnet::dumpstore($courseid,$sdom,$sname);
      if (@tmp && $tmp[0] =~ /^error/) {
          $returnstatus = 'error retrieving full student data';
          return $returnstatus;
- Line 998  sub update_full_student_data {
+ Line 816  sub update_full_student_data {
      while (my ($key,$value) = each(%studentdata)) {
          next if ($key =~ /^(\d+):(resource$|subnum$|keys:)/);
          my ($transaction,$symb,$parameter) = split(':',$key);
+ 	$symb = &unescape($symb);
+ 	$parameter = &unescape($parameter);
          my $symb_id = &get_symb_id($symb);
          if ($parameter eq 'timestamp') {
              # We can deal with 'timestamp' right away
- Line 1013  sub update_full_student_data {
+ Line 833  sub update_full_student_data {
              next;
          } elsif ($parameter eq 'version') {
              next;
-         } elsif ($parameter =~ /^resource\.(.*)\.(tries|
+ 	} elsif (&symb_is_for_task($symb)) {
+ 	    next if ($parameter !~ /^resource\.(.*)\.(award|
+ 						      awarded|
+ 						      solved|
+ 						      submission|
+ 						      portfiles|
+ 						      status|
+ 						      version|
+ 						      regrader)\s*$/x);
+ 	    my ($version_and_part_id, $field) = ($1,$2);
+ 	    next if ($version_and_part_id !~ /\./
+ 		     && $field ne 'regrader' && $field ne 'version');
+ 	    my ($version, $part, $instance) =
+ 		split(/\./,$version_and_part_id);
+ 	    #skip and instance dimension or criteria specific data
+ 	    next if (defined($instance)
+ 		     && $instance ne $field
+ 		     && $instance ne 'bridgetask');
+ 	    if (!defined($part)) {
+ 		$part = $version;
+ 	    }
+ 	    my $resp_id = &get_part_id('0');
+ 	    my $part_id = &get_part_id($part);
+ 	    if ($field eq 'version') {
+ 		# for tasks each version is an attempt at it thus
+ 		#     version -> tries
+ 		$partdata->{$symb_id}{$part_id}{$transaction}{'tries'}=
+ 		    $value;
+ 		# at new version time the record gets reset thus adding a
+ 		# virtual response awarddetail of 'new_version'
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'response_specific'}='status';
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'response_specific_value'}='new_version';
+ 	    } elsif ($field eq 'award' || $field eq 'awarded'
+ 		     || $field eq 'solved') {
+ 		$partdata->{$symb_id}{$part_id}{$transaction}{$field}=
+ 		    $value;
+ 	    } elsif ($field eq 'portfiles') {
+ 		# tasks only accepts portfolio submissions
+ 		$value = $dbh->quote($value);
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'submission'}=$value;
+ 	    } elsif ($field eq 'status') {
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'response_specific'}=$field;
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'response_specific_value'}=$value;
+ 	    } elsif ($field eq 'regrader') {
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'response_specific_2'}=$field;
+ 		$respdata->{$symb_id}{$part_id}{$resp_id}{$transaction}{'response_specific_value_2'}=$value;
+ 	    }
+ 	} elsif ($parameter =~ /^resource\.(.*)\.(tries|
                                                    award|
                                                    awarded|
                                                    previous|
- Line 1084  sub update_full_student_data {
+ Line 957  sub update_full_student_data {
          chop($store_command);
          $dbh->do($store_command);
          if ($dbh->err) {
-             $returnstatus = 'error storing part data';
+             $returnstatus = 'error saving part data';
              &Apache::lonnet::logthis('insert error '.$dbh->errstr());
              &Apache::lonnet::logthis("While attempting\n".$store_command);
          }
- Line 1111  sub update_full_student_data {
+ Line 984  sub update_full_student_data {
                               $transaction,
                               $data->{'awarddetail'},
                               $data->{'response_specific'},
-                              $data->{'response_specific_value'}).
+                              $data->{'response_specific_value'},
+                              $data->{'response_specific_2'},
+                              $data->{'response_specific_value_2'}).
                               "',".$submission."),";
                      $store_rows++;
                  }
- Line 1122  sub update_full_student_data {
+ Line 997  sub update_full_student_data {
          chop($store_command);
          $dbh->do($store_command);
          if ($dbh->err) {
-             $returnstatus = 'error storing response data';
+             $returnstatus = 'error saving response data';
              &Apache::lonnet::logthis('insert error '.$dbh->errstr());
              &Apache::lonnet::logthis("While attempting\n".$store_command);
          }
- Line 1134  sub update_full_student_data {
+ Line 1009  sub update_full_student_data {
          ($sname,$sdom,$courseid,
           &Apache::lonnet::convert_dump_to_currentdump(\%studentdata));
      if ($returnstatus eq 'okay' && $status ne 'okay') {
-         $returnstatus = 'error storing current data:'.$status;
+         $returnstatus = 'error saving current data:'.$status;
      } elsif ($status ne 'okay') {
-         $returnstatus .= ' error storing current data:'.$status;
+         $returnstatus .= ' error saving current data:'.$status;
      }
      ##
      ## Update the students time......
- Line 1153  sub update_full_student_data {
+ Line 1028  sub update_full_student_data {
      return $returnstatus;
  }
- ################################################
- ################################################
- =pod
- =item &update_student_data()
- Input: $sname, $sdom, $courseid
- Output: $returnstatus, \%student_data
- $returnstatus is a string describing any errors that occured.  'okay' is the
- default.
- \%student_data is the data returned by a call to lonnet::currentdump.
- This subroutine loads a students data using lonnet::currentdump and inserts
- it into the MySQL database.  The inserts are done on two tables,
- $performance_table and $parameters_table.  $parameters_table holds the data
- that is not included in $performance_table.  See the description of
- $performance_table elsewhere in this file.  The INSERT calls are made
- directly by this subroutine, not through lonmysql because we do a 'bulk'
- insert which takes advantage of MySQLs non-SQL compliant INSERT command to
- insert multiple rows at a time.  If anything has gone wrong during this
- process, $returnstatus is updated with a description of the error and
- \%student_data is returned.
- Notice we do not insert the data and immediately query it.  This means it
- is possible for there to be data returned this first time that is not
- available the second time.  CYA.
- =cut
- ################################################
- ################################################
  sub update_student_data {
      my ($sname,$sdom,$courseid) = @_;
      #
- Line 1200  sub update_student_data {
+ Line 1042  sub update_student_data {
      #
      # Download students data
      my $time_of_retrieval = time;
-     my @tmp = &Apache::lonnet::currentdump($courseid,$sdom,$sname);
+     my %student_data = &Apache::lonnet::currentdump($courseid,$sdom,$sname);
-     if ((scalar(@tmp) > 0) && ($tmp[0] =~ /^error:/)) {
+     if (&Apache::lonnet::error(%student_data)) {
          &Apache::lonnet::logthis('error getting data for '.
                                   $sname.':'.$sdom.' in course '.$courseid.
-                                  ':'.$tmp[0]);
+                                  ':'.(%student_data)[0]);
-         $returnstatus = 'error getting data';
+         $returnstatus =(%student_data)[0] ;
          return ($returnstatus,undef);
      }
-     if (scalar(@tmp) < 1) {
+     if (scalar(keys(%student_data)) < 1) {
          return ('no data',undef);
      }
-     my %student_data = @tmp;
      my @Results = &store_student_data($sname,$sdom,$courseid,\%student_data);
      #
      # Set the students update time
- Line 1241  sub store_updatetime {
+ Line 1082  sub store_updatetime {
      $dbh->do($request);
  }
+ my $requested_max_packet = 0;
+ my $max_allowed_packet;
  sub store_student_data {
      my ($sname,$sdom,$courseid,$student_data) = @_;
      #
- Line 1261  sub store_student_data {
+ Line 1105  sub store_student_data {
      my $starttime = Time::HiRes::time;
      my $elapsed = 0;
      my $rows_stored;
-     my $store_parameters_command  = 'INSERT IGNORE INTO '.$parameters_table.
+     my $store_parameters_prefix  = 'INSERT IGNORE INTO '.$parameters_table.
          ' VALUES '."\n";
      my $num_parameters = 0;
-     my $store_performance_command = 'INSERT IGNORE INTO '.$performance_table.
+     my $store_performance_prefix = 'INSERT IGNORE INTO '.$performance_table.
          ' VALUES '."\n";
      return ('error',undef) if (! defined($dbh));
+     unless ($requested_max_packet) {
+         (undef,$max_allowed_packet) = $dbh->selectrow_array(
+                                              qq{show variables LIKE ? },
+                                              undef,
+                                              "max_allowed_packet");
+         if ($max_allowed_packet !~ /^\d+$/) {
+             $max_allowed_packet = '';
+         }
+         $requested_max_packet = 1;
+     }
+     my @store_parameters_values = ();
+     my $curr_params_values = '';
+     my $curr_params_length = 0;
+     my @store_performance_values = ();
+     my $curr_perf_values = '';
+     my $curr_perf_length = 0;
+     my ($max_param,$max_perf);
+     if ($max_allowed_packet) {
+         $max_param = $max_allowed_packet - length($store_parameters_prefix);
+         $max_perf = $max_allowed_packet - length($store_performance_prefix);
+     }
      while (my ($current_symb,$param_hash) = each(%{$student_data})) {
          #
          # make sure the symb is set up properly
- Line 1279  sub store_student_data {
+ Line 1144  sub store_student_data {
                                                $symb_id,$student_id,
                                                $parameter)."',".
                                                    $dbh->quote($value)."),\n";
-                 $num_parameters ++;
                  if ($sql_parameter !~ /''/) {
-                     $store_parameters_command .= $sql_parameter;
+                     if ($max_param) {
+                         my $length = length($sql_parameter);
+                         if ($length > $max_param) {
+                             &Apache::lonnet::logthis("SQL parameter insert for student: $sname for parameter: $parameter would exceed max_allowed_packet size");
+                             &Apache::lonnet::logthis("symb_id: $symb_id");
+                             &Apache::lonnet::logthis("You may want to increase the max_allowed_packet size from the current: $max_allowed_packet");
+                             if ($parameter =~ /\.submission$/) {
+                                 $sql_parameter = "('".join("','",
+                                               $symb_id,$student_id,
+                                               $parameter)."',".
+                                                   $dbh->quote('WARNING: Submission too large -- see grading interface for actual submission')."),\n";
+                                 $length = length($sql_parameter);
+                                 &Apache::lonnet::logthis("Placeholder inserted instead of value of actual submission");
+                                 &Apache::lonnet::logthis("See grading interface for the actual submission");
+                             } else {
+                                 &Apache::lonnet::logthis("Skipping this item");
+                                 next;
+                             }
+                         }
+                         if ($length + $curr_params_length > $max_param) {
+                             push(@store_parameters_values,$curr_params_values);
+                             $curr_params_values = $sql_parameter;
+                             $curr_params_length = $length;
+                         } else {
+                             $curr_params_values .= $sql_parameter;
+                             $curr_params_length += $length;
+                         }
+                     } else {
+                         $curr_params_values .= $sql_parameter;
+                     }
                      #$rows_stored++;
+                     $num_parameters ++;
                  }
              }
          }
- Line 1290  sub store_student_data {
+ Line 1184  sub store_student_data {
          my %stored;
          while (my ($parameter,$value) = each(%$param_hash)) {
              next if ($parameter !~ /^resource\.(.*)\.(solved|awarded)$/);
-             my $part = $1;
+             my $part  = $1;
+ 	    my $which = $2;
  	    next if ($part =~ /\./);
              next if (exists($stored{$part}));
              $stored{$part}++;
              #
              my $part_id = &get_part_id($part);
              next if (!defined($part_id));
-             my $solved  = $value;
-             my $tries   = $param_hash->{'resource.'.$part.'.tries'};
+             my ($solved,$awarded);
-             my $awarded = $param_hash->{'resource.'.$part.'.awarded'};
+ 	    if ($which eq 'solved') {
+ 		$solved  = $value;
+ 		$awarded = $param_hash->{'resource.'.$part.'.awarded'};
+ 	    } else {
+ 		$solved  = $param_hash->{'resource.'.$part.'.solved'};
+ 		$awarded = $value;
+ 	    }
              my $award   = $param_hash->{'resource.'.$part.'.award'};
              my $awarddetail = $param_hash->{'resource.'.$part.'.awarddetail'};
              my $timestamp = $param_hash->{'timestamp'};
+ 	    my $tries   = $param_hash->{'resource.'.$part.'.tries'};
+ 	    if (&symb_is_for_task($current_symb)) {
+ 		$tries   = $param_hash->{'resource.'.$part.'.version'};
+ 	    }
              #
              $solved      = '' if (! defined($solved));
              $tries       = '' if (! defined($tries));
- Line 1313  sub store_student_data {
+ Line 1218  sub store_student_data {
                  "('".join("','",$symb_id,$student_id,$part_id,$part,
                                  $solved,$tries,$awarded,$award,
                                  $awarddetail,$timestamp)."'),\n";
-             $store_performance_command .= $sql_performance;
+             if ($max_perf) {
+                 my $length = length($sql_performance);
+                 if ($length > $max_perf) {
+                             &Apache::lonnet::logthis("SQL performance insert for student: $sname would exceed max_allowed_packet size");
+                             &Apache::lonnet::logthis("symb_id: $symb_id");
+                             &Apache::lonnet::logthis("Skipping this item.  You may want to increase the max_allowed_packet size from the current: $max_allowed_packet");
+                             next;
+                 } else {
+                     if ($length + $curr_perf_length > $max_perf) {
+                         push(@store_performance_values,$curr_perf_values);
+                         $curr_perf_values = $sql_performance;
+                         $curr_perf_length = $length;
+                     } else {
+                         $curr_perf_values .= $sql_performance;
+                         $curr_perf_length += $length;
+                     }
+                 }
+             } else {
+                 $curr_perf_values .= $sql_performance;
+             }
              $rows_stored++;
          }
      }
+     if ($curr_params_values ne '') {
+         push(@store_parameters_values,$curr_params_values);
+     }
+     if ($curr_perf_values ne '') {
+         push(@store_performance_values,$curr_perf_values);
+     }
      if (! $rows_stored) { return ($returnstatus, undef); }
-     $store_parameters_command =~ s|,\n$||;
-     $store_performance_command =~ s|,\n$||;
      my $start = Time::HiRes::time;
-     $dbh->do($store_performance_command);
+     foreach my $item (@store_performance_values) {
-     if ($dbh->err()) {
+         $item =~ s|,\n$||;
-         &Apache::lonnet::logthis('performance bigass insert error:'.
+         if ($item ne '') {
-                                  $dbh->errstr());
+             $dbh->do($store_performance_prefix.$item);
-         &Apache::lonnet::logthis('command = '.$/.$store_performance_command);
+             if ($dbh->err()) {
-         $returnstatus = 'error: unable to insert performance into database';
+                 &Apache::lonnet::logthis('performance insert error:'.
-         return ($returnstatus,$student_data);
+                                          $dbh->errstr());
+                 &Apache::lonnet::logthis('command = '.$/.$store_performance_prefix.$item);
+                 $returnstatus = 'error: unable to insert performance into database';
+                 return ($returnstatus,$student_data);
+             }
+         }
      }
-     $dbh->do($store_parameters_command) if ($num_parameters>0);
+     if ($num_parameters > 0) {
-     if ($dbh->err()) {
+         foreach my $item (@store_parameters_values) {
-         &Apache::lonnet::logthis('parameters bigass insert error:'.
+             $item =~ s|,\n$||;
-                                  $dbh->errstr());
+             if ($item ne '') {
-         &Apache::lonnet::logthis('command = '.$/.$store_parameters_command);
+                 $dbh->do($store_parameters_prefix.$item);
-         &Apache::lonnet::logthis('rows_stored = '.$rows_stored);
+                 if ($dbh->err()) {
-         &Apache::lonnet::logthis('student_id = '.$student_id);
+                      &Apache::lonnet::logthis('parameters insert error:'.
-         $returnstatus = 'error: unable to insert parameters into database';
+                                               $dbh->errstr());
-         return ($returnstatus,$student_data);
+                      &Apache::lonnet::logthis('command = '.$/.$store_parameters_prefix.$item);
+                      &Apache::lonnet::logthis('rows_stored = '.$rows_stored);
+                      &Apache::lonnet::logthis('student_id = '.$student_id);
+                      $returnstatus = 'error: unable to insert parameters into database';
+                      return ($returnstatus,$student_data);
+                 }
+             }
+         }
      }
      $elapsed += Time::HiRes::time - $start;
      return ($returnstatus,$student_data);
  }
- ######################################
- ######################################
- =pod
- =item &ensure_tables_are_set_up($courseid)
- Checks to be sure the MySQL tables for the given class are set up.
- If $courseid is omitted it will be obtained from the environment.
- Returns nothing on success and 'error' on failure
- =cut
- ######################################
- ######################################
  sub ensure_tables_are_set_up {
      my ($courseid) = @_;
      $courseid = $env{'request.course.id'} if (! defined($courseid));
- Line 1368  sub ensure_tables_are_set_up {
+ Line 1293  sub ensure_tables_are_set_up {
      #
      # if the tables do not exist, make them
      my @CurrentTable = &Apache::lonmysql::tables_in_db();
-     my ($found_symb,$found_student,$found_part,
+     my ($found_symb,$found_student,$found_groups,$found_groupnames,$found_part,
          $found_performance,$found_parameters,$found_fulldump_part,
          $found_fulldump_response,$found_fulldump_timestamp,
          $found_weight);
      foreach (@CurrentTable) {
          $found_symb        = 1 if ($_ eq $symb_table);
          $found_student     = 1 if ($_ eq $student_table);
+         $found_groups      = 1 if ($_ eq $students_groups_table);
+         $found_groupnames  = 1 if ($_ eq $groupnames_table);
          $found_part        = 1 if ($_ eq $part_table);
          $found_performance = 1 if ($_ eq $performance_table);
          $found_parameters  = 1 if ($_ eq $parameters_table);
- Line 1394  sub ensure_tables_are_set_up {
+ Line 1321  sub ensure_tables_are_set_up {
      }
  }
- ################################################
- ################################################
- =pod
- =item &ensure_current_data()
- Input: $sname, $sdom, $courseid
- Output: $status, $data
- This routine ensures the data for a given student is up to date.
- The $student_table is queried to determine the time of the last update.
- If the students data is out of date, &update_student_data() is called.
- The return values from the call to &update_student_data() are returned.
- =cut
- ################################################
- ################################################
  sub ensure_current_data {
      my ($sname,$sdom,$courseid) = @_;
      my $status = 'okay';   # return value
- Line 1423  sub ensure_current_data {
+ Line 1330  sub ensure_current_data {
      #
      # Get the update time for the user
      my $updatetime = 0;
+     my $getuserdir = 1;
      my $modifiedtime = &Apache::lonnet::GetFileTimestamp
-         ($sdom,$sname,$courseid.'.db',
+         ($sdom,$sname,$courseid.'.db',$getuserdir);
-          $Apache::lonnet::perlvar{'lonUsersDir'});
      #
+     if ($modifiedtime == -1) {
+ 	return ('no data',undef);
+     }
      my $student_id = &get_student_id($sname,$sdom);
+     &get_students_groupids($student_id);
      my @Result = &Apache::lonmysql::get_rows($student_table,
                                               "student_id ='$student_id'");
      my $data = undef;
      if (@Result) {
-         $updatetime = $Result[0]->[5];  # Ack!  This is dumb!
+         $updatetime = $Result[0]->[6];  # Ack!  This is dumb!
      }
      if ($modifiedtime > $updatetime) {
          ($status,$data) = &update_student_data($sname,$sdom,$courseid);
- Line 1440  sub ensure_current_data {
+ Line 1352  sub ensure_current_data {
      return ($status,$data);
  }
- ################################################
- ################################################
- =pod
- =item &ensure_current_full_data($sname,$sdom,$courseid)
- Input: $sname, $sdom, $courseid
- Output: $status
- This routine ensures the fulldata (the data from a lonnet::dump, not a
- lonnet::currentdump) for a given student is up to date.
- The $student_table is queried to determine the time of the last update.
- If the students fulldata is out of date, &update_full_student_data() is
- called.
- The return value from the call to &update_full_student_data() is returned.
- =cut
- ################################################
- ################################################
  sub ensure_current_full_data {
      my ($sname,$sdom,$courseid) = @_;
      my $status = 'okay';   # return value
- Line 1471  sub ensure_current_full_data {
+ Line 1361  sub ensure_current_full_data {
      &ensure_tables_are_set_up($courseid);
      #
      # Get the update time for the user
+     my $getuserdir = 1;
      my $modifiedtime = &Apache::lonnet::GetFileTimestamp
-         ($sdom,$sname,$courseid.'.db',
+         ($sdom,$sname,$courseid.'.db',$getuserdir);
-          $Apache::lonnet::perlvar{'lonUsersDir'});
      #
      my $student_id = &get_student_id($sname,$sdom);
+     &get_students_groupids($student_id);
      my @Result = &Apache::lonmysql::get_rows($student_table,
                                               "student_id ='$student_id'");
      my $updatetime;
      if (@Result && ref($Result[0]) eq 'ARRAY') {
-         $updatetime = $Result[0]->[6];
+         $updatetime = $Result[0]->[7];
      }
      if (! defined($updatetime) || $modifiedtime > $updatetime) {
          $status = &update_full_student_data($sname,$sdom,$courseid);
- Line 1488  sub ensure_current_full_data {
+ Line 1379  sub ensure_current_full_data {
      return $status;
  }
- ################################################
+ sub ensure_current_groups {
- ################################################
+     my ($courseid) = @_;
+     my ($cdom,$cnum);
- =pod
+     if (defined($courseid)) {
+         my %coursehash = &Apache::lonnet::coursedescription($courseid);
- =item &get_student_data_from_performance_cache()
+         $cdom = $coursehash{'domain'};
+         $cnum = $coursehash{'num'};
- Input: $sname, $sdom, $symb, $courseid
+     } elsif ($env{'request.course.id'}) {
+         $courseid = $env{'request.course.id'};
- Output: hash reference containing the data for the given student.
+         $cdom = $env{'course.'.$courseid.'.domain'};
- If $symb is undef, all the students data is returned.
+         $cnum = $env{'course.'.$courseid.'.num'};
+     }
+     if ($cdom eq '' || $cnum eq '') {
+         return 'error: invalid course';
+     }
+     &setup_table_names($courseid);
+     my @CurrentTables = &Apache::lonmysql::tables_in_db();
+     unless (grep(/^\Q$groupnames_table\E$/,@CurrentTables)) {
+         return;
+     }
+     # Get the update time for the groupnames table
+     my $getuserdir = 1;
+     my $modifiedtime = &Apache::lonnet::GetFileTimestamp
+         ($cdom,$cnum,'coursegroups.db',$getuserdir);
+     my %tableinfo = &Apache::lonmysql::table_information($groupnames_table);
+     my $updatetime;
+     if ($tableinfo{'Update_time'}) {
+         $updatetime = $tableinfo{'Update_time'};
+     }
+     if (! defined($updatetime) || $modifiedtime > $updatetime) {
+         my (%groups_in_sql,%removegroups,$addgroup);
+         my %curr_groups = &Apache::longroup::coursegroups($cdom,$cnum);
+         my @Result = &Apache::lonmysql::get_rows($groupnames_table);
+         foreach my $row (@Result) {
+             my ($id,$name) = @{$row};
+             unless (exists($curr_groups{$name})) {
+                 $groups_in_sql{$name}=$id;
+             } elsif ($id) {
+                 $removegroups{$id} = $name;
+             }
+         }
+         foreach my $group (keys(%curr_groups)) {
+             unless (exists($groups_in_sql{$group})) {
+                 $addgroup = 1;
+                 last;
+             }
+         }
+         if (keys(%removegroups)) {
+             my $dbh = &Apache::lonmysql::get_dbh();
+             foreach my $group_id (keys(%removegroups)) {
+                 my $command = 'DELETE FROM '.$groupnames_table.' WHERE group_id='.
+                               $group_id;
+                 $dbh->do($command);
+                 if ($dbh->err()) {
+                     &Apache::lonnet::logthis("error ".$dbh->errstr().
+                                              " occurred executing \n".
+                                              "SQL command: $command");
+                 }
+             }
+         }
+         if ($addgroup) {
+             &populate_groupnames_table($courseid);
+         }
+     }
+     return;
+ }
- This routine is the heart of the local caching system.  See the description
+ sub ensure_current_students_groups {
- of $performance_table, $symb_table, $student_table, and $part_table.  The
+     my ($courseid) = @_;
- main task is building the MySQL request.  The tables appear in the request
+     my ($cdom,$cnum);
- in the order in which they should be parsed by MySQL.  When searching
+     if (defined($courseid)) {
- on a student the $student_table is used to locate the 'student_id'.  All
+         my %coursehash = &Apache::lonnet::coursedescription($courseid);
- rows in $performance_table which have a matching 'student_id' are returned,
+         $cdom = $coursehash{'domain'};
- with data from $part_table and $symb_table which match the entries in
+         $cnum = $coursehash{'num'};
- $performance_table, 'part_id' and 'symb_id'.  When searching on a symb,
+     } elsif ($env{'request.course.id'}) {
- the $symb_table is processed first, with matching rows grabbed from
+         $courseid = $env{'request.course.id'};
- $performance_table and filled in from $part_table and $student_table in
+         $cdom = $env{'course.'.$courseid.'.domain'};
- that order.
+         $cnum = $env{'course.'.$courseid.'.num'};
+     }
+     &setup_table_names($courseid);
+     my @CurrentTables = &Apache::lonmysql::tables_in_db();
+     unless (grep(/^\Q$students_groups_table\E$/,@CurrentTables)) {
+         return;
+     }
+     # Get the update time for the groupnames table
+     my $getuserdir = 1;
+     my $modifiedtime = &Apache::lonnet::GetFileTimestamp
+         ($cdom,$cnum,'groupmembership.db',$getuserdir);
+     my %tableinfo = &Apache::lonmysql::table_information($students_groups_table);
+     my $updatetime;
+     if ($tableinfo{'Update_time'}) {
+         $updatetime = $tableinfo{'Update_time'};
+     }
+     if ((!defined($updatetime)) || ($modifiedtime > $updatetime)) {
+         if (&Apache::lonmysql::drop_table($students_groups_table)) {
+             if (&init_dbs($courseid)) {
+                 return "error creating $students_groups_table\n";
+             } else {
+                 &populate_students_groups_table($courseid);
+             }
+         }
+     }
+     return;
+ }
- Running 'EXPLAIN ' on the 'SELECT' statements generated can be quite
+ sub ensure_current_sections {
- interesting, especially if you play with the order the tables are listed.
+     my ($courseid) = @_;
+     my ($cdom,$cnum);
+     if (defined($courseid)) {
+         my %coursehash = &Apache::lonnet::coursedescription($courseid);
+         $cdom = $coursehash{'domain'};
+         $cnum = $coursehash{'num'};
+     } elsif ($env{'request.course.id'}) {
+         $courseid = $env{'request.course.id'};
+         $cdom = $env{'course.'.$courseid.'.domain'};
+         $cnum = $env{'course.'.$courseid.'.num'};
+     }
+     &setup_table_names($courseid);
+     my @CurrentTables = &Apache::lonmysql::tables_in_db();
+     unless (grep(/^\Q$student_table\E$/,@CurrentTables)) {
+         return;
+     }
+     # Get the update time for the student table
+     my $getuserdir = 1;
+     my $modifiedtime = &Apache::lonnet::GetFileTimestamp
+         ($cdom,$cnum,'classlist.db',$getuserdir);
+     my %tableinfo = &Apache::lonmysql::table_information($student_table);
+     my $updatetime;
+     if ($tableinfo{'Update_time'}) {
+         $updatetime = $tableinfo{'Update_time'};
+     }
+     if ((!defined($updatetime)) || ($modifiedtime > $updatetime)) {
+         &update_student_table($cdom,$cnum);
+     }
+     return;
+ }
- =cut
+ sub update_student_table {
+     my ($cdom,$cnum) = @_;
+     return unless (($cdom ne '') && ($cnum ne ''));
+     my (%roster,%sqldata);
+     my $classlist = &get_classlist($cdom,$cnum);
+     while (my ($student,$data) = each (%$classlist)) {
+         my ($section,$start,$end) = ($data->[&CL_SECTION()],
+                                      $data->[&CL_START()],
+                                      $data->[&CL_END()]);
+         if ($section eq '' || $section =~ /^\s*$/) {
+             $section = 'none';
+         }
+         if ($start eq '') { $start = 0; }
+         if ($end eq '')   { $end   = 0; }
+         $roster{$student}{'section'} = $section;
+         $roster{$student}{'start'} = $start;
+         $roster{$student}{'end'} = $end;
+     }
+     my $dbh = &Apache::lonmysql::get_dbh();
+     my $statement = "SELECT student_id,student,section,start,end FROM $student_table";
+     my $sth = $dbh->prepare($statement);
+     $sth->execute();
+     if ($sth->err()) {
+         &Apache::lonnet::logthis("Unable to execute MySQL request:");
+         &Apache::lonnet::logthis("\n".$statement."\n");
+         &Apache::lonnet::logthis("error is:".$sth->errstr());
+         return undef;
+     }
+     foreach my $row (@{$sth->fetchall_arrayref}) {
+         my ($id,$student,$section,$start,$end) = (@$row);
+         if (ref($roster{$student}) eq 'HASH') {
+             if (($roster{$student}{'section'} ne $section) ||
+                 ($roster{$student}{'start'} ne $start) ||
+                 ($roster{$student}{'end'} ne $end)) {
+                 $sqldata{$id} = {
+                                   section => $roster{$student}{'section'},
+                                   start   => $roster{$student}{'start'},
+                                   end     => $roster{$student}{'end'},
+                                 };
+             }
+         }
+     }
+     $sth->finish();
+     if (keys(%sqldata)) {
+         foreach my $id (sort { $a <=> $b } keys(%sqldata)) {
+             my $request = "UPDATE $student_table SET section='$sqldata{$id}{section}'".
+                           ", start='$sqldata{$id}{start}'".
+                           ", end='$sqldata{$id}{end}' WHERE student_id='$id'";
+             $dbh->do($request);
+         }
+     }
+     return;
+ }
- ################################################
- ################################################
  sub get_student_data_from_performance_cache {
      my ($sname,$sdom,$symb,$courseid)=@_;
      my $student = $sname.':'.$sdom if (defined($sname) && defined($sdom));
- Line 1602  sub get_student_data_from_performance_ca
+ Line 1653  sub get_student_data_from_performance_ca
      return $studentdata;
  }
- ################################################
- ################################################
- =pod
- =item &get_current_state()
- Input: $sname,$sdom,$symb,$courseid
- Output: Described below
- Retrieve the current status of a students performance.  $sname and
- $sdom are the only required parameters.  If $symb is undef the results
- of an &Apache::lonnet::currentdump() will be returned.
- If $courseid is undef it will be retrieved from the environment.
- The return structure is based on &Apache::lonnet::currentdump.  If
- $symb is unspecified, all the students data is returned in a hash of
- the form:
- (
-   symb1 => { param1 => value1, param2 => value2 ... },
-   symb2 => { param1 => value1, param2 => value2 ... },
- )
- If $symb is specified, a hash of
- (
-   param1 => value1,
-   param2 => value2,
- )
- is returned.
- If no data is found for $symb, or if the student has no performance data,
- an empty list is returned.
- =cut
- ################################################
- ################################################
  sub get_current_state {
      my ($sname,$sdom,$symb,$courseid,$forcedownload)=@_;
      #
- Line 1670  sub get_current_state {
+ Line 1684  sub get_current_state {
      } else {
          if ($status ne 'okay' && $status ne '') {
              &Apache::lonnet::logthis('status = '.$status);
-             return ();
+             return ('error: '.$status,undef);
          }
          my $returnhash = &get_student_data_from_performance_cache($sname,$sdom,
                                                        $symb,$courseid);
- Line 1679  sub get_current_state {
+ Line 1693  sub get_current_state {
      return ();
  }
- ################################################
- ################################################
- =pod
- =item &get_problem_statistics()
- Gather data on a given problem.  The database is assumed to be
- populated and all local caching variables are assumed to be set
- properly.  This means you need to call &ensure_current_data for
- the students you are concerned with prior to calling this routine.
- Inputs: $Sections, $status, $symb, $part, $courseid, $starttime, $endtime
- =over 4
- =item $Sections Array ref containing section names for students.
- 'all' is allowed to be the first (and only) item in the array.
- =item $status String describing the status of students
- =item $symb is the symb for the problem.
- =item $part is the part id you need statistics for
- =item $courseid is the course id, of course!
- =item $starttime and $endtime are unix times which to use to limit
- the statistical data.
- =back
- Outputs: See the code for up to date information.  A hash reference is
- returned.  The hash has the following keys defined:
- =over 4
- =item num_students The number of students attempting the problem
- =item tries The total number of tries for the students
- =item max_tries The maximum number of tries taken
- =item mean_tries The average number of tries
- =item num_solved The number of students able to solve the problem
- =item num_override The number of students whose answer is 'correct_by_override'
- =item deg_of_diff The degree of difficulty of the problem
- =item std_tries The standard deviation of the number of tries
- =item skew_tries The skew of the number of tries
- =item per_wrong The number of students attempting the problem who were not
- able to answer it correctly.
- =back
- =cut
- ################################################
- ################################################
  sub get_problem_statistics {
-     my ($Sections,$status,$symb,$part,$courseid,$starttime,$endtime) = @_;
+     my ($Sections,$Groups,$status,$symb,$part,$courseid,$starttime,$endtime) = @_;
      return if (! defined($symb) || ! defined($part));
      $courseid = $env{'request.course.id'} if (! defined($courseid));
      #
      &setup_table_names($courseid);
      my $symb_id = &get_symb_id($symb);
      my $part_id = &get_part_id($part);
-     my $stats_table = $courseid.'_problem_stats';
+     my $stats_table = &temp_table_name($courseid,'problem_stats');
      #
      my $dbh = &Apache::lonmysql::get_dbh();
      return undef if (! defined($dbh));
- Line 1768  sub get_problem_statistics {
+ Line 1719  sub get_problem_statistics {
          (defined($status)   && lc($status)        ne 'any')) {
          $request .= 'NATURAL LEFT JOIN '.$student_table.' AS b ';
      }
+     my ($groups_join,$group_limits) = &limit_by_group($Groups,'b','c','d');
+     if (defined($groups_join)) {
+         $request .= $groups_join;
+     }
      $request .= ' WHERE a.symb_id='.$symb_id.' AND a.part_id='.$part_id;
      #
      # Limit the students included to those specified
-     if (defined($Sections) && lc($Sections->[0]) ne 'all') {
+     my ($section_limits,$enrollment_limits)=
-         $request .= ' AND ('.
+         &limit_by_section_and_status($Sections,$status,'b');
-             join(' OR ', map { "b.section='".$_."'" } @$Sections
-                  ).')';
-     }
-     if (defined($status) && lc($status) ne 'any') {
-         $request .= " AND b.status='".$status."'";
-     }
      #
      # Limit by starttime and endtime
      my $time_requirements = undef;
- Line 1793  sub get_problem_statistics {
+ Line 1742  sub get_problem_statistics {
      if (defined($time_requirements)) {
          $request .= ' AND '.$time_requirements;
      }
+     if (defined($section_limits)) {
+         $request .= ' AND '.$section_limits;
+     }
+     if (defined($enrollment_limits)) {
+         $request .= ' AND '.$enrollment_limits;
+     }
+     # Limit by group, as required
+     if (defined($group_limits)) {
+         $request .= ' AND '.$group_limits;
+     }
      #
      # Finally, execute the request to create the temporary table
      $dbh->do($request);
- Line 1806  sub get_problem_statistics {
+ Line 1765  sub get_problem_statistics {
      #
      $request = 'SELECT MAX(tries),MIN(tries) FROM '.$stats_table.
          ' WHERE awarded>0';
-     if (defined($time_requirements)) {
-         $request .= ' AND '.$time_requirements;
-     }
      my ($max,$min) = &execute_SQL_request($dbh,$request);
      #
      $request = 'SELECT SUM(awarded) FROM '.$stats_table;
-     if (defined($time_requirements)) {
-         $request .= ' AND '.$time_requirements;
-     }
      my ($Solved) = &execute_SQL_request($dbh,$request);
      #
      $request = 'SELECT SUM(awarded) FROM '.$stats_table.
          " WHERE solved='correct_by_override'";
-     if (defined($time_requirements)) {
-         $request .= ' AND '.$time_requirements;
-     }
      my ($solved) = &execute_SQL_request($dbh,$request);
      #
      $Solved -= $solved;
- Line 1884  sub execute_SQL_request {
+ Line 1834  sub execute_SQL_request {
      return ();
  }
- ######################################################
- ######################################################
- =pod
- =item &populate_weight_table
- =cut
- ######################################################
- ######################################################
  sub populate_weight_table {
      my ($courseid) = @_;
      if (! defined($courseid)) {
- Line 1947  sub populate_weight_table {
+ Line 1887  sub populate_weight_table {
      $dbh->do($request);
      if ($dbh->err()) {
          &Apache::lonnet::logthis("error ".$dbh->errstr().
-                                  " occured executing \n".
+                                  " occurred executing \n".
                                   $request);
      }
      return;
  }
- ##########################################################
- ##########################################################
- =pod
- =item &limit_by_start_end_times
- Build SQL WHERE condition which limits the data collected by the start
- and end times provided
- Inputs: $starttime, $endtime, $table
- Returns: $time_limits
- =cut
- ##########################################################
- ##########################################################
  sub limit_by_start_end_time {
      my ($starttime,$endtime,$table) = @_;
      my $time_requirements = undef;
- Line 1985  sub limit_by_start_end_time {
+ Line 1907  sub limit_by_start_end_time {
      return $time_requirements;
  }
- ##########################################################
- ##########################################################
- =pod
- =item &limit_by_section_and_status
- Build SQL WHERE condition which limits the data collected by section and
- student status.
- Inputs: $Sections (array ref)
-     $enrollment (string: 'any', 'expired', 'active')
-     $tablename The name of the table that holds the student data
- Returns: $student_requirements,$enrollment_requirements
- =cut
- ##########################################################
- ##########################################################
  sub limit_by_section_and_status {
      my ($Sections,$enrollment,$tablename) = @_;
      my $student_requirements = undef;
- Line 2013  sub limit_by_section_and_status {
+ Line 1916  sub limit_by_section_and_status {
              join(' OR ', map { $tablename.".section='".$_."'" } @$Sections
                   ).')';
      }
-     #
      my $enrollment_requirements=undef;
      if (defined($enrollment) && $enrollment ne 'Any') {
-         $enrollment_requirements = $tablename.".status='".$enrollment."'";
+ 	my $now = time();
+ 	if ( $enrollment eq 'Future' ) {
+ 	    $enrollment_requirements =
+ 		"( $tablename.start > $now AND ".
+ 		"( $tablename.end = 0 OR $tablename.end > $now))";
+ 	} elsif ( $enrollment eq 'Active' ) {
+ 	    $enrollment_requirements =
+ 		"(( $tablename.start = 0 OR $tablename.start < $now )  AND ".
+ 		" ( $tablename.end   = 0 OR $tablename.end   > $now ))";
+ 	} elsif ( $enrollment eq 'Expired' ) {
+ 	    $enrollment_requirements =
+ 		"(( $tablename.start < $now )  AND ".
+ 		" ( $tablename.end   < $now ))";
+ 	}
      }
      return ($student_requirements,$enrollment_requirements);
  }
- ######################################################
- ######################################################
- =pod
- =item rank_students_by_scores_on_resources
- Inputs:
+ sub limit_by_group {
-     $resources: array ref of hash ref.  Each hash ref needs key 'symb'.
+     my ($Groups,$stutable,$grptable,$stugrptab) = @_;
-     $Sections: array ref of sections to include,
+     my $groups_join = undef;
-     $enrollment: string,
+     my $group_limits = undef;
-     $courseid (may be omitted)
+     if ( (defined($Groups) && $Groups->[0] ne 'all')) {
-     $starttime (may be omitted)
+         $groups_join =
-     $endtime (may be omitted)
+           ' LEFT JOIN '.$students_groups_table.
-     $has_award_for (may be omitted)
+                      ' AS '.$stugrptab.' ON '.
+                      $stugrptab.'.student_id = '.$stutable.'.student_id'.
- Returns; An array of arrays.  The sub arrays contain a student name and
+           ' LEFT JOIN '.$groupnames_table.
- their score on the resources. $starttime and $endtime constrain the
+                      ' AS '.$grptable.' ON '.
- list to awards obtained during the given time limits. $has_score_on
+                      $stugrptab.'.group_id = '.$grptable.'.group_id ';
- constrains the list to those students who at least attempted the
+         $group_limits =
- resource identified by the given symb, which is used to filter out
+           ' ('.
- such students for statistics that would be adversely affected by such
+              join(' OR ', map {  "$grptable.groupname='".$_."'" } @$Groups
- students.
+            ).')';
+     }
+     return ($groups_join,$group_limits);
+ }
- =cut
- ######################################################
- ######################################################
  sub RNK_student { return 0; };
  sub RNK_score   { return 1; };
  sub rank_students_by_scores_on_resources {
-     my ($resources,$Sections,$enrollment,$courseid,$starttime,$endtime,$has_award_for) = @_;
+     my ($resources,$Sections,$Groups,$enrollment,$courseid,$starttime,$endtime,
+         $has_award_for) = @_;
      return if (! defined($resources) || ! ref($resources) eq 'ARRAY');
      if (! defined($courseid)) {
          $courseid = $env{'request.course.id'};
- Line 2063  sub rank_students_by_scores_on_resources
+ Line 1974  sub rank_students_by_scores_on_resources
      my $dbh = &Apache::lonmysql::get_dbh();
      my ($section_limits,$enrollment_limits)=
          &limit_by_section_and_status($Sections,$enrollment,'b');
+     my ($groups_join,$group_limits) = &limit_by_group($Groups,'b','c','d');
      my $symb_limits = '('.join(' OR ',map {'a.symb_id='.&get_symb_id($_);
                                         } @$resources
                                 ).')';
- Line 2079  sub rank_students_by_scores_on_resources
+ Line 1991  sub rank_students_by_scores_on_resources
          ."$award_col FROM $performance_table AS a ".
          "NATURAL LEFT JOIN $weight_table AS w ".
          "LEFT JOIN $student_table AS b ON a.student_id=b.student_id ".
-         "$award_join WHERE ";
+         "$award_join $groups_join ";
+     my $limits;
      if (defined($section_limits)) {
-         $request .= $section_limits.' AND ';
+         $limits .= $section_limits.' AND ';
      }
      if (defined($enrollment_limits)) {
-         $request .= $enrollment_limits.' AND ';
+         $limits .= $enrollment_limits.' AND ';
      }
      if (defined($time_limits)) {
-         $request .= $time_limits.' AND ';
+         $limits .= $time_limits.' AND ';
      }
      if ($symb_limits ne '()') {
-         $request .= $symb_limits.' AND ';
+         $limits .= $symb_limits.' AND ';
+     }
+     if (defined($group_limits)) {
+         $limits .= $group_limits.' AND ';
      }
-     $request =~ s/( AND )$//;   # Remove extra conjunction
+     if ($limits) {
-     $request =~ s/( WHERE )$//; # In case there were no limits placed on it
+         $limits =~ s/( AND )$//;   # Remove extra conjunction
-     $request .= " $award_clause GROUP BY a.student_id ORDER BY score";
+         $request .= "WHERE $limits";
+     }
+     $request .= " $award_clause GROUP BY a.student_id ORDER BY score, b.student";
      #&Apache::lonnet::logthis('request = '.$/.$request);
      my $sth = $dbh->prepare($request) or die "Can't prepare $request";
      $sth->execute();
- Line 2102  sub rank_students_by_scores_on_resources
+ Line 2020  sub rank_students_by_scores_on_resources
      return ($rows);
  }
- ########################################################
- ########################################################
- =pod
- =item &get_sum_of_scores
- Inputs: $resource (hash ref, needs {'symb'} key),
- $part, (the part id),
- $students (array ref, contents of array are scalars holding 'sname:sdom'),
- $courseid
- Returns: the sum of the score on the problem part over the students and the
-    maximum possible value for the sum (taken from the weight table).
- =cut
- ########################################################
- ########################################################
  sub get_sum_of_scores {
      my ($symb,$part,$students,$courseid,$starttime,$endtime) = @_;
      if (! defined($courseid)) {
- Line 2164  sub get_sum_of_scores {
+ Line 2063  sub get_sum_of_scores {
      return ($rows->[0],$rows->[1]);
  }
- ########################################################
- ########################################################
- =pod
- =item &score_stats
- Inputs: $Sections, $enrollment, $symbs, $starttime,
-         $endtime, $courseid
- $Sections, $enrollment, $starttime, $endtime, and $courseid are the same as
- elsewhere in this module.
- $symbs is an array ref of symbs
- Returns: minimum, maximum, mean, s.d., number of students, and maximum
-   possible of student scores on the given resources
- =cut
- ########################################################
- ########################################################
  sub score_stats {
-     my ($Sections,$enrollment,$symbs,$starttime,$endtime,$courseid)=@_;
+     my ($Sections,$Groups,$enrollment,$symbs,$starttime,$endtime,$courseid)=@_;
      if (! defined($courseid)) {
          $courseid = $env{'request.course.id'};
      }
- Line 2196  sub score_stats {
+ Line 2075  sub score_stats {
      #
      my ($section_limits,$enrollment_limits)=
          &limit_by_section_and_status($Sections,$enrollment,'b');
+     my ($groups_join,$group_limits) = &limit_by_group($Groups,'b','c','d');
      my $time_limits = &limit_by_start_end_time($starttime,$endtime,'a');
      my @Symbids = map { &get_symb_id($_); } @{$symbs};
      #
-     my $stats_table = $courseid.'_problem_stats';
+     my $stats_table = &temp_table_name($courseid,'problem_stats');
      my $symb_restriction = join(' OR ',map {'a.symb_id='.$_;} @Symbids);
      my $request = 'DROP TABLE '.$stats_table;
      $dbh->do($request);
- Line 2210  sub score_stats {
+ Line 2090  sub score_stats {
          $performance_table.' AS a '.
          'NATURAL LEFT JOIN '.$weight_table.' AS w '.
          'LEFT JOIN '.$student_table.' AS b ON a.student_id=b.student_id '.
-         'WHERE ('.$symb_restriction.')';
+         $groups_join;
+     my $limit = ' WHERE ('.$symb_restriction.')';
      if ($time_limits) {
-         $request .= ' AND '.$time_limits;
+         $limit .= ' AND '.$time_limits;
      }
      if ($section_limits) {
-         $request .= ' AND '.$section_limits;
+         $limit .= ' AND '.$section_limits;
      }
      if ($enrollment_limits) {
-         $request .= ' AND '.$enrollment_limits;
+         $limit .= ' AND '.$enrollment_limits;
      }
-     $request .= ' GROUP BY a.student_id';
+     if ($group_limits) {
+         $limit .= ' AND '.$group_limits;
+     }
+     $request .= $limit.' GROUP BY a.student_id';
  #    &Apache::lonnet::logthis('request = '.$/.$request);
      my $sth = $dbh->prepare($request);
      $sth->execute();
- Line 2238  sub score_stats {
+ Line 2122  sub score_stats {
  }
- ########################################################
- ########################################################
- =pod
- =item &count_stats
- Inputs: $Sections, $enrollment, $symbs, $starttime,
-         $endtime, $courseid
- $Sections, $enrollment, $starttime, $endtime, and $courseid are the same as
- elsewhere in this module.
- $symbs is an array ref of symbs
- Returns: minimum, maximum, mean, s.d., and number of students
-   of the number of items correct on the given resources
- =cut
- ########################################################
- ########################################################
  sub count_stats {
-     my ($Sections,$enrollment,$symbs,$starttime,$endtime,$courseid)=@_;
+     my ($Sections,$Groups,$enrollment,$symbs,$starttime,$endtime,$courseid)=@_;
      if (! defined($courseid)) {
          $courseid = $env{'request.course.id'};
      }
- Line 2270  sub count_stats {
+ Line 2134  sub count_stats {
      #
      my ($section_limits,$enrollment_limits)=
          &limit_by_section_and_status($Sections,$enrollment,'b');
+     my ($groups_join,$group_limits) = &limit_by_group($Groups,'b','c','d');
      my $time_limits = &limit_by_start_end_time($starttime,$endtime,'a');
      my @Symbids = map { &get_symb_id($_); } @{$symbs};
      #
-     my $stats_table = $courseid.'_problem_stats';
+     my $stats_table = &temp_table_name($courseid,'problem_stats');
      my $symb_restriction = join(' OR ',map {'a.symb_id='.$_;} @Symbids);
      my $request = 'DROP TABLE '.$stats_table;
      $dbh->do($request);
- Line 2283  sub count_stats {
+ Line 2148  sub count_stats {
          'SUM(a.awarded) AS count FROM '.
          $performance_table.' AS a '.
          'LEFT JOIN '.$student_table.' AS b ON a.student_id=b.student_id '.
-         'WHERE ('.$symb_restriction.')';
+         $groups_join;
+     my $limit =  ' WHERE ('.$symb_restriction.')';
      if ($time_limits) {
-         $request .= ' AND '.$time_limits;
+         $limit .= ' AND '.$time_limits;
      }
      if ($section_limits) {
-         $request .= ' AND '.$section_limits;
+         $limit .= ' AND '.$section_limits;
      }
      if ($enrollment_limits) {
-         $request .= ' AND '.$enrollment_limits;
+         $limit .= ' AND '.$enrollment_limits;
      }
-     $request .= ' GROUP BY a.student_id';
+     if ($group_limits) {
+         $limit .= ' AND '.$group_limits;
+     }
+     $request .= $limit.' GROUP BY a.student_id';
  #    &Apache::lonnet::logthis('request = '.$/.$request);
      my $sth = $dbh->prepare($request);
      $sth->execute();
- Line 2305  sub count_stats {
+ Line 2174  sub count_stats {
      return($min,$max,$ave,$std,$count);
  }
- ######################################################
- ######################################################
- =pod
- =item get_student_data
- =cut
- ######################################################
- ######################################################
  sub get_student_data {
      my ($students,$courseid) = @_;
      $courseid = $env{'request.course.id'} if (! defined($courseid));
- Line 2347  sub get_student_data {
+ Line 2206  sub get_student_data {
      }
  }
- sub RD_student_id    { return 0; }
+ sub RD_student_id      { return 0; }
- sub RD_awarddetail   { return 1; }
+ sub RD_awarddetail     { return 1; }
- sub RD_response_eval { return 2; }
+ sub RD_response_eval   { return 2; }
- sub RD_submission    { return 3; }
+ sub RD_response_eval_2 { return 3; }
- sub RD_timestamp     { return 4; }
+ sub RD_submission      { return 4; }
- sub RD_tries         { return 5; }
+ sub RD_timestamp       { return 5; }
- sub RD_sname         { return 6; }
+ sub RD_tries           { return 6; }
+ sub RD_sname           { return 7; }
  sub get_response_data {
-     my ($Sections,$enrollment,$symb,$response,$courseid) = @_;
+     my ($Sections,$Groups,$enrollment,$symb,$response,$courseid) = @_;
      return undef if (! defined($symb) ||
                 ! defined($response));
      $courseid = $env{'request.course.id'} if (! defined($courseid));
- Line 2378  sub get_response_data {
+ Line 2238  sub get_response_data {
      #
      my ($student_requirements,$enrollment_requirements) =
          &limit_by_section_and_status($Sections,$enrollment,'d');
+     my ($groups_join,$group_limits) = &limit_by_group($Groups,'d','e','f');
      my $request = 'SELECT '.
          'a.student_id, a.awarddetail, a.response_specific_value, '.
-         'a.submission, b.timestamp, c.tries, d.student '.
+         'a.response_specific_value_2, a.submission, b.timestamp, '.
+ 	'c.tries, d.student '.
          'FROM '.$fulldump_response_table.' AS a '.
          'LEFT JOIN '.$fulldump_timestamp_table.' AS b '.
          'ON a.symb_id=b.symb_id AND a.student_id=b.student_id AND '.
- Line 2390  sub get_response_data {
+ Line 2252  sub get_response_data {
          'a.part_id=c.part_id AND a.transaction = c.transaction '.
          'LEFT JOIN '.$student_table.' AS d '.
          'ON a.student_id=d.student_id '.
-         'WHERE '.
+         $groups_join;
+     my $limit = ' WHERE '.
          'a.symb_id='.$symb_id.' AND a.response_id='.$response_id;
-     if (defined($student_requirements) || defined($enrollment_requirements)) {
+     if (defined($student_requirements)) {
-         $request .= ' AND ';
+         $limit .= ' AND '.$student_requirements;
-         if (defined($student_requirements)) {
-             $request .= $student_requirements.' AND ';
-         }
-         if (defined($enrollment_requirements)) {
-             $request .= $enrollment_requirements.' AND ';
-         }
-         $request =~ s/( AND )$//;
      }
-     $request .= ' ORDER BY b.timestamp';
+     if (defined($enrollment_requirements)) {
+         $limit .= ' AND '.$enrollment_requirements;
+     }
+     if (defined($group_limits)) {
+         $limit .= ' AND '.$group_limits;
+     }
+     $request .= $limit.' ORDER BY b.timestamp';
  #    &Apache::lonnet::logthis("request =\n".$request);
      my $sth = $dbh->prepare($request);
      $sth->execute();
- Line 2415  sub get_response_data {
+ Line 2277  sub get_response_data {
      if (ref($dataset) eq 'ARRAY' && scalar(@$dataset)>0) {
          # Clear the \'s from around the submission
          for (my $i =0;$i<scalar(@$dataset);$i++) {
-             $dataset->[$i]->[3] =~ s/(\'$|^\')//g;
+             $dataset->[$i]->[&RD_submission()] =~ s/(\'$|^\')//g;
          }
          return $dataset;
      }
  }
- sub RDs_awarddetail   { return 3; }
+ sub RDs_awarddetail     { return 3; }
- sub RDs_submission    { return 2; }
+ sub RDs_submission      { return 2; }
- sub RDs_timestamp     { return 1; }
+ sub RDs_timestamp       { return 1; }
- sub RDs_tries         { return 0; }
+ sub RDs_tries           { return 0; }
- sub RDs_awarded       { return 4; }
+ sub RDs_awarded         { return 4; }
+ sub RDs_response_eval   { return 5; }
+ sub RDs_response_eval_2 { return 6; }
+ sub RDs_part_award      { return 7; }
  sub get_response_data_by_student {
      my ($student,$symb,$response,$courseid) = @_;
- Line 2444  sub get_response_data_by_student {
+ Line 2309  sub get_response_data_by_student {
      my $dbh = &Apache::lonmysql::get_dbh();
      return undef if (! defined($dbh));
      my $request = 'SELECT '.
-         'c.tries, b.timestamp, a.submission, a.awarddetail, e.awarded '.
+         'c.tries, b.timestamp, a.submission, a.awarddetail, c.awarded, '.
+ 	'a.response_specific_value, a.response_specific_value_2, c.award '.
          'FROM '.$fulldump_response_table.' AS a '.
          'LEFT JOIN '.$fulldump_timestamp_table.' AS b '.
          'ON a.symb_id=b.symb_id AND a.student_id=b.student_id AND '.
- Line 2472  sub get_response_data_by_student {
+ Line 2338  sub get_response_data_by_student {
      if (ref($dataset) eq 'ARRAY' && scalar(@$dataset)>0) {
          # Clear the \'s from around the submission
          for (my $i =0;$i<scalar(@$dataset);$i++) {
-             $dataset->[$i]->[2] =~ s/(\'$|^\')//g;
+             $dataset->[$i]->[&RDs_submission] =~ s/(\'$|^\')//g;
          }
          return $dataset;
      }
- Line 2485  sub RT_tries      { return 2; }
+ Line 2351  sub RT_tries      { return 2; }
  sub RT_timestamp  { return 3; }
  sub get_response_time_data {
-     my ($sections,$enrollment,$symb,$part,$courseid) = @_;
+     my ($sections,$groups,$enrollment,$symb,$part,$courseid) = @_;
      return undef if (! defined($symb) ||
                       ! defined($part));
      $courseid = $env{'request.course.id'} if (! defined($courseid));
- Line 2506  sub get_response_time_data {
+ Line 2372  sub get_response_time_data {
      return undef if (! defined($dbh));
      my ($student_requirements,$enrollment_requirements) =
          &limit_by_section_and_status($sections,$enrollment,'d');
+     my ($groups_join,$group_limits) = &limit_by_group($groups,'d','e','f');
      my $request = 'SELECT '.
          'a.student_id, a.awarded, a.tries, b.timestamp '.
          'FROM '.$fulldump_part_table.' AS a '.
- Line 2514  sub get_response_time_data {
+ Line 2381  sub get_response_time_data {
          'a.transaction = b.transaction '.
          'LEFT JOIN '.$student_table.' as d '.
          'ON a.student_id=d.student_id '.
-         'WHERE '.
+         $groups_join;
+     my $limit = ' WHERE '.
          'a.symb_id='.$symb_id.' AND a.part_id='.$part_id;
-     if (defined($student_requirements) || defined($enrollment_requirements)) {
+     if (defined($student_requirements)) {
-         $request .= ' AND ';
+         $limit .= ' AND '.$student_requirements;
-         if (defined($student_requirements)) {
+     }
-             $request .= $student_requirements.' AND ';
+     if (defined($enrollment_requirements)) {
-         }
+         $limit .= ' AND '.$enrollment_requirements;
-         if (defined($enrollment_requirements)) {
+     }
-             $request .= $enrollment_requirements.' AND ';
+     if (defined($group_limits)) {
-         }
+         $limit .= ' AND '.$group_limits;
-         $request =~ s/( AND )$//;
      }
-     $request .= ' ORDER BY b.timestamp';
+     $request .= $limit.' ORDER BY b.timestamp';
  #    &Apache::lonnet::logthis("request =\n".$request);
      my $sth = $dbh->prepare($request);
      $sth->execute();
- Line 2542  sub get_response_time_data {
+ Line 2409  sub get_response_time_data {
  }
- ################################################
- ################################################
- =pod
- =item &get_student_scores($Sections,$Symbs,$enrollment,$courseid)
- =cut
- ################################################
- ################################################
  sub get_student_scores {
-     my ($sections,$Symbs,$enrollment,$courseid,$starttime,$endtime) = @_;
+     my ($sections,$groups,$Symbs,$enrollment,$courseid,$starttime,$endtime) = @_;
      $courseid = $env{'request.course.id'} if (! defined($courseid));
      &setup_table_names($courseid);
      my $dbh = &Apache::lonmysql::get_dbh();
      return (undef) if (! defined($dbh));
-     my $tmptable = $courseid.'_temp_'.time;
+     my $tmptable = &temp_table_name($courseid,'temp_'.time);
      my $request = 'DROP TABLE IF EXISTS '.$tmptable;
  #    &Apache::lonnet::logthis('request = '.$/.$request);
      $dbh->do($request);
- Line 2576  sub get_student_scores {
+ Line 2432  sub get_student_scores {
      my ($student_requirements,$enrollment_requirements) =
          &limit_by_section_and_status($sections,$enrollment,'b');
      #
+     my ($groups_join,$group_limits) = &limit_by_group($groups,'b','d','e');
      my $time_requirements = &limit_by_start_end_time($starttime,$endtime,'a');
      ##
      $request = 'CREATE TEMPORARY TABLE IF NOT EXISTS '.$tmptable.
- Line 2583  sub get_student_scores {
+ Line 2440  sub get_student_scores {
          $performance_table.' AS a ';
      $request .= "LEFT JOIN ".$weight_table.' AS c ON a.symb_id=c.symb_id AND a.part_id=c.part_id ';
      if (defined($student_requirements) || defined($enrollment_requirements)) {
-         $request .= ' LEFT JOIN '.$student_table.' AS b ON a.student_id=b.student_id';
+         $request .= ' LEFT JOIN '.$student_table.' AS b ON a.student_id=b.student_id ';
+     }
+     if (defined($groups_join)) {
+         $request .= $groups_join;
      }
-     if (defined($symb_requirements)      ||
+     if (defined($symb_requirements)       ||
-         defined($student_requirements)   ||
+         defined($student_requirements)    ||
-         defined($enrollment_requirements) ) {
+         defined($enrollment_requirements) ||
+         defined($group_limits) ) {
          $request .= ' WHERE ';
      }
      if (defined($symb_requirements)) {
- Line 2625  sub get_student_scores {
+ Line 2486  sub get_student_scores {
      return $dataset;
  }
- ################################################
- ################################################
- =pod
- =item &setup_table_names()
- input: course id
- output: none
- Cleans up the package variables for local caching.
- =cut
- ################################################
- ################################################
  sub setup_table_names {
      my ($courseid) = @_;
      if (! defined($courseid)) {
- Line 2659  sub setup_table_names {
+ Line 2505  sub setup_table_names {
          $have_read_student_table = 0;
          undef(%ids_by_student);
          undef(%students_by_id);
+         $have_read_groupnames_table = 0;
+         undef(%ids_by_groupname);
          #
          $current_course = $courseid;
      }
      #
      # Set up database names
-     my $base_id = $courseid;
+     my $base_id = 'md5_'.&Digest::MD5::md5_hex($courseid);
-     $symb_table        = $base_id.'_'.'symb';
+     $symb_table               = $base_id.'_'.'symb';
-     $part_table        = $base_id.'_'.'part';
+     $part_table               = $base_id.'_'.'part';
-     $student_table     = $base_id.'_'.'student';
+     $student_table            = $base_id.'_'.'student';
-     $performance_table = $base_id.'_'.'performance';
+     $groupnames_table         = $base_id.'_'.'groupnames';
-     $parameters_table  = $base_id.'_'.'parameters';
+     $students_groups_table    = $base_id.'_'.'studentgroups';
+     $performance_table        = $base_id.'_'.'performance';
+     $parameters_table         = $base_id.'_'.'parameters';
      $fulldump_part_table      = $base_id.'_'.'partdata';
      $fulldump_response_table  = $base_id.'_'.'responsedata';
      $fulldump_timestamp_table = $base_id.'_'.'timestampdata';
- Line 2679  sub setup_table_names {
+ Line 2529  sub setup_table_names {
                 $symb_table,
                 $part_table,
                 $student_table,
+                $groupnames_table,
+                $students_groups_table,
                 $performance_table,
                 $parameters_table,
                 $fulldump_part_table,
- Line 2689  sub setup_table_names {
+ Line 2541  sub setup_table_names {
      return;
  }
- ################################################
+ sub temp_table_name {
- ################################################
+     my ($courseid,$affix) = @_;
+     my $base_id = 'md5_'.&Digest::MD5::md5_hex($courseid);
- =pod
+     return $base_id.'_'.$affix;
+ }
- =back
- =item End of Local Data Caching Subroutines
- =cut
- ################################################
- ################################################
  } # End scope of table identifiers
- ################################################
- ################################################
- =pod
- =head3 Classlist Subroutines
- =item &get_classlist();
- Retrieve the classist of a given class or of the current class.  Student
- information is returned from the classlist.db file and, if needed,
- from the students environment.
- Optional arguments are $cdom, and $cnum (course domain,
- and course number, respectively).  If either is ommitted the course
- will be taken from the current environment ($env{'request.course.id'},
- $env{'course.'.$cid.'.domain'}, and $env{'course.'.$cid.'.num'}).
- Returns a reference to a hash which contains:
-  keys    '$sname:$sdom'
-  values  [$sdom,$sname,$end,$start,$id,$section,$fullname,$status,$type,$lockedtype]
- The constant values CL_SDOM, CL_SNAME, CL_END, etc. can be used
- as indices into the returned list to future-proof clients against
- changes in the list order.
- =cut
- ################################################
- ################################################
  sub CL_SDOM     { return 0; }
  sub CL_SNAME    { return 1; }
- Line 2746  sub CL_FULLNAME { return 6; }
+ Line 2562  sub CL_FULLNAME { return 6; }
  sub CL_STATUS   { return 7; }
  sub CL_TYPE     { return 8; }
  sub CL_LOCKEDTYPE   { return 9; }
+ sub CL_CREDITS  { return 10; }
+ sub CL_INSTSEC { return 11; }
+ sub CL_GROUP    { return 12; }
+ sub CL_PERMANENTEMAIL { return 13; }
+ sub CL_ROLE     { return 14; }
+ sub CL_EXTENT   { return 15; }
+ sub CL_PHOTO   { return 16; }
+ sub CL_THUMBNAIL { return 17; }
+ sub CL_AUTHORQUOTA { return 18; }
+ sub CL_AUTHORUSAGE { return 19; }
  sub get_classlist {
      my ($cdom,$cnum) = @_;
- Line 2765  sub get_classlist {
+ Line 2591  sub get_classlist {
          }
          my ($sname,$sdom) = split(/:/,$student);
          my @Values = split(/:/,$info);
-         my ($end,$start,$id,$section,$fullname,$type,$lockedtype);
+         my ($end,$start,$id,$section,$fullname,$type,$lockedtype,$credits,$instsec);
          if (@Values > 2) {
-             ($end,$start,$id,$section,$fullname,$type,$lockedtype) = @Values;
+             ($end,$start,$id,$section,$fullname,$type,$lockedtype,$credits,$instsec) = @Values;
          } else { # We have to get the data ourselves
              ($end,$start) = @Values;
              $section = &Apache::lonnet::getsection($sdom,$sname,$cid);
- Line 2802  sub get_classlist {
+ Line 2628  sub get_classlist {
          if(((!$end) || $now < $end) && ((!$start) || ($now > $start))) {
              $status='Active';
          }
+         if(($now < $start) && ((!$end) || $now < $end )) {
+             $status='Future';
+         }
          $classlist{$student} =
-             [$sdom,$sname,$end,$start,$id,$section,$fullname,$status,$type,$lockedtype];
+             [$sdom,$sname,$end,$start,$id,$section,$fullname,$status,$type,
+              $lockedtype,$credits,$instsec];
      }
      if (wantarray()) {
          return (\%classlist,['domain','username','end','start','id',
-                              'section','fullname','status','type','lockedtype']);
+                              'section','fullname','status','type',
+                              'lockedtype','credits','instsec']);
      } else {
          return \%classlist;
      }
  }
+ sub get_group_memberships {
+     my ($classlist,$keylist,$cdom,$cnum) = @_;
+     return ({},{}) if (!ref($classlist) || !ref($keylist));
+     my $cid = $cdom.'_'.$cnum;
+     if (!defined($cdom) || !defined($cnum)) {
+         $cid =  $env{'request.course.id'};
+         $cdom = $env{'course.'.$cid.'.domain'};
+         $cnum = $env{'course.'.$cid.'.num'};
+     }
+     my (%classgroups,%studentgroups);
+     my $now = time;
+     my $access_end = $env{'course.'.$cid.'.default_enrollment_end_date'};
+     my %curr_groups =&Apache::longroup::coursegroups($cdom,$cnum);
+     if (%curr_groups) {
+         my $grpindex = &CL_GROUP();
+         my %groupmemberhash =
+ 	    &Apache::lonnet::get_group_membership($cdom,$cnum);
+         foreach my $student (keys(%{$classlist})) {
+             %{$classgroups{$student}} = ();
+             my $hasgroup = 0;
+             foreach my $status ('previous','future','active','aftercourse') {
+                 %{$classgroups{$student}{$status}} = ();
+             }
+             foreach my $group (keys(%curr_groups)) {
+                 if (defined($groupmemberhash{$group.':'.$student})) {
+                     my ($end,$start) = split(/:/,$groupmemberhash{$group.':'.
+                                                                     $student});
+                     if ($start == -1) {
+                         next;
+                     } else {
+                         $studentgroups{$group} ++;
+                         $hasgroup = 1;
+                         if ($end && $end < $now) {
+                             $classgroups{$student}{'previous'}{$group} =
+                                          $groupmemberhash{$group.':'.$student};
+                             if ($classlist->{$student}[&CL_STATUS()] eq 'Expired') {
+                                 if ($access_end && $access_end < $now) {
+                                     if ($access_end - $end < 86400) {
+                                         $classgroups{$student}{'aftercourse'}{$group} = $groupmemberhash{$group.':'.$student};
+                                     }
+                                 }
+                             }
+                         } elsif ($now > $start) {
+                             if (!$end || $end > $now) {
+                                 $classgroups{$student}{'active'}{$group} =
+                                          $groupmemberhash{$group.':'.$student};
+                             }
+                         } else {
+                             $classgroups{$student}{'future'}{$group} =
+                                          $groupmemberhash{$group.':'.$student};
+                         }
+                     }
+                 }
+             }
+             if (!$hasgroup) {
+                 $studentgroups{'none'} ++;
+             } else {
+                 $classlist->{$student}->[$grpindex] = join(',',
+                               sort(keys(%{$classgroups{$student}{'active'}})));
+             }
+         }
+     }
+     return (\%classgroups,\%studentgroups);
+ }
+ sub get_students_groups {
+     my ($student,$enrollment_status,$classgroups) = @_;
+     my @studentsgroups = ();
+     if (ref($$classgroups{$student}{'active'}) eq 'HASH') {
+         push(@studentsgroups,keys(%{$$classgroups{$student}{'active'}}));
+     }
+     if ($enrollment_status eq 'Any') {
+         foreach my $status ('previous','future') {
+             if (ref($$classgroups{$student}{$status}) eq 'HASH') {
+                 push(@studentsgroups,keys(%{$$classgroups{$student}{$status}}));
+             }
+         }
+     } else {
+         if (ref($$classgroups{$student}{'aftercourse'}) eq 'HASH') {
+             push(@studentsgroups,keys(%{$$classgroups{$student}{'aftercourse'}}));
+         }
+     }
+     return @studentsgroups;
+ }
  # ----- END HELPER FUNCTIONS --------------------------------------------
 ;
  __END__
+ =pod
+ =head1 NAME
+ Apache::loncoursedata
+ =head1 SYNOPSIS
+ Set of functions that download and process student and course information.
+ =head1 PACKAGES USED
+   Apache::lonnet
+   Apache::longroup
+   Time::HiRes
+   Apache::lonmysql
+   LONCAPA
+   Digest::MD5
+ =head1 DOWNLOAD INFORMATION
+ This section contains all the functions that get data from other servers
+ and/or itself.
+ =head1 LOCAL DATA CACHING SUBROUTINES
+ The local caching is done using MySQL.  There is no fall-back implementation
+ if MySQL is not running.
+ The programmers interface is to call &get_current_state() or some other
+ primary interface subroutine (described below).  The internals of this
+ storage system are documented here.
+ There are six tables used to store student performance data (the results of
+ a dumpcurrent).  Each of these tables is created in MySQL with a name of
+ $courseid_*****, where ***** is 'symb', 'part', or whatever is appropriate
+ for the table.  The tables and their purposes are described below.
+ Some notes before we get started.
+ Each table must have a PRIMARY KEY, which is a column or set of columns which
+ will serve to uniquely identify a row of data.  NULL is not allowed!
+ INDEXes work best on integer data.
+ JOIN is used to combine data from many tables into one output.
+ lonmysql.pm is used for some of the interface, specifically the table creation
+ calls.  The inserts are done in bulk by directly calling the database handler.
+ The SELECT ... JOIN statement used to retrieve the data does not have an
+ interface in lonmysql.pm and I shudder at the thought of writing one.
+ =head2 Table Descriptions
+ =over 4
+ =head2 Tables used to store meta information
+ The following tables hold data required to keep track of the current status
+ of a students data in the tables or to look up the students data in the tables.
+ =over 4
+ =item C<$symb_table>
+ The symb_table has two columns.  The first is a 'symb_id' and the second
+ is the text name for the 'symb' (limited to 64k).  The 'symb_id' is generated
+ automatically by MySQL so inserts should be done on this table with an
+ empty first element.  This table has its PRIMARY KEY on the 'symb_id'.
+ =item C<$part_table>
+ The part_table has two columns.  The first is a 'part_id' and the second
+ is the text name for the 'part' (limited to 100 characters).  The 'part_id' is
+ generated automatically by MySQL so inserts should be done on this table with
+ an empty first element.  This table has its PRIMARY KEY on the 'part' (100
+ characters) and a KEY on 'part_id'.
+ =item C<$student_table>
+ The student_table has 7 columns.  The first is a 'student_id' assigned by
+ MySQL.  The second is 'student' which is username:domain.  The third through
+ fifth are 'section', 'status' (enrollment status), and 'classification'
+ (to be used in the future).  The sixth and seventh ('updatetime' and
+ 'fullupdatetime') contain the time of last update and full update of student
+ data.  This table has its PRIMARY KEY on the 'student_id' column and is indexed
+ on 'student', 'section', and 'status'.
+ =item C<$groupnames_table>
+ The groupnames_table has 2 columns.  The first is a 'group_id' assigned by
+ MySQL.  The second is 'groupname' which is the name of the group in the course.
+ =item C<$students_groups_table>
+ The students_groups_table has 2 columns.  The first is the 'student_id', and the
+ second is the 'group_id'. These two columns comprise the PRIMARY KEY for this
+ table, as an individual student may be affiliated with more than one group at
+ any time. This table is indexed on both student_id and group_id.
+ =back
+ =head2 Tables used to store current status data
+ The following tables store data only about the students current status on
+ a problem, meaning only the data related to the last attempt on a problem.
+ =over 4
+ =item C<$performance_table>
+ The performance_table has 9 columns.  The first three are 'symb_id',
+ 'student_id', and 'part_id'.  These comprise the PRIMARY KEY for this table
+ and are directly related to the $symb_table, $student_table, and $part_table
+ described above.  MySQL does better indexing on numeric items than text,
+ so we use these three "index tables".  The remaining columns are
+ 'solved', 'tries', 'awarded', 'award', 'awarddetail', and 'timestamp'.
+ These are either the MySQL type TINYTEXT or various integers ('tries' and
+ 'timestamp').  This table has KEYs of 'student_id' and 'symb_id'.
+ For use of this table, see the functions described below.
+ =item C<$parameters_table>
+ The parameters_table holds the data that does not fit neatly into the
+ performance_table.  The parameters table has four columns: 'symb_id',
+ 'student_id', 'parameter', and 'value'.  'symb_id', 'student_id', and
+ 'parameter' comprise the PRIMARY KEY for this table.  'parameter' is
+ limited to 255 characters.  'value' is limited to 64k characters.
+ =back
+ =head2 Tables used for storing historic data
+ The following tables are used to store almost all of the transactions a student
+ has made on a homework problem.  See loncapa/docs/homework/datastorage for
+ specific information about each of the parameters stored.
+ =over 4
+ =item C<$fulldump_response_table>
+ The response table holds data (documented in loncapa/docs/homework/datastorage)
+ associated with a particular response id which is stored when a student
+ attempts a problem.  The following are the columns of the table, in order:
+ 'symb_id','part_id','response_id','student_id','transaction','tries',
+ 'awarddetail', 'response_specific', 'response_specific_value',
+ 'response_specific_2', 'response_specific_value_2', and 'submission
+ (the text of the students submission).  The primary key is based on the
+ first five columns listed above.
+ =item C<$fulldump_part_table()>
+ The part table holds data (documented in loncapa/docs/homework/datastorage)
+ associated with a particular part id which is stored when a student attempts
+ a problem.  The following are the columns of the table, in order:
+ 'symb_id','part_id','student_id','transaction','tries','award','awarded',
+ and 'previous'.  The primary key is based on the first five columns listed
+ above.
+ =item C<$fulldump_timestamp_table()>
+ The timestamp table holds the timestamps of the transactions which are
+ stored in $fulldump_response_table and $fulldump_part_table.  This data is
+ about both the response and part data.  Columns: 'symb_id','student_id',
+ 'transaction', and 'timestamp'.
+ The primary key is based on the first 3 columns.
+ =item C<$weight_table()>
+ The weight table holds the weight for the problems used in the class.
+ Whereas the weight of a problem can vary by section and student the data
+ here is applied to the class as a whole.
+ Columns: 'symb_id','part_id','response_id','weight'.
+ =back
+ =head1 IMPORTANT SUBROUTINES
+ Here is a brief overview of the subroutines which are likely to be of
+ interest:
+ =over 4
+ =item C<&get_current_state()>
+ programmers interface.
+ =item C<&init_dbs()>
+ table creation
+ =item C<&update_student_data()>
+ data storage calls
+ =item C<&get_student_data_from_performance_cache()>
+ data retrieval
+ =back
+ =head1 OTHER SUBROUTINES
+ =over 4
+ =item C<&make_into_hash($values)>
+ Returns a reference to a hash as described by $values.  $values is
+ assumed to be the result of
+     join(':',map {&escape($_)} %orighash);
+ This is a helper function for get_current_state.
+ =item C<&init_dbs()>
+ Input: course id
+ Output: 0 on success, positive integer on error
+ This routine issues the calls to lonmysql to create the tables used to
+ store student data.
+ item C<&delete_caches()>
+ This routine drops all the tables associated with a course from the
+ MySQL database.
+ Input: course id (optional, determined by environment if omitted)
+ Returns: nothing
+ =item C<&get_part_id()>
+ Get the MySQL id of a problem part string.
+ Input: $part
+ Output: undef on error, integer $part_id on success.
+ =item C<&get_part()>
+ Get the string describing a part from the MySQL id of the problem part.
+ Input: $part_id
+ Output: undef on error, $part string on success.
+ =item C<&get_symb_id()>
+ Get the MySQL id of a symb.
+ Input: $symb
+ Output: undef on error, integer $symb_id on success.
+ =item C<&get_symb()>
+ Get the symb associated with a MySQL symb_id.
+ Input: $symb_id
+ Output: undef on error, $symb on success.
+ =item C<&get_student_id()>
+ Get the MySQL id of a student.
+ Input: $sname, $dom
+ Output: undef on error, integer $student_id on success.
+ =item C<&get_student()>
+ Get student username:domain associated with the MySQL student_id.
+ Input: $student_id
+ Output: undef on error, string $student (username:domain) on success.
+ =item C<&clear_internal_caches()>
+ Causes the internal caches used in get_student_id, get_student,
+ get_symb_id, get_symb, get_part_id, and get_part to be undef'd.
+ Needs to be called before the first operation with the MySQL database
+ for a given Apache request.
+ =item C<&update_full_student_data($sname,$sdom,$courseid)>
+ Does a lonnet::dump on a student to populate the courses tables.
+ Input: $sname, $sdom, $courseid
+ Output: $returnstatus
+ $returnstatus is a string describing any errors that occurred.  'okay' is the
+ default.
+ This subroutine loads a students data using lonnet::dump and inserts
+ it into the MySQL database.  The inserts are done on three tables,
+ $fulldump_response_table, $fulldump_part_table, and $fulldump_timestamp_table.
+ The INSERT calls are made directly by this subroutine, not through lonmysql
+ because we do a 'bulk'insert which takes advantage of MySQLs non-SQL
+ compliant INSERT command to insert multiple rows at a time.
+ If anything has gone wrong during this process, $returnstatus is updated with
+ a description of the error.
+ Once the "fulldump" tables are updated, the tables used for chart and
+ spreadsheet (which hold only the current state of the student on their
+ homework, not historical data) are updated.  If all updates have occurred
+ successfully, $student_table is updated to reflect the time of the update.
+ Notice we do not insert the data and immediately query it.  This means it
+ is possible for there to be data returned this first time that is not
+ available the second time.  CYA.
+ =item C<&update_student_data()>
+ Input: $sname, $sdom, $courseid
+ Output: $returnstatus, \%student_data
+ $returnstatus is a string describing any errors that occurred.  'okay' is the
+ default.
+ \%student_data is the data returned by a call to lonnet::currentdump.
+ This subroutine loads a students data using lonnet::currentdump and inserts
+ it into the MySQL database.  The inserts are done on two tables,
+ $performance_table and $parameters_table.  $parameters_table holds the data
+ that is not included in $performance_table.  See the description of
+ $performance_table elsewhere in this file.  The INSERT calls are made
+ directly by this subroutine, not through lonmysql because we do a 'bulk'
+ insert which takes advantage of MySQLs non-SQL compliant INSERT command to
+ insert multiple rows at a time.  If anything has gone wrong during this
+ process, $returnstatus is updated with a description of the error and
+ \%student_data is returned.
+ Notice we do not insert the data and immediately query it.  This means it
+ is possible for there to be data returned this first time that is not
+ available the second time.  CYA.
+ =item &ensure_tables_are_set_up($courseid)
+ Checks to be sure the MySQL tables for the given class are set up.
+ If $courseid is omitted it will be obtained from the environment.
+ Returns nothing on success and 'error' on failure
+ =item C<&ensure_current_data()>
+ Input: $sname, $sdom, $courseid
+ Output: $status, $data
+ This routine ensures the data for a given student is up to date.
+ The $student_table is queried to determine the time of the last update.
+ If the students data is out of date, &update_student_data() is called.
+ The return values from the call to &update_student_data() are returned.
+ =item C<&ensure_current_full_data($sname,$sdom,$courseid)>
+ Input: $sname, $sdom, $courseid
+ Output: $status
+ This routine ensures the fulldata (the data from a lonnet::dump, not a
+ lonnet::currentdump) for a given student is up to date.
+ The $student_table is queried to determine the time of the last update.
+ If the students fulldata is out of date, &update_full_student_data() is
+ called.
+ The return value from the call to &update_full_student_data() is returned.
+ =item C<&get_student_data_from_performance_cache()>
+ Input: $sname, $sdom, $symb, $courseid
+ Output: hash reference containing the data for the given student.
+ If $symb is undef, all the students data is returned.
+ This routine is the heart of the local caching system.  See the description
+ of $performance_table, $symb_table, $student_table, and $part_table.  The
+ main task is building the MySQL request.  The tables appear in the request
+ in the order in which they should be parsed by MySQL.  When searching
+ on a student the $student_table is used to locate the 'student_id'.  All
+ rows in $performance_table which have a matching 'student_id' are returned,
+ with data from $part_table and $symb_table which match the entries in
+ $performance_table, 'part_id' and 'symb_id'.  When searching on a symb,
+ the $symb_table is processed first, with matching rows grabbed from
+ $performance_table and filled in from $part_table and $student_table in
+ that order.
+ Running 'EXPLAIN ' on the 'SELECT' statements generated can be quite
+ interesting, especially if you play with the order the tables are listed.
+ =item C<&get_current_state()>
+ Input: $sname,$sdom,$symb,$courseid
+ Output: Described below
+ Retrieve the current status of a students performance.  $sname and
+ $sdom are the only required parameters.  If $symb is undef the results
+ of an &Apache::lonnet::currentdump() will be returned.
+ If $courseid is undef it will be retrieved from the environment.
+ The return structure is based on &Apache::lonnet::currentdump.  If
+ $symb is unspecified, all the students data is returned in a hash of
+ the form:
+ (
+   symb1 => { param1 => value1, param2 => value2 ... },
+   symb2 => { param1 => value1, param2 => value2 ... },
+ )
+ If $symb is specified, a hash of
+ (
+   param1 => value1,
+   param2 => value2,
+ )
+ is returned.
+ If no data is found for $symb, or if the student has no performance data,
+ an empty list is returned.
+ =item C<&get_problem_statistics()>
+ Gather data on a given problem.  The database is assumed to be
+ populated and all local caching variables are assumed to be set
+ properly.  This means you need to call &ensure_current_data for
+ the students you are concerned with prior to calling this routine.
+ Inputs: $Sections, Groups, $status, $symb, $part, $courseid, $starttime,
+         $endtime
+ =over 4
+ =item $Sections Array ref containing section names for students.
+ 'all' is allowed to be the first (and only) item in the array.
+ =item $Groups Array ref containing group names for students.
+ 'all' is allowed to be the first (and only) item in the array.
+ =item $status String describing the status of students
+ =item $symb is the symb for the problem.
+ =item $part is the part id you need statistics for
+ =item $courseid is the course id, of course!
+ =item $starttime and $endtime are unix times which to use to limit
+ the statistical data.
+ =back
+ Outputs: See the code for up to date information.  A hash reference is
+ returned.  The hash has the following keys defined:
+ =over 4
+ =item * num_students
+ The number of students attempting the problem
+ =item tries
+ The total number of tries for the students
+ =item max_tries
+ The maximum number of tries taken
+ =item mean_tries
+ The average number of tries
+ =item num_solved T
+ he number of students able to solve the problem
+ =item num_override
+ The number of students whose answer is 'correct_by_override'
+ =item deg_of_diff
+ The degree of difficulty of the problem
+ =item std_tries
+ The standard deviation of the number of tries
+ =item skew_tries
+ The skew of the number of tries
+ =item per_wrong
+ The number of students attempting the problem who were not
+ able to answer it correctly.
+ =back
+ =item C<&populate_weight_table()>
+ =item C<&limit_by_start_end_times()>
+ Build SQL WHERE condition which limits the data collected by the start
+ and end times provided
+ Inputs: $starttime, $endtime, $table
+ Returns: $time_limits
+ =item C<&limit_by_section_and_status()C>
+ Build SQL WHERE condition which limits the data collected by section and
+ student status.
+ Inputs: $Sections (array ref)
+     $enrollment (string: 'any', 'expired', 'active')
+     $tablename The name of the table that holds the student data
+ Returns: $student_requirements,$enrollment_requirements
+ =item C<&limit_by_group()>
+ Build SQL LEFT JOIN statement to include students_groups and groupnames tables and SQL WHERE condition which limits the data collected by group.
+ Inputs: $Groups (array ref)
+     $stutable   The name of the table which holds the student data.
+     $grptable   The name of the table which maps group_id to groupname.
+     $stugrptab  The name of the table which holds student group affiliations.
+ Returns: $groups_join,$group_limits
+    $groups_join  JOIN part of SQL statement (to include group related tables)
+    $group_limits SQL WHERE condition limiting to requested groups
+ =item C<rank_students_by_scores_on_resources()>
+ Inputs:
+     $resources: array ref of hash ref.  Each hash ref needs key 'symb'.
+     $Sections: array ref of sections to include,
+     $Groups: array ref of groups to include.
+     $enrollment: string,
+     $courseid (may be omitted)
+     $starttime (may be omitted)
+     $endtime (may be omitted)
+     $has_award_for (may be omitted)
+ Returns; An array of arrays.  The sub arrays contain a student name and
+ their score on the resources. $starttime and $endtime constrain the
+ list to awards obtained during the given time limits. $has_score_on
+ constrains the list to those students who at least attempted the
+ resource identified by the given symb, which is used to filter out
+ such students for statistics that would be adversely affected by such
+ students.
+ =item C<&get_sum_of_scores>
+ Inputs: $resource (hash ref, needs {'symb'} key),
+ $part, (the part id),
+ $students (array ref, contents of array are scalars holding 'sname:sdom'),
+ $courseid
+ Returns: the sum of the score on the problem part over the students and the
+    maximum possible value for the sum (taken from the weight table).
+ =item C<&score_stats()>
+ Inputs: $Sections, $enrollment, $symbs, $starttime,
+         $endtime, $courseid
+ $Sections, $enrollment, $starttime, $endtime, and $courseid are the same as
+ elsewhere in this module.
+ $symbs is an array ref of symbs
+ Returns: minimum, maximum, mean, s.d., number of students, and maximum
+   possible of student scores on the given resources
+ =item C<&count_stats()>
+ Inputs: $Sections, $Groups, $enrollment, $symbs, $starttime,
+         $endtime, $courseid
+ $Sections, $Groups $enrollment, $starttime, $endtime, and $courseid are the
+ same as elsewhere in this module.
+ $symbs is an array ref of symbs
+ Returns: minimum, maximum, mean, s.d., and number of students
+   of the number of items correct on the given resources
+ =item C<get_student_data()>
+ =item C<&get_student_scores($Sections,$Groups,$Symbs,$enrollment,$courseid)>
+ =item C<&setup_table_names()>
+ input: course id
+ output: none
+ =back
+ =head3 End of Local Data Caching Subroutines
+ =head3 Classlist Subroutines
+ =over
+ =item &get_classlist();
+ Retrieve the classist of a given class or of the current class.  Student
+ information is returned from the classlist.db file and, if needed,
+ from the students environment.
+ Optional arguments are $cdom, and $cnum (course domain,
+ and course number, respectively).  If either is ommitted the course
+ will be taken from the current environment ($env{'request.course.id'},
+ $env{'course.'.$cid.'.domain'}, and $env{'course.'.$cid.'.num'}).
+ Returns a reference to a hash which contains:
+  keys    '$sname:$sdom'
+  values  [$sdom,$sname,$end,$start,$id,$section,$fullname,$status,$type,
+           $lockedtype,$credits,$instsec]
+ The constant values CL_SDOM, CL_SNAME, CL_END, etc. can be used
+ as indices into the returned list to future-proof clients against
+ changes in the list order.
+ =back
+ =cut

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>

Removed from v.1.155
changed lines
	Added in v.1.201.2.4