CVS log for pvfs2/src/server/final-response.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

changing llu handle references to strings

merged in changes from summer at LANL

*** empty log message ***

merging hints/events code into trunk.

reverse merge from HEAD.

reverse merge of changes to HEAD since branch.  Includes small file changes.

Standardize remote vs. local calling convention.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

Reverse merged and ported to HEAD.

hints and events.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

stubbed in state machine to handle refilling precreate pools

stubbed in new request type: batch_create.  The plan is for this to react
like create, except that it can create many handles of the same type (on
one server) in one protocol exchange

merging s_req_resp_type_map[] ordering bug fix from trunk to
small-file-branch

merging s_req_resp_type_map[] table order bug fix from trunk to
pvfs-2-7-branch

bugfix: the LISTATTR request type was listed in the wrong order in the
s_req_resp_type_map[] table.  This made some gossip error messages
confusingly incorrect for about half of the existing request types.

minor change allows state actions to not be declared at the top of *.sm files.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

Cleanup of performance counter code...

move request scheduler load start/stop directly around state machines

Created a better interface for load values in the performance counter.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

Added in help for lock.sm.

add hints to job layer

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

privatize odd struct for server debug messages

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Reverse merges from HEAD..

Move PVFS_MAX_SERVER_OP into the enum and audit places where it was used in comparisons.  Requiring manual adjustment of that value has bugged me for some time.

updates to parallel state machine code that get a basic test working
This test is still included in get_config.sm and needs to be removed.

added some state machine debugging

Everything compiles.   Crap!  Now I have to see if it runs!!!

correcting compile errors

updated to support SM control block interface

Reverse merge from trunk to lockd-branch...

reverse merge of branch

reverse merge of trunk to branch

consolidate all extern declarations of server perf counter

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

changes from head merge

changes to perfbase benchmarks and addition of perf counter queue fields

merging changes from HEAD to dbpf branch

fix for the read request size being sent in small_io operation.  The displayed error this fixes is:  pint_request.c line 89: PINT_process_request: NULL segmax or bytemax!

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

replaced zero-fill code in client io state machine with one that works.  We:

* zero the buffer according to the memory request
* do flow
* calculate return size:
 - check that the ub of the file request is within file data (not a hole or past eof) -> return_size = sizeof(filereq)
 - if that fails, search through retreived datafiles for offset > ub_file_req -> return_size = sizeof(filereq)
 - if that fails, get all the datafiles and search through them for offset > ub_file_req -> return_size = sizeof(filereq)
 - if that fails, calculate correct return size from file req and eof
  * while(next_contig_segment(file_req).ub < eof)
      return_size += sizeof(current_contig_segment(file_req))
    if(current_contig_segment(file_req).lb < eof)
      return_size += (eof - current_contig_segment(file_req).lb)

---

* Added a test for holes and appropriate zero-fill behavior
* Also fixed error reporting for ping to display individual server errors.
* Fixed a bug in PINT_dist_copy

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

added extended attribute support

[pcarns]: do a couple things here:
- implements the retry and timeout configuration file options
( http://www.beowulf-underground.org/pipermail/pvfs2-developers/2005-June/001287.html )

- It turns out that it wasn't necessary for clients to choose a value if
  multiple file systems disagree on timeouts.  Instead they just store
  independent settings for each FS.

- This patch adds --client-job-timeout and --server-job-timeout options to
  genconfig (each toggle both flow and bmi timeouts)

- A few operations were multiplying the default client timeout by 5 on the
  premise that those particular operations may take a while.  I removed those
  special cases and instead increased the default client timeout values for
  everything.  My reasoning is that if any one operation can take a long time,
  then by extension any other operation could take an equally long time because
  of queueing behind the slow ones.

cleanup for extended attribute stuff.  removed non-list server ops
in favor of list type and renamed list type.  cleaned up a few items in
request debug programs.

dded delete extended attribute feature

updates for adding extended attributes

adjustments to gossip output.

- added a client/server operation and mgmt method
  (PVFS_mgmt_get_dirdata_handle/PVFS_imgmt_get_dirdata_handle) for
  retrieving the dirdata handle of a specified 'parent' object (if
  any)
- fixed a crash bug (race) in trove that could return an object on
  queue addition that has already been serviced (since we've signaled
  the trove worker thread before returning the id in the first place);
  should not be a problem if we used safe_register for the ids, but
  otherwise not worth it

- apply Nathan's format specifier patch referenced here:

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-August/000779.html

with slight modification

- fixed typo in the mgmt-remove-object request fill
- added some stubs for mgmt-remove-dirent

- fixed bug in pint-dev's test method that didn't clear the outcount
  on error.  this caused the pvfs2-client-core to get short read
  errors, which were handled gracefully (by reposting the dev unexp),
  but it's better if it doesn't see them at all in that case (since
  there really was nothing read).
- started adding some debugging hooks for printing server response
  types (i only did getattr for now since that's the one that appears to
  be causing trouble)
- misc cleanups

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

server now reference counts bmi addresses

- set a server side timeout on how long it takes to transmit responses; for
  now it is #defined to 30 seconds, should really be a config file option
- still no timeout on flows, need something more clever there

add timeout arguments to bmi and flow related job calls, not used yet

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

formatting.

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

made server side nested machines print out what overall operation they
are a part of
- probably stuck the op->string mapping function in a dumb place; we
  can move it later, I guess...

fixed final_response so that it filters out subsystem specific masks from
the error code before propigating it to the client; this should make
final_response safe to use directly as an error transition out of job failures
in most cases.

crdirent state machine now uses final_response nested machine to handle
sending responses and releasing the scheduler

fleshed out just a little more

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

filled in some content; scheduler release, encode, and send response

added stubs for another nested state machine