CVS log for pvfs2/src/server/pvfs2-server.c
	Help

Request diff between arbitrary revisions / Display revisions graphically

Added debugging of capabilities and credentials. Use security, access and access_detail flags.

Merged robust security code into Orange-Branch.

added new uid management app under src/apps/admin/pvfs2-get-uid
also included are a server and client side sm and the uid mgmt implementation files

merge with Orange-Branch

update error if segv-backtrace not configured correctly to disable functionality

1. bump up TROVE to version 0.1.6, add a data migration function migrate_collection_0_1_5.
Will create a single dirdata object in distributed directory structure. More dirdata handles will be requested when calling split function.
2. some code cleanup in mgmt-create-root-dir.sm and pvfs2-req-proto.h.

changing llu handle references to strings

adding handle printing files

Handle printing functionality added for client, server, and other utilities

trove/bmi version change, start working through handle changes from server-side file system creation

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

Removed MIRROR_HANDLE_INIT from server startup.
Files affected:
	pvfs2-server.c
	pvfs2-server.h

Committed Windows client code to Orange-Branch.

Moved the definition and storage allocation of MIRROR_HANDLE_INIT to mirror_initialize() in the
server.  This variable only needs to be set once for the duration of a server.
Files affected:
	create-immutable-copies.sm
	pvfs2-server.c
	pvfs2-server.h

uuid converstation, PVFS_x_position changes, server/client side first pass done

change to #defined calls to uuids

migrate from uint64_t type assignments to uuids

fixed stupid break

Modifications to perf counter system
Minor formatting and documentation edits

compiler warning cleanup

compiler warning cleanup

compiler warning cleanup

compiler warning cleanup

don't need that line

pvfs2-server alias option takes an argument

don't try to shutdown the server if the help argument is passed to pvfs2-server

add backtrace handler to SIGABRT

add extra checks during trove cleanup and initialization

define _GNU_SOURCE in CFLAGS in place of __USE_GNU throughout the code. Using the internal glibc macro of __USE_GNU isn't correct and in the case of newer platforms was the cause of backtrace functionality not building correct.

 friendly message after log re-opened

re-open server log file on SIGHUP

merge with Orange-Branch

clean pvfs2-server.c

distributed directory structure for root and lost+found directory. In pvfs2-server.c, it will check if the root directory is set. Need 10~20 seconds for creation when the servers are launched at first time.

remove gossip_err line

remove absolute path restriction on pvfs2-server. get cwd on startup, build full path for user

Added functionality to the SIGHUP feature allowing the debug mask to be reset when SIGHUP signal is received
by the server.  Changes submitted by Bart Taylor from Acxiom.  In addition, the user must now specify a full
path for the pvfs2-fs.conf file when starting the server.  In this way, SIGHUP will always be able to find
the file.

Added DeferSyncToShutdown option, which completely diables any possible
calls to fdatasync or db->sync() while PVFS is running.

merge with Orange-Branch

In the call to trove_collection_lookup() from server_initialize_subsystems(), I passed in the address
of a local variable, orig_fsid, instead of the address of cur_fs->coll_id, because this variable
is modified in the lookup at the same time the trove_thread_function() tries to read it,
causing a data race condition.

Additional changes from merging with Orange-Branch

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

merge with Orange-Branch

initial transactional bdb work

Modified server-state-machine-complete function.  Reversed execution of functions BMI_set_info and
BMI_unexpected_free to prevent a memory leak.  If BMI_set_info removes the bmi addr information
from the cur_ref_list, then BMI_unexpected_free is unable to execute the deallocate function stored
in the bmi addr information.

Next round of distributed directory changes.

update to multiple pre-create pool implementation, only create pools for servers of the types they have to give (no I/O handles from meta only servers)

1. add dist-dir-struct to root handle and lost+found handle
2. add dirdata bucket selection to several state machines, however, sys-rename and sys-readdir are not finished yet. There might be other state machines missing.
3. compilable, not expect to run correctly yet.
4. more to change.

incorporate dist-dir-struct to the source tree. It's compilable but not runnable. still have a lot to change and add. Commit as a touch base.

*** empty log message ***

per-dataspace handle type precreate pools implementation

added distributed directory struct and util functions, includes a md5 implementation. First try, more to change.

merging Orange Branch changes in

initial merge with Orange-Branch. much will be broken

lots of little fixes that have been in the works for a while

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

commiting changes from merge of Orange and Blue into test branch.

merge David Metheny's eventlogging reload on sighup patch from trunk to 2-8

patch from David Metheny to reload event logging mask on sighup

merge double free fix from trunk to 2-8 branch

fixed a double free in one of the server startup failure paths

merging patch from trunk: Bart Taylor's fix for segfault on SIGHUP

patch from Bart Taylor to prevent segfault on SIGHUP

merge Michael's shutdown memory leak fixes from trunk to 2.8

patch from Michael Moore to clean up minor memory leaks on server shutdown

bug and memory fixes to getvalue operation

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:pcarns:20090727185444

merge from HEAD:

patch from David Metheny to allow servers to reload configuration values
related to squashing, read only, and trusted network settings when a HUP
signal is received

Merge with work from this summer.  Metadata and data storage spaces can now be located on different disks.  Fixed a small bug in string encoding.  Updated test code to reflect new Trove dual-storage space API.  Genconfig updated as well as affected admin apps. In the correct branch this time.

Undo accidental commit to MAIN.  Reverted each file to pre-commit version.

Merge with work from this summer.  Metadata and data storage spaces can now be located on different disks.  Fixed a small bug in string encoding.  Updated test code to reflect new Trove dual-storage space API.  Genconfig updated as well as affected admin apps.

merged in changes from summer at LANL

patches from Dries Kimpe:
- Added check before redefining USE_GNU in pvfs2-server.c
- Made use of endecode_fields_xxx macro's consistent and solved warning
in case they were compiled out.  The macro should be used as a function
call, and thus have a semicolumn at the end.

Merged Phil's tcp-peer-comm-branch that allows simultaneous server to server communications.
Added basic allreduce module:
  - implemented a recursive distance doubling algorithm.
  - works only when the # of servers are power of 2.
  - when activated, servers simply exchange their id to each other.

TODO
  - handle arbitrary sizes of buffers of various data types.

initial support for solaris.  doesn't include /dev/epoll support.  See trac ticket #97.

patch from David Metheny to allow servers to reload configuration values
related to squashing, read only, and trusted network settings when a HUP
signal is received

- Change the Trove version to 0.1.4
- Implemented trove_migrate() to update database from 0.1.3 to 0.1.4
- Migration from 0.1.3 to 0.1.4 stats existing bstreams and
  updates the b_size attribute in the database.

properly cancel bmi unexpected jobs and cleanup unexpected state machines on shutdown.

Properly shutdown trove collections that have different methods than the global method.  For example, filesystems specified with directio need to call the directio cleanup to shutdown the threads involved.

update precreation code to handle the case of multiple file systems hosted
on the same pvfs2-server daemon

adding config options for direct io stuff.

trivial freopen() warnings from newer tool chains

fix more warning crankiness in newer tool chains by looking at write() and
freopen() error codes

fix warning

merging hints/events code into trunk.

merge of direct IO code to PVFS HEAD.

reverse merge from HEAD.

reverse merge from HEAD to directio branch

fixed bug in precreation that caused servers to tend to pick the same layout
for datafiles 2 through N:
https://trac.mcs.anl.gov/projects/pvfs/ticket/63

Fix bug in BMI reference counting.  Previously the server was responsible
for both incrementing and decrementing count while working on a state
machine, but this leaves a window of time between completion of
testunexpected() and the start of a state machine.  If a connection was
reset during this window it could cause state machine to try to use
non-existent BMI address, with some odd results.

This patch adds a flag that the server can use to have BMI automatically
increment reference count on any unexpected message before it is passed to
the server for processing.

reverse merge of changes to HEAD since branch.  Includes small file changes.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

dspace, keyval, coll_attr dbs are now created within the environment.
A whole lot of other things try to make replication work

fixes from Phil's comments for timeout values and cond signal/braodcast.

merge trunk updates down to small-file-branch.  Passes basic tests but needs
some double checking of pint-cached-config and sys-create conflicts.

minor tweak to how shm key is generated; incorporate fsid of first file
system in configuration (allows two unrelated servers to both use shm
in bdb on same box as long as they have different fsids)

Change the replication group to be dynamically configurable. Add dbpf_db_reptab_t in dbpf_sync.h. Add dbpf_collection_find_dbreptab in dbpf_collection.c

Add a seperate thread to handle the db_rep_send sm. Add the server db replication sm.

set is_rep_master in config to 2 to indicate no replication for mkspace

db-rep-send state machine added

updated all references in include/, src/client/, src/common/, src/io/,
src/proto/, and src/server/ to use the new PVFS_credential in place of the
old PVFS_credentials. the admin apps in particular need to be updated to use
the new API.

Reverse merged and ported to HEAD.

reverse merge of trunk changes since branch.

mo' changes, bug fixes to hints/events.  Fixed encoding bug for hints.

reverse merge latest trunk changes to small-file-branch and update msgpair
usage where needed

latest changes to tracing/hints code.

Changed coll_env into transactional. Added db_log_buffer_size and db_log_directory as configurable options. Generate_shm_key has an error.

quiet warning

Fixed a bug in the lookup state machine (nlmills).  Moved code around in check.h/.c.  Fixed spacing in server.c.

patch from Nick Mills to make server error out (rather than just log
messages) if configuration file is invalid

improved configuration and error handling. fixed bug in server startup.

reverse merge of latest changes from HEAD to he branch.

added security module initialize/finalize to sever startup/shutdown code. fixed a bug in keystore parsing.

Updated branch with code from HEAD

hints and events.

syncing small-file-branch back up with trunk at small-file-branch-point2 tag
(reverse merge)

removing some duplicate function definitions

fixes the migrate collection utility to work with the single config file format.

fix for broken migrate collection admin tool since unified config file changes.  The migrate collection tool requires the alias to be passed when parsing the config file.

merging various sm fixes from trunk to small-file-branch

yet another op field to remember to set when an sm transitions from an
unexpected machine to a machine for a specific request type

some server side error handling cleanups.  If refiller fails, sleep for a
bit and try again later rather than giving up entirely.  On startup mainly
just log errors and shutdown gracefully; those are crticial
misconfiguration problems.

Make precreate pools safe for split io server / meta server configurations.
Only meta servers are supposed to spool up precreated handles.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

function comments

Cleanup to cached config.  All the functions took a server_config struct, but none of them actually used it (using the cached config instead).  There's a bunch of other cleanup that needs to be done between the server_config_mgr, the config_fetch/get state machines, and the rest of the config code, this is just a small start.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

shuffled some things around

Preliminary cut at servers asking each other for handles.  Request is
exchanged, but doesn't do anything yet.  Also some tweaks to how precreate
pool information is assembled at startup.

stubbed in state machine to handle refilling precreate pools

Start an api for keeping up with the precreate pools.  For now all it does
is find the pools on startup or create them if they don't exist.  All
of the API functions are static in pvfs2-server.c for the moment.  Running
with the assumption we will track allocated handles as keyvals.

merging minor tweaks to db hint fn

minor tweaks to db hint function

merging db bug fix and heartbeat updates from trunk; I may want to reuse
some logic from the former to identify peer servers on startup

merging patches for heartbeat docs and db failover corruption bug fix to
pvfs-2-7-branch
http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-November/003740.html

accepting patch for db corruption after failover:
http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-November/003740.html

inode stuffing operation.  also added a keyval_remove_list.

stubbed in new request type: batch_create.  The plan is for this to react
like create, except that it can create many handles of the same type (on
one server) in one protocol exchange

minor change allows state actions to not be declared at the top of *.sm files.

commit of Murali's unifying config file patch.  Also adding a convert script for converting to the new single fs.conf format.

commit of phil's patches:

pvfs2-error-cleanup.patch:
--------------------------
This patch updates a gossip_debug() message in trove to be a gossip_err() in order to help diagnose configuration problems that can happen if your configuration file is out of sync with your storage space.  It also changes several gossip_lerr() calls in pvfs2-server.c to be gossip_err() calls instead.  Most of those are generic startup/configuration problems that don't really need line or backtrace information.

pvfs2-syslog.patch:
---------------------------
This patch brings the syslog backend of gossip up to working order. There were a few bugs in the implementation.  You can now enable syslog for servers by adding "LogType syslog" to the configuration file.  You can also enable sylog for pvfs2-client by using the "--logtype syslog" command line argument.  The default behavior has not changed.

Let dist free its structures when the server exits.

Merge HEAD changes to TAS-branch.

fix of segfault if versions mismatch.  reported by Kyle.

server SM build warnings

get rid of overly verbose debug msg

This patch fixes the errors seen when the attr cache was enabled with a non-zero timeout.  The problem was that getattr was completing immediately, which the client-core code didn't handle properly.  Now it does (I hope).  Also, the smcb was still being added to the completion list even on immediate completion, and once the client-core code cleaned up the smcb, pulling it off the completion list later and trying to service it was causing memory errors.

To fix it, I added a PINT_state_machine_continue call, which is a wrapper around PINT_state_machine_next, and calls terminate (which then calls the terminate callback, and adds the smcb to the completion list).  This allows us to call PINT_state_machine_next from PINT_state_machine_start, without calling terminate if the state machine completes immediately.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

merge phil's patches from trunk.

patches from phil:  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-April/003300.html

merge recent fixes from HEAD

[phil]: patches from phil.  Committed with mods.  See:

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-March/003255.html

commit of changes to acl and extended attr checking code.  Hopefully this is will be easier to extend in the future.

Upgrade to current CVS version..

make gossip_debug_fp print to stderr with the right type of timestamp the way I want it to.

fix the broken gossip bits.

adding server starting/ready printouts.  Required changes to gossip

Cleanup of performance counter code...

move request scheduler load start/stop directly around state machines

Trove load added. (Number of avg. pending trove operations (meta + data)).

Created a better interface for load values in the performance counter.

Added real load statistics for bmi, flow and trove operations.

Update migration branch to current CVS version

Upgrade to current CVS version

merge latest changes in 2.6 to head.

fix --enable-fast build bug.

 cleaned out some old code, made sure "completed" gets set on
 terminating SMs.

Added -n flag which simplifies startup of multiple servers with mpi.
Therefore, hostname is added to server_conf like pvfs2-genconfig does.

reverse merge of HEAD to WALT3 branch.

commit of trove method capabilities.  We should now be able to support multiple trove methods.  The TroveMethod option to the config file can set the method to something other than the default (dbpf).

moved unexpected message handling to a state machine.  reworked sop list
so unexpected messages do not start new server ops after a halt signal.
compiles and executes simple test.

Added in help for lock.sm.

Allow to start the event logging automatically if start_event_logging is specified in EventLogging list.

Adapt event managment (more to come) for mpe for our needs.
Also add stuff from atop to allow to get detailed system statistics.
Performance counter now logs additional stuff: load, cpu usage, network and disk usage (more possible, thx atop). These values are given to the event managment and loggend in mpe.

backmerging of HEAD to branch...

fixes from 1.5 branch that never made it to trunk.

* added some bits to gossip that print out the thread id for each debug
message.  This can be set with the threads logstamp.

* fixed bug in dbpf sync coalesce code that wasn't ever syncing and
completing operations that got queued in the ready-to-be-synced state,
but the following operations would error (EEXISTS for example),
so the sync never happened.
* removed all the DBPF_OP_NEEDS_SYNC return codes in dspace and keyval
service functions that modify the db.  The syncing code can figure out
what needs to be synced and what doesn't from now on.

* replace DBPF_OP_IS_* macros that do range checking with logical or checks.
This ends up being easier for me to follow and update.

* crdirent gets posted to the request scheduler with the handle of the
directory (not the dirent handle).  In this case, the operation
on the directory handle is read-only.  Not setting the readonly_flag
for this case was causing a hang with other modifying operations on the
directory (setattr for example).

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Merge from HEAD to posix-extensions-branch...
This should facilitate merging the posix branch to HEAD...

Implement root squash, read only and all squash export options. All of these are server-side
settings and set in the fs.conf files.. Perhaps the docs need to be pushed out to the pvfs.org website
after this check in?
Wild cards and mask-based addresses are allowed as part of the fs.conf export settings..

Also added a mount time option to honor setuid bit in files, mount -o suid...

Removed the -c option from pvfs2-ping and made the config file checks default.

[pcarns]:
tacl-xattr-homedir.patch:
-------------------------
This makes tacl-xattr.sh slightly more portable.  Some Linux distributions have adduser utilities that do not create the home directory for you.  This patch
explicitly does a mkdir -p and chmod after adduser to make sure that required home directories exist.

tacl-xattr-symlink.patch:
-------------------------
This is an important fix to the test script.  It was using the -L argument to getfattr to traverse symbolic links when dumping the ACLs from the test directory.
This
leads to unpredictable results because there was no way to tell whether getfattr would traverse a real directory or its symbolic link first (it depends on the
dirent
order) and the other would always be left out.  ACLs are not supported on symbolic links anyway, so it was just adding noise to the test script.

xattr-symlink.patch:
-------------------------
This is the only change to PVFS2 itself.  Like most Linux file systems, PVFS2 does not support xattrs on symbolic links (despite what is implied by man pages).
This
is due to the fact that symbolic links have 777 permissions by default that would allow anyone on the system to store xattrs in any symbolic link.  This patch
updates
PVFS2 semantics slightly, however, to be more in line with how other file systems implement this.  In particular listxattr() is now allowed (it just returns that
there are zero entries), and setxattr() is implemented to return EPERM rather than EOPNOTSUPP.

pread-pwrite.patch:
-------------------
This fixes a bug in a patch that I submitted earlier to provide a simple alternate AIO implementation.  It defines _GNU_SOURCE in a limited area for dbpf so that
we
can get proper definitions of pread() and pwrite() on Linux.  I tried using _XOPEN_SOURCE=500, but it will break any .c file that includes dbpf.h due to
incompatibilities with Berkeley DB.

zero-dfile.patch:
-------------------
This fixes a bug in the getattr handling on pvfs2-server if it happens to find an attribute structure with the dfile array zeroed out.  In this case, it needs to
set
the attr flag appropriately to prevent the response encoder from segfaulting while processing the array in the response structure.  This condition is very hard to
trigger, but the server should be able to gracefully report the error rather than crashing.

bind-specific.patch:
--------------------
This patch adds a new config file option (TCPBindSpecific) that if enabled tells the server to bind only to its specific IP address rather than using INADDR_ANY.
This is particularly helpful in failover scenarios where you would like one physical machine to assume two ip addresses (and two servers) to run simultaneously
when
another server crashes.  Without this patch you would need to select a different port on each server to prevent collisions.

Added patch from migration branch which allows server to add hostname if -n parameter is given on startup. Useful to start server with MPI on a shared homedirectory.

Alternative Trove implementation branch with Trove transaction support for db >= 4.3.

Modification for migration added, some functions still need to be adapted after moving to a new branch...

PVFS2-hint support + minor changes (verbose request scheduler)

Added request statistics to scheduler, reference from datafiles to
parent metafile (needed for migration) and auto migration tool(unfinished).

Reverse merges from HEAD..

updates to parallel state machine code that get a basic test working
This test is still included in get_config.sm and needs to be removed.

Backmerge with HEAD

Allow user to add a the flag -n on server startup which automatically
adds the hostname as prefix to the server_conf. (useful to startup with
MPI on a shared home directory).

Migration of datafiles between different servers works. Uses msgpairray.sm,
transfers only one datafile between two servers and rewrites metadata.
Todo: Adaption of request scheduler to ensure that migration and write
I/O do not happen concurrently on source dataserver.
Also allow users to specify command line parameter which supresses
transfer of pvfs2-hints.

PINT_decode does some inplace decoding, so we need to keep the unexpected buffer around until the state machine completes.

Don't assert on malloc failures!

migration fixes for problems seen on bgl:

1. trove_keyval_write requires all keys to be null terminated, and for the
specified length to include the null terminator.

2. need to check that the return value of translate_keyval_key, to make
sure that the keyval isn't a component filename.

Added bmi unexpected free callbacks to tcp, gm and ib to indicate when unexpected buffers
are done. All they do right now in these methods is to free these buffers.
Move the call to bmi_unexpected_free() earlier (right after the decode is done)
Cleaned up test programs and server to use the new api..
Also cleaned up tcp and gm bmi structures to init structures the c standard way..

Added tool which prints file mapping to server aliases. Added
experimental stubs for migration tools. Added kernel tgid to request ID.
Bugfix of PINT_cached_config_get_one_server_str.

Added PVFS-hint, changed tests for pwrite/odirect/transactions/aio dbpf
 support to allow users to explicitly enable or disable the functionality.
In the aio implementation is a bug since somebody else modified it.

reverse merge from trunk.  working for now.

Whew.. Final bits of the acl puzzle (hopefully!).
Clearly, ACLs require support at the server. (what was I thinking earlier? :))
Patch adds a PINT_check_acls() function that does that. the pvfs2_acl_entry structure is moved to a common
header file shared between kernel module and server.
When PINT_check_mode() throws up its hand in despair, ACL checks are done. NOTE: ACLs dont impose a performance
overhead when permission checks go through properly. When they fail, we post a job_trove_keyval_read()
and when that completes, we check if ACLs allow the operation to proceed.
Added a lot more debug statements.
It is unfortunately not enough to change only prelude sm, but lookup sm also had to be modified
to check for acls. sigh..

Hopefully, this does not break anything else. Simple preliminary tests dont seem to indicate any borkennes.
All but 2 of the LTP acl tests seem to be working now I think...

[phil]: flow-proto-tuning: This patch adds "FlowBufferSizeBytes" and "FlowBuffersPerFlow" options to the configuration file format.  They allow you to specify the buffer size that the default flow protocol will use as well as the maximum number of buffers to use per flow.  Note that if you change either of these parameters, then you need to remount any active clients so that they pick up the configuration change before performing any I/O.

[phil]: max-aio: This patch adds "TroveMaxConcurrentIO" to the configuration file format.  It allows you to specify the maximum number of I/O operations that trove will allow to proceed concurrently (currently 16).  Note from the previous email regarding AIO that depending on your access pattern, AIO may queue all of your operations anyway regardless of this setting.  It probably doesn't have much effect unless you are accessing more than one file at a time, or if you are using an alternative to the stock AIO implementation.

[phil]: alternative lio_listio implementation that spawns a thread and does pwrite/pread.

[phil]: bmi-socket-close: This fixes a bug in the new BMI_set_info(... BMI_TCP_CLOSE_SOCKET ...) mechanism, which is used to reconnect the socket to the initial configuration server if new socket buffer sizes are specified in the config file.  I didn't follow the code path find the exact problem, but at a high level it wasn't being thorough enough in cleaning out the old socket.  This showed up when using epoll and specifying socket buffer sizes in the server configuration- in this case the client will often fail to mount with a cryptic "not a directory" error and leave some epoll() errors in the pvfs2-client.log file.  I think a stale (or possibly reused) file descriptor was being left in the epoll fd set.  At any rate, the fix is to use a different set of functions for tearing down the entire address etc. so that it is reconnected on the next BMI addr lookup.  This path is already used by the server to discard old BMI addresses after critical errors on addresses that cannot be reconnected.  It is triggered from bmi.c without entering the bmi_tcp module, so this patch also adds a check to make sure we don't bother for non-tcp methods.

[phil]: bmi-test-overflow: One of the bmi bandwidth test programs was using types that might overflow if testing large enough transfers.  The fix is to convert to doubles and drop in several type casts to be cautious when performing the computation that was causing trouble.

[phil]: cancel-bugs: The biggest fix here is a change to the job timer code.  It was performing some pointer operations in the wrong order, which could lead to job timers failing to trigger in some cases. This would prevent some operations from ever timing out.  A secondary fix is a minor cleanup in BMI to catch potential race conditions in cancellation where a lock wasn't being held while checking to see if the target operation is complete.

[phil]: flow-post-error: This patch adds checks in the client side I/O state machine to test for failure at post time for flow operations.  This type of error is uncommon unless the flow parameters are faulty, but it should have checked anyway to be safe.

[phil]: dfile-config-check: This is a safety test.  The problem here is that there was no bounds checking for the DefaultNumDFiles option in the config file.  This made it possible to select -1 (which in PVFS1 meant "use the default number").  In PVFS2 this number gets passed verbatim to the client and would cause malloc failures and various other odd results when used. The patch just checks at parse time to make sure the value isn't negative.

[phil]: bmi-flow-logging: This patch just adds a little bit more gossip logging output to BMI and the flow protocol.  In particular, it adds tag information to messages about BMI operations, and adds pointer information to the flow protocol to help match posts and completions of flows in log files.

[phil]: server-freopen: This patch just removes duplicate calls to fdreopen(stdin/stdout/stderr) that were being performed on server startup when the server is run in the background.  The first set should be sufficient.

Reverse merge changes from HEAD to posix-extensions-branch.

Drain the server pipeline before exiting. (i.e. finish the requests that were being
serviced before exiting)
Patch adds all server s_op structures into one of 2 lists (unexp list or exp list)
At the time of the signal, if the exp list is empty all requests have been serviced and
it is ok to exit, else we wait until those ops have been serviced or they time out.
This is necessary to prevent FS corruption (admittedly a rare occurrence ;))

updated term code, SM union remval and statecomp update for pjmp states
compiles clean, beginning test

configure option --enable-aio determines if aio or dbpf-threaded (new)
is used... dbpf-bstream-threaded.c contains threaded I/O version.
This file need some revision... It will be used for testing O_DIRECT.

revert style changes back to previous versions.

run maint/pvfs2indent-80col.sh on all .c files to correct style :)

minor changes

extensive reworking of the state machine interface
smcb_alloc now sets op and locates state machine
smcb_set_op resets op and state machine on existing smcb
state_machine_next loops automatically until the SM ends or hits a
deferred action, state_machine_start first executes a state action, then
does a state_machine_next
this cleaned up a whole slew of things in both client and state machine.
this version compiles and correctly passes the basic tests. - WBL

added some state machine debugging

compiles - about to to return code mod

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

new state-machine-invoke function for cleaner interface

branch for unified state machine code supporting concurrent SMs

reverse merge of recent bug fixes from trunk

commit of keyval handle info changes.  This allows info to be stored on a per handle basis in a 'null key' inside the database.  Right now we just use it to store the count of directory entries, so that we don't have to do a read/modify/write in crdirent and rmdirent.

Also includes changes to the keyval strings to use #define values instead of the actual strings.  This just makes that code lest prone to bugs in the future.

invalid length for keyval entries.

[phil] see http://www.beowulf-underground.org/pipermail/pvfs2-developers/2006-June/002128.html.

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

reverse merge of trunk to branch

apply const to non-modifiable strings

mark two data structures static

consolidate all extern declarations of server perf counter

fix the BMI_set_info calls for send/recv buffer to pass in an actual pointer.  Fix the unused variable warning tcp_set_info.

* [Phil] removed db stat left over from keyval-per-db days
* [Phil] added stranded bstream checking
* [Phil] added comparison function for dspace db
* [Phil] added db cache size and type config options

* [Julian] added tcp send/recv buffer size options

[pcarns]: print "starting" message to stdout and log file.  also do not
truncate log files.

[pcarns]: fix two bugs in performance monitoring path.
- fill in timestamps correctly when multiple timestamps requests
- accumulate metadata values.  other tools will do the math for instantaneous
  resluts.

Includes reverse merges from trunk
in addition, it includes fixes for readdir and readdirplus to have a variable length
number of directory entries passed in from client-core using a trailer page (currently).
However, there is nothing that prevents us from sharing those pages with the kernel using vmap
and friends which is the next logical step.
So client-core now does a readdir/readdirplus and writev's the trailer page (which is essentially
an encoded version of the readdir/readdirplus response) to the kmod.
The kernel module decodes that and copies it to the user-space app (which issued the getdents/getdents_plus
system call)
Added/edited the getdents.c test program in the test/posix sub directory to issue
getdents/getdents64/getdents_plus/getdents64_plus system calls
Also fixed a critical bug that was somehow never triggered in readdir if the buffer size
provided by glibc/user was not sufficient enough and we advance the f_pos token beyond where we stopped.

avoid unattractive cast

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

code to coalesce db syncs.

added versioning test.  There's some nasty hacks with set/getenv to be able to test it properly without modifying sysint signatures.  Those will have to be removed at some point.

*** empty log message ***

Added a few more missing files and fixes from HEAD to lockd-branch.
lockd-branch should build cleanly now.

commit of versioning stuff to branch

when creating or removing a storage space, do not give the impression that
pvfs2-server is still running.

remove duplicate definition.

onekeyval merge

fix merge bug

reverse merge from trunk.  see if sio tests pass now as well.

merge from trunk

[pcarns]: add logic to server side enforcing attributes into a 'supported' name
space (currently: system, user, trusted, security).  Reserved keys live in
'system.pvfs2' instead of top level.  No change to wire or on-disk formats:
server does translation.

changes from head merge

changes to perfbase benchmarks and addition of perf counter queue fields

keyval_array_size isn't needed/used.

missing key string

mostly working onekey impl

[pcarns]: reorder the cleanup steps for pvfs2-server:
	- put trove after flow so any pending flows can complete trove ops
	- shut down perf counters last so that other components can record info

merging changes from HEAD to dbpf branch

generate backtraces on x86_64

Phil's new tcache and the acache changes to match.  Also includes Phil's performance counter changes.  See http://www.beowulf-underground.org/pipermail/pvfs2-developers/2005-December/001704.html.  Thanks Phil!

merging small I/O changes to trunk.  The majority of this commit includes the
new small I/O state machines on client and server, and the modifications to
the client IO state machine.  It also includes mods to the zero-fill code
to perform zero-fill of holes only if they exist instead of zero-ing the entire
memory region initially.  Also, some debug stuff has been added to the
state machine processing code to print enter/exit of state machines if sm
debug is enabled.  The genconfig now includes ioports/metaports for running
multiple servers on the same host for testing.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

[pcarns]: Added safety check and verbose error message for case in which server
is started with a configuration file that does not match the storage space (in
particular, if the fsid's do not match)

small I/O changes in sio branch for now

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

Attached patches implements trusted ports and networks similar to
what PVFS1 implemented.
It adds a new <Security> </Security> tag to the fs.conf config file
and parse utilities accordingly. We can edit these later if there
is a strong need.

Within these tags, you have to specify TrustedPorts <Range of ports> and
TrustedNetwork tcp://<network> tcp://<mask>.
I dont know how to support this cleanly on  non tcp/ip bmi
architectures, so currently this only enables it for the bmi_tcp module
alone. (i guess we could do it for bmi_ib module as well but not done
until someone says that is the right thing to do...)

Also adds support for a --trusted argument to the pvfs2-genconfig script
that will prompt for the trusted port ranges and the network address/masks and write it out
to the fs.conf and server.conf file.

Configure now supports a --enable-trusted to enable all these features.
(NOTE that the fs.conf tags (or lack of one) still allow you to not
select these features even if configure was asked to compile with this
option).

this patch also cleans up all the warnings that arises with gcc
(3.4) and --enable-strict build mode.

append to the pvfs2 server log file, instead of overwriting

one last fix for error handling: if we can't open the file, return.

if the file does not exist, access(2) doesn't do what i wanted

log any logfile-related errors before we redirect stdin/stdout/stderr

[pcarns]: open the server log file earlier.  If something bad happens, we now
get an error at the command line.

updates to my event changes to bring them inline with trunk

merged acache changes to trunk.

set getattr op to complete.  also modified state machine code a little to allow clients to call _machine_locate.  Created static state machine tables for client SMs, and turned PINT_server_req_table into a static table instead of doing all the init dynamically.

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]: add an additional logging statement when the server starts up

[pcarns]: must use absolute path for pidfile or server will lose track of it on
chroot

added extended attribute support

[pcarns]: make all error paths return error codes.  reduce the number of
functions that call exit, making them instead return an error code and
funneling the results to a single exit point.

cleanup for extended attribute stuff.  removed non-list server ops
in favor of list type and renamed list type.  cleaned up a few items in
request debug programs.

dded delete extended attribute feature

Added support for Filesystem config option DefaultNumDFiles and Distribution
configuration sections.

Also performed simple cleanup for simple_stripe distribution.

Applied Phil Carns patch to improve the dot conf parser error messages and
type checking.

updates for adding extended attributes

Merge in all of phil's patches fixing up permissions checking as described
here:
http://www.parl.clemson.edu/pipermail/pvfs2-developers/2005-June/001258.html

The last of the early june phil patches.  "Make PVFS2 honor directory
executable bit permissions properly"

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

always write logfile from scratch, do not append

pcarns:
This patch adds the ability to time stamp log messages with a date and
time rather than the default hr:min:sec:usec format.  This is useful for
production systems if you are more worried about the approximate time
that an error occurred rather than debugging performance problems.

The time stamp format is controlled by a new server config file variable
called "LogStamp".  It has three valid values: "none", "usec", or
"datetime", which correspond to the following formats respectively:

Initialization completed successfully.

[20:10:46.631851] Initialization completed successfully.

[02/25 20:10] Initialization completed successfully.

eliminate global src/server include

- make sure that our error routines strip any known class error bits
  instead of deciding that they're unrecognized if set
- improved trove error handling
- improved pvfs2-server error handling
- fixed crash bug in our config parser that tried to strdup strings
  that might be NULL
- pvfs2-types header file simplification and cleanups

- cosmetic changes due to timestamped logging

- added a PVFS_util method to get the current system and return it as
  a valid PVFS_time type
- added arguments to the crdirent/chdirent/rmdirent operations that
  now can take optional parent atime/mtime/ctime fields
- added support on server *dirent operations that update the parent
  time attributes (if specified) on successful completion of the
  *dirent operation; null/zero time fields are not updated
- modified sys-create/sys-mkdir/sys-symlink/sys-rename/sys-remove to
  work with the new operations properly; in general,
  create/remove/rename operations updates the mtime and ctime of the
  parent directory, which is now properly supported
- moved some common server side attribute handling code into a macro
  to make it re-useable across several server state machines
- updated en{de}coding of modified operations
- misc cleanups

- moved to fully 64 bit unsigned gossip masks
- setparam mgmt method now takes uint64_t values instead of int64_t
- added object type printing to acache debugging
- fixed bug in shared-state-machine getattr that could allow
  attributes to come out of the acache with incorrect attr
  types/fields
- fixed bug is msgpair setup for create/mkdir/symlink
- gossip mask change to uint64_t and many gossip cleanups
- misc cleanups

add a few safety checks to server_state_machine_complete() so that it is
safe to call for server state machines that are not launched by an incoming
request

- added a client/server operation and mgmt method
  (PVFS_mgmt_get_dirdata_handle/PVFS_imgmt_get_dirdata_handle) for
  retrieving the dirdata handle of a specified 'parent' object (if
  any)
- fixed a crash bug (race) in trove that could return an object on
  queue addition that has already been serviced (since we've signaled
  the trove worker thread before returning the id in the first place);
  should not be a problem if we used safe_register for the ids, but
  otherwise not worth it

split pidfile creation to allow relative-path pidfiles to work

- fixed logging format

- added configuration option TroveSyncMeta (can be 'yes' or 'no')
- added configuration option TroveSyncData (can be 'yes' or 'no')
- replaced TroveSyncMode configuration option with separate
  TroveSyncMeta and TroveSyncData options.  the defaults are yes to
  both.  this is a change because previously data was NOT being sync'd
  after reads/writes.  for now, only writes are sync'd if
  TroveSyncData is 'yes'.
- updated pvfs2-genconfig to reflect config option changes
- added top-level and flowproto-multiqueue understanding of the
  data sync mode option via setinfo
- added hook to server to pass data sync mode option to the flow
  implementation via setinfo on a per fs/coll_id basis
- added fp_multiqueue tracking of this to be able to pass the
  appropriate data sync flag to trove on a per fs basis
- misc cleanups

create pidfile after fork() else atexit() happens and wrong pid recorded

- added a client/server operation for removing a directory entry under
  a specified parent object without prejudice
- modified pvfs2-remove-object to have working options to use that
  functionality

- use the new sm and stop relying on the old remove

bugfix: use PVFS_perror_gossip() instead of strerror; the latter doesn't
understand pvfs2 error codes.  Also make formatting of a few pvfs2-server.c
level error messages a little more uniform.

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- pvfs2-server cleanups, including better error handling in many
  places
- organize command line options into a single struct to avoid growing
  number of declared static vars in pvfs2-server
- fix the pidfile usage (which didn't work at all for me, particularly
  using the -p option that getopt_long couldn't possibly have
  understood);  now you can do the standard 'kill `cat
  pvfs2-server.pidfile`' to properly terminate the server
- remove broken url from comment and just state the problem and refer
  to the thread in question

remember to use the right enumeration type so the compiler can help with
warnings for new operations

add a flag to request parameters structure that indicates if the prelude
state machine should expect to find attributes for the target object

folding pointers to state machines into unified table of request parameters on
server

start consolidating pvfs2 server request parameters into a single table to
make it easier to experiment with new requests; so far table only includes
op type, name, and permission requirements; will include more information later

break manual state machine launcher function into two parts, so that there
is an opportunity to set fields in state structure before launching the
machine

shuffle declaration of server_state_machine_start_noreq() around so that it
is available for use within other server state machines

initialize cached config interface in the server; convenient for future user
in resolving addresses during inter-server communication

add a pidfile option to pvfs2-server for integration with init scripts

Changing distributions to require initialization before use.  Giving
distributions an additional hook/callback to be called once at registration
time for the distribution to allow it to register its parameters and set
parameter defaults as neccesary.

- added a chdirent server operation useful during renames of target files
  that already exist.
- still working on cleaning up rename, as it does not properly remove some
  data that is left to be removed

add a mechanism for a server to send a generic response back to the client
when it receives a request with a bad version number or encoding type, so
that the user gets a clear error message at the application level too

Make error messages about protocol version / format errors more specific and
clear.  Now looks something like this on server side:

Error: poorly formatted protocol message received.
   Protocol version mismatch: received version 103 when expecting version
   104.  Please verify your PVFS2 installation and make sure that the version
   is consistent.
server_state_machine_start: Protocol error

add a server state machine (driven by a request scheduler timer) to check
for jobs that have timed out once per second; still not doing anything
because timeouts are set to inf. for now

added event logging instrumentation that tracks the beginning and end of
each full server state machine

- compile fix

- return decode errors if we get them

- keep the server shutdown logging pretty

back to catching SIGINT and SIGQUIT, change behavior of SIGPIPE and SIGUSR
so that they are ignored and not logged, make sure we only handle one of
each type of signal if we get to shutdown path

whoops- maybe not such a swell idea :)  removing these signals again for the
moment

what the heck, catch SIGQUIT too (generated by ctrl backslash by default)

catch SIGINT so that we are graceful when ctrl-c'd

- extend the job interface to take a flags argument, which is passed to various
  trove operations, rather than hard-coding TROVE_SYNC
- make changes to server to pass in the mode value matching the config option
- update test programs (hardcode TROVE_SYNC)

- renamed handle purgatory to handle recycle timeout seconds
- added a trove sync mode to the config to use for the default trove sync
  flags (not hooked up to job interface yet)
- replaced gossip_lerr in the config parser with gossip_err
- set default encoder at config parse time
- update pvfs2-genconfig with support for new keywords

move the PINT_event_initialize and finalize into the server startup/shutdown

- cleanup the storage mode string (aio-threaded,threaded,non-threaded)
- make the storage mode string displayed when the version is requested
- made a more direct exit when the server is only asked for the version

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

trivial typo

- okay, fix backgrounded server due to popular demand

- make sure server prints error message to stderr instead of to log file
  if it fails during startup (such as if storage space doesn't exist)
- added some shutdown server debugging, if enabled to help discover hangs

assign the "server_controlling_pid" after we daemonize

- fix pvfs2-server's fds.  I recently broke them
- cosmetic fix

- changelog version bump
- configure version bump
- some server cleanups

- limit signal handling to one thread in thread libraries that all get the
  same signal handler for each thread

make sure that the segv handler (if segv backtracing enabled) cannot be
triggered more than once

- clean up pvfs2-server command line options and offer long alternatives

added --enable-segv-backtrace option to configure, turns on server side
behavior of dumping a stack trace if a seg fault is received

make sure that SIGPIPE, SIGUSR1, and SIGUSR2 are all noted in logs but
don't hurt server, removed old signal() call not needed any more

exit a bit cleaner on segfault

listing signals once is probably enough :)

replaced signal() with sigaction(), necessary to hook into some signal
debugging stuff that I want to use.  Looks like we need to fix a bug or two
here as well...

um...one semi-colon will do, though there's really no global shortage of them.

This is a full merge of my pvfs2-dbpf-attr-cache-branch, so this is
the cumulative changelog (based on existing cvs messages).

- initial dbpf-attr-cache implementation and cleanups
- added all the hooks to grab configurable cacheable metadata keys
  from a config file.  re-arranged the config file by separating out
  fs specific hints into a StorageHints area.  updated genconfig to
  account for these changes.
- initial working implementation of the unified dbpf-attr and
  keyval/data pair cache.  holy cow this thing is fast... much work to
  be done still.
- initial locking added
- replaced debug level with a new DBPF_ATTRCACHE_DEBUG level
- added safe to/from cache data copies
- added write-through caching for keyval data (still need to do generic attrs)
- updated simple.conf
- allow config file to have multiple AttrCacheKeywords lines
- fix some cache parts that allow symlinks to work (i.e. be cached)
- added a verbose warning for a particular kind of configuration error
  for a user to see if it should happen
- some more robust configure checks
- more configure related changes
  disable vis build if sdl is detected, but sdl-ttf isn't
- fix broken ifdef nesting that caused compilation failure on non
  aio-threaded configurations
- safe copy of attributes (to/from, preparing for write-through impl)
- server cleanups (making way for pre-caching of a number of attrs on server
  startup since it reads them from disk during handle iterate anyway)
- update create test program to report human readable creation error
- put some quick human readable error reporting in pvfs2-client-core
- added some quickhash method we've been wanting but never had
- implemented policy to limit previously unbounded size of attr cache
- adjusted simple.conf to have some good defaults
- adjusted pvfs2-genconfig to match those defaults
- fixed some htable cleanups
- started write-through attr updates
- update changelog and version info, preparing for mainline merge
- implement write-through of dbpf-resize operation
- make sys-truncate return -PVFS_EINVAL if truncate is called on an object
  that can't be truncated
- fix error code return value of sys-truncate
- cosmetic changes to truncate test program output
- added some comments
- updated showconfig with copyright info
- updated sample configs for showconfig
- added a default handle purgatory value if a value of zero is specified
  (e.g. not specified in config file at all)
- for now, remove the cached dbpf-attr object on i/o writes
- slightly robustify the glibc aio callback detection

- safe copy of attributes (to/from, preparing for write-through impl)
- server cleanups (making way for pre-caching of a number of attrs on server
  startup since it reads them from disk during handle iterate anyway)
- bumped up dbcache fd value from 16 to 1024
- misc cleanups

added all the hooks to grab configurable cacheable metadata keys from a config
file.  re-arranged the config file by separating out fs specific hints into
a StorageHints area.  updated genconfig to account for these changes.

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

added an opposite equivalent of pvfs2_mkspace called pvfs2_rmspace that tears
down a complete filesystem.  it can be invoked by running the server with a
-r option.

take two of the previous patch.

bugfix; nathan reported running server with no args can show chowdered output
when we try to parse the config file with options that aren't valid

exit() immediately if we catch a segfault

added stubs for server mgmt operation to retrieve event log

fix asymmetry on the one odd request type so that struct name, operation enum name, and item name match

configure time option (--disable-perf-counters) that can turn off the
server side performance counters

added handlers for more signals; even if we don't do anything clever
with them yet, we should at least note their presence in the logs...

filled in request protocol structures for new mgmt function to retrieve
low level dspace information, including encode/decode routines.
- untested
- still need true scheduler support

seperated/renamed variables/#defines that controls how many jobs we test for
at a time- number of initial unexpected jobs is now independent of the
number of jobs that we will test for at any given time...

added a preliminary implementation of an fs dumping utility- right now
just lists all handles from each server (not tested hard yet); fixed
a few bugs along the way

started implementing server request that will retrieve performance
statistics using a mgmt operation

a little bit of progress on perf counting infrastructure, untested

variables named "ret" being used for anything besides int return values
make my brain hurt

added a state machine that runs continuously in the back ground to
manage performance statistics (doesn't actually do anything yet), along
the way added support for starting state machines that are not
triggered by protocol requests

cleaned up signal handling slightly- added non fatal handler for sigpipe

more consistently use typedef for PINT_llist

a new config file option (HandlePurgatory) will give us some control over how
long freed handles wait before being returned to the system.

renamed all llist_XXX functions to PINT_llist_XXX, first step in fixing
most obvious name space problems (those that collide with pvfs1)

started on an admin tool that shows statfs-ish type stuff, not finished
yet

ugly hack for valgrind snuck in on the truncate bus.  sorry about that.

. landing a new and improved client-side truncate.
. completing the landing of a fully-functioning server-side truncate.

don't clear dist info on i/o write; remove broken statfs stubs (Phil: did you
forget to add server/statfs.??)

added statfs request type, and a trivial stub for server state machine

added a "FlowModules" config file option (mandatory) that tells the
server what flowprotocols to initialize

added a manditory server config file option (BMIModules) that lists
which bmi modules the server should initialize, also fixed a bug in
BMI's parsing of module list

add server startup banner featuring the version number

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

another step closer to getting symlink specific attribute data written to
(and able to be read from) disk

- added pvfs2-mkspace and pvfs2-showcoll for removal on make clean
- added versioning information to the configure setup so apps can know
  which version to print if they care (i.e. pvfs2-ls, pvfs2-ping, etc)
  NOTE: I've arbitrarily picked the version start of 0.0.1 -- if we
  don't like this version for the friendly release, be sure to update
  the version in configure.in
- declare the strnlen prototype in pvfs2-showcoll, as the warning seems
  to have re-appeared; fixed for good now

scattered server cleanups; most notable change is the dynamic allocation
of the job structures (if less than a hard max)

move mtrace call to server if enabled.

- implemented job_trove_dspace_verify
- implemented ROOT_CHECK parameter in admin interface
- finished up pvfs2-ping and marked "NEEDS TESTING" in todo file

temporarily disabling flowproto_dump_offsets

minor usage correction

noop support on the server side

server now prints out some useful messages and continues to operate if a
bogus request is received.

first steps toward getting the server to gracefully handle bogus requests