CVS log for pvfs2/src/server/pvfs2-server.h
	Help

Request diff between arbitrary revisions / Display revisions graphically

Added debugging of capabilities and credentials. Use security, access and access_detail flags.

Merged robust security code into Orange-Branch.

added new uid management app under src/apps/admin/pvfs2-get-uid
also included are a server and client side sm and the uid mgmt implementation files

merge with Orange-Branch

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

Removed MIRROR_HANDLE_INIT from server startup.
Files affected:
	pvfs2-server.c
	pvfs2-server.h

Committed Windows client code to Orange-Branch.

Moved the definition and storage allocation of MIRROR_HANDLE_INIT to mirror_initialize() in the
server.  This variable only needs to be set once for the duration of a server.
Files affected:
	create-immutable-copies.sm
	pvfs2-server.c
	pvfs2-server.h

changes to PVFS_handle_X

merge with Orange-Branch

distributed directory structure for root and lost+found directory. In pvfs2-server.c, it will check if the root directory is set. Need 10~20 seconds for creation when the servers are launched at first time.

Modified pjmp-machines.sm and pvfs2-server.h to correct a problem where the local remove request,
generated by tree-communication, was not protected by the job scheduler.

Correct dbpf_open_cache_remove bug.

redesigned lookup code to work with security

reverted to Orange-Branch version of lookup code. many modifications will
be needed before the code will compile again.

Commit changes make by Michael in cu-security-branch to fix the incorrect use of accessing two different members of union.

change PINT_server_op variable used by tree_communicate to use a more descriptive name

change PINT_server_op variable used by tree_communicate to use a more descriptive name

Porting job files

add dirdata_handle_rebuild in server remove.

Clean up memory leaks.

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

merge with Orange-Branch

Fixes to lookup.

Partial fix to lookup problems.

Next round of distributed directory changes.

Corrected memory leaks associated with using the PJMP state machine command.
Files affected:
	src/common/misc/state-machine-fns.c
	src/server/create-immutable-copies.sm
	src/server/get-attr.sm
	src/server/list-attr.sm
	src/server/module.mk.in
	src/server/precreate-pool-refiller.sm
	src/server/pvfs2-server.c
	src/server/pvfs2-server.h
	src/server/set-eattr.sm
	src/server/tree-communicate.sm

update to multiple pre-create pool implementation, only create pools for servers of the types they have to give (no I/O handles from meta only servers)

1. add dist-dir-struct to root handle and lost+found handle
2. add dirdata bucket selection to several state machines, however, sys-rename and sys-readdir are not finished yet. There might be other state machines missing.
3. compilable, not expect to run correctly yet.
4. more to change.

incorporate dist-dir-struct to the source tree. It's compilable but not runnable. still have a lot to change and add. Commit as a touch base.

*** empty log message ***

per-dataspace handle type precreate pools implementation

*** empty log message ***

added distributed directory struct and util functions, includes a md5 implementation. First try, more to change.

merging Orange Branch changes in

Pass multiple dirent handles between server and client.

removed unused field in PINT_server_getattr_op

initial merge with Orange-Branch. much will be broken

*** empty log message ***

*** empty log message ***

cleaned up credential verification

moved security types into main pvfs2-types.h header

removed anything and everything related to certificates

revert cu-security-branch to before the attempted merge with Orange-Branch

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

Added code allowing multiple local datahandles in the tree-code.`
 Modified Files:
  Tag: B2O-Merge-Test
 	pvfs2-server.h tree-communicate.sm

Added "grep" operator into pipeline.sm.
Updated the grep test program in the test/active_storage directory.

checkpoint

Merge branches

checkpoint

*** empty log message ***

Added a feature that allows a user to pass k-means related hints (e.g., number of data points, number of coordinates, number of clusters, etc) to the server.

implementation of parent information in keyval databases and changes to get-path to use it

Updates and Fixes.

Merge of Orange-Branch, Orange-FailOver-Becky, and version 1.164 of sys-io.sm (do not unstuff on a read).
This version of Orange-FailOver-Becky supports failover for large files only.  The next iteration will
support small-io failover.

merged in changes from latest Orange-Branch

*** empty log message ***

get path state machine work

Change server struct for get-path request

intermim commit for get-path/get-value implementation

Phase 2 runs without crashing

get-path work, get-value changes to call get-path, dbpf-keyval debugging

state machine implemenation work

merged in changes from summer at LANL

get-path implemenation work

initial get-path state machine and associated changes

*** empty log message ***

Initial commit of parallel fsck code

Merge range query changes and other fixes

Completed most of the getattr changes.  Not all done just yet.

Improved stripe alignment algorithm, especially for the 1st and last stripes.
Increased the BMI buffer size: the default was 16MB

Initial import of branch supporting keyval attribute/value lookup

Completed changes to PVFS_object_attr.  Changes to get-attr.sm are still underday

Revised the algorithm for checking misaligned bytes for each stripe.

Fixed the bugs when there are more than 4 servers performing stripe alignment and allreduce operations.

Fixed a bug in bcast.sm.

Fixed a bug in allreduce.sm by spawning send/recv nested sm concurrently
within allreduce.sm.

Implemented kmeans algorithm based on the state machine, kmeans.sm.
There are two supporting state machines for this: allreduce.sm and bcast.sm.
The simple1_unaligned.c file is updated for testing kmeans.
Sample input file (edge17695.bin) for testing kmeans algorithm is added.

Merged ORANGE-BECKY-BRANCH - Last of mirror changes.

Implemented kmeans.sm with bcast state.

The buffer_size value in pipeline.sm is now from the fs_config,
and the default size is 1MB.
Added a skeleton for the kmeans operation, which is based on the nested state machine.

Split the previous allreduce.sm into send and recv nested state machines using pjmp.
Changed pipeline.sm to setup allreduce.sm in either SEND or RECV type.

Updated the buffer copy method from the nested state machine to parent.

Used PINT_sm_frame(smcb->parent_smcb, 0) to refer parent smcbs in pipeline.sm.
Removed possible double free() in io.sm.

fixed invalid free() in io.sm.

Revised the allreduce.sm and other related codes accordingly.

changes to interface.

Merged Phil's tcp-peer-comm-branch that allows simultaneous server to server communications.
Added basic allreduce module:
  - implemented a recursive distance doubling algorithm.
  - works only when the # of servers are power of 2.
  - when activated, servers simply exchange their id to each other.

TODO
  - handle arbitrary sizes of buffers of various data types.

Modified struct PINT_servre_create_copies_op

commit of active attributes stuff.

*** empty log message ***

*** empty log message ***

Added the server to server communication module within pipeline.sm
to pull the small portion of data when the data is not aligned on strip boundaries.

Changed the default file stripe size to 256KB (originally 64KB)
so that pipeline unit is equal to it.

Setting up Orange Branch

Merged read.sm and write.sm into a single file, pipeline.sm.

This file has been renamed to pipeline.h.

Cleaned the data structures related to flow_d.

Initial version of parallel read/write sms is implemented.

*** empty log message ***

*** empty log message ***

*** empty log message ***

*** empty log message ***

properly cancel bmi unexpected jobs and cleanup unexpected state machines on shutdown.

get rid of sm_p in server-side create_file processing.

*** empty log message ***

Updated to new state machine code.

new server state machine to handle getcred requests

merging hints/events code into trunk.

tree communication implementation

reverse merge from HEAD.

*** empty log message ***

reverse merge of changes to HEAD since branch.  Includes small file changes.

Standardize remote vs. local calling convention.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

dspace, keyval, coll_attr dbs are now created within the environment.
A whole lot of other things try to make replication work

Bugfix in ACL init.

db-replication state machine which processes the incomming replication message

Updated lookup, remove, getattr, create client side.  Comments in lookup server side.  Check code updated for new file creation.  Server cleaned up unnecessary runtime variables.  Remove serverside updated to use capabilities.

Pass the fs_id with the replication message

Reverse merged and ported to HEAD.

reverse merge latest trunk changes to small-file-branch and update msgpair
usage where needed

removed extra prototype to prevent compiler warning

Merge from HEAD.  Fleshed out credential.  Encode/decode added for credential and cleaned up for capability.  Added capability/credential into request/response structures.  Fleshed out comments in security module.

defined read, write, execute, and setattr capabilities

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

fix bug in partitioning of local and remote handles; fill in all remaining
error handling paths (needs further testing)

track local and remote handles through lifetime of the state machine to make
later cleanup easier

preserve error codes in create path before attempting cleanup steps

store complete layout struct (encoded) for stuffed files

add support for LAYOUT_LIST and LAYOUT_RANDOM to create state machine; needs
more testing

*** empty log message ***

reverse merge of latest changes from HEAD to he branch.

added security module initialize/finalize to sever startup/shutdown code. fixed a bug in keystore parsing.

Updated branch with code from HEAD

*** empty log message ***

fix nasty header deps in msgpairarray.  We now can use the smcb code to push a msgpairarray op structure onto the fram before jumping to msgpairarray.  This makes s2s using msgpairarray easier.

adjusting default parameters for precreate

hints and events.

syncing small-file-branch back up with trunk at small-file-branch-point2 tag
(reverse merge)

Push use of named enumerations for request scheduling parameters all the
way down into request-scheduler.c rather than converting to integer flags.
This fixes a bug that allowed server modification operations to proceed
concurrently as if they were read only.

Nested implementation of set-attr.

merged listattr refactoring to small-file-branch; client side readdirplus
still needs adjustment to understand stuffed files properly

re-implement listattr using a parallel jump to the nested getattr machine

some sanity checking on layout type before unstuffing; simplifying the
get_handles process a bit if we are using the normal round robin layout.

Updated get-attr.sm to use the same heuristic that unstuff.sm does to detect
stuffed files.  Also added a state to immediately capture the results of
reading the distribution keyval rather than trying to preserve
s_op->val.buffer all the way through the state machine.

reverting keyval_read_list() consolidation from getattr for the time being
to simplify file stuffing testing/debugging

commit of unstuff code that got missed before.

set up for create-file mod - stubbed out stuff, etc.
a few small bug fixes to statecomp and statemachine stuff

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

readonly symbolic

Change access_type functions from 0/1 to use the enum
PINT_server_req_access_type.  There may have been some backward
conditionals in here.  This at least makes them all visible,
and fixes a problem with ReadOnly exports where only reads
(but not writes) were forbidden.

config file control of some precreate parameters.  rather conservative
default parameters for now

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

persistent storage of precreated handles, more tweaks to allow binary keys

get batch create working (sequentially for now)

Preliminary cut at servers asking each other for handles.  Request is
exchanged, but doesn't do anything yet.  Also some tweaks to how precreate
pool information is assembled at startup.

stubbed in state machine to handle refilling precreate pools

Start an api for keeping up with the precreate pools.  For now all it does
is find the pools on startup or create them if they don't exist.  All
of the API functions are static in pvfs2-server.c for the moment.  Running
with the assumption we will track allocated handles as keyvals.

inode stuffing operation.  also added a keyval_remove_list.

stubbed in new request type: batch_create.  The plan is for this to react
like create, except that it can create many handles of the same type (on
one server) in one protocol exchange

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

Removed lock_id and added lock_node_p.

commit of changes to acl and extended attr checking code.  Hopefully this is will be easier to extend in the future.

Remove errant line noise from Phil patch.

fixup berkeley db invocations to be backward compatible with older versions of bdb

move request scheduler load start/stop directly around state machines

Update migration branch to current CVS version

merge changes from trunk to 2.6 branch: this fix uses the keyval_iterate_keys to see if any entries exist in a directory, instead of looking at the directory count.  This is a workaround in case the directory count becomes invalid somehow (as seen on bgl).

Upgrade to current CVS version

change remove dir check for non-empty dir.  We used to get the count, now we do a keyval iterate with a max count of 1.  If entries exist, the iterate will return a count of 1.  This allows the rmdir to succeed if a dirent count gets out of sync somehow.  dirent counts are only used by getattr now.

reverse merge of HEAD to WALT3 branch.

moved unexpected message handling to a state machine.  reworked sop list
so unexpected messages do not start new server ops after a halt signal.
compiles and executes simple test.

Added in help for lock.sm.

- Support for immutable files
- Cleanups in the pvfs2-bufmap interfaces (replace int with size_t)
- Utility for displaying extended attributes and setting them
- Support for disabling atime updates at the object level, file system level (mount time)
  and directories.

backmerging of HEAD to branch...

* added some bits to gossip that print out the thread id for each debug
message.  This can be set with the threads logstamp.

* fixed bug in dbpf sync coalesce code that wasn't ever syncing and
completing operations that got queued in the ready-to-be-synced state,
but the following operations would error (EEXISTS for example),
so the sync never happened.
* removed all the DBPF_OP_NEEDS_SYNC return codes in dspace and keyval
service functions that modify the db.  The syncing code can figure out
what needs to be synced and what doesn't from now on.

* replace DBPF_OP_IS_* macros that do range checking with logical or checks.
This ends up being easier for me to follow and update.

* crdirent gets posted to the request scheduler with the handle of the
directory (not the dirent handle).  In this case, the operation
on the directory handle is read-only.  Not setting the readonly_flag
for this case was causing a hang with other modifying operations on the
directory (setattr for example).

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Changed behavior of migration: now the client initiates the source servers I/O. This allows concurrent migrations of a metafile's datafiles.
TODO: add another message exchange to modify a files (metadata) datafiles before the file is deleted. Also added (working) eclipse project properties.

Modification for migration added, some functions still need to be adapted after moving to a new branch...

Added request statistics to scheduler, reference from datafiles to
parent metafile (needed for migration) and auto migration tool(unfinished).

Reverse merges from HEAD..

Move PVFS_MAX_SERVER_OP into the enum and audit places where it was used in comparisons.  Requiring manual adjustment of that value has bugged me for some time.

Backmerge with HEAD

Migration of datafiles between different servers works. Uses msgpairray.sm,
transfers only one datafile between two servers and rewrites metadata.
Todo: Adaption of request scheduler to ensure that migration and write
I/O do not happen concurrently on source dataserver.
Also allow users to specify command line parameter which supresses
transfer of pvfs2-hints.

Added tool which prints file mapping to server aliases. Added
experimental stubs for migration tools. Added kernel tgid to request ID.
Bugfix of PINT_cached_config_get_one_server_str.

reverse merge from trunk.  working for now.

Whew.. Final bits of the acl puzzle (hopefully!).
Clearly, ACLs require support at the server. (what was I thinking earlier? :))
Patch adds a PINT_check_acls() function that does that. the pvfs2_acl_entry structure is moved to a common
header file shared between kernel module and server.
When PINT_check_mode() throws up its hand in despair, ACL checks are done. NOTE: ACLs dont impose a performance
overhead when permission checks go through properly. When they fail, we post a job_trove_keyval_read()
and when that completes, we check if ACLs allow the operation to proceed.
Added a lot more debug statements.
It is unfortunately not enough to change only prelude sm, but lookup sm also had to be modified
to check for acls. sigh..

Hopefully, this does not break anything else. Simple preliminary tests dont seem to indicate any borkennes.
All but 2 of the LTP acl tests seem to be working now I think...

Reverse merge changes from HEAD to posix-extensions-branch.

Drain the server pipeline before exiting. (i.e. finish the requests that were being
serviced before exiting)
Patch adds all server s_op structures into one of 2 lists (unexp list or exp list)
At the time of the signal, if the exp list is empty all requests have been serviced and
it is ok to exit, else we wait until those ops have been serviced or they time out.
This is necessary to prevent FS corruption (admittedly a rare occurrence ;))

updated term code, SM union remval and statecomp update for pjmp states
compiles clean, beginning test

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

new state-machine-invoke function for cleaner interface

branch for unified state machine code supporting concurrent SMs

merge of murali's changes from trunk.

[pcarns]: Patch to allow modifying distribution, distribution parameters and number of
data files to use within a given directory. Such setting will only impact creation of future files
within the same directory. It does not change existing files' parameters. All such hints
are inherited by newly created directories as well..
[pcarns]: Allow trove to continue to read a list of xattrs/keyvals
without stopping if some subset of keys dont exist in the keyval db. Modifies the keyval_read_list() function
to return an array of completely independant error codes for each key that is read.
[pcarns]: adjust encode_string() macro to make it safe to call on NULL strings
[pcarns]: bug in the getattr state machine that would cause it to read dirent_count most of the time
regardless of whether the attribute mask requested it or not.
[slang]: fix a couple of bugs in the get-attr server-side state machine whereby the dirent_count was being read
using the dirent handle instead of the dirdata handle. Fixed a return value from dbpf_keyval_get_handle_info_op_svc
so that get-attr would not needlessly hang..
[vilayann]: fixes for the viewdist utility yet again, since the keys for the xattrs have changed...

reverse merge of recent bug fixes from trunk

commit of keyval handle info changes.  This allows info to be stored on a per handle basis in a 'null key' inside the database.  Right now we just use it to store the count of directory entries, so that we don't have to do a read/modify/write in crdirent and rmdirent.

Also includes changes to the keyval strings to use #define values instead of the actual strings.  This just makes that code lest prone to bugs in the future.

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

reverse merge of trunk to branch

apply const to non-modifiable strings

fix deadlock bugs in coalescing code.  add proper state machine debugging.

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

added versioning test.  There's some nasty hacks with set/getenv to be able to test it properly without modifying sysint signatures.  Those will have to be removed at some point.

get branch to build

Added a few more missing files and fixes from HEAD to lockd-branch.
lockd-branch should build cleanly now.

commit of versioning stuff to branch

change distribution storage encode/decode to use wire encoding.

onekeyval merge

fixes to make sure keyvals get remove properly.

reverse merge from trunk.  see if sio tests pass now as well.

get dirent count correct.  Thanks Phil!

if aio threaded isn't supported (or the configure check for libc version fails for some other reason), the non-threaded aio impl doesn't set the result size in the case of reads.  So if the result size doesn't equal the expected bytes found from process_request, we just set the size to the expected bytes for now.

merge from trunk

2 bug fixes:

* fix the readdir code to return the error code to the client instead of assuming that an error just means the directory is empty.

* fix the init-to-zero of dirdata_size for new directories (root dir and lost+found).

[pcarns]: add logic to server side enforcing attributes into a 'supported' name
space (currently: system, user, trusted, security).  Reserved keys live in
'system.pvfs2' instead of top level.  No change to wire or on-disk formats:
server does translation.

forgot to decrement value

keyval_array_size isn't needed/used.

mostly working onekey impl

merging small I/O changes to trunk.  The majority of this commit includes the
new small I/O state machines on client and server, and the modifications to
the client IO state machine.  It also includes mods to the zero-fill code
to perform zero-fill of holes only if they exist instead of zero-ing the entire
memory region initially.  Also, some debug stuff has been added to the
state machine processing code to print enter/exit of state machines if sm
debug is enabled.  The genconfig now includes ioports/metaports for running
multiple servers on the same host for testing.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

small I/O changes in sio branch for now

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

get things compiling again with --enable-fast

updates to my event changes to bring them inline with trunk

merged acache changes to trunk.

set getattr op to complete.  also modified state machine code a little to allow clients to call _machine_locate.  Created static state machine tables for client SMs, and turned PINT_server_req_table into a static table instead of doing all the init dynamically.

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

added extended attribute support

[pcarns]: do a couple things here:
- implements the retry and timeout configuration file options
( http://www.beowulf-underground.org/pipermail/pvfs2-developers/2005-June/001287.html )

- It turns out that it wasn't necessary for clients to choose a value if
  multiple file systems disagree on timeouts.  Instead they just store
  independent settings for each FS.

- This patch adds --client-job-timeout and --server-job-timeout options to
  genconfig (each toggle both flow and bmi timeouts)

- A few operations were multiplying the default client timeout by 5 on the
  premise that those particular operations may take a while.  I removed those
  special cases and instead increased the default client timeout values for
  everything.  My reasoning is that if any one operation can take a long time,
  then by extension any other operation could take an equally long time because
  of queueing behind the slow ones.

cleanup for extended attribute stuff.  removed non-list server ops
in favor of list type and renamed list type.  cleaned up a few items in
request debug programs.

dded delete extended attribute feature

[pcarns]: "This adds a new directory attribute field that indicates how many
entries (of any type) are in a given directory.  The server already had the
ability to gather this information, but it just wasn't being reported.   A
client can read this with getattr(), but it can't be modified directly."

updates for adding extended attributes

Merge in all of phil's patches fixing up permissions checking as described
here:
http://www.parl.clemson.edu/pipermail/pvfs2-developers/2005-June/001258.html

The last of the early june phil patches.  "Make PVFS2 honor directory
executable bit permissions properly"

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

doxygen-related checkins.

- fixed test program ls.c and admin program pvfs2-ls.c to not issue
  an extra call to readdir on every run by fixing a slight logic bug
- changed some readdir debugging
- added a directory version that is passed back from the server to
  the client on each successful readdir call (happens to be the
  directory's mtime encoded as an opaque uint64_t)
- updated client and server response structures and encoder to
  accomodate this
- NOTE: the versioning is not perfect since retrieving the directory
  mtime and iterating the handle list isn't an atomic operation with
  respect to the overall readdir op; i.e. there's still a window where
  a difference may not be detected.  then again, since the mtime is in
  seconds granularity, this should not present a major detectable
  problem anyway.  can use a different version number if problems
  arise often in practice

break primary steps of mkdr sm out into a seperate nested machine to allow
for reuse

- modify server side remove state machine to check the k_size of the
  dirdata object being removed if it's a directory to report early
  back to the client that the directory is not empty (if it's not)
- remove the readdir checks in the client remove path now that the
  server-side remove can tell us directly if a directory is not empty
- modified PVFS specific error routines to print the value of
  unrecognized error codes if encountered
- some cleanups

forgot extern sm declaration in pvfs2-server.h

break core steps of remove operation into a nested machine to enable reuse
elsewhere

- added a PVFS_util method to get the current system and return it as
  a valid PVFS_time type
- added arguments to the crdirent/chdirent/rmdirent operations that
  now can take optional parent atime/mtime/ctime fields
- added support on server *dirent operations that update the parent
  time attributes (if specified) on successful completion of the
  *dirent operation; null/zero time fields are not updated
- modified sys-create/sys-mkdir/sys-symlink/sys-rename/sys-remove to
  work with the new operations properly; in general,
  create/remove/rename operations updates the mtime and ctime of the
  parent directory, which is now properly supported
- moved some common server side attribute handling code into a macro
  to make it re-useable across several server state machines
- updated en{de}coding of modified operations
- misc cleanups

add extern for getattr work nested state machine

break the main work steps of server side getattr operation into a nested
state machine so that they can be easily reused as part of other state
machines if needed

- moved to fully 64 bit unsigned gossip masks
- setparam mgmt method now takes uint64_t values instead of int64_t
- added object type printing to acache debugging
- fixed bug in shared-state-machine getattr that could allow
  attributes to come out of the acache with incorrect attr
  types/fields
- fixed bug is msgpair setup for create/mkdir/symlink
- gossip mask change to uint64_t and many gossip cleanups
- misc cleanups

- extend server side mkdir state machine to create the dirdata dspace
  when creating the directory (rather than waiting for crdirent to
  create it if it doesn't already exist).  this puts the fs in a
  better state for tools like fsck to analyze and moves the work into
  mkdir rather than crdirent (which makes more sense since the dirdata
  object is integral to our directories functioning properly)
- better error handling in mkdir (no longer blow away all error codes
  with -1)

- added a client/server operation and mgmt method
  (PVFS_mgmt_get_dirdata_handle/PVFS_imgmt_get_dirdata_handle) for
  retrieving the dirdata handle of a specified 'parent' object (if
  any)
- fixed a crash bug (race) in trove that could return an object on
  queue addition that has already been serviced (since we've signaled
  the trove worker thread before returning the id in the first place);
  should not be a problem if we used safe_register for the ids, but
  otherwise not worth it

- added a client/server operation for removing a directory entry under
  a specified parent object without prejudice
- modified pvfs2-remove-object to have working options to use that
  functionality

- use the new sm and stop relying on the old remove

- fixed bug in pint-dev's test method that didn't clear the outcount
  on error.  this caused the pvfs2-client-core to get short read
  errors, which were handled gracefully (by reposting the dev unexp),
  but it's better if it doesn't see them at all in that case (since
  there really was nothing read).
- started adding some debugging hooks for printing server response
  types (i only did getattr for now since that's the one that appears to
  be causing trouble)
- misc cleanups

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

added a msgpair stub to the server state machine structure (for single
message exchanges), similar to that found in client state machine structure

typo

modify crdirent state functions so that they operate on op structure fields
rather than request fields; will allow code to be reused for other requests

naming update

add a flag to request parameters structure that indicates if the prelude
state machine should expect to find attributes for the target object

folding pointers to state machines into unified table of request parameters on
server

start consolidating pvfs2 server request parameters into a single table to
make it easier to experiment with new requests; so far table only includes
op type, name, and permission requirements; will include more information later

added fix for warning seen with some compilers during dependency
building step for msgpairarray

break manual state machine launcher function into two parts, so that there
is an opportunity to set fields in state structure before launching the
machine

shuffle declaration of server_state_machine_start_noreq() around so that it
is available for use within other server state machines

link msgpairarray code into server

renaming msgpairarray state machine

initialize cached config interface in the server; convenient for future user
in resolving addresses during inter-server communication

Changing distributions to require initialization before use.  Giving
distributions an additional hook/callback to be called once at registration
time for the distribution to allow it to register its parameters and set
parameter defaults as neccesary.

- added a chdirent server operation useful during renames of target files
  that already exist.
- still working on cleaning up rename, as it does not properly remove some
  data that is left to be removed

add a mechanism for a server to send a generic response back to the client
when it receives a request with a bad version number or encoding type, so
that the user gets a clear error message at the application level too

tested and enabled server side timeouts for flow operations.  For now the
server defaults to this behavior: if a flow goes more than 30 seconds
without moving any data at all, it will be cancelled.  Still no timeouts on
the client side yet; coming soon...

- set a server side timeout on how long it takes to transmit responses; for
  now it is #defined to 30 seconds, should really be a config file option
- still no timeout on flows, need something more clever there

add a server state machine (driven by a request scheduler timer) to check
for jobs that have timed out once per second; still not doing anything
because timeouts are set to inf. for now

- server side get-attr path re-work and cleanups
- added getattr gossip flag specific to the getattr operation

Move from bmi_addr_t to PVFS_BMI_addr_t for consistency.

move the PINT_event_initialize and finalize into the server startup/shutdown

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

remove some more now unnecessary attribute retrievals

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

a more sane lookup implementation that appears to work (at least in prelim
testing) properly across multiple meta-data servers.  cleanups will
follow, as well as additional functionality (such as symlink following).

locally, I've got a 2 meta-server setup that runs pts and all sysint test
programs properly!

added stubs for server mgmt operation to retrieve event log

fix asymmetry on the one odd request type so that struct name, operation enum name, and item name match

filled in request protocol structures for new mgmt function to retrieve
low level dspace information, including encode/decode routines.
- untested
- still need true scheduler support

added a preliminary implementation of an fs dumping utility- right now
just lists all handles from each server (not tested hard yet); fixed
a few bugs along the way

stubbed in server operation for iterate handles

started implementing server request that will retrieve performance
statistics using a mgmt operation

a little bit of progress on perf counting infrastructure, untested

added a state machine that runs continuously in the back ground to
manage performance statistics (doesn't actually do anything yet), along
the way added support for starting state machines that are not
triggered by protocol requests

. landing a new and improved client-side truncate.
. completing the landing of a fully-functioning server-side truncate.

added statfs request type, and a trivial stub for server state machine

another step closer to getting symlink specific attribute data written to
(and able to be read from) disk

scattered server cleanups; most notable change is the dynamic allocation
of the job structures (if less than a hard max)

noop support on the server side

started adding hooks for a NOOP request

An experimental server cleanup attempt.  Things to look for:  Is this cleanup
method cleaner than the server_level_init method?  Is this cleanup correct?
(I'm not convinced the server_level_init stuff was correct, but hey...)
This is based on a brief discussion with RobR using a bit field to determine
what has been initialized so we know what to teardown.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

phew... we now have a PVFS_sys_flush operation on both the client and server
sides.

put a few pieces in place for mgmt_setparam operation, hopefully located
strategically so as not to conflict on merge with the flush...

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

made server side nested machines print out what overall operation they
are a part of
- probably stuck the op->string mapping function in a dumb place; we
  can move it later, I guess...

initial support for "flush"

cleaned up set-attr.sm a little by relying on attributes read by prelude
state machine

simplified rmdirent by making it use attributes read in advance
by prelude nested sm

sortof cleaned up remove state machine by taking advantage of attributes
read in advance by prelude.  Maybe just made things more confusing- will
come back and clean up later...

simplified readdir state machine by taking advantage of attributes
read during prelude

simplified the crdirent state machine by taking advantage of
attributes that have already been read by the prelude state machine

started taking little steps towards getting the prelude state machine to
do simple permission checking: added attr struct to server op struct,
exposed a helper util for deriving the target handle from a given
request structure, started adding states to prelude

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

crdirent state machine now uses final_response nested machine to handle
sending responses and releasing the scheduler

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

integrated nested prelude machine into create as a test case - it works
only with a hack in place for now.  There is a bug that prevents a nested
machine from being used in the first state of a top level machine...

Cleaner still.  And still compiles fine.

Slowly removing typedefs in an attempt to resolve this circular struct issue and make nested state machines work again.

Avoid a memory allocation during server side set-attr by adding scratch
space (an attr object) to the server op structure.

more random stuff :).

Still cleaning up the state machine code so I can reuse it...

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More misc. cleanup.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

Trying to clean up the state machine code so I can move it into the client.  Harder than it should be.

added an initialize and finalize function to the encoding interface; doesn't
do anything yet

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

got rid of some extra typedefs, cleaned up some naming conventions

cleaned up PVFS_ds_keyval type (mainly just naming convention) and moved
definition to pvfs2-storage.h

moved the io state machine specific part of the server_op structure into
a server_io_op struct within the union

last patch for nested state machines - added state stack to server op

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

Remove can now knock out directories, and will handle the case where the dirdata object was never created.

Added additional states to remove state machine to handle datafile removal; because datafiles don't have attributes stored in a
keyval, remove was failing.  Fixed this by falling back to a dspace_getattr() which verifies that it is a datafile before allowing
removal.

Machinations to work around cross-dependencies between state-machine.h and pvfs2-server.h.

Moved "strsize" member into getconfig specific scratch space.

More documentation/cleanup in pvfs2-server, bug fix on lookup.c to address failures in lookup process.  Can return 0 segments now...is that an error?

Documentation and formatting changes.

Major cleanup on rmdirent.  Works.

warning removal.

Cleanup on lookup; removed v_a and k_a values in lookup scratch space that weren't needed.

More cleanup, focus on remove.

Fixed empty directory problems with readdir, some lookup cleanup (needs more).

Cleanup on crdirent.

Reduced amount of data in the lookup-specific structure, relying on req and resp structures instead.

Many modifications to lookup.c; not done yet, but it works as well as if not better than before.

Made the unexpected_bmi_buff member in the server op struct contained within the
actual structure and does not require a seperate malloc.  Updated references to
this in each state machine and to the while(1) main loop, and the statemachine
processor.

Fixed a nasty bug pointed out by robl with concurrent unexpected requests.  The
while(1) loop was updated and comments surround the problem.

Fixed a double free bug inside of the create dirent state machine. (Thanks phil).

completed io state machine and tested some

started working on how to start a flow from inside a server state
machine, but ran into some questions...

Declared a type for the server_level_init enum and changed the
prototype of server_shutdown(...) to take it instead of an int.  Also
added a default server_level_init value (STATUS_UNKNOWN = 0).  Should not
change any functionality.

Attempted to cleanup the littering of "state_level_init++" lines and have
slightly less confusing error handling in the main server body code.  In
theory, functionality should be affected at all by this.

finished name change of unexpected_info -> BMI_unexpected_info and
propigated into server code

COMMENTS!

added some readability changes

Making things readable

Moved union from pvfs2-server.h to state-machine.h
Fixed get attrib to do proper encoding...
Comments!

dw

added in core server code and rules to build it