CVS log for pvfs2/include/pvfs2-sysint.h
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

Added pvfs2-join.c under src/apps/admin dir.

merge with Orange-Branch

updates to user interface code:
all posix wrappers implemented
socket wrappers added
sendfile implemented
sys-lookup modified to return a path on failure
other files are minor formatting changes

uuid converstation, PVFS_x_position changes, server/client side first pass done

move PVFS_ds_position to different types when appropriate, pass a flag around instead of using special canary values in the handle. Things will be broken for a while.

Modifications to perf counter system
Minor formatting and documentation edits

fixed pvfs2-fs-dump.

add --distribution option to pvfs2-genconfig.

local round-robin layouts for Hadoop.

initial merge with Orange-Branch. much will be broken

Preliminary work for distributed directories - making client aware of DIR_ENT_KEY.

*** empty log message ***

*** empty log message ***

moved security types into main pvfs2-types.h header

*** empty log message ***

removed anything and everything related to certificates

revert cu-security-branch to before the attempted merge with Orange-Branch

checkpoint

checkpoint

Merge from HEAD: fix additional parameter passing

Added a feature that allows a user to pass k-means related hints (e.g., number of data points, number of coordinates, number of clusters, etc) to the server.

Merge of Orange-Branch, Orange-FailOver-Becky, and version 1.164 of sys-io.sm (do not unstuff on a read).
This version of Orange-FailOver-Becky supports failover for large files only.  The next iteration will
support small-io failover.

merged in changes from latest Orange-Branch

merged in changes from summer at LANL

query multiple metadata servers per client call

merged Orange with Blue - first merging

Merge range query changes and other fixes

Completed most of the getattr changes.  Not all done just yet.

Initial import of branch supporting keyval attribute/value lookup

Merged changes in Blue branch into Orange branch

Merged Phil's tcp-peer-comm-branch that allows simultaneous server to server communications.
Added basic allreduce module:
  - implemented a recursive distance doubling algorithm.
  - works only when the # of servers are power of 2.
  - when activated, servers simply exchange their id to each other.

TODO
  - handle arbitrary sizes of buffers of various data types.

Added an additional parameter to PVFS_isys_write and PVFS_isys read that is passed to PVFS_isys_io.

Cleaned the data structures related to flow_d.

Kernal code update, getting credentials still needs thought.

completed getcred client state machine

began work on sys-getcred client state machine

merging hints/events code into trunk.

reverse merge from HEAD.

reverse merge of changes to HEAD since branch.  Includes small file changes.

Added new blksize field to system attributes returned by PVFS_sys_getattr().
Modified kernel stat function to return this value as the st_blksize field
in the stat() system call.  blksize is distribution specific and normally
indicates the strip size for the file.

updated all references in include/, src/client/, src/common/, src/io/,
src/proto/, and src/server/ to use the new PVFS_credential in place of the
old PVFS_credentials. the admin apps in particular need to be updated to use
the new API.

Reverse merged and ported to HEAD.

modified getattr client sm to handle credential and capability. compiles
but doesn't actually work yet. for starters, all the apps that call
PVFS_[i]sys_getattr need to be fixed.

*** empty log message ***

Updated branch with code from HEAD

*** empty log message ***

hints and events.

syncing small-file-branch back up with trunk at small-file-branch-point2 tag
(reverse merge)

add system interface hooks to set and get basic options at runtime;
for now only supports ncache and acache timeout options

set up for create-file mod - stubbed out stuff, etc.
a few small bug fixes to statecomp and statemachine stuff

from HEAD: export PVFS_isys_{read,write} routines

get the arguments correct for the isys_read and isys_write macros

We already expose PVFS_isys_io, but not PVFS_isys_read and PVFS_isys_write.
fixed.

new layout parameter for PVFS_[i]sys_create

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

Modified response structure and changed interface.

Update migration branch to current CVS version

Upgrade to current CVS version

Moved PVFS_[sys|mgmt]_release functions to internal.  Added some prototypes and placeholders for PVFS_*_cancel.

reverse merge of HEAD to WALT3 branch.

PVFS_sys_lock interface changes.

Added PVFS_sysresp_lock_s and the PVFS_isys_lock and PVFS_sys_lock prototypes.

- Support for immutable files
- Cleanups in the pvfs2-bufmap interfaces (replace int with size_t)
- Utility for displaying extended attributes and setting them
- Support for disabling atime updates at the object level, file system level (mount time)
  and directories.

backmerging of HEAD to branch...

Modified parent handle reference for datafiles behavior.
Also allow client to get parent handle in getattr sm.

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

apply const to all credentials structs passed in via client interface for correctness and optimization

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Based on RobR, Sam and RobL's suggestions, add a flag to the sys_mntent structure to
enable/disable checking of the config files integrity.
Right now, client-core sets this to 0, effectively skipping the fs.conf integrity checks
and the pvfs2- system interface users (namely ping, etc) set this to 1 by default to enable
the fs.conf integrity checks...
Also fixed a few warnings on x86

PVFS2-hint support + minor changes (verbose request scheduler)

Added PVFS-hint, changed tests for pwrite/odirect/transactions/aio dbpf
 support to allow users to explicitly enable or disable the functionality.
In the aio implementation is a bug since somebody else modified it.

Reverse merge changes from HEAD to posix-extensions-branch.

merge of murali's changes from trunk.

[pcarns]: Patch to allow modifying distribution, distribution parameters and number of
data files to use within a given directory. Such setting will only impact creation of future files
within the same directory. It does not change existing files' parameters. All such hints
are inherited by newly created directories as well..
[pcarns]: Allow trove to continue to read a list of xattrs/keyvals
without stopping if some subset of keys dont exist in the keyval db. Modifies the keyval_read_list() function
to return an array of completely independant error codes for each key that is read.
[pcarns]: adjust encode_string() macro to make it safe to call on NULL strings
[pcarns]: bug in the getattr state machine that would cause it to read dirent_count most of the time
regardless of whether the attribute mask requested it or not.
[slang]: fix a couple of bugs in the get-attr server-side state machine whereby the dirent_count was being read
using the dirent handle instead of the dirdata handle. Fixed a return value from dbpf_keyval_get_handle_info_op_svc
so that get-attr would not needlessly hang..
[vilayann]: fixes for the viewdist utility yet again, since the keys for the xattrs have changed...

Reverse merge from trunk to lockd-branch...

reverse merge of branch

reverse merge of trunk to branch

*** empty log message ***

added testsome,wait, and release functions to external interfaces.

Includes reverse merges from trunk
in addition, it includes fixes for readdir and readdirplus to have a variable length
number of directory entries passed in from client-core using a trailer page (currently).
However, there is nothing that prevents us from sharing those pages with the kernel using vmap
and friends which is the next logical step.
So client-core now does a readdir/readdirplus and writev's the trailer page (which is essentially
an encoded version of the readdir/readdirplus response) to the kmod.
The kernel module decodes that and copies it to the user-space app (which issued the getdents/getdents_plus
system call)
Added/edited the getdents.c test program in the test/posix sub directory to issue
getdents/getdents64/getdents_plus/getdents64_plus system calls
Also fixed a critical bug that was somehow never triggered in readdir if the buffer size
provided by glibc/user was not sufficient enough and we advance the f_pos token beyond where we stopped.

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

Added a few more missing files and fixes from HEAD to lockd-branch.
lockd-branch should build cleanly now.

commit of versioning stuff to branch

[pcarns]: implement mount as a nonblocking routine.  fixes a problem with
multiple mounted file systems where a hanging mount while servicing one file
system would make all other VFS requests hang as well.

[pcarns]: add a nonblocking version of statfs and make pvfs2-client-core use
it.  This addresses a problem with multiple pvfs2 file systems:  one file
system's problems could affect access to other file systems.

added PVFS2_ prefix to ALIGN_VAR

fixes for 32/64 bit issues

meant for it to be like this

change WORDSIZE checks to INTPTR_MIN == INT32_MIN, which is more portable.

commit of murali's 64bit fixes.

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

Murali's patch to add VFS support for extended attributes.

added extended attribute support

cleanup for extended attribute stuff.  removed non-list server ops
in favor of list type and renamed list type.  cleaned up a few items in
request debug programs.

dded delete extended attribute feature

[pcarns]: "This adds a new directory attribute field that indicates how many
entries (of any type) are in a given directory.  The server already had the
ability to gather this information, but it just wasn't being reported.   A
client can read this with getattr(), but it can't be modified directly."

Added num_dfiles as a tabfile option.  Added PVFS_util_get_mntent interface
to pvfs2-util.h, a top level include.

updates for adding extended attributes

more doxygen

more doxygen work.

doxygen-related checkins.

- sysint error code type conversion to PVFS_error (instead of int)

- fixed test program ls.c and admin program pvfs2-ls.c to not issue
  an extra call to readdir on every run by fixing a slight logic bug
- changed some readdir debugging
- added a directory version that is passed back from the server to
  the client on each successful readdir call (happens to be the
  directory's mtime encoded as an opaque uint64_t)
- updated client and server response structures and encoder to
  accomodate this
- NOTE: the versioning is not perfect since retrieving the directory
  mtime and iterating the handle list isn't an atomic operation with
  respect to the overall readdir op; i.e. there's still a window where
  a difference may not be detected.  then again, since the mtime is in
  seconds granularity, this should not present a major detectable
  problem anyway.  can use a different version number if problems
  arise often in practice

add core multihome support

- moved to fully 64 bit unsigned gossip masks
- setparam mgmt method now takes uint64_t values instead of int64_t
- added object type printing to acache debugging
- fixed bug in shared-state-machine getattr that could allow
  attributes to come out of the acache with incorrect attr
  types/fields
- fixed bug is msgpair setup for create/mkdir/symlink
- gossip mask change to uint64_t and many gossip cleanups
- misc cleanups

- some type movement

- move mgmt decls to proper header file

- added a client/server operation for removing a directory entry under
  a specified parent object without prejudice
- modified pvfs2-remove-object to have working options to use that
  functionality

- added credential check to PINT_init_sysint_credentials to make sure
  they're valid before trying to copy (and returning -PVFS_EINVAL if
  they're not)
- added low level server/client mgmt operation to remove a pvfs2
  object
  without prejudice
- added src/apps/admin/pvfs2-remove-object tool to remove a pvfs2
  object using the new mgmt operation  (dirent removal is not yet
  supported, but currently arbitrary handles can be removed properly)
  [ this is a low level tool only useful if you know exactly what
  you're doing ]
- modify pvfs2-fs-dump to not crash if it finds a handle it expected
  to get information on isn't available

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

Moving msgpair retry constants to msgpairarray.h to avoid including sysint
in server code.

move definition of msgpair state structure out of client header
and into generic msgpairarray header, and rename accordingly

- merging in the pvfs2-nm-nb-branch with the main tree
  see ChangeLog for details, or browse the cvs history of the branch
  for full details

NOTE THAT THIS IS A BRANCH COMMIT (tagged as nm-nb-branch).  Feel free
to ignore it completey as it's a snapshot of a work in progress and it
will crash your computer and reformat your hard drive.

This is almost the initial draft of the pvfs2-client-core that
operates in a non-blocking manner.  While it runs, there are still
issues that need to be resolved (that are keeping it out of the main
CVS trunk).  Many other changes were made along the way, so it's more
than just that.

- added compile time option for disabling thread-safety in the client
  library (enabled by default; --disable-thread-safety to disable)
- improved configure summary information emitted at configure time
- added missing non-blocking sysint declarations to sysint header
- re-wrote pvfs2-client-core to use sysint non-blocking operations
  where possible
- made sysint test and testsome() calls more useable from a user point
  of view
- merged dev unexp polling/handling with system interface
- added PINT_sys_dev_unexp call that allows posting unexpected device
  messages so that they can be returned from the sysint testsome
  method in addition to completed sysint operations
- many memory leaks fixed -- many more to go (started adding macros
  for freeing the server response objects)
- added a id_gen_fast_unregister macro that is a no op, to make the
  api more consistent with the id_gen_safe_* calls
- server-config-mgr: report mutex still in use if it is, but also make
  sure not to unlock an already unlocked mutex (valgrind complains)
- many assertions added
- many formatting changes
- many ptr assignments to NULL after freeing in the job interface;
  done while tracking down a problem
- modified the pint-dev device interface to make sure it can handle
  the pvfs2 device in a non-blocking manner
- pint-dev was using buffers larger than it needed across the device;
  fixed them to be the right size
- added a method to free the memory region mapped into the kernel
  through the device
- freed that mapped memory region on pvfs2-client-core shutdown
  (valgrind complained)
- modified device driver to work properly in non-blocking mode from
  userspace by implementing the character device poll method
- modified pint-dev test function to properly handle non-blocking
  responses from the device driver
- modified PINT_flow_reset to not allocate a new mutex unless the old
  one was destroyed
- modified all job uses of the id-generator to use the safe, rather
  than fast, methods (useful for several reasons including safe
  cancellation)
- modified cancellation methods to be able to handle ops that have
  already completed
- modified the job_dev_unexp method to have (and honor) the no
  immediate completion flag if passed (used in the pvfs2-client-core)
- modified the job completion callbacks to make sure to NOT add a
  completed job desc to the completion queue if it's already been
  added (by checking a flag, not scanning).  this is a safety and
  should only be used when a non-thread-safe client library is being
  used in a thread safe env, but we should handle it gracefully anyway
- replaced all kernel allocations of ops through the slab allocator
  handled op_cache to be replaced by a wrapper method (op_alloc(), as
  suggested by Murali) -- the other cache allocations will probably be
  replaced later.  this allows the removal of the extern op_cache
  declaration
- added macros for freeing some of the most heavily used server
  response messages (readdir, lookup, getattr) -- the others are
  coming later
- freed dirents coming out of sysint response object in the
  pvfs2-client

- wrap id-generator code to avoid including id-generator.h directly from
  pvfs2-sysint.h

- implement non-blocking operations for all sysint and mgmt operations
- refactor blocking operations in terms of non-blocking operations
- change all sysint credential arguments to credential references
- memory leaks, bug fixes, cleanups

NOTE: this is a non-blocking first draft, updates are sure to come

Removing erroneous const in PVFS_sys_dist.

- properly set the inode address operations on file open as expected
- replace request_semaphore operations with interruptible operations
- added a macro that can fill in the sys_attr objects when creating
  objects with default values (doesn't require an inode to copy from)
- fix kernel oops on cancelled object creation (file,dir,symlink)
- return more specific error codes in a few places (using translation)
- fix annoying quicklist compile error added a few days ago
- several comment updates
- kernel debugging output cleanups

Adding dist name to PVFS_sys_dist struct.

- some cleanups, error handling, warning removal

Adding interface and stubs to expose distribution interface to clients.

- a start of the msgpair restarts and recv unposts
- needs work and doesn't handle all cases properly

- add a retry flag to each msgpair object -- it's unused now but will be used
  in the msgpair code as an indicator if an operation can be retried or not on
  a communication failure
- some cleanups

added in stub for dist argument to sys_create(), must be set to NULL for
now.  None of the other dist changes are in place, but I wanted to go ahead
and get this in for the next point release so we don't break compatibility
adding the sys_create() arg later

rename PVFS_pinode_reference -> PVFS_object_ref in system interface, a few
related variable name updates

- add a first draft cut of the ability to dynamically add and remove
  mount entries to the system
- add a simple test program that removes and adds some mount entries -- only
  useful if you have a carefully crafted setup (as I do)
- this *might* work, but needs more testing and work

shuffle tab file structure over to utils header now that system interface
isn't bound to it

added hooks to add file systems _after_ system interface initialization,
updated apps accordingly, made pvfs2-ping more verbose using the extra info
that we now have access to.  still not very robust.  here are some
(temporary) limitations:
- client must initialize all compiled in flowprotocols and bmi modules from
  the beginning, no hooks to init them as we go
- can't remove file systems that have been added
- probably can't yet add file systems that didn't come from parse_fstab() call

modified PVFS_util_parse_pvfstab to return a const pointer to a tab
structure managed within the library; user no longer responsible for
allocating or freeing any of it

cleaned out some stale comments, removed a declaration I meant to get rid of
last time

renamed pvfs_mntlist -> PVFS_util_tab, also renamed some of its fields.
This is going to look a little weird until some other updates catch up...

stash the fs id in the mnt structure once it is resolved, will lead to
better mapping later

renamed pvfs_mntent -> PVFS_sys_mntent

- test/client/sysint/ls size fix for symlinks
- test/client/sysint/io-test fix for allowing i/o on symlinks that are metafiles
- test/client/sysint/get-info for showing symlink targets on symlink objs

add PVFS2_ prefix to the LOOKUP_LINK_{,NO_}FOLLOW constants

- added sysint parameter to sys_lookup that dictates if the final
  object on lookup should be resolved if it's a symlink

parse wire encoding method from fstab entry

made system interface level attributes expose datafile count (and
added corresponding attribute mask), dfile count may be reported in
getattr, and used at create time to specify number of datafiles to create.
re-enabled command line option to set number of datafiles in pvfs2-import.

iterater special value cleanups: renamed PVFS2_READDIR_START ->
PVFS_READDIR_START, defined PVFS_ITERATOR_START/END at top level types file,
redifined TROVE equivalents in terms of them, used INT32_MAX rather than
INT_MAX to reflect range of type used for position values

reorganized some code for accessing list of physical servers for a given
file system (will be used in multiple places soon), added mgmt interface hook
to get number of servers, added prototypes for statfs function and mgmt statfs
function (that returns array of stats from all servers), not implemented yet

removed an unused structure definition

code format update

added ability to parse out manually specified flow protocol from tab file,
not doing anything useful with the result yet

cleaning house; removed several struct and function definitions for
non-existant functionality

WARNING: pvfs2tab format has changed- I'll post to list about it in a minute

PVFS_util_parse_pvfstab() cleanup:
- renamed some mnt struct fields
- rewrote parsing, using getmntent() and friends
- removed filename argument to function
- added prioritized searching for entries in PVFSTAB_FILE, /etc/fstab,
  /etc/pvfs2tab, ./pvfs2tab
- changed format of pvfs2tab to be a little cleaner and to fit in nicer
  with system fstab
- corrected the way that tab file options are represented

Finally...symlinks work.  We can store and retrieve the target paths properly,
encode/decode them, whatever.  the only thing left to be done is the lookup
for proper symlink resolution as far as symlinks are concerned. (I think)

started making progress toward getting symlink support.  got sidetracked
along the way trying to work out an easy method to unify sys-create,
sys-symlink, and sys-mkdir all at once.  create should work as normal, but
the rest of the changes shouldn't affect functionality (i.e. symlink doesn't
remotely work, sys-mkdir, isn't there yet).

Put remaining code in place to handle memory datatypes, not ready yet
(pending a fix to some corner case in req processing).  Everything should
still work as it was prior to this commit.  When we do enable memory
datatypes, I'll need to go back and uncomment some code and adjust all
PVFS_sys_{read/write} calls to pass in the right datatype (they are just
taking NULL for now).  PVFS_sys_{read/write} API is in its final form
(I think...) for now, at least.  For now this code is dependent upon
the fact that the file datatype isn't implicitly tiled yet, need to get
memory datatypes going before relaxing that constraint...

changed reference to datatypes called "io_req..." to "file_req...", to make
the distinction clearer between file datatypes and memory datatypes once the
latter is added

Removed all references to PVFS_SYS_IO_{READ,WRITE}, instead using PVFS_IO_xxx versions exclusively.

phew... we now have a PVFS_sys_flush operation on both the client and server
sides.

header shuffling- you probably have to do a distclean to build after
updating