CVS log for pvfs2/src/io/trove/trove.c
	Help

Request diff between arbitrary revisions / Display revisions graphically

initial working versionof usrint code
auto configures usrint on an ucache off (not working yet)
new acl repimplemented - old one has #def
some minor format edits

uuid converstation, PVFS_x_position changes, server/client side first pass done

checkpoint on the uuid road

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

merge with Orange-Branch

Bump Trove DBPF version for new precreate-handle storage format. Include migration to 0.1.5. Add job_trove_fs_del_eattr() function and associated lower level calls

bug and memory fixes to getvalue operation

checkpoint

checkpoint

merged in changes from summer at LANL

get-path implemenation work

initial get-path state machine and associated changes

Merge range query changes and other fixes

Initial import of branch supporting keyval attribute/value lookup

merging hints/events code into trunk.

reverse merge from HEAD.

reverse merge of changes to HEAD since branch.  Includes small file changes.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

dspace, keyval, coll_attr dbs are now created within the environment.
A whole lot of other things try to make replication work

db-replication state machine which processes the incomming replication message

Add a seperate thread to handle the db_rep_send sm. Add the server db replication sm.

db-rep-send state machine added

Reverse merged and ported to HEAD.

grouped transaction, checkpoint thread

Fixed a bug of setting the db log directory. Added an option in mpi-md-test to test different clinets open different files.

use transactional db. db_log_buffer_size and db_log_directory are added into configuration options. shm_key has an error

dspace_remove_list operation for trove; untested

hints and events.

a completely untested dspace_create_list() operation to allow multiple
handles to be created with one trove call

persistent storage of precreated handles, more tweaks to allow binary keys

inode stuffing operation.  also added a keyval_remove_list.

Created a better interface for load values in the performance counter.

Update migration branch to current CVS version

Upgrade to current CVS version

reverse merge of HEAD to WALT3 branch.

commit of trove method capabilities.  We should now be able to support multiple trove methods.  The TroveMethod option to the config file can set the method to something other than the default (dbpf).

hint is given down to Trove

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Reverse merges from HEAD..

reverse merge from trunk.  working for now.

[phil]: flow-proto-tuning: This patch adds "FlowBufferSizeBytes" and "FlowBuffersPerFlow" options to the configuration file format.  They allow you to specify the buffer size that the default flow protocol will use as well as the maximum number of buffers to use per flow.  Note that if you change either of these parameters, then you need to remount any active clients so that they pick up the configuration change before performing any I/O.

[phil]: max-aio: This patch adds "TroveMaxConcurrentIO" to the configuration file format.  It allows you to specify the maximum number of I/O operations that trove will allow to proceed concurrently (currently 16).  Note from the previous email regarding AIO that depending on your access pattern, AIO may queue all of your operations anyway regardless of this setting.  It probably doesn't have much effect unless you are accessing more than one file at a time, or if you are using an alternative to the stock AIO implementation.

[phil]: alternative lio_listio implementation that spawns a thread and does pwrite/pread.

[phil]: bmi-socket-close: This fixes a bug in the new BMI_set_info(... BMI_TCP_CLOSE_SOCKET ...) mechanism, which is used to reconnect the socket to the initial configuration server if new socket buffer sizes are specified in the config file.  I didn't follow the code path find the exact problem, but at a high level it wasn't being thorough enough in cleaning out the old socket.  This showed up when using epoll and specifying socket buffer sizes in the server configuration- in this case the client will often fail to mount with a cryptic "not a directory" error and leave some epoll() errors in the pvfs2-client.log file.  I think a stale (or possibly reused) file descriptor was being left in the epoll fd set.  At any rate, the fix is to use a different set of functions for tearing down the entire address etc. so that it is reconnected on the next BMI addr lookup.  This path is already used by the server to discard old BMI addresses after critical errors on addresses that cannot be reconnected.  It is triggered from bmi.c without entering the bmi_tcp module, so this patch also adds a check to make sure we don't bother for non-tcp methods.

[phil]: bmi-test-overflow: One of the bmi bandwidth test programs was using types that might overflow if testing large enough transfers.  The fix is to convert to doubles and drop in several type casts to be cautious when performing the computation that was causing trouble.

[phil]: cancel-bugs: The biggest fix here is a change to the job timer code.  It was performing some pointer operations in the wrong order, which could lead to job timers failing to trigger in some cases. This would prevent some operations from ever timing out.  A secondary fix is a minor cleanup in BMI to catch potential race conditions in cancellation where a lock wasn't being held while checking to see if the target operation is complete.

[phil]: flow-post-error: This patch adds checks in the client side I/O state machine to test for failure at post time for flow operations.  This type of error is uncommon unless the flow parameters are faulty, but it should have checked anyway to be safe.

[phil]: dfile-config-check: This is a safety test.  The problem here is that there was no bounds checking for the DefaultNumDFiles option in the config file.  This made it possible to select -1 (which in PVFS1 meant "use the default number").  In PVFS2 this number gets passed verbatim to the client and would cause malloc failures and various other odd results when used. The patch just checks at parse time to make sure the value isn't negative.

[phil]: bmi-flow-logging: This patch just adds a little bit more gossip logging output to BMI and the flow protocol.  In particular, it adds tag information to messages about BMI operations, and adds pointer information to the flow protocol to help match posts and completions of flows in log files.

[phil]: server-freopen: This patch just removes duplicate calls to fdreopen(stdin/stdout/stderr) that were being performed on server startup when the server is run in the background.  The first set should be sufficient.

Reverse merge changes from HEAD to posix-extensions-branch.

revert style changes back to previous versions.

run maint/pvfs2indent-80col.sh on all .c files to correct style :)

merge of murali's changes from trunk.

[pcarns]: Patch to allow modifying distribution, distribution parameters and number of
data files to use within a given directory. Such setting will only impact creation of future files
within the same directory. It does not change existing files' parameters. All such hints
are inherited by newly created directories as well..
[pcarns]: Allow trove to continue to read a list of xattrs/keyvals
without stopping if some subset of keys dont exist in the keyval db. Modifies the keyval_read_list() function
to return an array of completely independant error codes for each key that is read.
[pcarns]: adjust encode_string() macro to make it safe to call on NULL strings
[pcarns]: bug in the getattr state machine that would cause it to read dirent_count most of the time
regardless of whether the attribute mask requested it or not.
[slang]: fix a couple of bugs in the get-attr server-side state machine whereby the dirent_count was being read
using the dirent handle instead of the dirdata handle. Fixed a return value from dbpf_keyval_get_handle_info_op_svc
so that get-attr would not needlessly hang..
[vilayann]: fixes for the viewdist utility yet again, since the keys for the xattrs have changed...

reverse merge of recent bug fixes from trunk

commit of keyval handle info changes.  This allows info to be stored on a per handle basis in a 'null key' inside the database.  Right now we just use it to store the count of directory entries, so that we don't have to do a read/modify/write in crdirent and rmdirent.

Also includes changes to the keyval strings to use #define values instead of the actual strings.  This just makes that code lest prone to bugs in the future.

[phil] see http://www.beowulf-underground.org/pipermail/pvfs2-developers/2006-June/002128.html.

Reverse merge from trunk to lockd-branch...

reverse merge of branch

reverse merge of trunk to branch

* [Phil] removed db stat left over from keyval-per-db days
* [Phil] added stranded bstream checking
* [Phil] added comparison function for dspace db
* [Phil] added db cache size and type config options

* [Julian] added tcp send/recv buffer size options

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

commit of versioning stuff to branch

merging changes from HEAD to dbpf branch

Phil's new tcache and the acache changes to match.  Also includes Phil's performance counter changes.  See http://www.beowulf-underground.org/pipermail/pvfs2-developers/2005-December/001704.html.  Thanks Phil!

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

added extended attribute support

updates for adding extended attributes

more doxygen work.

Eliminated use of trove-autogen.pl, since this code hasn't changed in ages.