CVS log for pvfs2/src/io/job/job.h
	Help

Request diff between arbitrary revisions / Display revisions graphically

merge with Orange-Branch

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

Committed Windows client code to Orange-Branch.

uuid converstation, PVFS_x_position changes, server/client side first pass done

move PVFS_ds_position to different types when appropriate, pass a flag around instead of using special canary values in the handle. Things will be broken for a while.

Porting job code

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

merge with Orange-Branch

Bump Trove DBPF version for new precreate-handle storage format. Include migration to 0.1.5. Add job_trove_fs_del_eattr() function and associated lower level calls

*** empty log message ***

per-dataspace handle type precreate pools implementation

initial merge with Orange-Branch. much will be broken

revert cu-security-branch to before the attempted merge with Orange-Branch

bug and memory fixes to getvalue operation

checkpoint

merge from HEAD:

make sure that js_p->actual_size and *out_size_p agree for
job_trove_bstream_read/write_list().  Also remove unused (and not working
properly) read_at() and write_at() functions at the job level.

merged in changes from latest Orange-Branch

merged in changes from summer at LANL

get-path implemenation work

initial get-path state machine and associated changes

merged Orange with Blue - first merging

Merge range query changes and other fixes

Initial import of branch supporting keyval attribute/value lookup

Merged changes in Blue branch into Orange branch

Applied Phil's patch that fixes size mismatches between js_p->actual_size and *out_size_p in the job_trove_bstream_{read,write}_list call, and revised pipeline.sm accordingly.

make sure that js_p->actual_size and *out_size_p agree for
job_trove_bstream_read/write_list().  Also remove unused (and not working
properly) read_at() and write_at() functions at the job level.

properly cancel bmi unexpected jobs and cleanup unexpected state machines on shutdown.

merging hints/events code into trunk.

reverse merge from HEAD.

reverse merge from HEAD to directio branch

fixed bug in precreation that caused servers to tend to pick the same layout
for datafiles 2 through N:
https://trac.mcs.anl.gov/projects/pvfs/ticket/63

reverse merge of changes to HEAD since branch.  Includes small file changes.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

dspace, keyval, coll_attr dbs are now created within the environment.
A whole lot of other things try to make replication work

db-replication state machine which processes the incomming replication message

Reverse merged and ported to HEAD.

implemented function to iterate through handles contained within precreate
pool.  Not tested yet, but intended for use in fsck and similar tools to
prevent precreated handles from showing up as stranded objects.

fix bug in partitioning of local and remote handles; fill in all remaining
error handling paths (needs further testing)

dspace_remove_list operation for trove; untested

add flag field to control trove sync in get_handles() for completeness

Updated branch with code from HEAD

hints and events.

syncing small-file-branch back up with trunk at small-file-branch-point2 tag
(reverse merge)

Push use of named enumerations for request scheduling parameters all the
way down into request-scheduler.c rather than converting to integer flags.
This fixes a bug that allowed server modification operations to proceed
concurrently as if they were read only.

job function to go with new trove_dspace_create_list()

added a mechanism for pool fillers to signal the job interface when they
think that they will not be able make progress soon.  This prevents callers
waiting on handles from hanging indefinitely.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

pool refillers wake back up now if the pool gets low

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

Use setparam state machine as a temporary test harness for get_handles()
interface to precreate pools.  Added test program to trigger it.  Stub for
get_handles() function.

first cut at function to block until precreate pool count gets low.  Mostly
works, except the pool count isn't being intialized properly

shuffled some things around

first cut at job functionality to fill precreate pools; untested

inode stuffing operation.  also added a keyval_remove_list.

move request scheduler load start/stop directly around state machines

Created a better interface for load values in the performance counter.

Added real load statistics for bmi, flow and trove operations.

Update migration branch to current CVS version

Added blocking job for lock waiting.

hint is given down to Trove

add hints to job layer

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Reverse merge changes from HEAD to posix-extensions-branch.

merge of murali's changes from trunk.

[pcarns]: Patch to allow modifying distribution, distribution parameters and number of
data files to use within a given directory. Such setting will only impact creation of future files
within the same directory. It does not change existing files' parameters. All such hints
are inherited by newly created directories as well..
[pcarns]: Allow trove to continue to read a list of xattrs/keyvals
without stopping if some subset of keys dont exist in the keyval db. Modifies the keyval_read_list() function
to return an array of completely independant error codes for each key that is read.
[pcarns]: adjust encode_string() macro to make it safe to call on NULL strings
[pcarns]: bug in the getattr state machine that would cause it to read dirent_count most of the time
regardless of whether the attribute mask requested it or not.
[slang]: fix a couple of bugs in the get-attr server-side state machine whereby the dirent_count was being read
using the dirent handle instead of the dirdata handle. Fixed a return value from dbpf_keyval_get_handle_info_op_svc
so that get-attr would not needlessly hang..
[vilayann]: fixes for the viewdist utility yet again, since the keys for the xattrs have changed...

reverse merge of recent bug fixes from trunk

commit of keyval handle info changes.  This allows info to be stored on a per handle basis in a 'null key' inside the database.  Right now we just use it to store the count of directory entries, so that we don't have to do a read/modify/write in crdirent and rmdirent.

Also includes changes to the keyval strings to use #define values instead of the actual strings.  This just makes that code lest prone to bugs in the future.

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

fixes to immediate completion code.

merging small I/O changes to trunk.  The majority of this commit includes the
new small I/O state machines on client and server, and the modifications to
the client IO state machine.  It also includes mods to the zero-fill code
to perform zero-fill of holes only if they exist instead of zero-ing the entire
memory region initially.  Also, some debug stuff has been added to the
state machine processing code to print enter/exit of state machines if sm
debug is enabled.  The genconfig now includes ioports/metaports for running
multiple servers on the same host for testing.

small I/O changes in sio branch for now

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

added extended attribute support

updates for adding extended attributes

reduce flow.h include usage

merging in declaration for job_null() (forgot to before)

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- fix a typo
- replace scaling timeout and write_ack_transfer_timeout with pre-posting
  with an infinite timeout and adjusting it after the flow completes

added job function that lets the caller reset the timeout on a previously
posted job; untested

- merging in the pvfs2-nm-nb-branch with the main tree
  see ChangeLog for details, or browse the cvs history of the branch
  for full details

NOTE THAT THIS IS A BRANCH COMMIT (tagged as nm-nb-branch).  Feel free
to ignore it completey as it's a snapshot of a work in progress and it
will crash your computer and reformat your hard drive.

This is almost the initial draft of the pvfs2-client-core that
operates in a non-blocking manner.  While it runs, there are still
issues that need to be resolved (that are keeping it out of the main
CVS trunk).  Many other changes were made along the way, so it's more
than just that.

- added compile time option for disabling thread-safety in the client
  library (enabled by default; --disable-thread-safety to disable)
- improved configure summary information emitted at configure time
- added missing non-blocking sysint declarations to sysint header
- re-wrote pvfs2-client-core to use sysint non-blocking operations
  where possible
- made sysint test and testsome() calls more useable from a user point
  of view
- merged dev unexp polling/handling with system interface
- added PINT_sys_dev_unexp call that allows posting unexpected device
  messages so that they can be returned from the sysint testsome
  method in addition to completed sysint operations
- many memory leaks fixed -- many more to go (started adding macros
  for freeing the server response objects)
- added a id_gen_fast_unregister macro that is a no op, to make the
  api more consistent with the id_gen_safe_* calls
- server-config-mgr: report mutex still in use if it is, but also make
  sure not to unlock an already unlocked mutex (valgrind complains)
- many assertions added
- many formatting changes
- many ptr assignments to NULL after freeing in the job interface;
  done while tracking down a problem
- modified the pint-dev device interface to make sure it can handle
  the pvfs2 device in a non-blocking manner
- pint-dev was using buffers larger than it needed across the device;
  fixed them to be the right size
- added a method to free the memory region mapped into the kernel
  through the device
- freed that mapped memory region on pvfs2-client-core shutdown
  (valgrind complained)
- modified device driver to work properly in non-blocking mode from
  userspace by implementing the character device poll method
- modified pint-dev test function to properly handle non-blocking
  responses from the device driver
- modified PINT_flow_reset to not allocate a new mutex unless the old
  one was destroyed
- modified all job uses of the id-generator to use the safe, rather
  than fast, methods (useful for several reasons including safe
  cancellation)
- modified cancellation methods to be able to handle ops that have
  already completed
- modified the job_dev_unexp method to have (and honor) the no
  immediate completion flag if passed (used in the pvfs2-client-core)
- modified the job completion callbacks to make sure to NOT add a
  completed job desc to the completion queue if it's already been
  added (by checking a flag, not scanning).  this is a safety and
  should only be used when a non-thread-safe client library is being
  used in a thread safe env, but we should handle it gracefully anyway
- replaced all kernel allocations of ops through the slab allocator
  handled op_cache to be replaced by a wrapper method (op_alloc(), as
  suggested by Murali) -- the other cache allocations will probably be
  replaced later.  this allows the removal of the extern op_cache
  declaration
- added macros for freeing some of the most heavily used server
  response messages (readdir, lookup, getattr) -- the others are
  coming later
- freed dirents coming out of sysint response object in the
  pvfs2-client

add timeout arguments to bmi and flow related job calls, not used yet

fix the api to job_flow_cancel()

implement job_bmi_cancel(), some minor job cancel api changes, seems to work
fine in some small msgpair test cases induced with gdb

- extend the job interface to take a flags argument, which is passed to various
  trove operations, rather than hard-coding TROVE_SYNC
- make changes to server to pass in the mode value matching the config option
- update test programs (hardcode TROVE_SYNC)

Move from bmi_addr_t to PVFS_BMI_addr_t for consistency.

combo PVFS_id_gen_t -> PVFS_BMI_addr_t, formatting, PVFS error code patch.
Ugly.  Sorry if my formatting pisses someone off; at least I'm not using >
80 columns any more :).

- add hooks to the job interface for various cancellation methods

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

shuffled PVFS_aint in pvfs2-types.h over to job_aint in job.h, hopefully
fixing one of the kernel space type problems

augmented request scheduler with an extra "index" argument that allows the
caller to specify which handle in an array of list operation should be
scheduled

filled in server side of iterate handles, untested

job level hooks for trove iterate handles function

job level hooks for request scheduler timers

converted msgpairarray and io client side state machines to use new
status_user_tag feature of job interface, removed id field from status,
modified job code to fill in status_user_tag in status on immediate
completion as well as at test time

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

some minor work towards supporting status_user_tag in the job api, just
checking in so I can switch to looking at something else for a bit...

sorry. forgot to compile before checking in. errors fixed.

add job_trove_{keyval,bstream}_flush

job status structure now also reports the id of the operation

Made necessary changes for sending handle ranges over to the server.  This
required changes to the trove_dspace_create call, so a lot of stuff had to
change to make this compile.  mkdir is broken and may be for a while.

Server receives handle ranges now for creates, but doesn't do anything with
them yet.  That's to come.

Renamed id_gen_t to PVFS_id_gen_t and moved it definition into pvfs2-types.h.
Also took the #include for id-generator.h out of header files and into .c
files where possible.

stubbed in job level hooks to device interface

got rid of some extra typedefs, cleaned up some naming conventions

clean up of ds_attributes (naming convention)

got rid of the PVFS_coll_id type- it was a duplicate of PVFS_fs_id.

cleaned up vtag type

cleaned up PVFS_ds_keyval type (mainly just naming convention) and moved
definition to pvfs2-storage.h

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

got rid of PVFS_flag type and bmi_flag_t.  need to come back later and
convert the buffer_flag variables in BMI to named enumerations; they are
ints for now

Removed bitmask arguments along the trove_dspace_create call path; needs
testing; I'll be looking into this.

Filled in missing bits of true context support at the job level.
Still not using it in any useful manner in higher level code yet.

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

reformatting

just moved around some comments

modified the job interface so that it takes _pointers_ to vtags as
arguments.  Adjusted everywhere that uses these functions to pass
in NULL for now.

added support for trove keyval read list

updated job_test() function to new implementation that handles timeouts
correctly

moved new job_testsome() implementation (with correct timeout semantics)
into position; updated all code that calls it accordingly, tried out
several test cases to verify behavior, fixed some bugs

got rid of job_waitworld and added timeout argument to
job_testworld.  It isn't implemented right yet, but I wanted to
get the API change out of the way so that I don't have to modify
the server again later

finished adding extra argument to job_bmi_unexp(); propigated changes
into server code and test programs that use that function.

implemented the flags argument to job_bmi_unexp(), which can be used
to prevent the function from completing immediately.  For now it is
hidden with a #define trick until the server code is ready for the API
change.

renamed struct unexpected_info to struct BMI_unexpected_info, so that
it is more consistent with the rest of the API

added in job code; added in request processor code where empty
files were accidentally checked in last time