CVS log for pvfs2/src/io/bmi/bmi.c
	Help

Request diff between arbitrary revisions / Display revisions graphically

merge with Orange-Branch

updates to user interface code:
all posix wrappers implemented
socket wrappers added
sendfile implemented
sys-lookup modified to return a path on failure
other files are minor formatting changes

Committed Windows client code to Orange-Branch.

Testing BMI TCP

Porting BMI TCP

Porting BMI sockets

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

merging Orange Branch changes in

initial merge with Orange-Branch. much will be broken

revert cu-security-branch to before the attempted merge with Orange-Branch

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

merging bmi-experimental-branch to trunk

commiting changes from merge of Orange and Blue into test branch.

Merged failover with tree code.
Modified Files:
  Tag: Orange-Branch
 	prepare src/apps/kernel/linux/pvfs2-client-core.c
 	src/client/sysint/client-state-machine.c
 	src/client/sysint/client-state-machine.h
 	src/client/sysint/finalize.c
	src/client/sysint/sys-getattr.sm
 	src/client/sysint/sys-io.sm
	src/client/sysint/sys-small-io.sm
 	src/common/gossip/gossip.h
	src/common/misc/msgpairarray.sm
 	src/common/misc/pint-event.c
	src/common/misc/pint-event.h
 	src/io/bmi/bmi.c
	src/io/dev/pint-dev.c
	src/io/job/job.c
 	src/kernel/linux-2.6/devpvfs2-req.c
 	src/kernel/linux-2.6/file.c
	src/kernel/linux-2.6/pvfs2-mod.c
 	src/proto/PINT-le-bytefield.c
	src/proto/pvfs2-req-proto.h
 	src/server/create-immutable-copies.sm
	src/server/get-attr.sm
 	src/server/small-io.sm
	src/server/tree-communicate.sm
 	src/server/request-scheduler/request-scheduler.c
 	src/server/request-scheduler/request-scheduler.h

Patch that gives expected/unexpected its own method_plan.  Fixes test_unexepected/test_context race.  From Dries Kimpe.

Completed coding and debugging of small-io failover.

The remaining pieces of BMI ZOID support.

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:pcarns:20090930154442

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:harms:20090610160748

merging in "no polling" flag support from trunk to bmi-experimental-branch

Add a flags field to bmi method ops structure.  BMI_METHOD_FLAG_NO_POLLING
can be used to disable implicit busy polling on a particular BMI method.

merged in changes from latest Orange-Branch

merged in changes from summer at LANL

merged Orange with Blue - first merging

Add yet another argument in the unexpected functions.  The "class" will be
used to differentiate different classes of unexpected messages that arrive
at the same host but destined for separate components.  Default API is
unchanged and assumes PVFS uses class value 0.  No method support yet.

simplifying redundant methods a little further

Drop non-list versions of send, recv, and sendunexpected from the method api
(but leave in the BMI api for now).  Only implemented for tcp for now, need
to update other methods.

Merged changes in Blue branch into Orange branch

Added more gossip messages.

Revised the allreduce.sm and other related codes accordingly.

Changed bmi-tcp so that an accept failure due to no resources available, that no error is propogarted so the BMI thread doesn't terminate. The accept failure for the no resource condition will also signal BMI to drop all addresses that have a resource count of zero, which may free resources to let future accepts to complete.

Merged Phil's tcp-peer-comm-branch that allows simultaneous server to server communications.
Added basic allreduce module:
  - implemented a recursive distance doubling algorithm.
  - works only when the # of servers are power of 2.
  - when activated, servers simply exchange their id to each other.

TODO
  - handle arbitrary sizes of buffers of various data types.

Added bmi initialize ref count for Nawab.

unused header inclusion

How did this stuff sneak back in?  Remove unecessary memory alignment stuff
from bmi.c and flowproto-multiqueue.c

merging hints/events code into trunk.

bmi changes to allow separate bmi packaging

Move aligned allocation call to bmi-tcp.  Remove padding stuff from flow.

reverse merge of direct IO changes to he-branch.

merge of direct IO code to PVFS HEAD.

reverse merge from HEAD.

reverse merge from HEAD to directio branch

Fix bug in BMI reference counting.  Previously the server was responsible
for both incrementing and decrementing count while working on a state
machine, but this leaves a window of time between completion of
testunexpected() and the start of a state machine.  If a connection was
reset during this window it could cause state machine to try to use
non-existent BMI address, with some odd results.

This patch adds a flag that the server can use to have BMI automatically
increment reference count on any unexpected message before it is passed to
the server for processing.

reverse merge of changes to HEAD since branch.  Includes small file changes.

merge trunk updates down to small-file-branch.  Passes basic tests but needs
some double checking of pint-cached-config and sys-create conflicts.

fixes to get alignment right and an attempt at aligning unaligned accesses

Make sure that BMI initializes id generator before using.  The
id_gen_safe_initialize() is safe to call multiple times with matching
finalize calls.

This fixes server crash on startup with new BMI changes.

experimental fixes to some bmi address mgmt performance problems:
- Search by addr now uses existing id_gen_t field rather than scanning
  linked list.  Switched to id_gen_safe* to remain safe in case we search
  for an addr that has been destroyed.
- Search by method addr now uses a back pointer to parent addr structure rather
  than scanning linked list.

Reverse merged and ported to HEAD.

hints and events.

fix bad commit with endpoint parsing

bmi cleanups.

commit of Phil's patches for the halloween bug fixes.  See the pvfs2-dev for details.

merge halloween bug fixes to 2.6 branch.

my fix for the halloween bug.

id_gen_safe is only needed for ids that get passed out to the user through the system interfaces.  We can use id_gen_fast for the bmi address value.

Initial addition of portals BMI method.

Free one bit of memory on normal exit.

fix warning

fix id_gen_safe static allocation leakages.  Note that the job interface (to return op ids through the system interface) is really the only user of the id_gen_safe code.  The bmi layers call it as well, but only from _test and _testsome, which aren't ever used at presnet in PVFS.

Also increase the number of buckets in the id_gen_safe hashable.  67 gives a loading factor of about 50...we're probably going to want to allow more operations in-progress than that in todays world.

Merge HEAD changes to TAS-branch.

fixes to let bmi methods only get the listen addr they care about.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

merge recent fixes from HEAD

[phil]: This patch corrects a variety of error code problems.

See: http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-March/003253.html

Update migration branch to current CVS version

Sync hint-branch to current CVS version

Initial commit of BMI MX implementation provided by Scott and Myricom.  Thanks Scott!

Synchronization with HEAD

explain a bit more what the bmi method_addr_reg callback is for

* ib: cache explict BMI_memalloc registrations; huge latency improvement
* bmi: avoid gettimeofday() to determine poll vs block for multi-method
    scenario
* ib: rely on bmi to make poll vs block decision
* ib: only check for new connections and async events during blocking periods

merge pw's fixes from HEAD that quiet warnings in non pthread scenario and include some headers that get missed otherwise.  Also fix some warnings that crept in.

include some headers explicitly that are pulled by pthread only in the posix locking case

reverse merge of HEAD to WALT3 branch.

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

trivial const fix

Merge from HEAD to posix-extensions-branch...
This should facilitate merging the posix branch to HEAD...

Implement root squash, read only and all squash export options. All of these are server-side
settings and set in the fs.conf files.. Perhaps the docs need to be pushed out to the pvfs.org website
after this check in?
Wild cards and mask-based addresses are allowed as part of the fs.conf export settings..

Also added a mount time option to honor setuid bit in files, mount -o suid...

Removed the -c option from pvfs2-ping and made the config file checks default.

Reverse merges from HEAD..

Backmerge with HEAD

Added bmi unexpected free callbacks to tcp, gm and ib to indicate when unexpected buffers
are done. All they do right now in these methods is to free these buffers.
Move the call to bmi_unexpected_free() earlier (right after the decode is done)
Cleaned up test programs and server to use the new api..
Also cleaned up tcp and gm bmi structures to init structures the c standard way..

reverse merge from trunk.  working for now.

[phil]: alternative lio_listio implementation that spawns a thread and does pwrite/pread.

[phil]: bmi-socket-close: This fixes a bug in the new BMI_set_info(... BMI_TCP_CLOSE_SOCKET ...) mechanism, which is used to reconnect the socket to the initial configuration server if new socket buffer sizes are specified in the config file.  I didn't follow the code path find the exact problem, but at a high level it wasn't being thorough enough in cleaning out the old socket.  This showed up when using epoll and specifying socket buffer sizes in the server configuration- in this case the client will often fail to mount with a cryptic "not a directory" error and leave some epoll() errors in the pvfs2-client.log file.  I think a stale (or possibly reused) file descriptor was being left in the epoll fd set.  At any rate, the fix is to use a different set of functions for tearing down the entire address etc. so that it is reconnected on the next BMI addr lookup.  This path is already used by the server to discard old BMI addresses after critical errors on addresses that cannot be reconnected.  It is triggered from bmi.c without entering the bmi_tcp module, so this patch also adds a check to make sure we don't bother for non-tcp methods.

[phil]: bmi-test-overflow: One of the bmi bandwidth test programs was using types that might overflow if testing large enough transfers.  The fix is to convert to doubles and drop in several type casts to be cautious when performing the computation that was causing trouble.

[phil]: cancel-bugs: The biggest fix here is a change to the job timer code.  It was performing some pointer operations in the wrong order, which could lead to job timers failing to trigger in some cases. This would prevent some operations from ever timing out.  A secondary fix is a minor cleanup in BMI to catch potential race conditions in cancellation where a lock wasn't being held while checking to see if the target operation is complete.

[phil]: flow-post-error: This patch adds checks in the client side I/O state machine to test for failure at post time for flow operations.  This type of error is uncommon unless the flow parameters are faulty, but it should have checked anyway to be safe.

[phil]: dfile-config-check: This is a safety test.  The problem here is that there was no bounds checking for the DefaultNumDFiles option in the config file.  This made it possible to select -1 (which in PVFS1 meant "use the default number").  In PVFS2 this number gets passed verbatim to the client and would cause malloc failures and various other odd results when used. The patch just checks at parse time to make sure the value isn't negative.

[phil]: bmi-flow-logging: This patch just adds a little bit more gossip logging output to BMI and the flow protocol.  In particular, it adds tag information to messages about BMI operations, and adds pointer information to the flow protocol to help match posts and completions of flows in log files.

[phil]: server-freopen: This patch just removes duplicate calls to fdreopen(stdin/stdout/stderr) that were being performed on server startup when the server is run in the background.  The first set should be sufficient.

Reverse merge changes from HEAD to posix-extensions-branch.

revert style changes back to previous versions.

run maint/pvfs2indent-80col.sh on all .c files to correct style :)

[pete] add a connection close set_info option to bmi-tcp so that we can reset the buffer sizes for the connection to the mountpoint server.
[slang] set the defaults for the buffer sizes to 0 in the config file so they
don't get set on new connections unless the config file requests it.

Reverse merge from trunk to lockd-branch...

reverse merge of branch

reverse merge of trunk to branch

extraneous breaks; fix logic if open_context failed in some method

Fixes for the latencies noticed in BMI (by Julian) when the main thread is waiting in the poll/epoll_wait
and another thread indicates that data is ready on another socket which is not being watched.
In the poll case, the fix (suggested by pete) is to create a pipe and add the pipe fd
to the poll set being watched. Adding a socket for watch also sends a 1 byte message over the pipe.
This in turn caused the poll() thread to break out of the poll and cause it to retry with the updated poll
sock set.
For the epoll case it is perfectly safe for the thread adding the socket to the watch list to call
epoll_ctl() directly since the kernel takes care of necessary serialization for that.

merging small I/O changes to trunk.  The majority of this commit includes the
new small I/O state machines on client and server, and the modifications to
the client IO state machine.  It also includes mods to the zero-fill code
to perform zero-fill of holes only if they exist instead of zero-ing the entire
memory region initially.  Also, some debug stuff has been added to the
state machine processing code to print enter/exit of state machines if sm
debug is enabled.  The genconfig now includes ioports/metaports for running
multiple servers on the same host for testing.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

small I/O changes in sio branch for now

Attached patches implements trusted ports and networks similar to
what PVFS1 implemented.
It adds a new <Security> </Security> tag to the fs.conf config file
and parse utilities accordingly. We can edit these later if there
is a strong need.

Within these tags, you have to specify TrustedPorts <Range of ports> and
TrustedNetwork tcp://<network> tcp://<mask>.
I dont know how to support this cleanly on  non tcp/ip bmi
architectures, so currently this only enables it for the bmi_tcp module
alone. (i guess we could do it for bmi_ib module as well but not done
until someone says that is the right thing to do...)

Also adds support for a --trusted argument to the pvfs2-genconfig script
that will prompt for the trusted port ranges and the network address/masks and write it out
to the fs.conf and server.conf file.

Configure now supports a --enable-trusted to enable all these features.
(NOTE that the fs.conf tags (or lack of one) still allow you to not
select these features even if configure was asked to compile with this
option).

this patch also cleans up all the warnings that arises with gcc
(3.4) and --enable-strict build mode.

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]: add a new bmi method that can resolve connections from unexpected
requests into hostnames

fix debug: not a context_id, it is an op id

fixed error reporting for gethostbyname in BMI-TCP
also fixed simple omissions and errors in related code

more doxygen work.

doxygen-related checkins.

label where the mysterious message comes from

adaptive polling for bmi

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- merging in the pvfs2-nm-nb-branch with the main tree
  see ChangeLog for details, or browse the cvs history of the branch
  for full details

(branch commit): some mutex protection of the active method count in bmi, so
that runtime addition of bmi methods will work safely if a threaded job
library is also being used

fail quietly on unrecognized BMI method types in pvfs2tab

bmi_tcp now safely destroys failed address references which cannot be
reconnected

bug fix to address cleanup, still can't enable because of a race condition
in bmi_tcp

- started hacking on a way to let servers release bmi resources for addresses
  that cannot be reconnected; commented out for now until I have more time to
  work on it
- also a bug fix on socket cleanup path, making sure not to call close on a
  socket that is already set to -1

removed bogus comment

on second thought, handle reference counting at the bmi level rather than
handing it off to individual modules

added bmi hint to reference count addresses; not doing anything with it yet

- make all of bmi use the id-safe-register methods instead of the fast ones
  to disassociate the ids from the data

make old c compilers happy

initialize needed methods dynamically, not at startup

fix null pointer in rarely-used multi-method testsome

- consolidate the string split functions in flow/bmi by adding a new
  str util function that does what they did
- some cleanups

reapply earlier updates to allow BMI to initialize all modules by default
if none are explicitly listed by caller

- better error translation function (doesn't assume -errno values)
- gm error translation started

- cleanup the error codes in bmi by translating them and making them
  more pvfs friendly.  this only fixes the generic bmi and bmi-tcp parts,
  as the other modules will need some similar translation
- bug fix that will not add an invalid socket if detected

give BMI the optional ability to initialize all compiled in modules at once

Move from bmi_addr_t to PVFS_BMI_addr_t for consistency.

here comes more bmi debugging output

consistent BMI assertions

tightening the bmi_tcp error handling a bit, still needs work

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

demoted some gossip_lerr messages to gossip_err

hooks to start tinkering with a cancel operation function in BMI

step one of some lock reorganization- get rid of "one big mutex" that
serializes all BMI calls, and put burden of thread safety on individual
modules.  for now this means that each module has its own big mutex to
serialize things, but this can be optimized on a per module basis later.

this is a necessary step towards making pvfs2 work under nptl; the fact that
we hold mutex locks for too long seems to exacerbate starvation problems
in that environment...

added some more descriptive printing in cases in which BMI (and in particular,
bmi_gm) can't resolve a given address string

added a simple reverse lookup function

moved BMI addr length limit to somewhere globally accessable (although
it should probably reside somewhere else long term?) and made BMI
enforce it

Change BMI send functions to accept const buffer pointers.
    Change BMI list functions to promise not to modify the lists themselves.
    Some casts in BMI methods are to stick non-const pointers in const pointer
    storage to permit the use of generic functions for both send and recv.

add function to determine method name from a scheme,
requires moving static methods out of init; add IB method option too

remove unnecessary +1

added a manditory server config file option (BMIModules) that lists
which bmi modules the server should initialize, also fixed a bug in
BMI's parsing of module list

Renamed id_gen_t to PVFS_id_gen_t and moved it definition into pvfs2-types.h.
Also took the #include for id-generator.h out of header files and into .c
files where possible.

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

got rid of PVFS_flag type and bmi_flag_t.  need to come back later and
convert the buffer_flag variables in BMI to named enumerations; they are
ints for now

Consolidated some redundant types in BMI and converted to a named
enumeration.  Required changing the name of the flags passed in to
bmi_memalloc() and bmi_memfree() all over the place, unfortunately.

fixed long standing segfault bug that would occur if GM initialization
failed

rewrote BMI_testsome() for case in which support for more than one
method is compiled in.  Still not very clever, but much less ugly than
the original implementation.

added a new debugging mask (BMI_DEBUG_OFFSETS) that dumps all of the
size and offset information about buffers passing through BMI as they
are posted

added a get_info() option to BMI that can be used to retrieve a void*
pointer to the method specific address structure for a bmi address.
Could be used by a flow protocol that wants to use the same addressing
scheme as BMI.

whoops- realized that I should probably be nice and hand back a list of
completed ids in testcontext() :)

implemented testcontext() at the BMI multiplexer level; untested; no
module level support yet

contexts are now handled correctly at the bmi multiplexing level; still
not doing anything with them at the modules level

started putting hooks in place to pass context information down to the
module level in BMI

modified existing BMI API functions to support the concept of
"contexts" and added hooks to create and destroy them.  This is just a
placeholder, contexts aren't actually implemented yet.

#if 0'd some awful code and replaced it with an ENOSYS (should force me
to reimplement it when needed, I hope- I don't want to work on it right
now, but I also don't want to forget about it later :)

modified all of the BMI code to fall in line with the code formatting
style that we want to use now.  Used the pvfs2-mangle-in-place-80col.sh
script (teamed up with a find -exec command) to do so.

vim formatting line fixes, using pvfs2fixbadformat.pl.

moved reference-list code from llist to quicklist (this was the last
place in the I/O path using llist)

made all testXXX functions return 1 if anything completes

free 'requested_method' array after using it

renamed struct unexpected_info to struct BMI_unexpected_info, so that
it is more consistent with the rest of the API

fixed nomenclature in BMI api, so that timeout is now
max_idle_timeout.  At the time of this checkin, still need to
update modules accordingly.

big changes to various I/O API's- there are no longer any "wait"
functions in the BMI, flow, or request scheduling APIs.  In the
request scheduler case, they have simply been discarded.  In BMI
and flows, the testXXX() functions have been augmented with a
timeout argument so that they may perform the role of the
former waitXXX() functions.

renamed a bunch of files in the bmi subdirectory to use dashes instead
of underscores

merged in bmi code