CVS log for pvfs2/include/pvfs2-debug.h
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

initial working versionof usrint code
auto configures usrint on an ucache off (not working yet)
new acl repimplemented - old one has #def
some minor format edits

merge with Orange-Branch

Committed Windows client code to Orange-Branch.

Windows client: added debug options, made debugging use gossip, added message file for event logging

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

rcache and readdir fix.

merge with orange branch

Modified sys-getattr.sm:getattr_acache_lookup.  When mirroring was implemented, getattr was
modified to always return the mirrored handles whenever the primary data handles are retrieved.
If the metadata handle does not have mirrors, then getattr returns an attrmask with
PVFS_ATTR_META_MIRROR_DFILES removed.  Thus, when the acache_lookup function compares the
attributes in the acache with the requested attributes and finds that the mirrors are missing,
it was always reporting a miss in the cache.  So, in the case where mirrors are not
available, the acache lookup will now ignore the mirror attribute when it makes its comparison.

Modified pvfs2-types-debug.h to print PVFS_ATTR_META_MIRROR_DFILES when parsing the
attrmask.

Re-added "mirror" to the list of gossip debug entries in pvfs2-debug.h.  Somehow it got removed.

merging Orange Branch changes in

initial merge with Orange-Branch. much will be broken

removed anything and everything related to certificates

revert cu-security-branch to before the attempted merge with Orange-Branch

added bmi and trove timing in I/O pipeline.

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

commiting changes from merge of Orange and Blue into test branch.

fix some compile errors on Fedora 11 (Tinderbox Stan build)

Added functionality to the kernel module and client-core allowing the gossip debug
masks for either the client or the kernel to be modified AFTER the client-core is started.
Modified Files:
 	include/pvfs2-debug.h
 	src/apps/kernel/linux/pvfs2-client-core.c
 	src/common/misc/pvfs2-debug.c src/io/dev/pint-dev.c
 	src/io/dev/pint-dev.h src/kernel/linux-2.6/dcache.c
 	src/kernel/linux-2.6/devpvfs2-req.c
 	src/kernel/linux-2.6/pvfs2-dev-proto.h
 	src/kernel/linux-2.6/pvfs2-mod.c
 	src/kernel/linux-2.6/pvfs2-proc.c
 	src/kernel/linux-2.6/pvfs2-utils.c
 	src/kernel/linux-2.6/upcall.h

These changes give us the ability to modify the gossip-debug-mask for the client-core or the
kernel module dynamically via the proc system.  The following files were modified:
 	include/pvfs2-debug.h
 	src/apps/kernel/linux/pvfs2-client-core.c
 	src/common/misc/pvfs2-debug.c src/io/dev/pint-dev.c
 	src/io/dev/pint-dev.h src/kernel/linux-2.6/dcache.c
 	src/kernel/linux-2.6/devpvfs2-req.c
 	src/kernel/linux-2.6/pvfs2-dev-proto.h
 	src/kernel/linux-2.6/pvfs2-mod.c
 	src/kernel/linux-2.6/pvfs2-proc.c
 	src/kernel/linux-2.6/pvfs2-utils.c
 	src/kernel/linux-2.6/upcall.h

merged in changes from latest Orange-Branch

merged in changes from summer at LANL

initial get-path state machine and associated changes

Initial import of branch supporting keyval attribute/value lookup

Setting up Orange Branch

new server state machine to handle getcred requests

reverse merge of direct IO changes to he-branch.

merge of direct IO code to PVFS HEAD.

adding threaded odirect impl

db-rep-send state machine added

Reverse merged and ported to HEAD.

changes to trove layer to add direct I/O funcionality (O_DIRECT).

Added debug code and log messages.

Sam's prelude and scheduler updates

separate user space device debugging mask from kernel space device debugging
mask; take the former out of the "verbose" mask so that it doesn't
continuously clutter logging output
https://trac.mcs.anl.gov/projects/pvfs/ticket/1

Initial addition of portals BMI method.

Merge HEAD changes to TAS-branch.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

Upgrade to current CVS version..

data corruption fixes caused by sync coalescing zeroing the error code (EEXIST) for failed crdirents getting added to the coalesce queue.  Also added some debug statements.

Update migration branch to current CVS version

Sync hint-branch to current CVS version

Initial commit of BMI MX implementation provided by Scott and Myricom.  Thanks Scott!

Synchronization with HEAD

add pvfs2-validate to head.

Upgrade to current CVS version

reverse merge of HEAD to WALT3 branch.

[phil]: added ACCESS_DEBUG option to print hostnames instead of IP addresses in debug output.
[slang]: fix to previous commit of debug statement in remove.sm

Added GOSSIP_LOCK_DEBUG.

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Alternative Trove implementation branch with Trove transaction support for db >= 4.3.

Reverse merges from HEAD..

reverse merge from trunk.  working for now.

Gossip'ized kmod as well so that we dont have 2 separate calls to printing diagnostics.
gossip for the kmod is fairly primitive and is handled simply by means of a macro in gossip.h.
Replaced and removed pvfs2_print and pvfs2_error with gossip_debug.
Edited quickstart to include comments on the new kmod debug parameters as well.

Checkin of previous patches

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

reverse merge of trunk to branch

apply const to non-modifiable strings

fix deadlock bugs in coalescing code.  add proper state machine debugging.

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

Added a few more missing files and fixes from HEAD to lockd-branch.
lockd-branch should build cleanly now.

commit of versioning stuff to branch

onekeyval merge

removed all db refs in open db.  changed keyval debug stuff a little.

commit includes:

* a new keyval-pcache component in dbpf that keeps track of positions returned from readdir.  The keys of the cache are handle+position and the values are the key names in the dbpf-keyval DB.  This cache requires some changes to be made to tcache (which its based on), such as entries that don't have a timeout.  If the cache lookup fails for some reason (server failover, etc.), the position is the index of the entry in the directory, so the fallback is to step through the entire list of entries up to that point.

* Murali's patch to the kernel module to fix readdir retries.  Previously the module was ignoring the token (position) returned from previous readdir calls and waiting for count to be 0 (ignored TOKEN_END).  This forced 1 more readdir operation (2x for directories with <32 entries).  Now we keep track of the token returned and pass that in to the next call.

* Changed the keyval code to not use the open-cache.  Now we just initialize the DB once and keep a reference to it.  This also needs to be done for the dspace code, and the open-cache should probably be cleaned up as well.

* fixes to iterate bugs

merging changes from HEAD to dbpf branch

Phil's new tcache and the acache changes to match.  Also includes Phil's performance counter changes.  See http://www.beowulf-underground.org/pipermail/pvfs2-developers/2005-December/001704.html.  Thanks Phil!

merging small I/O changes to trunk.  The majority of this commit includes the
new small I/O state machines on client and server, and the modifications to
the client IO state machine.  It also includes mods to the zero-fill code
to perform zero-fill of holes only if they exist instead of zero-ing the entire
memory region initially.  Also, some debug stuff has been added to the
state machine processing code to print enter/exit of state machines if sm
debug is enabled.  The genconfig now includes ioports/metaports for running
multiple servers on the same host for testing.

small I/O changes in sio branch for now

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

added extended attribute support

dded delete extended attribute feature

set_eattr, get_eattr, and get_eattr_list working

adding new varstrip distribution contributed by Heidelberg folks.

updates for adding extended attributes

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- added a setattr debugging mask and changed most detailed setattr
  debugging to use it
- added a mkdir debugging mask and changed most detailed mkdir
  debugging to use it
- added some inlined methods in PVFS_util (as they need to be used on
  both the server and the client) for getting the current time in
  PVFS_time format, encoding a PVFS_time as a version (finer grained
  than a 'normal' PVFS_time since we can use the high 32 bits), and
  decoding the version as a PVFS_time
- added a compatibility hack that _should_ allow no noticeable
  breakage on existing storage space, but will eventually migrate to
  the slightly new storage format over time
- modified client side sys-mkdir, sys-create, and sys-symlink to
  encode the mtime as a version when passing it to the server (so it's
  transparent from the server perspective)
- modified server side get-attr to decode the version read from disk
  back into an mtime (so it's transparent from the client perspective)
- modified mkspace method to properly version newly created root and
  lost+found directories
- modified the server side mkdir operation to return -PVFS_EINVAL if
  the object attr type is not a directory object
- misc debugging changes and cleanups

- add a clientcore_timing debug mask that only logs the operation
  timings from the pvfs2-client-core
- clean up the ncache and make it useable by the lookup/remove calls
- increase ncache timeout from 5 seconds to 30 seconds by default
  (NOTE: the ncache has almost no effect from the VFS standpoint, as
  the kernel's dcache obviates it after the first successful lookup)

- typo fix

- break out clientcore debugging into its own debugging mask
- fixed error code returned by trove on open-cache-get failures on bstream
  flushes

- split msgpair debugging into it's own class (no longer part of 'client')

- keeps kernel modulecompilation working after recent changes

- replaced temporary test values

- moved to fully 64 bit unsigned gossip masks
- setparam mgmt method now takes uint64_t values instead of int64_t
- added object type printing to acache debugging
- fixed bug in shared-state-machine getattr that could allow
  attributes to come out of the acache with incorrect attr
  types/fields
- fixed bug is msgpair setup for create/mkdir/symlink
- gossip mask change to uint64_t and many gossip cleanups
- misc cleanups

add abilitity to negate log categories in PVFS_debug_eventlog_to_mask, e.g. PVFS2_DEBUGMASK="verbose,-request"

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

added new debugging mask ("cancel" string, GOSSIP_CANCEL_DEBUG value) that
shows a little bit more descriptive information when timeouts occur

- merge in as much as possible from the main tree to keep up to date

- add some permission checking debugging to help narrow down problems as
  we find them (with a 'permission' debug mask and keyword)
- enhance printing of possible keywords in pvfs2-set-debugmask by spacing
  them better

- improve human readable size reporting
- allow human readable size reporting to use si units
- fix job-dev test program due to job dev api change
- fix memory leak of unexpected sm pointers
- add --si option to pvfs2-ls (similar to ls's --si option)
- add -H option to pvfs2-statfs (similar to du's -H option)
- merge with cvs head

debugging mask for dbpf-open-cache, should have been in last commit

- added a trove_op debugging keyword/mask.  useful for low level logging of
  all metadata operations being handled in trove (for now only when the trove
  thread is enabled, which is default)

- server and client side io sm cleanups
- give io it's own debugmask
- improved error handling on the client side

- give readdir operation its own debug mask
- cleanups

- server side get-attr path re-work and cleanups
- added getattr gossip flag specific to the getattr operation

- misc cleanups and modification while tracking down a bug during 'rm -rf'
- added a remove specific gossip keyword

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

- test program cleanups
- use no debugging as the default mask for some test programs
- use properly generated credentials in a couple more places
- bug fixes to lookup test program

- a complete re-write of the client side lookup operation that can properly
  handle '.' and '..' segments, as well as recursive relative/absolute
  symlinks across multiple metadata servers
- addition of a lookup debugging keyword, specific to the lookup operation
- minor changes to test programs to aid development
- added two shell scripts in the test/client/sysint directory to give a
  casual look at some trickier cases that lookup can handle (and to help
  make sure future versions can still handle)

- implemented readahead mmap/execution caching in pvfs2-client-core
  to optimize the vfs' small block behaviour that normally hurts
  performance on pvfs2 significantly.  this is the initial implementation
  and probably needs work.

This is a full merge of my pvfs2-dbpf-attr-cache-branch, so this is
the cumulative changelog (based on existing cvs messages).

- initial dbpf-attr-cache implementation and cleanups
- added all the hooks to grab configurable cacheable metadata keys
  from a config file.  re-arranged the config file by separating out
  fs specific hints into a StorageHints area.  updated genconfig to
  account for these changes.
- initial working implementation of the unified dbpf-attr and
  keyval/data pair cache.  holy cow this thing is fast... much work to
  be done still.
- initial locking added
- replaced debug level with a new DBPF_ATTRCACHE_DEBUG level
- added safe to/from cache data copies
- added write-through caching for keyval data (still need to do generic attrs)
- updated simple.conf
- allow config file to have multiple AttrCacheKeywords lines
- fix some cache parts that allow symlinks to work (i.e. be cached)
- added a verbose warning for a particular kind of configuration error
  for a user to see if it should happen
- some more robust configure checks
- more configure related changes
  disable vis build if sdl is detected, but sdl-ttf isn't
- fix broken ifdef nesting that caused compilation failure on non
  aio-threaded configurations
- safe copy of attributes (to/from, preparing for write-through impl)
- server cleanups (making way for pre-caching of a number of attrs on server
  startup since it reads them from disk during handle iterate anyway)
- update create test program to report human readable creation error
- put some quick human readable error reporting in pvfs2-client-core
- added some quickhash method we've been wanting but never had
- implemented policy to limit previously unbounded size of attr cache
- adjusted simple.conf to have some good defaults
- adjusted pvfs2-genconfig to match those defaults
- fixed some htable cleanups
- started write-through attr updates
- update changelog and version info, preparing for mainline merge
- implement write-through of dbpf-resize operation
- make sys-truncate return -PVFS_EINVAL if truncate is called on an object
  that can't be truncated
- fix error code return value of sys-truncate
- cosmetic changes to truncate test program output
- added some comments
- updated showconfig with copyright info
- updated sample configs for showconfig
- added a default handle purgatory value if a value of zero is specified
  (e.g. not specified in config file at all)
- for now, remove the cached dbpf-attr object on i/o writes
- slightly robustify the glibc aio callback detection

- replaced debug level with a new DBPF_ATTRCACHE_DEBUG level
- added safe to/from cache data copies
- added write-through caching for keyval data (still need to do generic attrs)
- updated simple.conf

renamed dcache to ncache since it's a name cache and
ncache is a more appropriate name for it.

CVS removed:
src/client/sysint/pint-dcache.h
src/client/sysint/pint-dcache.c
src/client/sysint/lookup-dcache.sm
test/client/sysint/dcache-torture.c
test/client/sysint/test-pint-dcache.c

CVS added:
src/client/sysint/ncache.h
src/client/sysint/ncache.c
src/client/sysint/lookup-ncache.sm
test/client/sysint/ncache-torture.c
test/client/sysint/test-pint-ncache.c

Also some header file cleanups

renamed pcache to acache since it's an attribute cache and
acache is a more appropriate name for it.

CVS removed:
src/client/sysint/getattr-pcache.sm
src/client/sysint/pcache.h
src/client/sysint/pcache.c
test/client/sysint/pcache-test.h
test/client/sysint/pcache-test.c
test/client/sysint/pcache-torture.c

CVS added:
src/client/sysint/getattr-acache.sm
src/client/sysint/acache.h
src/client/sysint/acache.c
test/client/sysint/acache-torture.c

Also some header file cleanups

add another bmi debug flag, for IB

moved PVFS_debug_ methods into src/common/misc/pvfs2-debug.c
Added a method to iterate available keywords.  Integrated that method into
pvfs2-set-debugmask program so that it can show available keyword/masks.

modified pvfs2-set-debugmask to use symbolic names for mask levels,
added "none" as an explicit name in PVFS_debug_eventlog_to_mask() to
be able to set mask to 0 nicely

add __KERNEL__ preproc lines to have kernel space ignore the pvfs2-debug.h
if it gets included.  Also added the start of a pvfs2-ls utility.  It's mostly
functional, but needs lots of tweaking (output formatting).  Also, the paths
are munged and non-sensical right now.

moved static var to local in static function

added a DIST_DEBUG flag for gossip; gossipified some fprintfs in dist
code; added a line in the server to log what it's logging.

Corrected the custom strstr implementation; moved the
PINT_config_get_debug_mask function from server-config.c into pvfs2-debug.h as
PVFS_debug_eventlog_to_mask; fixed broken configure check for strstr

okay, properly gossipified trove -- seems to work now

reformatted the way the enum values are assigned (use bitshifting now);
makes it easier to pick the next value, and easier to tell how much room
is left in the debug mask bitspace.

step one of header cleanup: ran pvfs2-mangle-in-place-80col.sh on these
files to get the code format closer to standard

added a new debugging mask (BMI_DEBUG_OFFSETS) that dumps all of the
size and offset information about buffers passing through BMI as they
are posted

added a field in the enum for the pcache for gossip love

added a new debugging flag (REQ_SCHED_DEBUG), and added some output with
that key in the request scheduler.  It now can show when requests are
posted, scheduled, and released- to aid in debugging.

added CLIENT_DEBUG mask so the system interface can enjoy gossip love

finished rudimentary dcache test harness; changed the type of a flag
argument to dcache_remove from an unsigned char to an int to be more
consistent with some of our other api's

Imported sources

Initial revision