CVS log for pvfs2/src/server/readdir.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

1.put back permission check in {cr|ch|rm}dirent.sm and readdir.sm.
2. change set-attr.sm, pass the change of UID|GID|PERM of a directory to its dirdata.
3. in tree-setattr, add a state to call prelude sm.
4. display dirdata count of a directory in pvfs2-stat.

changing llu handle references to strings

move PVFS_ds_position to different types when appropriate, pass a flag around instead of using special canary values in the handle. Things will be broken for a while.

Add a "type" field to keys in the keyval database so both directory
entries and attributes can be stored for dirdata handles.

turn off permission check and little tree_setattr fix.

rcache and readdir fix.

Preliminary work for distributed directories - making client aware of DIR_ENT_KEY.

merged in changes from summer at LANL

merging hints/events code into trunk.

Reverse merged and ported to HEAD.

Updated security checking in state machines.  Added flags where needed.

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

hints and events.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

minor change allows state actions to not be declared at the top of *.sm files.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

Changed PVFS_ds_position from int32_t to uint64_t, allowing us to remove DB_RECNUM flag from our databases.

Also changed the dspace comparison function to read the dspace db front to back instead of back to front.

Also changed the dspace_iterate_handles code to use DB_MULTIPLE to improve
server startup times.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

Upgrade to current CVS version

reverse merge of HEAD to WALT3 branch.

Commit of Murali's patch that removes the directory version (mtime) check between readdir calls in the kernel module.  This may cause delete files to show up after they've been deleted on longer listings, but seems better than seeing duplicates.

Also commit a fix to the debug message in readdir: the token is an integer value (index of the entry in the directory).

add hints to job layer

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

Reverse merge changes from HEAD to posix-extensions-branch.

Simple pint/ls/cp/diff test works

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

updated to support SM control block interface

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

fix deadlock bugs in coalescing code.  add proper state machine debugging.

onekeyval merge

removed all the keyval type stuff.  Also my prefix callback doesn't seem to be working..

fixes to make sure keyvals get remove properly.

2 bug fixes:

* fix the readdir code to return the error code to the client instead of assuming that an error just means the directory is empty.

* fix the init-to-zero of dirdata_size for new directories (root dir and lost+found).

mostly working onekey impl

fixes to immediate completion code.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- added a setattr debugging mask and changed most detailed setattr
  debugging to use it
- added a mkdir debugging mask and changed most detailed mkdir
  debugging to use it
- added some inlined methods in PVFS_util (as they need to be used on
  both the server and the client) for getting the current time in
  PVFS_time format, encoding a PVFS_time as a version (finer grained
  than a 'normal' PVFS_time since we can use the high 32 bits), and
  decoding the version as a PVFS_time
- added a compatibility hack that _should_ allow no noticeable
  breakage on existing storage space, but will eventually migrate to
  the slightly new storage format over time
- modified client side sys-mkdir, sys-create, and sys-symlink to
  encode the mtime as a version when passing it to the server (so it's
  transparent from the server perspective)
- modified server side get-attr to decode the version read from disk
  back into an mtime (so it's transparent from the client perspective)
- modified mkspace method to properly version newly created root and
  lost+found directories
- modified the server side mkdir operation to return -PVFS_EINVAL if
  the object attr type is not a directory object
- misc debugging changes and cleanups

- fixed test program ls.c and admin program pvfs2-ls.c to not issue
  an extra call to readdir on every run by fixing a slight logic bug
- changed some readdir debugging
- added a directory version that is passed back from the server to
  the client on each successful readdir call (happens to be the
  directory's mtime encoded as an opaque uint64_t)
- updated client and server response structures and encoder to
  accomodate this
- NOTE: the versioning is not perfect since retrieving the directory
  mtime and iterating the handle list isn't an atomic operation with
  respect to the overall readdir op; i.e. there's still a window where
  a difference may not be detected.  then again, since the mtime is in
  seconds granularity, this should not present a major detectable
  problem anyway.  can use a different version number if problems
  arise often in practice

- added a PVFS_util method to get the current system and return it as
  a valid PVFS_time type
- added arguments to the crdirent/chdirent/rmdirent operations that
  now can take optional parent atime/mtime/ctime fields
- added support on server *dirent operations that update the parent
  time attributes (if specified) on successful completion of the
  *dirent operation; null/zero time fields are not updated
- modified sys-create/sys-mkdir/sys-symlink/sys-rename/sys-remove to
  work with the new operations properly; in general,
  create/remove/rename operations updates the mtime and ctime of the
  parent directory, which is now properly supported
- moved some common server side attribute handling code into a macro
  to make it re-useable across several server state machines
- updated en{de}coding of modified operations
- misc cleanups

- make pvfs2-ls show the mtime for long listings instead of the atime
- make set-info test program update the mtime, rather than atime
- fixed bug on I/O that didn't do an opencache 'put' to release the
  descriptor in use from the cache (pointed out by Avery)
- replace tabs in dbpf-keyval.c
- misc cleanups

- error code sanity checks, better error handling, and cleanups on the
  sysint, le-encoder, and related server operations

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- give readdir operation its own debug mask
- cleanups

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

updated to sm syntax: state machines now allow a special "terminate"
transition to indicate the final state in a state machine.  updated
all state machines accordingly.  statecomp will generate a warning if a
statemachine does not contain at least one "terminate" or "return"
transition.  other related cleanups and sanity checks.

This is a full merge of my pvfs2-dbpf-attr-cache-branch, so this is
the cumulative changelog (based on existing cvs messages).

- initial dbpf-attr-cache implementation and cleanups
- added all the hooks to grab configurable cacheable metadata keys
  from a config file.  re-arranged the config file by separating out
  fs specific hints into a StorageHints area.  updated genconfig to
  account for these changes.
- initial working implementation of the unified dbpf-attr and
  keyval/data pair cache.  holy cow this thing is fast... much work to
  be done still.
- initial locking added
- replaced debug level with a new DBPF_ATTRCACHE_DEBUG level
- added safe to/from cache data copies
- added write-through caching for keyval data (still need to do generic attrs)
- updated simple.conf
- allow config file to have multiple AttrCacheKeywords lines
- fix some cache parts that allow symlinks to work (i.e. be cached)
- added a verbose warning for a particular kind of configuration error
  for a user to see if it should happen
- some more robust configure checks
- more configure related changes
  disable vis build if sdl is detected, but sdl-ttf isn't
- fix broken ifdef nesting that caused compilation failure on non
  aio-threaded configurations
- safe copy of attributes (to/from, preparing for write-through impl)
- server cleanups (making way for pre-caching of a number of attrs on server
  startup since it reads them from disk during handle iterate anyway)
- update create test program to report human readable creation error
- put some quick human readable error reporting in pvfs2-client-core
- added some quickhash method we've been wanting but never had
- implemented policy to limit previously unbounded size of attr cache
- adjusted simple.conf to have some good defaults
- adjusted pvfs2-genconfig to match those defaults
- fixed some htable cleanups
- started write-through attr updates
- update changelog and version info, preparing for mainline merge
- implement write-through of dbpf-resize operation
- make sys-truncate return -PVFS_EINVAL if truncate is called on an object
  that can't be truncated
- fix error code return value of sys-truncate
- cosmetic changes to truncate test program output
- added some comments
- updated showconfig with copyright info
- updated sample configs for showconfig
- added a default handle purgatory value if a value of zero is specified
  (e.g. not specified in config file at all)
- for now, remove the cached dbpf-attr object on i/o writes
- slightly robustify the glibc aio callback detection

formatting.

initial dbpf-attr-cache implementation and cleanups

boring 64-bit printf changes

first draft sysint readdir implementation as a state-machine.  replaces
non-state-machine version by default.  added check for a special case in
the server's readdir that can skip a lot of work if no entries were requested
in a readdir operation.  fixed a bug in the readdir sysint test program that
called readdir 1 too many times in the general case.

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

simplified readdir state machine by taking advantage of attributes
read during prelude

added error transition out of prelude states for each machine; this should
allow us to properly handle scheduler errors and eventually permission
checking failures

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

readdir with a nested state machine

Created a common function that can be used to end execution of a state
machine, and started using it in all existing machines.  In particular, this
cleans up any resources that were allocated before the machine started
executing, so that the state machines don't have to worry about resources
that they didn't allocate...

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

Updated all existing server state machines to use new "prelude" nested
machine.  In theory, when we add permission checking we can do it by only
modifying that one nested machine.  Note, however, that most state machines
don't have an error transition from the initial state yet, so there is no way
to detect scheduling errors or permission failures until we clean that up...

the nomenclature in these machines was driving me bananas;
changed name of job_status_t* variables from "ret" to "js_p", changed
integer variables with names like "job_post_ret" to "ret"

even though phill pointed out how to do this in .vimrc, add the modelines

Added mode lines to emacs directives so emacs knows what to do with these files.

Cleaned up naming of state machines in server so we don't have the "clashing with syscall names" problem any more.

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More random cleanup.  Still working on making the state machine code movable.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

more sanity checking of request protocol boundaries, mostly on server side.
Also elected to remove config buffer size field from getconfig request, and
instead have that boundary dictated by protocol headers

Modified the function names in these files to use the same verbage for the release states.

readdir req/resp cleanup

got rid of the PVFS_token type and replaced with PVFS_ds_position where
appropriate

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

renamed statemachine files