CVS log for pvfs2/src/server/crdirent.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

1.put back permission check in {cr|ch|rm}dirent.sm and readdir.sm.
2. change set-attr.sm, pass the change of UID|GID|PERM of a directory to its dirdata.
3. in tree-setattr, add a state to call prelude sm.
4. display dirdata count of a directory in pvfs2-stat.

changing llu handle references to strings

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

changes to PVFS_handle_X

1.add entry verification in rmdirent.sm. 2. add crdirent&rmdirent retry to sys-remove.sm when dirdata mismatch.

1.fix rmdir. 2. adjust remove_rebuild to send the dist_dir_attr to the rebuilt dirdata objects. 3. modify crdirent.sm to check if the incoming dirent belongs to its dirdata. more change to come for chdirent.sm, rmdirent.sm, etc. 4. modify sys-mkdir to retry getattr if it's contacting the wrong dirdata object. more change to come for sys-create, sys-remove, sys-rename, sys-lookup, etc.

Add a "type" field to keys in the keyval database so both directory
entries and attributes can be stored for dirdata handles.

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

turn off permission check and little tree_setattr fix.

Fixes to lookup.

merging Orange Branch changes in

initial merge with Orange-Branch. much will be broken

Preliminary work for distributed directories - making client aware of DIR_ENT_KEY.

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

commiting changes from merge of Orange and Blue into test branch.

merged Bart Taylor's mtime versioning bug fix from trunk to 2-8 branch

patch from Bart Taylor to fix bug in mtime versioning on server

merged in changes from summer at LANL

*** empty log message ***

merging hints/events code into trunk.

*** empty log message ***

Updated client and server state machines to work with capabilities.  Lookup needs some cleaning server side since it does unneeded permission checking.

Reverse merged and ported to HEAD.

more work on state machine permission checks

more work on state machine permission checks

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

*** empty log message ***

hints and events.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

minor change allows state actions to not be declared at the top of *.sm files.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

reverse merge of HEAD to WALT3 branch.

add hints to job layer

backmerging of HEAD to branch...

* added some bits to gossip that print out the thread id for each debug
message.  This can be set with the threads logstamp.

* fixed bug in dbpf sync coalesce code that wasn't ever syncing and
completing operations that got queued in the ready-to-be-synced state,
but the following operations would error (EEXISTS for example),
so the sync never happened.
* removed all the DBPF_OP_NEEDS_SYNC return codes in dspace and keyval
service functions that modify the db.  The syncing code can figure out
what needs to be synced and what doesn't from now on.

* replace DBPF_OP_IS_* macros that do range checking with logical or checks.
This ends up being easier for me to follow and update.

* crdirent gets posted to the request scheduler with the handle of the
directory (not the dirent handle).  In this case, the operation
on the directory handle is read-only.  Not setting the readonly_flag
for this case was causing a hang with other modifying operations on the
directory (setattr for example).

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

reverse merge from trunk.  working for now.

Reverse merge changes from HEAD to posix-extensions-branch.

- Misc. cleanups (replaced a big macro with an inline function)
- Removed parent_*time from the *dirent request protocol since time stamps are now
kept at the server and do not need to be sent over the wire (This is the reason why
major version of the wire protocol has been bumped)
- Adjust macros for the *dirent protocol accordingly.
- Add a few more debugging statements to kmod

Patch fixes the atime, mtime and ctime problems that people have
reported with PVFS2.
Patch changes the notion of who sets these times. Earlier behavior was the
clients providing the times to the server. New behavior is to have the
server set these times except in  the implementation of utimes() system
call whereby clients explicitly set the time (atime and mtime).
NOTE: We dont allow the clients to dictate the ctime setting at all since
that is consistent with all file systems.
So what we have is:
read, readdir -> atime update required
write, truncate -> mtime update required
ctime is updated on every setattr implicitly on the server.

Since there is no need for parent_*time fields in the server request for chdirent,
crdirent and rmdirent, the next step is to remove those fields, fix the endecode
macros and bump up the major version of the protocol.

Patch also implements these times lazily by flushing them only at the time
of a close rather than after every read/write which could be pretty
expensive.

added some state machine debugging

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

updated to support SM control block interface

reverse merge of recent bug fixes from trunk

commit of keyval handle info changes.  This allows info to be stored on a per handle basis in a 'null key' inside the database.  Right now we just use it to store the count of directory entries, so that we don't have to do a read/modify/write in crdirent and rmdirent.

Also includes changes to the keyval strings to use #define values instead of the actual strings.  This just makes that code lest prone to bugs in the future.

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

fix deadlock bugs in coalescing code.  add proper state machine debugging.

merge client_test fix from trunk into branch.  Also for each of the state machines that have multiple trove_keyval_write ops, remove TROVE_SYNC from all but the last (usually only 2).

fix 64bit warnings

onekeyval merge

removed all the keyval type stuff.  Also my prefix callback doesn't seem to be working..

fixes to make sure keyvals get remove properly.

get dirent count correct.  Thanks Phil!

fix warning.

use correct key flags for keyval_read/write

mostly working onekey impl

fixes to immediate completion code.

immediate completion now turns on with: ImmediateCompletion yes option in fs config

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

Many users reported timestamp problems when listing files mounted on a PVFS2 volume,
This was traced down to the versioned mtime patch that was added to detect race conditions
in readdir and retry. However, the approach was deemed unclean since sometimes the client
was aware of the versioned mtime and sometimes not.
This patch attempts to avoid that drawback by moving it all to the server.
The client does not see of any the versioned mtime stuff at all now.

fixes after branch point shift

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

more or less David Metheny's suggested fix for handling nonprintable filenames
- first, like the linux VFS, allow any character in a file name except '/'
- second, if someone does attempt to create a file with a / in the name, make
  sure the request says marked as 'scheduled' so that it gets properly cleaned
  up (otherwise the server will wait for the bad request to finish and no other
  operations will be serviced).

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- added a setattr debugging mask and changed most detailed setattr
  debugging to use it
- added a mkdir debugging mask and changed most detailed mkdir
  debugging to use it
- added some inlined methods in PVFS_util (as they need to be used on
  both the server and the client) for getting the current time in
  PVFS_time format, encoding a PVFS_time as a version (finer grained
  than a 'normal' PVFS_time since we can use the high 32 bits), and
  decoding the version as a PVFS_time
- added a compatibility hack that _should_ allow no noticeable
  breakage on existing storage space, but will eventually migrate to
  the slightly new storage format over time
- modified client side sys-mkdir, sys-create, and sys-symlink to
  encode the mtime as a version when passing it to the server (so it's
  transparent from the server perspective)
- modified server side get-attr to decode the version read from disk
  back into an mtime (so it's transparent from the client perspective)
- modified mkspace method to properly version newly created root and
  lost+found directories
- modified the server side mkdir operation to return -PVFS_EINVAL if
  the object attr type is not a directory object
- misc debugging changes and cleanups

- increased mmap-ra-small buf size from 4K to 16K to avoid many small
  allocations (causes larger mem footprint of client-core)
- adjusted client sysint setattr operation to work in a write-through
  manner with the acache (when enabled)
- fixed client sysint setattr to invalidate the acache entry (if any)
- added some debugging
- misc cleanups

- added a PVFS_util method to get the current system and return it as
  a valid PVFS_time type
- added arguments to the crdirent/chdirent/rmdirent operations that
  now can take optional parent atime/mtime/ctime fields
- added support on server *dirent operations that update the parent
  time attributes (if specified) on successful completion of the
  *dirent operation; null/zero time fields are not updated
- modified sys-create/sys-mkdir/sys-symlink/sys-rename/sys-remove to
  work with the new operations properly; in general,
  create/remove/rename operations updates the mtime and ctime of the
  parent directory, which is now properly supported
- moved some common server side attribute handling code into a macro
  to make it re-useable across several server state machines
- updated en{de}coding of modified operations
- misc cleanups

removed gossip debug statement that generates a warning

- add PVFS_{i}mgmt_create_dirent mgmt calls
- make request sceduler honor the mgmt-remove-* operations even while
  in admin mode by setting the appropriate flag

- error code sanity checks, better error handling, and cleanups on the
  sysint, le-encoder, and related server operations

- apply Nathan's format specifier patch referenced here:

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-August/000779.html

with slight modification

- added configuration option TroveSyncMeta (can be 'yes' or 'no')
- added configuration option TroveSyncData (can be 'yes' or 'no')
- replaced TroveSyncMode configuration option with separate
  TroveSyncMeta and TroveSyncData options.  the defaults are yes to
  both.  this is a change because previously data was NOT being sync'd
  after reads/writes.  for now, only writes are sync'd if
  TroveSyncData is 'yes'.
- updated pvfs2-genconfig to reflect config option changes
- added top-level and flowproto-multiqueue understanding of the
  data sync mode option via setinfo
- added hook to server to pass data sync mode option to the flow
  implementation via setinfo on a per fs/coll_id basis
- added fp_multiqueue tracking of this to be able to pass the
  appropriate data sync flag to trove on a per fs basis
- misc cleanups

- more error handling and cleanups

- added an internal ref_type member to the client state machine object
  so that the callers of the sm_shared_getattr_* methods (which use
  the object_ref or parent_ref internal members) can enforce the
  attribute type before doing the operation they're trying to do.  for
  example, in the error case of trying to do a readdir on a metafile
  (rather than a directory) the previous readdir operation would do a
  getattr and then issue the readdir operation to the server (and get
  an error back from the server because the object type is wrong).
  with this simple type checking in place, the getattr will fail with
  an appropriate error message (in this example case -PVFS_ENOTDIR) so
  that we can avoid the second network round-trip all together.  the
  same goes for I/O and truncate operations (as the objects must be
  metafiles), and create/symlink/mkdir (as the parent MUST be a
  directory).  How are symlinks handled?  they're not -- our sysint
  API calls expect actual handles, never links.  That means that
  callers (i.e. the pvfs2-client, ROMIO, the test programs, etc) will
  have to resolve the link with the appropriate lookup option
  before-hand (this is consistent with what we've been doing so far
  anyway, as added support for symlink type checking doesn't work
  since all operations are not symlink aware -- nor should they be,
  IMO).
- added error handling to crdirent, which would crash the server if a
  non-directory type was passed to it
- misc cleanups on both server and client side

- enforce a handle allocation of the (hidden/internal) dspace created on the
  server side during a crdirent to be within the metadata range that the server
  is managing.  this is a bugfix to bad behaviour that allowed metadata to
  live in the data handle ranges all this time

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

removed an unecessary assertion (new handle doesn't need to be known when
just checking for existance of an old entry); corrected an old function
comment

modify crdirent state functions so that they operate on op structure fields
rather than request fields; will allow code to be reused for other requests

naming update

refactor crdirent state machine a little
- break the collection of steps that check for the existance of the entry
  into a seperate nested machine
- allows the above steps to possibly be reused elsewhere
- simplifies primary state machine into "check existance, write entry, send
  response"
- a few error handling cleanups/clarifications along the way

- implement non-blocking operations for all sysint and mgmt operations
- refactor blocking operations in terms of non-blocking operations
- change all sysint credential arguments to credential references
- memory leaks, bug fixes, cleanups

NOTE: this is a non-blocking first draft, updates are sure to come

- more error handling
- more cleanups
- fixed the dbcache try_remove method after the last set of changes to it
  (must remove even if it's not in the cache)

- okay, some more cleanups and removals of hardcoded TROVE_SYNC flags

- added some debugging to get an idea of how the trove metadata syncs are
  affecting us
- enforced the trove_sync_mode flag in a few more places, but being careful
  to sync some operations that *must* be sync'd
- some cleanups

- extend the job interface to take a flags argument, which is passed to various
  trove operations, rather than hard-coding TROVE_SYNC
- make changes to server to pass in the mode value matching the config option
- update test programs (hardcode TROVE_SYNC)

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

updated to sm syntax: state machines now allow a special "terminate"
transition to indicate the final state in a state machine.  updated
all state machines accordingly.  statecomp will generate a warning if a
statemachine does not contain at least one "terminate" or "return"
transition.  other related cleanups and sanity checks.

boring 64-bit printf changes

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

simplified the crdirent state machine by taking advantage of
attributes that have already been read by the prelude state machine

added error transition out of prelude states for each machine; this should
allow us to properly handle scheduler errors and eventually permission
checking failures

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

crdirent state machine now uses final_response nested machine to handle
sending responses and releasing the scheduler

Created a common function that can be used to end execution of a state
machine, and started using it in all existing machines.  In particular, this
cleans up any resources that were allocated before the machine started
executing, so that the state machines don't have to worry about resources
that they didn't allocate...

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

Updated all existing server state machines to use new "prelude" nested
machine.  In theory, when we add permission checking we can do it by only
modifying that one nested machine.  Note, however, that most state machines
don't have an error transition from the initial state yet, so there is no way
to detect scheduling errors or permission failures until we clean that up...

the nomenclature in these machines was driving me bananas;
changed name of job_status_t* variables from "ret" to "js_p", changed
integer variables with names like "job_post_ret" to "ret"

even though phill pointed out how to do this in .vimrc, add the modelines

crdirent doesn't crash anymore

Added mode lines to emacs directives so emacs knows what to do with these files.

Cleaned up naming of state machines in server so we don't have the "clashing with syscall names" problem any more.

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More random cleanup.  Still working on making the state machine code movable.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

Modified the function names in these files to use the same verbage for the release states.

got rid of the "generic" response structure, cleaned up several response
structure definitions

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

Replaced many instances of the bucket/mask stuff with just handle stuff.

Err...I accidently updated only the generated .c files; took me a while to
realize what was going wrong.

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

renamed statemachine files