CVS log for pvfs2/src/server/set-attr.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

Added pvfs2-join.c under src/apps/admin dir.

1.put back permission check in {cr|ch|rm}dirent.sm and readdir.sm.
2. change set-attr.sm, pass the change of UID|GID|PERM of a directory to its dirdata.
3. in tree-setattr, add a state to call prelude sm.
4. display dirdata count of a directory in pvfs2-stat.

changing llu handle references to strings

store distdir struct in dirdata handle keyval space.

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

turn off permission check and little tree_setattr fix.

Next round of distributed directory changes.

merging Orange Branch changes in

initial merge with Orange-Branch. much will be broken

cleaned up credential verification

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

commiting changes from merge of Orange and Blue into test branch.

merged Bart Taylor's mtime versioning bug fix from trunk to 2-8 branch

patch from Bart Taylor to fix bug in mtime versioning on server

merged in changes from summer at LANL

merging hints/events code into trunk.

*** empty log message ***

fixed setattr permission checks wrt changing a file's owner and group

Updated setattr to set the owner of new files correctly.

Reverse merged and ported to HEAD.

state machine permission checking code cleanup

Updated security checking in state machines.  Added flags where needed.

defined read, write, execute, and setattr capabilities

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

*** empty log message ***

*** empty log message ***

hints and events.

Nested version of set-attr.sm

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

minor change allows state actions to not be declared at the top of *.sm files.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

reverse merge of HEAD to WALT3 branch.

add hints to job layer

backmerging of HEAD to branch...

* added some bits to gossip that print out the thread id for each debug
message.  This can be set with the threads logstamp.

* fixed bug in dbpf sync coalesce code that wasn't ever syncing and
completing operations that got queued in the ready-to-be-synced state,
but the following operations would error (EEXISTS for example),
so the sync never happened.
* removed all the DBPF_OP_NEEDS_SYNC return codes in dspace and keyval
service functions that modify the db.  The syncing code can figure out
what needs to be synced and what doesn't from now on.

* replace DBPF_OP_IS_* macros that do range checking with logical or checks.
This ends up being easier for me to follow and update.

* crdirent gets posted to the request scheduler with the handle of the
directory (not the dirent handle).  In this case, the operation
on the directory handle is read-only.  Not setting the readonly_flag
for this case was causing a hang with other modifying operations on the
directory (setattr for example).

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

Override Murali's changes with my own that just copies the function.  Let the edit wars commence!

reverse merge from trunk.  working for now.

Reverse merge changes from HEAD to posix-extensions-branch.

Fix a bug found by LTP for the setattr operation on a symlink
(lchown system call -> setattr -> ..)

Patch fixes the atime, mtime and ctime problems that people have
reported with PVFS2.
Patch changes the notion of who sets these times. Earlier behavior was the
clients providing the times to the server. New behavior is to have the
server set these times except in  the implementation of utimes() system
call whereby clients explicitly set the time (atime and mtime).
NOTE: We dont allow the clients to dictate the ctime setting at all since
that is consistent with all file systems.
So what we have is:
read, readdir -> atime update required
write, truncate -> mtime update required
ctime is updated on every setattr implicitly on the server.

Since there is no need for parent_*time fields in the server request for chdirent,
crdirent and rmdirent, the next step is to remove those fields, fix the endecode
macros and bump up the major version of the protocol.

Patch also implements these times lazily by flushing them only at the time
of a close rather than after every read/write which could be pretty
expensive.

Simple pint/ls/cp/diff test works

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

updated to support SM control block interface

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

reverse merge of trunk to branch

asserting unsigned variable non-negative is always true

fix deadlock bugs in coalescing code.  add proper state machine debugging.

merge client_test fix from trunk into branch.  Also for each of the state machines that have multiple trove_keyval_write ops, remove TROVE_SYNC from all but the last (usually only 2).

change distribution storage encode/decode to use wire encoding.

removed all the keyval type stuff.  Also my prefix callback doesn't seem to be working..

mostly working onekey impl

immediate completion now turns on with: ImmediateCompletion yes option in fs config

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

[pcarns]: create symlinks with permission 0777 and then make sure no one can
change the permissions to something else.  Symlinks should always have 777
permissions.

Many users reported timestamp problems when listing files mounted on a PVFS2 volume,
This was traced down to the versioned mtime patch that was added to detect race conditions
in readdir and retry. However, the approach was deemed unclean since sometimes the client
was aware of the versioned mtime and sometimes not.
This patch attempts to avoid that drawback by moving it all to the server.
The client does not see of any the versioned mtime stuff at all now.

updates to my event changes to bring them inline with trunk

merged acache changes to trunk.

intermediate commit of acache changes to branch.

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- added a setattr debugging mask and changed most detailed setattr
  debugging to use it
- added a mkdir debugging mask and changed most detailed mkdir
  debugging to use it
- added some inlined methods in PVFS_util (as they need to be used on
  both the server and the client) for getting the current time in
  PVFS_time format, encoding a PVFS_time as a version (finer grained
  than a 'normal' PVFS_time since we can use the high 32 bits), and
  decoding the version as a PVFS_time
- added a compatibility hack that _should_ allow no noticeable
  breakage on existing storage space, but will eventually migrate to
  the slightly new storage format over time
- modified client side sys-mkdir, sys-create, and sys-symlink to
  encode the mtime as a version when passing it to the server (so it's
  transparent from the server perspective)
- modified server side get-attr to decode the version read from disk
  back into an mtime (so it's transparent from the client perspective)
- modified mkspace method to properly version newly created root and
  lost+found directories
- modified the server side mkdir operation to return -PVFS_EINVAL if
  the object attr type is not a directory object
- misc debugging changes and cleanups

- added a PVFS_util method to get the current system and return it as
  a valid PVFS_time type
- added arguments to the crdirent/chdirent/rmdirent operations that
  now can take optional parent atime/mtime/ctime fields
- added support on server *dirent operations that update the parent
  time attributes (if specified) on successful completion of the
  *dirent operation; null/zero time fields are not updated
- modified sys-create/sys-mkdir/sys-symlink/sys-rename/sys-remove to
  work with the new operations properly; in general,
  create/remove/rename operations updates the mtime and ctime of the
  parent directory, which is now properly supported
- moved some common server side attribute handling code into a macro
  to make it re-useable across several server state machines
- updated en{de}coding of modified operations
- misc cleanups

- error code sanity checks, better error handling, and cleanups on the
  sysint, le-encoder, and related server operations

- added configuration option TroveSyncMeta (can be 'yes' or 'no')
- added configuration option TroveSyncData (can be 'yes' or 'no')
- replaced TroveSyncMode configuration option with separate
  TroveSyncMeta and TroveSyncData options.  the defaults are yes to
  both.  this is a change because previously data was NOT being sync'd
  after reads/writes.  for now, only writes are sync'd if
  TroveSyncData is 'yes'.
- updated pvfs2-genconfig to reflect config option changes
- added top-level and flowproto-multiqueue understanding of the
  data sync mode option via setinfo
- added hook to server to pass data sync mode option to the flow
  implementation via setinfo on a per fs/coll_id basis
- added fp_multiqueue tracking of this to be able to pass the
  appropriate data sync flag to trove on a per fs basis
- misc cleanups

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- pvfs2-server cleanups, including better error handling in many
  places
- organize command line options into a single struct to avoid growing
  number of declared static vars in pvfs2-server
- fix the pidfile usage (which didn't work at all for me, particularly
  using the -p option that getopt_long couldn't possibly have
  understood);  now you can do the standard 'kill `cat
  pvfs2-server.pidfile`' to properly terminate the server
- remove broken url from comment and just state the problem and refer
  to the thread in question

- improve human readable size reporting
- allow human readable size reporting to use si units
- fix job-dev test program due to job dev api change
- fix memory leak of unexpected sm pointers
- add --si option to pvfs2-ls (similar to ls's --si option)
- add -H option to pvfs2-statfs (similar to du's -H option)
- merge with cvs head

possible fix for Nathan's dfile_count/dist_size bug?
- preserve incoming mask on setattr rather than trimming to common subset
- key writing of dist/dfiles off of DIST/DFILE mask rather than just TYPE mask

- clean up translation of mode to pvfs2 permissions function
- make use of the function in other places
- add some debugging
- make sure setattr sets meta fields on disk to 0 if not a metafile
- modify getattr/setattr in the following way:

  the server now allows the getattr/setattr operation to pass
  permission checks in the additional cases that the file owner is ==
  credentials.uid OR the credentials.uid == 0.  this will certainly
  need modification, as it doesn't allow things like root_squashing.
  i've modified this behaviour because other file systems allow
  setattr/getattr operations on objects that do not have write/read
  permissions by the user (or at all in fact), which we previously
  required.  this doesn't take into account parent dirs/access
  controls -- as the vfs takes care of a good deal of that.  a simple
  example of where we fail is when a tarball unpacks/creates a new
  file with permissions 444 (r--r--r--) and a utime is issued after
  it.  the utime would fail because we don't have user write access on
  the file, so we'd get all kinds of warnings.

- for non-blocking the passed in sys_attr structures need to be copied
  to avoid stack corruption/bunk values (similar to credential problem;
  there are probably more instances just like this, and they'll be fixed
  as they come)

Updates for distribution renamings.

- okay, some more cleanups and removals of hardcoded TROVE_SYNC flags

- extend the job interface to take a flags argument, which is passed to various
  trove operations, rather than hard-coding TROVE_SYNC
- make changes to server to pass in the mode value matching the config option
- update test programs (hardcode TROVE_SYNC)

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

updated to sm syntax: state machines now allow a special "terminate"
transition to indicate the final state in a state machine.  updated
all state machines accordingly.  statecomp will generate a warning if a
statemachine does not contain at least one "terminate" or "return"
transition.  other related cleanups and sanity checks.

This is a full merge of my pvfs2-dbpf-attr-cache-branch, so this is
the cumulative changelog (based on existing cvs messages).

- initial dbpf-attr-cache implementation and cleanups
- added all the hooks to grab configurable cacheable metadata keys
  from a config file.  re-arranged the config file by separating out
  fs specific hints into a StorageHints area.  updated genconfig to
  account for these changes.
- initial working implementation of the unified dbpf-attr and
  keyval/data pair cache.  holy cow this thing is fast... much work to
  be done still.
- initial locking added
- replaced debug level with a new DBPF_ATTRCACHE_DEBUG level
- added safe to/from cache data copies
- added write-through caching for keyval data (still need to do generic attrs)
- updated simple.conf
- allow config file to have multiple AttrCacheKeywords lines
- fix some cache parts that allow symlinks to work (i.e. be cached)
- added a verbose warning for a particular kind of configuration error
  for a user to see if it should happen
- some more robust configure checks
- more configure related changes
  disable vis build if sdl is detected, but sdl-ttf isn't
- fix broken ifdef nesting that caused compilation failure on non
  aio-threaded configurations
- safe copy of attributes (to/from, preparing for write-through impl)
- server cleanups (making way for pre-caching of a number of attrs on server
  startup since it reads them from disk during handle iterate anyway)
- update create test program to report human readable creation error
- put some quick human readable error reporting in pvfs2-client-core
- added some quickhash method we've been wanting but never had
- implemented policy to limit previously unbounded size of attr cache
- adjusted simple.conf to have some good defaults
- adjusted pvfs2-genconfig to match those defaults
- fixed some htable cleanups
- started write-through attr updates
- update changelog and version info, preparing for mainline merge
- implement write-through of dbpf-resize operation
- make sys-truncate return -PVFS_EINVAL if truncate is called on an object
  that can't be truncated
- fix error code return value of sys-truncate
- cosmetic changes to truncate test program output
- added some comments
- updated showconfig with copyright info
- updated sample configs for showconfig
- added a default handle purgatory value if a value of zero is specified
  (e.g. not specified in config file at all)
- for now, remove the cached dbpf-attr object on i/o writes
- slightly robustify the glibc aio callback detection

initial dbpf-attr-cache implementation and cleanups

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

boring 64-bit printf changes

got rid of some irritating messages, we don't need this printing all the time

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

attempts to get symlink targets written and read from disk.

another step closer to getting symlink specific attribute data written to
(and able to be read from) disk

properly handle situation when we don't have valid symlink fields in the
request attr object.

started making progress toward getting symlink support.  got sidetracked
along the way trying to work out an easy method to unify sys-create,
sys-symlink, and sys-mkdir all at once.  create should work as normal, but
the rest of the changes shouldn't affect functionality (i.e. symlink doesn't
remotely work, sys-mkdir, isn't there yet).

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

Properly write out the dfile_count and dist_size on meta files instead of
finding it chowdered on the next getattr.  Also try to honor individual
attr masks when writing.  added some gossip too.

minor setattr changes; still trying to track down a bug

cleaned up set-attr.sm a little by relying on attributes read by prelude
state machine

added error transition out of prelude states for each machine; this should
allow us to properly handle scheduler errors and eventually permission
checking failures

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

final response nested machine for set-attr

Created a common function that can be used to end execution of a state
machine, and started using it in all existing machines.  In particular, this
cleans up any resources that were allocated before the machine started
executing, so that the state machines don't have to worry about resources
that they didn't allocate...

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

Updated all existing server state machines to use new "prelude" nested
machine.  In theory, when we add permission checking we can do it by only
modifying that one nested machine.  Note, however, that most state machines
don't have an error transition from the initial state yet, so there is no way
to detect scheduling errors or permission failures until we clean that up...

the nomenclature in these machines was driving me bananas;
changed name of job_status_t* variables from "ret" to "js_p", changed
integer variables with names like "job_post_ret" to "ret"

even though phill pointed out how to do this in .vimrc, add the modelines

More attribute related fixes (along the lines of the last changes)

Made quite a bit of progress in trying to no longer ignore the attributes
of meta files, at least when it comes to the dfiles and dist information.
Much more attr progress will be needed, but I think this is a good start in
the right direction.  the set-info program properly updates the access time
of the specified file on disk and get-info can retrieve it.

Use a consistent error code of -1, rather than 1 to make the sysint properly
acknowledge the error when it occurs.

If attributes for a handle/object exist on disk and we get a set-attr,
we now check to make sure that the request isn't trying to change the
object type.  This might break stuff.

Avoid a memory allocation during server side set-attr by adding scratch
space (an attr object) to the server op structure.

Added mode lines to emacs directives so emacs knows what to do with these files.

Cleaned up naming of state machines in server so we don't have the "clashing with syscall names" problem any more.

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More random cleanup.  Still working on making the state machine code movable.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

got rid of the "generic" response structure, cleaned up several response
structure definitions

cleaned up the metafile attributes structure

next phase of attribute cleanup:
- added mask field to obj attribute structure
- remove seperate mask value from pinode struct and several functions,
  including sysint functions
- made server store mask value properly and report it back- even though it
  still ignores which mask fields are set in getattr and setattr requests
- for now, made it so that number of datafiles _cannot_ be specified at
  sys_create time by setting fields in the attributes structure directly.
  Until this issue is resolved, there is currently no way to pick the
  number of datafiles at create time- it is always chosen by system.
- misc. other related cleanup

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

changed PVFS_time typedef so that it is based on int64_t rather than
time_t (getting rid of dependance on system specific type).

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

renamed statemachine files