CVS log for pvfs2/src/server/prelude.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Added debugging of capabilities and credentials. Use security, access and access_detail flags.

Merged robust security code into Orange-Branch.

added new uid management app under src/apps/admin/pvfs2-get-uid
also included are a server and client side sm and the uid mgmt implementation files

merge with Orange-Branch

changing llu handle references to strings

change to #defined calls to uuids

migrate from uint64_t type assignments to uuids

Modifications to perf counter system
Minor formatting and documentation edits

compiler warning cleanup

Applied changes for RootSquash functionality that were missed when blue and
orange were merged several months ago.  Thanks to Bart Taylor for providing
the patches.
Modified Files:
  Tag: Orange-Branch
     src/server/mkdir.sm
     src/server/prelude.sm

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

*** empty log message ***

merging Orange Branch changes in

pieced back in root squash code to server permission checks

added code to skip permission checks for subordinate server ops

initial merge with Orange-Branch. much will be broken

lots of little fixes that have been in the works for a while

cleaned up credential verification

revert cu-security-branch to before the attempted merge with Orange-Branch

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

fix RootSquash bug reported by Bart Taylor

fix root squash bug.  Patch provided by Bart Taylor.

commiting changes from merge of Orange and Blue into test branch.

Merge branches

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:pcarns:20090721184004

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:pcarns:20090630160011

merges from HEAD

merge fix from HEAD

merged in changes from latest Orange-Branch

merged in changes from summer at LANL

merged Orange with Blue - first merging

patch from Bart Taylor; moves permission checking code back to common
directory in order to add safety checks for truncate operation permission

Initial import of branch supporting keyval attribute/value lookup

Merged changes in Blue branch into Orange branch

Fix bug reported by Bart Taylor.  If permission checks fall through to ACL
checking on the server side, then ENOENT could be returned if ACL attributes
weren't present.  It should have been EACCES instead.

https://trac.mcs.anl.gov/projects/pvfs/ticket/104

Merged Phil's tcp-peer-comm-branch that allows simultaneous server to server communications.
Added basic allreduce module:
  - implemented a recursive distance doubling algorithm.
  - works only when the # of servers are power of 2.
  - when activated, servers simply exchange their id to each other.

TODO
  - handle arbitrary sizes of buffers of various data types.

*** empty log message ***

fixing another problem uncovered in setattr permission path; iozone triggers
this by opening a new file with permissions set to zero and then truncating
it

Short term fix for symlink problem that showed up with previous commit.
Symlink object creation uses a separate create (batch_create) request and
setattr request, so we need special permission handling for setattr in
that case.  A typo in the old logic inadvertently prevented this problem
in the past.

fix problem in secondary group permission check for setattr; that specific
case had a hand coded test that did not go to secondary groups.  Fixes a
problem reported by David Metheny

Setting up Orange Branch

merging hints/events code into trunk.

*** empty log message ***

*** empty log message ***

Standardize remote vs. local calling convention.

made getattr conform to current conventions. fixed bugs in capability
verification and hash lookup.

invented the concept of a null capability and implemented it on the server
and the client

Fleshed out ACL retrieval in getattr SM.  Removed old code from prelude SM.

Reverse merged and ported to HEAD.

Temporarily bypass capability verification.

Removed warnings due to use of int64_t in several files. Using lld() mask to print int64_t numbers.

implemented validation with credentials. fixed bug in verify.

basic framework for validation in prelude

added workaround for attribute checking

Moved code around from predule sm into check.c to prepare for new code going in.  NOTE:  All checks skipped at this point.

hints and events.

syncing small-file-branch back up with trunk at small-file-branch-point2 tag
(reverse merge)

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

readonly symbolic

Change access_type functions from 0/1 to use the enum
PINT_server_req_access_type.  There may have been some backward
conditionals in here.  This at least makes them all visible,
and fixes a problem with ReadOnly exports where only reads
(but not writes) were forbidden.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

stubbed in state machine to handle refilling precreate pools

backing out a few client side things

inode stuffing operation.  also added a keyval_remove_list.

merge warning fixups with --enable-fast.

fix a bunch of warnings for unused and uninit vars

[on behalf of Phil]:

This patch adds a new configuration parameter called "RootSquashExceptions".  It has the same value syntax as the existing RootSquash parameter, but it allows you to list hosts or subnets that are exempt from root squashing.

This is helpful if you want to root squash all clients (or at least a large set of them) but still allow root access from particular admin or power user clients.  That was possible before, but probably tedious unless the node you wanted to exclude just happened to fall in a convenient subnet.

minor change allows state actions to not be declared at the top of *.sm files.

Move some server-only functions out of src/common/misc into src/server.
These functions happen not to compile on clients without getpwuid support.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

merge recent fixes from HEAD

Patches from Phil.  Thanks Phil!

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-March/003250.html

Cleanup of performance counter code...

move request scheduler load start/stop directly around state machines

Created a better interface for load values in the performance counter.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

reverse merge of HEAD to WALT3 branch.

add hints to job layer

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

printf warning fixes

Merge from HEAD to posix-extensions-branch...
This should facilitate merging the posix branch to HEAD...

Implement root squash, read only and all squash export options. All of these are server-side
settings and set in the fs.conf files.. Perhaps the docs need to be pushed out to the pvfs.org website
after this check in?
Wild cards and mask-based addresses are allowed as part of the fs.conf export settings..

Also added a mount time option to honor setuid bit in files, mount -o suid...

Removed the -c option from pvfs2-ping and made the config file checks default.

Reverse merges from HEAD..

Backmerge with HEAD

Migration of datafiles between different servers works. Uses msgpairray.sm,
transfers only one datafile between two servers and rewrites metadata.
Todo: Adaption of request scheduler to ensure that migration and write
I/O do not happen concurrently on source dataserver.
Also allow users to specify command line parameter which supresses
transfer of pvfs2-hints.

Whew.. Final bits of the acl puzzle (hopefully!).
Clearly, ACLs require support at the server. (what was I thinking earlier? :))
Patch adds a PINT_check_acls() function that does that. the pvfs2_acl_entry structure is moved to a common
header file shared between kernel module and server.
When PINT_check_mode() throws up its hand in despair, ACL checks are done. NOTE: ACLs dont impose a performance
overhead when permission checks go through properly. When they fail, we post a job_trove_keyval_read()
and when that completes, we check if ACLs allow the operation to proceed.
Added a lot more debug statements.
It is unfortunately not enough to change only prelude sm, but lookup sm also had to be modified
to check for acls. sigh..

Hopefully, this does not break anything else. Simple preliminary tests dont seem to indicate any borkennes.
All but 2 of the LTP acl tests seem to be working now I think...

Simple pint/ls/cp/diff test works

Everything compiles.   Crap!  Now I have to see if it runs!!!

correcting compile errors

updated to support SM control block interface

Reverse merge from trunk to lockd-branch...

reverse merge of branch

reverse merge of trunk to branch

consolidate all extern declarations of server perf counter

[pcarns]: The logic in getattr and prelude state machines was flawed w/ regard
   to how to interpret error codes from reading attributes, and how to
   interpret attributes that aren't filled in properly.  The first case caused
   it to ignore some types of error codes that were properly reported by trove.
   The second case amplified the problem by allow attributes to pass through
   the system that don't have a valid type field set.  The server should not be
   able to read attributes without a type field set (this should be reported as
   an error).  Another minor cleanup is to detect datafiles by looking at the
   type field, rather than assuming that any object with a mask of 0 is a
   datafile.

CVS/time stamp brain damage. :)

changes from head merge

changes to perfbase benchmarks and addition of perf counter queue fields

fixes to immediate completion code.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

added extended attribute support

cleanup for extended attribute stuff.  removed non-list server ops
in favor of list type and renamed list type.  cleaned up a few items in
request debug programs.

updates for adding extended attributes

Merge in all of phil's patches fixing up permissions checking as described
here:
http://www.parl.clemson.edu/pipermail/pvfs2-developers/2005-June/001258.html

The last of the early june phil patches.  "Make PVFS2 honor directory
executable bit permissions properly"

Phil redid the permission checking logic:
"A rewrite of the main pvfs2 permission checking function to clear up a
few things:
- fixes a bug in the permission checking logic related in particular to
  group permissions
- more thorough logging (with PERMISSIONS_DEBUG mask), allows you to see
  exactly what causes access to be granted or denied
- takes a new access_type argument rather than overloading the use of
  defines like PVFS_U_READ etc.
- moved to common location in case it is useful for non-server uses

avoid line breaks in debugging messages

Modified permission checking to allow for reading of attributes of an object
any time someone has obtained the handle.  Permission checking is actually
performed on the parent directory for these cases.

Adjusted gossip output.

missed a header conversion

- fixes the bug where removing a file on client A is seen on client B with
  bad attributes.  this is caused by a bug in trove that allows getattr
  operations to complete even if the bstream or keyval doesn't exist yet
  because it can't distinguish between a valid ENOENT error and when it's
  ok because it just hasn't been created yet
- to resolve this, i've hacked around it for now in the VFS by making sure that
  we properly handle attributes of the wrong type (i.e. bad attributes have
  object type DATAFILE because the server assumes anything untyped is a
  DATAFILE)
- misc cleanups

- properly grab the credentials from the vfs for each operation, not earlier
  as they may contain garbage or values from a previous user! (this should
  solve Ti's permission problem along with the below group fix)
- fixed prelude.sm to allow getattr permissions if the group matches
  (two users of the same group can list each other's file)
- misc cleanups

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

add a flag to request parameters structure that indicates if the prelude
state machine should expect to find attributes for the target object

start consolidating pvfs2 server request parameters into a single table to
make it easier to experiment with new requests; so far table only includes
op type, name, and permission requirements; will include more information later

removed some debugging messages that simply duplicate assertions that are
triggered later; I had added this when trying to track down one of Nathan's
bugs a while back

- merge in as much as possible from the main tree to keep up to date

- add some permission checking debugging to help narrow down problems as
  we find them (with a 'permission' debug mask and keyword)
- enhance printing of possible keywords in pvfs2-set-debugmask by spacing
  them better

- improve human readable size reporting
- allow human readable size reporting to use si units
- fix job-dev test program due to job dev api change
- fix memory leak of unexpected sm pointers
- add --si option to pvfs2-ls (similar to ls's --si option)
- add -H option to pvfs2-statfs (similar to du's -H option)
- merge with cvs head

added some warning messages, checking validity of dist_size and dfile_count
in prelude before get-attr.sm is executed

- clean up translation of mode to pvfs2 permissions function
- make use of the function in other places
- add some debugging
- make sure setattr sets meta fields on disk to 0 if not a metafile
- modify getattr/setattr in the following way:

  the server now allows the getattr/setattr operation to pass
  permission checks in the additional cases that the file owner is ==
  credentials.uid OR the credentials.uid == 0.  this will certainly
  need modification, as it doesn't allow things like root_squashing.
  i've modified this behaviour because other file systems allow
  setattr/getattr operations on objects that do not have write/read
  permissions by the user (or at all in fact), which we previously
  required.  this doesn't take into account parent dirs/access
  controls -- as the vfs takes care of a good deal of that.  a simple
  example of where we fail is when a tarball unpacks/creates a new
  file with permissions 444 (r--r--r--) and a utime is issued after
  it.  the utime would fail because we don't have user write access on
  the file, so we'd get all kinds of warnings.

- implement non-blocking operations for all sysint and mgmt operations
- refactor blocking operations in terms of non-blocking operations
- change all sysint credential arguments to credential references
- memory leaks, bug fixes, cleanups

NOTE: this is a non-blocking first draft, updates are sure to come

- some gossip and minor changes

- added a chdirent server operation useful during renames of target files
  that already exist.
- still working on cleaning up rename, as it does not properly remove some
  data that is left to be removed

add a mechanism for a server to send a generic response back to the client
when it receives a request with a bad version number or encoding type, so
that the user gets a clear error message at the application level too

server now reference counts bmi addresses

fix warnings from adding new state machine

- server side get-attr path re-work and cleanups
- added getattr gossip flag specific to the getattr operation

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

first draft client sysint lookup implementation as a state machine (only
partially complete).  seems to be about as good as the previous lookup, so
the original sys_lookup is replaced by this one by default as of this checkin.
fixed a getattr bug that forced a size read query on the same server, even
if multiple servers are being used; added a dmkdir test program for testing
distributed lookup (in progress); updated changelog to add some previously
added features (in prep for next release); probably some other things

added stubs for server mgmt operation to retrieve event log

fix asymmetry on the one odd request type so that struct name, operation enum name, and item name match

augmented request scheduler with an extra "index" argument that allows the
caller to specify which handle in an array of list operation should be
scheduled

filled in request protocol structures for new mgmt function to retrieve
low level dspace information, including encode/decode routines.
- untested
- still need true scheduler support

gave setparam the ability to report back the original value of the parameter
for each server, not utilizing anywhere yet

stubbed in server operation for iterate handles

allow read only operations to pass through while in admin mode

started implementing server request that will retrieve performance
statistics using a mgmt operation

added a state machine that runs continuously in the back ground to
manage performance statistics (doesn't actually do anything yet), along
the way added support for starting state machines that are not
triggered by protocol requests

. landing a new and improved client-side truncate.
. completing the landing of a fully-functioning server-side truncate.

added statfs request type, and a trivial stub for server state machine

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

added mgmt-noop code that I forgot to commit last time (whoops), modified
msgpair nested machine to allow caller to set the server address
to contact independently of the fs_id/handle if needed

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

phew... we now have a PVFS_sys_flush operation on both the client and server
sides.

put a few pieces in place for mgmt_setparam operation, hopefully located
strategically so as not to conflict on merge with the flush...

made server side nested machines print out what overall operation they
are a part of
- probably stuck the op->string mapping function in a dumb place; we
  can move it later, I guess...

minor setattr changes; still trying to track down a bug

minor fix to the prelude to skip permission checking on removes when we don't have attributes (datafiles).

server properly gathers size data when it's requested now; still need to glue
up several places in the client to have it reported properly.

cleaned up some error code filtering

removed a sanity check that was a little bit too aggressive for the way
we store metadata right now

added TODO note

Server side permission checking is now in place for operations that need it

more progress in the prelude state machine; it retrieves common attributes
now when possible; not doing anything with them yet.  Also removed
PVFS_ERROR_FS error mask.

started taking little steps towards getting the prelude state machine to
do simple permission checking: added attr struct to server op struct,
exposed a helper util for deriving the target handle from a given
request structure, started adding states to prelude

I guess we have to put something in here besides comments, right?

added stubs for a nested state machine to use at the beginning of most
server operations; posts to request scheduler and checks permissions