CVS log for pvfs2/src/server/lookup.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

changing llu handle references to strings

changes to PVFS_handle_X

redesigned lookup code to work with security

Additional changes for adding a type field to keys in the keyval database

reverted to Orange-Branch version of lookup code. many modifications will
be needed before the code will compile again.

fixed pvfs2-fs-dump.

in lookup, compare server names to decide local or remote instead of rely on dist_dir_attr.server_no.

Clean up memory leaks.

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

fix bug with passing s_op instead of smcb

fix bug with passing s_op instead of smcb

Corrected call to job_trove_keyval_read.  Passed in smcb instead of s_op.

Fixes to lookup.

Partial fix to lookup problems.

Next round of distributed directory changes.

merging Orange Branch changes in

initial merge with Orange-Branch. much will be broken

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

commiting changes from merge of Orange and Blue into test branch.

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:pcarns:20090721184004

merged in changes from summer at LANL

patch from Bart Taylor; moves permission checking code back to common
directory in order to add safety checks for truncate operation permission

merging hints/events code into trunk.

Updated lookup, remove, getattr, create client side.  Comments in lookup server side.  Check code updated for new file creation.  Server cleaned up unnecessary runtime variables.  Remove serverside updated to use capabilities.

Updated lookup/getattr on server/client.

Updated client and server state machines to work with capabilities.  Lookup needs some cleaning server side since it does unneeded permission checking.

updated all references in include/, src/client/, src/common/, src/io/,
src/proto/, and src/server/ to use the new PVFS_credential in place of the
old PVFS_credentials. the admin apps in particular need to be updated to use
the new API.

Reverse merged and ported to HEAD.

reverse merge of trunk changes since branch.

reverted to an earlier version of the lookup code

reverse merge latest trunk changes to small-file-branch and update msgpair
usage where needed

ACL checks done before standard permissions.  check.c updated to determine failure mode (if invalid/missing ACL return EIO, if permission denied return EACCES)

state machine permission checking code cleanup

Updated security checking in state machines.  Added flags where needed.

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

Merge from HEAD

bug fix in server lookup state machine

Fixed a bug in the lookup state machine (nlmills).  Moved code around in check.h/.c.  Fixed spacing in server.c.

hints and events.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

minor change allows state actions to not be declared at the top of *.sm files.

Move some server-only functions out of src/common/misc into src/server.
These functions happen not to compile on clients without getpwuid support.

murali's fix of lookup bug.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

Sync hint-branch to current CVS version

merge 2.6 branch changes to head

fixed an error where a SM frame is incorrectly freed

quiet some general warnings

reverse merge of HEAD to WALT3 branch.

add hints to job layer

backmerging of HEAD to branch...

* added some bits to gossip that print out the thread id for each debug
message.  This can be set with the threads logstamp.

* fixed bug in dbpf sync coalesce code that wasn't ever syncing and
completing operations that got queued in the ready-to-be-synced state,
but the following operations would error (EEXISTS for example),
so the sync never happened.
* removed all the DBPF_OP_NEEDS_SYNC return codes in dspace and keyval
service functions that modify the db.  The syncing code can figure out
what needs to be synced and what doesn't from now on.

* replace DBPF_OP_IS_* macros that do range checking with logical or checks.
This ends up being easier for me to follow and update.

* crdirent gets posted to the request scheduler with the handle of the
directory (not the dirent handle).  In this case, the operation
on the directory handle is read-only.  Not setting the readonly_flag
for this case was causing a hang with other modifying operations on the
directory (setattr for example).

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

Reverse merges from HEAD..

Backmerge with HEAD

PINT_decode does some inplace decoding, so we need to keep the unexpected buffer around until the state machine completes.

Don't assert on malloc failures!

Whew.. Final bits of the acl puzzle (hopefully!).
Clearly, ACLs require support at the server. (what was I thinking earlier? :))
Patch adds a PINT_check_acls() function that does that. the pvfs2_acl_entry structure is moved to a common
header file shared between kernel module and server.
When PINT_check_mode() throws up its hand in despair, ACL checks are done. NOTE: ACLs dont impose a performance
overhead when permission checks go through properly. When they fail, we post a job_trove_keyval_read()
and when that completes, we check if ACLs allow the operation to proceed.
Added a lot more debug statements.
It is unfortunately not enough to change only prelude sm, but lookup sm also had to be modified
to check for acls. sigh..

Hopefully, this does not break anything else. Simple preliminary tests dont seem to indicate any borkennes.
All but 2 of the LTP acl tests seem to be working now I think...

Reverse merge changes from HEAD to posix-extensions-branch.

minor update

Simple pint/ls/cp/diff test works

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

updated to support SM control block interface

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

fix deadlock bugs in coalescing code.  add proper state machine debugging.

onekeyval merge

removed all the keyval type stuff.  Also my prefix callback doesn't seem to be working..

mostly working onekey impl

fixes to immediate completion code.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

[pcarns]: fix bogus error messages

Merge in all of phil's patches fixing up permissions checking as described
here:
http://www.parl.clemson.edu/pipermail/pvfs2-developers/2005-June/001258.html

The last of the early june phil patches.  "Make PVFS2 honor directory
executable bit permissions properly"

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- make pvfs2-ls show the mtime for long listings instead of the atime
- make set-info test program update the mtime, rather than atime
- fixed bug on I/O that didn't do an opencache 'put' to release the
  descriptor in use from the cache (pointed out by Avery)
- replace tabs in dbpf-keyval.c
- misc cleanups

- error code sanity checks, better error handling, and cleanups on the
  sysint, le-encoder, and related server operations

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

fix 64-bit compilation warnings

- remove unused extern variable in kernel code
- cleanup some gossip and comments in server side lookup

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

updated to sm syntax: state machines now allow a special "terminate"
transition to indicate the final state in a state machine.  updated
all state machines accordingly.  statecomp will generate a warning if a
statemachine does not contain at least one "terminate" or "return"
transition.  other related cleanups and sanity checks.

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

boring 64-bit printf changes

bugfixes; cleanups; more lookup work -- now we support full symlink resolution
by default (link semantics will change later) up to an arbitrary depth of
8 links deep

a more sane lookup implementation that appears to work (at least in prelim
testing) properly across multiple meta-data servers.  cleanups will
follow, as well as additional functionality (such as symlink following).

locally, I've got a 2 meta-server setup that runs pts and all sysint test
programs properly!

first draft client sysint lookup implementation as a state machine (only
partially complete).  seems to be about as good as the previous lookup, so
the original sys_lookup is replaced by this one by default as of this checkin.
fixed a getattr bug that forced a size read query on the same server, even
if multiple servers are being used; added a dmkdir test program for testing
distributed lookup (in progress); updated changelog to add some previously
added features (in prep for next release); probably some other things

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

okay symlink creation (i.e. setattr) and getattr on the type seems to work now.
lookup still needs work to make them useful.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

added error transition out of prelude states for each machine; this should
allow us to properly handle scheduler errors and eventually permission
checking failures

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

lookup now uses final_response nested state machine

Created a common function that can be used to end execution of a state
machine, and started using it in all existing machines.  In particular, this
cleans up any resources that were allocated before the machine started
executing, so that the state machines don't have to worry about resources
that they didn't allocate...

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

Updated all existing server state machines to use new "prelude" nested
machine.  In theory, when we add permission checking we can do it by only
modifying that one nested machine.  Note, however, that most state machines
don't have an error transition from the initial state yet, so there is no way
to detect scheduling errors or permission failures until we clean that up...

the nomenclature in these machines was driving me bananas;
changed name of job_status_t* variables from "ret" to "js_p", changed
integer variables with names like "job_post_ret" to "ret"

even though phill pointed out how to do this in .vimrc, add the modelines

changed some error handling behavior.  I don't think this is right, but is
better than what we had before...

Added mode lines to emacs directives so emacs knows what to do with these files.

Cleaned up naming of state machines in server so we don't have the "clashing with syscall names" problem any more.

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More random cleanup.  Still working on making the state machine code movable.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

last bit of req proto limit enforcement for now; mostly server side

cleaned up lookup_path response structure- in particular made the size of
the handle and attribute arrays explicit

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

Replaced str_utils with str-utils before it gets out of hand.

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

renamed statemachine files