CVS log for pvfs2/src/server/mkdir.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

merge with Orange-Branch

1.put back permission check in {cr|ch|rm}dirent.sm and readdir.sm.
2. change set-attr.sm, pass the change of UID|GID|PERM of a directory to its dirdata.
3. in tree-setattr, add a state to call prelude sm.
4. display dirdata count of a directory in pvfs2-stat.

changing llu handle references to strings

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

uuid converstation, PVFS_x_position changes, server/client side first pass done

changes to PVFS_handle_X

merge with Orange-Branch

Applied changes for RootSquash functionality that were missed when blue and
orange were merged several months ago.  Thanks to Bart Taylor for providing
the patches.
Modified Files:
  Tag: Orange-Branch
     src/server/mkdir.sm
     src/server/prelude.sm

1.fix rmdir. 2. adjust remove_rebuild to send the dist_dir_attr to the rebuilt dirdata objects. 3. modify crdirent.sm to check if the incoming dirent belongs to its dirdata. more change to come for chdirent.sm, rmdirent.sm, etc. 4. modify sys-mkdir to retry getattr if it's contacting the wrong dirdata object. more change to come for sys-create, sys-remove, sys-rename, sys-lookup, etc.

Store dirdata attributes in keyval database for dirdata handles.

1. sys-getattr only contact active dirdata servers. 2. cleanup mkdir. 3. fix a memory leak in PINT_free_object_attr.

1. add >InitNumDirdataHandles< field to fs_config and default value set to 2. 2. add command line options >-n, --init-num-dirdata< to pvfs2-mkdir to set initial number of dirdata handles for a directory. (not yet applied to kernel module.)

adjust mkdir.

fixed pvfs2-fs-dump.

add dirdata_handle_rebuild in server remove.

Clean up memory leaks.

turn off permission check and little tree_setattr fix.

Fixes to lookup.

replace write_keyval_to_local_dirdata_handle with write_keyval_to_meta_handle.

Next round of distributed directory changes.

adjust format of gossip bitmap in some process.

1. add dist-dir-struct to root handle and lost+found handle
2. add dirdata bucket selection to several state machines, however, sys-rename and sys-readdir are not finished yet. There might be other state machines missing.
3. compilable, not expect to run correctly yet.
4. more to change.

incorporate dist-dir-struct to the source tree. It's compilable but not runnable. still have a lot to change and add. Commit as a touch base.

*** empty log message ***

fix root squash bug with mkdir

cleaned up credential verification

implementation of parent information in keyval databases and changes to get-path to use it

storing parent de implementation

merged in changes from summer at LANL

fix server bugs due to merge.

merging hints/events code into trunk.

Reverse merged and ported to HEAD.

Added TROVE_SYNC in job_trove_keyval_write

Updated security checking in state machines.  Added flags where needed.

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

hints and events.

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

minor change allows state actions to not be declared at the top of *.sm files.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

reverse merge of HEAD to WALT3 branch.

add hints to job layer

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

move frequently used extern declaration into a header file

Override Murali's changes with my own that just copies the function.  Let the edit wars commence!

reverse merge from trunk.  working for now.

Reverse merge changes from HEAD to posix-extensions-branch.

Patch fixes the atime, mtime and ctime problems that people have
reported with PVFS2.
Patch changes the notion of who sets these times. Earlier behavior was the
clients providing the times to the server. New behavior is to have the
server set these times except in  the implementation of utimes() system
call whereby clients explicitly set the time (atime and mtime).
NOTE: We dont allow the clients to dictate the ctime setting at all since
that is consistent with all file systems.
So what we have is:
read, readdir -> atime update required
write, truncate -> mtime update required
ctime is updated on every setattr implicitly on the server.

Since there is no need for parent_*time fields in the server request for chdirent,
crdirent and rmdirent, the next step is to remove those fields, fix the endecode
macros and bump up the major version of the protocol.

Patch also implements these times lazily by flushing them only at the time
of a close rather than after every read/write which could be pretty
expensive.

Simple pint/ls/cp/diff test works

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

updated to support SM control block interface

reverse merge of recent bug fixes from trunk

commit of keyval handle info changes.  This allows info to be stored on a per handle basis in a 'null key' inside the database.  Right now we just use it to store the count of directory entries, so that we don't have to do a read/modify/write in crdirent and rmdirent.

Also includes changes to the keyval strings to use #define values instead of the actual strings.  This just makes that code lest prone to bugs in the future.

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

fix deadlock bugs in coalescing code.  add proper state machine debugging.

merge client_test fix from trunk into branch.  Also for each of the state machines that have multiple trove_keyval_write ops, remove TROVE_SYNC from all but the last (usually only 2).

onekeyval merge

removed all the keyval type stuff.  Also my prefix callback doesn't seem to be working..

2 bug fixes:

* fix the readdir code to return the error code to the client instead of assuming that an error just means the directory is empty.

* fix the init-to-zero of dirdata_size for new directories (root dir and lost+found).

missing key string

mostly working onekey impl

immediate completion now turns on with: ImmediateCompletion yes option in fs config

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- added a setattr debugging mask and changed most detailed setattr
  debugging to use it
- added a mkdir debugging mask and changed most detailed mkdir
  debugging to use it
- added some inlined methods in PVFS_util (as they need to be used on
  both the server and the client) for getting the current time in
  PVFS_time format, encoding a PVFS_time as a version (finer grained
  than a 'normal' PVFS_time since we can use the high 32 bits), and
  decoding the version as a PVFS_time
- added a compatibility hack that _should_ allow no noticeable
  breakage on existing storage space, but will eventually migrate to
  the slightly new storage format over time
- modified client side sys-mkdir, sys-create, and sys-symlink to
  encode the mtime as a version when passing it to the server (so it's
  transparent from the server perspective)
- modified server side get-attr to decode the version read from disk
  back into an mtime (so it's transparent from the client perspective)
- modified mkspace method to properly version newly created root and
  lost+found directories
- modified the server side mkdir operation to return -PVFS_EINVAL if
  the object attr type is not a directory object
- misc debugging changes and cleanups

break primary steps of mkdr sm out into a seperate nested machine to allow
for reuse

- extend server side mkdir state machine to create the dirdata dspace
  when creating the directory (rather than waiting for crdirent to
  create it if it doesn't already exist).  this puts the fs in a
  better state for tools like fsck to analyze and moves the work into
  mkdir rather than crdirent (which makes more sense since the dirdata
  object is integral to our directories functioning properly)
- better error handling in mkdir (no longer blow away all error codes
  with -1)

- added configuration option TroveSyncMeta (can be 'yes' or 'no')
- added configuration option TroveSyncData (can be 'yes' or 'no')
- replaced TroveSyncMode configuration option with separate
  TroveSyncMeta and TroveSyncData options.  the defaults are yes to
  both.  this is a change because previously data was NOT being sync'd
  after reads/writes.  for now, only writes are sync'd if
  TroveSyncData is 'yes'.
- updated pvfs2-genconfig to reflect config option changes
- added top-level and flowproto-multiqueue understanding of the
  data sync mode option via setinfo
- added hook to server to pass data sync mode option to the flow
  implementation via setinfo on a per fs/coll_id basis
- added fp_multiqueue tracking of this to be able to pass the
  appropriate data sync flag to trove on a per fs basis
- misc cleanups

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- extend the job interface to take a flags argument, which is passed to various
  trove operations, rather than hard-coding TROVE_SYNC
- make changes to server to pass in the mode value matching the config option
- update test programs (hardcode TROVE_SYNC)

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

updated to sm syntax: state machines now allow a special "terminate"
transition to indicate the final state in a state machine.  updated
all state machines accordingly.  statecomp will generate a warning if a
statemachine does not contain at least one "terminate" or "return"
transition.  other related cleanups and sanity checks.

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

boring 64-bit printf changes

first draft of mkdir implementation as a state machine.  it is used by default
in the tree now.  fixed an annoyance in the test create program -- now it uses
getuid/getgid for creating new files.

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

added error transition out of prelude states for each machine; this should
allow us to properly handle scheduler errors and eventually permission
checking failures

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

mkdir with a nested state machine

Created a common function that can be used to end execution of a state
machine, and started using it in all existing machines.  In particular, this
cleans up any resources that were allocated before the machine started
executing, so that the state machines don't have to worry about resources
that they didn't allocate...

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

Updated all existing server state machines to use new "prelude" nested
machine.  In theory, when we add permission checking we can do it by only
modifying that one nested machine.  Note, however, that most state machines
don't have an error transition from the initial state yet, so there is no way
to detect scheduling errors or permission failures until we clean that up...

the nomenclature in these machines was driving me bananas;
changed name of job_status_t* variables from "ret" to "js_p", changed
integer variables with names like "job_post_ret" to "ret"

remove commented-out code

even though phill pointed out how to do this in .vimrc, add the modelines

mkdir is now mostly functional, while crdirent is broken.  Some misc cleanups
as well.

Added mode lines to emacs directives so emacs knows what to do with these files.

Cleaned up naming of state machines in server so we don't have the "clashing with syscall names" problem any more.

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More random cleanup.  Still working on making the state machine code movable.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

big nasty commit: converted all of the attribute masks to new
style, as found in pvfs2-types.h.  This required changing a
lot of code, most significantly in the pinode cache / pinode
helper area.  Also got rid of a pinode helper function that was
rendered unecessary after the attribute modifications.

System interface tests appear to behave just like before...

Things still aren't right, though!  But I think it will work at
least as well as before the commit, and is slightly more consistent.
Things that are still off:
- the mask should be stored inside the attributes structure,
  rather than kept in a seperate variable everywhere
- server currently ignores attribute masks entirely and does its
  own thing
- a lot of the client code, in particular getattr.c and the pinode
  helper/cache, make big assumptions about how attributes and
  masks can be handled.  In a lot of places they are just copied
  around as atomic units regardless of mask, for example.

I had to stop changing stuff somewhere...

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

Consolidated some redundant types in BMI and converted to a named
enumeration.  Required changing the name of the flags passed in to
bmi_memalloc() and bmi_memfree() all over the place, unfortunately.

Replaced many instances of the bucket/mask stuff with just handle stuff.

Err...I accidently updated only the generated .c files; took me a while to
realize what was going wrong.

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

renamed statemachine files