CVS log for pvfs2/src/server/create.sm
	Help

Request diff between arbitrary revisions / Display revisions graphically

Added debugging of capabilities and credentials. Use security, access and access_detail flags.

Merged robust security code into Orange-Branch.

Added optimization.  When a file is created and the requested number of dfiles is 1,
then we create the metadata without "stuffed" values, i.e., key="nd".  When data
is written to this file, there is no need to execute the "unstuff" server side
state machine.

Added optimization.  When a file is created and the requested number of dfiles is 1,
then we create the metadata without "stuffed" values, i.e., key="nd".  When data
is written to this file, there is no need to execute the "unstuff" server side
state machine.

merge with Orange-Branch

changing llu handle references to strings

client/server builds, much to fix

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

uuid converstation, PVFS_x_position changes, server/client side first pass done

changes to PVFS_handle_X

Remove debugging statement.

Removed debugging statement.

server side fix for the PINT_dist_encode segfault, check that the attribute sent with the request has the distribution bit set in the mask

local layout bug fix during hostname comparison.

bug fix in LOCAL layout.

bug fix for the LOCAL+two files case

local round-robin layouts for Hadoop.

comment out unused s_op to prevent compile warnings

merge with Orange-Branch

*** empty log message ***

Elaine pointed out backwards arguments 2 and 3 in the job_precreate_pool_get_handles() call

per-dataspace handle type precreate pools implementation

initial merge with Orange-Branch. much will be broken

cleaned up credential verification

revert cu-security-branch to before the attempted merge with Orange-Branch

checkpoint

merge from HEAD:

Corrects a double free during the error handling case. Changed so that u.create.datafile_handles is only freed within create.sm:cleanup. See Trac #89.

implementation of parent information in keyval databases and changes to get-path to use it

storing parent de implementation

Merge of Orange-Branch, Orange-FailOver-Becky, and version 1.164 of sys-io.sm (do not unstuff on a read).
This version of Orange-FailOver-Becky supports failover for large files only.  The next iteration will
support small-io failover.

merged in changes from latest Orange-Branch

*** empty log message ***

merged in changes from summer at LANL

Removed debugging statements.

Merged Phil's tcp-peer-comm-branch that allows simultaneous server to server communications.
Added basic allreduce module:
  - implemented a recursive distance doubling algorithm.
  - works only when the # of servers are power of 2.
  - when activated, servers simply exchange their id to each other.

TODO
  - handle arbitrary sizes of buffers of various data types.

*** empty log message ***

Setting up Orange Branch

Corrects a double free during the error handling case. Changed so that u.create.datafile_handles is only freed within create.sm:cleanup. See Trac #89.

*** empty log message ***

*** empty log message ***

merging hints/events code into trunk.

reverse merge from HEAD.

*** empty log message ***

added a credential to the setattr request message

removed short circuit on permission check

reverse merge of changes to HEAD since branch.  Includes small file changes.

*** empty log message ***

*** empty log message ***

Standardize remote vs. local calling convention.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

Updated client and server state machines to work with capabilities.  Lookup needs some cleaning server side since it does unneeded permission checking.

Reverse merged and ported to HEAD.

more work on state machine permission checks

modified permission checking to use callbacks instead of flags. added custom
checks to the io and small-io state machines.

test and fix bugs in cleanup path of stuffed create

fix bug in partitioning of local and remote handles; fill in all remaining
error handling paths (needs further testing)

track local and remote handles through lifetime of the state machine to make
later cleanup easier

preserve error codes in create path before attempting cleanup steps

add flag field to control trove sync in get_handles() for completeness

just updating comments; punting on stuffing arbitrary layouts for now.

store complete layout struct (encoded) for stuffed files

correct handling of dfile count if list layout overrides it

use server list correctly, some extra debugging messages

add support for LAYOUT_LIST and LAYOUT_RANDOM to create state machine; needs
more testing

protocol size limit for layout information; custom encode functions for
layout in order to enforce limit; remember to free decoding memory for
create request fields

*** empty log message ***

*** empty log message ***

*** empty log message ***

trying to be a little smarter about when to sync in the precreate path

removing extent ranges from create request

Reorganized some to add ability to create local datafile if needed
even if the file is not stuffed.  This could happen in some cases
either because of the requested layout or because stuffing is disabled.
Also explicitly disable LAYOUT_LIST support for the time being; will
add back in and test later.

Added ability to disable stuffing by using "FileStuffing no" setting in
<Filesystem> section; useful for benchmarking purposes

hints and events.

syncing small-file-branch back up with trunk at small-file-branch-point2 tag
(reverse merge)

fill in the num_dfiles_req field in create requests, was previously being
set to zero

cutting down on keyvals read by unstuff by relying on what getattr work
nested sm found where possible

make sure to store dfile count of 1 when creating stuffed files, also
minor variable cleanup.  Current setup allows reads and writes to
stuffed files again.

remove unecessary check of return value.

fixes to non stuffed create.

fixes to small file creates in the non-stuffed case (mds != ios).

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

Sam's prelude and scheduler updates

The normal path for a server request operation is to jump to the prelude state machine, which tries schedule the request based on the handle+fsid.  This meant that the request scheduler code had a switch/case returning the handle+fsid for each request we defined.  The function was also overloaded with a readonly parameter (whether the request modified any objects on the server), and a scheduling policy parameter (as the return code), which specified whether the request scheduling could be bypassed for this request.

I moved all that to a per-request parameter structure, that gets specified within the server state machine for the request.  Now there is one array of server requests (moved to pvfs2-server-req.c) which the request scheduler and prelude state machine can poke at to get parameters (defined in the state machine for that request), and based decisions on.

The patch includes some other minor changes:

* A fix to the BMI addr ref count increment/decrement.  Previously, the addr ref count was being incremented/decremented in the prelude/final-response state machines.  That requires that those state machines get invoked, which I don't want to have to depend on.  I've moved the INC call to the unexpected_sm, and the DEC call to where the state machine gets finished in the server.  That way we don't have to worry about whether we're using prelude/final-response.

* Cleanup of the request scheduler.  The way we were changing the mode from normal to admin was sort of embedded into the request scheduling code, so we had to hack the mode change from prelude_sm instead of in setparam.  Also, the request scheduler had to be aware of the setparam fields in the request.  I've moved the mode change code to separate request scheduler and job calls.  I think something similar could be done for the timer code (it doesn't really belong in the request scheduler), but I'll leave that for another patch.

* Changed the way we compile the server binary in Makefile.in to allow for multiple object files.

Start an api for keeping up with the precreate pools.  For now all it does
is find the pools on startup or create them if they don't exist.  All
of the API functions are static in pvfs2-server.c for the moment.  Running
with the assumption we will track allocated handles as keyvals.

minor change allows state actions to not be declared at the top of *.sm files.

Merge HEAD changes to TAS-branch.

merge of the WALT3 branch to HEAD.  This patch changes the way state actions
are represented as C structures (what statecomp generates).  It also changes
the main state action parameter from a s_op on the server or an sm_p on the client
to a unified smcb pointer (state machine control block) for all state actions (both client and server).  Finally, initial support for concurrent state machines has
been added to allow state actions to be invoked concurrently .. a first step for
server-to-server.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

update to current CVS version because statecomp modifications are neat for my work, thanks!

Remove forward declarations of states.

Update migration branch to current CVS version

reverse merge of HEAD to WALT3 branch.

add hints to job layer

backmerging of HEAD to branch...

Suppress printf warnings on opteron builds.

* added some bits to gossip that print out the thread id for each debug
message.  This can be set with the threads logstamp.

* fixed bug in dbpf sync coalesce code that wasn't ever syncing and
completing operations that got queued in the ready-to-be-synced state,
but the following operations would error (EEXISTS for example),
so the sync never happened.
* removed all the DBPF_OP_NEEDS_SYNC return codes in dspace and keyval
service functions that modify the db.  The syncing code can figure out
what needs to be synced and what doesn't from now on.

* replace DBPF_OP_IS_* macros that do range checking with logical or checks.
This ends up being easier for me to follow and update.

* crdirent gets posted to the request scheduler with the handle of the
directory (not the dirent handle).  In this case, the operation
on the directory handle is read-only.  Not setting the readonly_flag
for this case was causing a hang with other modifying operations on the
directory (setattr for example).

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

Modification for migration added, some functions still need to be adapted after moving to a new branch...

Added request statistics to scheduler, reference from datafiles to
parent metafile (needed for migration) and auto migration tool(unfinished).

Add pvfs2 client core / kernel modul support for hints.

Added PVFS-hint, changed tests for pwrite/odirect/transactions/aio dbpf
 support to allow users to explicitly enable or disable the functionality.
In the aio implementation is a bug since somebody else modified it.

Reverse merge changes from HEAD to posix-extensions-branch.

added some state machine debugging

Everything compiles.   Crap!  Now I have to see if it runs!!!

Julian's changes to coalescing and dbpf storage options:

- Changes the coalesce code, now we decide the mode (coalesce or not) during
the coalesce function call and not during the enque process.
- Add the coalesce high / low watermark to the dbpf_collection structure, so
colls can have different values.
- Put per default the TROVE_SYNC flag into trove from different statemachines
where we used the macro before. The macro is still there and should be
removed if we agree to this behavior.
- Removes the coalesce config option
- Moves SMALL_IO_MAX_REGIONS to internal and renames it to IO_MAX_REGIONS (we
had it at least 3 times defined somewhere).

correcting compile errors

updated to support SM control block interface

Reverse merge from trunk to lockd-branch...

reverse merge of branch

merge sync coalescing branch.

fix deadlock bugs in coalescing code.  add proper state machine debugging.

fixes to immediate completion code.

immediate completion now turns on with: ImmediateCompletion yes option in fs config

merging changes from HEAD to dbpf branch

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

[pcarns]:
- adds a new prefix ("A") for access style logging
- logs user, group, host for each operation if enabled
- triggered by a macro in the server (PINT_ACCESS_DEBUG) that checks gossip
  level before doing any work
- basic format looks like this (H is handle, S is s_op structure, crdirent is
  operation type, Success is response status sent):

[A 08/02 15:14] pcarns.users@localhost H=1048576 S=0x80f7d20: crdirent:
finish (Success)

- turning on "access" logging gives messages like the above on completion of
  operations, plus messages for some operations that give key information (like
  the path name for lookups, the object type for getattr, etc.)
- turning on "access_detail" logging adds additional messages that print when
  the request is received and when it starts processing

* added event logging for states
* moved state debug code to state-machine so all state start/end
points get debug messages automatically
* added textlog stuff.  This allows us to write the default event
queue out to a text file that can be converted to slog2 later.

- added configuration option TroveSyncMeta (can be 'yes' or 'no')
- added configuration option TroveSyncData (can be 'yes' or 'no')
- replaced TroveSyncMode configuration option with separate
  TroveSyncMeta and TroveSyncData options.  the defaults are yes to
  both.  this is a change because previously data was NOT being sync'd
  after reads/writes.  for now, only writes are sync'd if
  TroveSyncData is 'yes'.
- updated pvfs2-genconfig to reflect config option changes
- added top-level and flowproto-multiqueue understanding of the
  data sync mode option via setinfo
- added hook to server to pass data sync mode option to the flow
  implementation via setinfo on a per fs/coll_id basis
- added fp_multiqueue tracking of this to be able to pass the
  appropriate data sync flag to trove on a per fs basis
- misc cleanups

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

- okay, some more cleanups and removals of hardcoded TROVE_SYNC flags

- extend the job interface to take a flags argument, which is passed to various
  trove operations, rather than hard-coding TROVE_SYNC
- make changes to server to pass in the mode value matching the config option
- update test programs (hardcode TROVE_SYNC)

- rename all gossip masks to GOSSIP_xxx
- some misc cleanups

updated to sm syntax: state machines now allow a special "terminate"
transition to indicate the final state in a state machine.  updated
all state machines accordingly.  statecomp will generate a warning if a
statemachine does not contain at least one "terminate" or "return"
transition.  other related cleanups and sanity checks.

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

boring 64-bit printf changes

added an extra argument to all of the job API post fuctions: status_user_tag.
... similar to user_ptr semantically (set by caller at post time, returned
by API at test time), except that it is an integer type and is returned
through the status struct.  Not used yet, but intended to help in client
side state machines that have multiple transitions to the same state and
need a way to differentiate between them.

renamed PINT_server_job_context to server_job_context (only used in server).  Moved extern declaration into pvfs2-server.h,
and then removed all the externs from all the state machines.

General cleanup and reorganization in src/server.  Got rid of server-state-machine.c, merging the right stuff into pvfs2-server.c.

I've started using server_ for the prefix for functions used only in the server; seems like a good way to differentiate.

I'm still not happy with the error handling in the server; we need to do something about that.

unified debugging messages for state transitions under a single macro that
only takes a single string (function name) as an argument and formats
the rest on its own.  Updated all state machines (except for flush) to use
it.

added error transition out of prelude states for each machine; this should
allow us to properly handle scheduler errors and eventually permission
checking failures

trying to rename some of the state machine handling functions to be more
consistent and reflect what they actually do

create state machine now uses final_response nested machine to release from
scheduler and send response

Created a common function that can be used to end execution of a state
machine, and started using it in all existing machines.  In particular, this
cleans up any resources that were allocated before the machine started
executing, so that the state machines don't have to worry about resources
that they didn't allocate...

random cleanup; made response structure a static part of server op structure,
rather than dynamically allocating it for each operation

Updated all existing server state machines to use new "prelude" nested
machine.  In theory, when we add permission checking we can do it by only
modifying that one nested machine.  Note, however, that most state machines
don't have an error transition from the initial state yet, so there is no way
to detect scheduling errors or permission failures until we clean that up...

the nomenclature in these machines was driving me bananas;
changed name of job_status_t* variables from "ret" to "js_p", changed
integer variables with names like "job_post_ret" to "ret"

a glimpse into madness :) fixed a bug in the state machine code that prevented
it from handling state machines with initial states that point to nested
machines

integrated nested prelude machine into create as a test case - it works
only with a hack in place for now.  There is a bug that prevents a nested
machine from being used in the first state of a top level machine...

remove commented-out code

even though phill pointed out how to do this in .vimrc, add the modelines

blast out of tree builders!

Added mode lines to emacs directives so emacs knows what to do with these files.

Cleaned up naming of state machines in server so we don't have the "clashing with syscall names" problem any more.

Woohoo look at me!  state machine headers are now obtuse.  and in src/common/misc.  and theoretically reusable between client and server.
New server-state-machine.c has server-specific functions for state machine; we'll need some of those for the client, but different implementations.

More random cleanup.  Still working on making the state machine code movable.

now using pvfs error codes; cleaned up includes all around; still trying to make it possible to move the state machine code in a reasonable way.

modifications to the encoding interface; took encoding type out of argument
and put into structs that represent encoding/decoding state where possible,
other minor cleanups

turned the "bmi_flag_t buffer_flag" variable into a named enumeration
"enum bmi_buffer_type buffer_type"

Consolidated some redundant types in BMI and converted to a named
enumeration.  Required changing the name of the flags passed in to
bmi_memalloc() and bmi_memfree() all over the place, unfortunately.

Replaced many instances of the bucket/mask stuff with just handle stuff.

Err...I accidently updated only the generated .c files; took me a while to
realize what was going wrong.

Added contexts to the job API, and updated all job calls to reflect it.
Contexts aren't actually implemented yet.

renamed statemachine files