CVS log for pvfs2/src/proto/PINT-le-bytefield.c
	Help

Request diff between arbitrary revisions / Display revisions graphically

Merged robust security code into Orange-Branch.

added new uid management app under src/apps/admin/pvfs2-get-uid
also included are a server and client side sm and the uid mgmt implementation files

merge with Orange-Branch

remove extents, handle ranges, precreate pools and various other things that aren't needed in their current forms with UUIDs

Modifications to perf counter system
Minor formatting and documentation edits

distributed directory structure for root and lost+found directory. In pvfs2-server.c, it will check if the root directory is set. Need 10~20 seconds for creation when the servers are launched at first time.

redesigned lookup code to work with security

reverted to Orange-Branch version of lookup code. many modifications will
be needed before the code will compile again.

prevent memory leak in setattr encode_release.

1. fix two memory leaks in fee_object_attr and decode_release. 2. clean sys-mkdir and acache.

Merged in changes from Orange-Branch. There were a number of bugs fixed there since this branch was created.

Fixes to lookup.

Next round of distributed directory changes.

*** empty log message ***

merging Orange Branch changes in

fixes to make the new server state machines compile with robust security

encoding fixes

initial merge with Orange-Branch. much will be broken

*** empty log message ***

removed anything and everything related to certificates

revert cu-security-branch to before the attempted merge with Orange-Branch

merge of B2O-Blue-Sync,HEAD,and latest of Orange.

commiting changes from merge of Orange and Blue into test branch.

Merged failover with tree code.
Modified Files:
  Tag: Orange-Branch
 	prepare src/apps/kernel/linux/pvfs2-client-core.c
 	src/client/sysint/client-state-machine.c
 	src/client/sysint/client-state-machine.h
 	src/client/sysint/finalize.c
	src/client/sysint/sys-getattr.sm
 	src/client/sysint/sys-io.sm
	src/client/sysint/sys-small-io.sm
 	src/common/gossip/gossip.h
	src/common/misc/msgpairarray.sm
 	src/common/misc/pint-event.c
	src/common/misc/pint-event.h
 	src/io/bmi/bmi.c
	src/io/dev/pint-dev.c
	src/io/job/job.c
 	src/kernel/linux-2.6/devpvfs2-req.c
 	src/kernel/linux-2.6/file.c
	src/kernel/linux-2.6/pvfs2-mod.c
 	src/proto/PINT-le-bytefield.c
	src/proto/pvfs2-req-proto.h
 	src/server/create-immutable-copies.sm
	src/server/get-attr.sm
 	src/server/small-io.sm
	src/server/tree-communicate.sm
 	src/server/request-scheduler/request-scheduler.c
 	src/server/request-scheduler/request-scheduler.h

*** empty log message ***

bug and memory fixes to getvalue operation

keyval query memory cleanups

Merge branches

checkpoint

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:nlmills:20090716184559

merge from HEAD: http://www.pvfs.org/fisheye/changelog/PVFS/?cs=MAIN:nlmills:20090622193923

Merge of Orange-Branch, Orange-FailOver-Becky, and version 1.164 of sys-io.sm (do not unstuff on a read).
This version of Orange-FailOver-Becky supports failover for large files only.  The next iteration will
support small-io failover.

debugging cleanup

merged in changes from latest Orange-Branch

remove debugging cruft

interim commit

merged in changes from summer at LANL

initial get-path state machine and associated changes

*** empty log message ***

merged Orange with Blue - first merging

Add needed fix to MIRROR initialization so pvfs2-server runs

Initial commit of parallel fsck code

Merge range query changes and other fixes

Completed most of the getattr changes.  Not all done just yet.

edits made to get first blue/orange merge working

fixed le-bytefield decoding bug: unintended fallthrough in case statement

Initial import of branch supporting keyval attribute/value lookup

Completed changes to PVFS_object_attr.  Changes to get-attr.sm are still underday

Merged changes in Blue branch into Orange branch

Corrected memory leaks in the seteattr and geteattr calls. (lebf_decode_rel()).

Corrected memory leak with geteattr.

SETEATTR will now update the meta-hint flag if the user is changing the mirroring mode.  If the mirror-mode is
changed to NO_MIRRORING, then the mirror flag is turned off.  Otherwise, the mirror flag is turned on.

Corrected memory leak errors with the seteattr request.

Implemented kmeans algorithm based on the state machine, kmeans.sm.
There are two supporting state machines for this: allreduce.sm and bcast.sm.
The simple1_unaligned.c file is updated for testing kmeans.
Sample input file (edge17695.bin) for testing kmeans algorithm is added.

fixed PINT-le-bytefield encoding bug

Split the previous allreduce.sm into send and recv nested state machines using pjmp.
Changed pipeline.sm to setup allreduce.sm in either SEND or RECV type.

fixed invalid free() in io.sm.

*** empty log message ***

Added the server to server communication module within pipeline.sm
to pull the small portion of data when the data is not aligned on strip boundaries.

Changed the default file stripe size to 256KB (originally 64KB)
so that pipeline unit is equal to it.

Setting up Orange Branch

*** empty log message ***

fix memory leak from create response

bug fixes

new authentication protocol

merging hints/events code into trunk.

bmi changes to allow separate bmi packaging

*** empty log message ***

tree communication implementation

reverse merge from HEAD.

reverse merge from HEAD to directio branch

Fix bug in BMI reference counting.  Previously the server was responsible
for both incrementing and decrementing count while working on a state
machine, but this leaves a window of time between completion of
testunexpected() and the start of a state machine.  If a connection was
reset during this window it could cause state machine to try to use
non-existent BMI address, with some odd results.

This patch adds a flag that the server can use to have BMI automatically
increment reference count on any unexpected message before it is passed to
the server for processing.

added a credential to the setattr request message

reverse merge of changes to HEAD since branch.  Includes small file changes.

Merging small files branch to head.  Includes server side precreation of
data files and file stuffing.

db-replication state machine which processes the incomming replication message

Updated lookup/getattr on server/client.

db-rep-send state machine added

fixed size calculation and cleanup routines for le-bytefield encoding

Reverse merged and ported to HEAD.

More work on encode/decode.

Removed warnings due to use of int64_t in several files. Using lld() mask to print int64_t numbers.

fix silly union bug in decoder

protocol size limit for layout information; custom encode functions for
layout in order to enforce limit; remember to free decoding memory for
create request fields

debugging

removing extent ranges from create request

hints and events.

merging enum/int lebf_intialize() update from trunk

merging enum/int lebf_initialize() update from trunk

switched an int to an enum to make sure we get compile warnings if any
operations are neglected in lebf_initialize().  Added a harmless missing
operation.

unstuff works now for basic test cases

set up for create-file mod - stubbed out stuff, etc.
a few small bug fixes to statecomp and statemachine stuff

Big pile-o' changes to make Pete go mad.

The main changes are the implementation of the stuffed create path (now just the create state machine and operation), and the beginnings of the "unstuff" operation.

This also includes the request scheduler and state machine cleanups (required for using prelude with batch_remove) and whatever else I may have happened to change along the way.

This code is able to create files, but writing to them is almost guaranteed to break.

get batch create working (sequentially for now)

stubbed in state machine to handle refilling precreate pools

inode stuffing operation.  also added a keyval_remove_list.

stubbed in new request type: batch_create.  The plan is for this to react
like create, except that it can create many handles of the same type (on
one server) in one protocol exchange

commit of Murali's unifying config file patch.  Also adding a convert script for converting to the new single fs.conf format.

Remove random extern declaration.

new layout parameter for PVFS_[i]sys_create

add a finalize to lebf to cleanup array allocated in initialize.

Merge HEAD changes to TAS-branch.

walt3 reverse merge from head includes merges of pw's sm changes (no state declarations), cleanup of state machine code, and other general merging/fixes.

Fixed comment on decoding the lock_req.

merge recent fixes from HEAD

[phil]: changes to prevent decoding response trailers on errors.  See:

http://www.beowulf-underground.org/pipermail/pvfs2-developers/2007-March/003252.html

Thanks Phil!

Update migration branch to current CVS version

I think there is an error with the freeing of reqs here, but I will
leave it as is for now.

reverse merge of HEAD to WALT3 branch.

Added cases for PVFS_SERV_LOCK.

backmerging of HEAD to branch...

Merge HEAD into Walt's branch.
Rework new state machines to the new cleanups introduced by Walt.

hoist externs into appropriate header file

Merge posix-extensions-branch to HEAD
This branch implemented patches to the 2.6.16 kernel for the proposed POSIX I/O extensions and those patches are under
the patches subdirectory.
It also implements the PVFS2 specific hooks for these system calls.

Tools that may be of immediate use to the pvfs2 general audience is the pvfs2-lsplus utility in src/apps/admin
that should be noticeably faster than the pvfs2-ls utility if there are a lot of objects in a single directory.
Other features are left out by configure and are not even built if the kernels do not support those features
and callbacks.

Based on RobR, Sam and RobL's suggestions, add a flag to the sys_mntent structure to
enable/disable checking of the config files integrity.
Right now, client-core sets this to 0, effectively skipping the fs.conf integrity checks
and the pvfs2- system interface users (namely ping, etc) set this to 1 by default to enable
the fs.conf integrity checks...
Also fixed a few warnings on x86

printf warning fixes

Merge from HEAD to posix-extensions-branch...
This should facilitate merging the posix branch to HEAD...

Modification for migration added, some functions still need to be adapted after moving to a new branch...

PVFS2-hint support + minor changes (verbose request scheduler)

explain protocol error message a bit better; remove assert() that kills server when request buffer mismatch happens

Added request statistics to scheduler, reference from datafiles to
parent metafile (needed for migration) and auto migration tool(unfinished).

Reverse merges from HEAD..

Move PVFS_MAX_SERVER_OP into the enum and audit places where it was used in comparisons.  Requiring manual adjustment of that value has bugged me for some time.

Migration of datafiles between different servers works. Uses msgpairray.sm,
transfers only one datafile between two servers and rewrites metadata.
Todo: Adaption of request scheduler to ensure that migration and write
I/O do not happen concurrently on source dataserver.
Also allow users to specify command line parameter which supresses
transfer of pvfs2-hints.

Added tool which prints file mapping to server aliases. Added
experimental stubs for migration tools. Added kernel tgid to request ID.
Bugfix of PINT_cached_config_get_one_server_str.

Added PVFS-hint, changed tests for pwrite/odirect/transactions/aio dbpf
 support to allow users to explicitly enable or disable the functionality.
In the aio implementation is a bug since somebody else modified it.

Reverse merge changes from HEAD to posix-extensions-branch.

revert style changes back to previous versions.

run maint/pvfs2indent-80col.sh on all .c files to correct style :)

merge of murali's changes from trunk.

[pcarns]: Patch to allow modifying distribution, distribution parameters and number of
data files to use within a given directory. Such setting will only impact creation of future files
within the same directory. It does not change existing files' parameters. All such hints
are inherited by newly created directories as well..
[pcarns]: Allow trove to continue to read a list of xattrs/keyvals
without stopping if some subset of keys dont exist in the keyval db. Modifies the keyval_read_list() function
to return an array of completely independant error codes for each key that is read.
[pcarns]: adjust encode_string() macro to make it safe to call on NULL strings
[pcarns]: bug in the getattr state machine that would cause it to read dirent_count most of the time
regardless of whether the attribute mask requested it or not.
[slang]: fix a couple of bugs in the get-attr server-side state machine whereby the dirent_count was being read
using the dirent handle instead of the dirdata handle. Fixed a return value from dbpf_keyval_get_handle_info_op_svc
so that get-attr would not needlessly hang..
[vilayann]: fixes for the viewdist utility yet again, since the keys for the xattrs have changed...

First cut of the sys_readdirplus implementation where clients can issue a single
system interface call to get the directory entries as well as the attribute information
of the directory entries.
What this means is that we do not need to call a sys_getattr of the individual directory entries
anymore.
(The kernel version is still untested and won't work until the VFS patch is in place! :))

On the server, there is now a new request/state machine called listattr(poor name, i know!)
which essentially does a vectored getattr for a list of handles as opposed to a getattr
which only does for 1. Perhaps, we need to consolidate them into 1 at a later point of time after
others have given this code a look-see!
On the client, there is a new system interface, and a corresponding statemachine implementation
called readdirplus which does the following algorithm
a) Step 1 - Do a regular readdir
b) Step 2 - Collate all meta handles (obtained from the readdir) and send out 1 listattr
to every server that owns atleast 1 handle from the set.
c) Step 3 - Handles that refer to metafiles (determined by the attributes of the response)
need an additional step to retrieve their file sizes. So we collate all data file handles
of all metafiles, and send out 1 listattr to every server that owns atleast 1 handle from
the set.
d) Step 4 - Copy object attributes obtained for all objects into the sys response structure and
also compute file sizes of regular files since we know the individual dfile handles' file sizes
from Step 3.

Clearly, this is a huge win in terms of performance of listing directories which have large number of directory
entries.
Next step is to hammer out the VFS patches and coreutils patch for ls to get the same benefit.
Also added a new utility called pvfs2-lsplus in src/apps/admin that uses this API.

added versioning test.  There's some nasty hacks with set/getenv to be able to test it properly without modifying sysint signatures.  Those will have to be removed at some point.

commit of versioning stuff to branch

merging changes from HEAD to dbpf branch

add more info, and a newline

merging small I/O changes to trunk.  The majority of this commit includes the
new small I/O state machines on client and server, and the modifications to
the client IO state machine.  It also includes mods to the zero-fill code
to perform zero-fill of holes only if they exist instead of zero-ing the entire
memory region initially.  Also, some debug stuff has been added to the
state machine processing code to print enter/exit of state machines if sm
debug is enabled.  The genconfig now includes ioports/metaports for running
multiple servers on the same host for testing.

fixes to small io stuff to get test-zero-fill working again.  Also committing updates from HEAD to make things easier for merge.

committing lld/llu fixes to 1.3.1 branch

moved the llu and lld macros to an internal header so that we don't have to include pvfs2-config.h in the installed pvfs2-types.h header.

* malloc.h is obsolete in freebsd, check for it
* No error.h in freebsd
* linux allows Lu,Lx, and Ld to be interchangeable with llu,llx, and
lld.  Other platforms (solaris, bsd) don't.  Replaced all the L*
with ll*.
* bsd doesn't have mntent.h.  use fstab.h and our own funcs.
* bsd doesn't know about ssize_t.  Use size_t instead.
* bsd doesn't have asm errnos.  Check for each of those
errno macros and if not defined define to -1
* check for db3/db.h db4/db.h header if db path is specified using
--with-db
* darwin has aio support builtin, doesn't need -lrt
* remove kernel stuff in pvfs2-types.h that sets WORDSIZE.  gcc
defines __LP64__ on archs with 64bit longs so we use that instead.
* darwin's open doesn't accept O_LARGEFILE.  off_t is an int64 by
default.  define O_LARGEFILE if it hasn't been.
* need to run ranlib on darwin after libpvfs2.a gets installed to regenerate
table of contents

small I/O changes in sio branch for now

This patch adds a new system interface (listeattr) to list the extended attributes
of a file.
Also adds support in the VFS kernel module to support the listxattr
system call and a bunch of test programs to debug these.

Updated the FAQ to reflect usage of xattrs through VFS and acls through
VFS.

updates to my event changes to bring them inline with trunk

merge changes from trunk since branch point into branch.  Effectively moves the branchpoint to the latest trunk.

Murali's patch to add VFS support for extended attributes.

added extended attribute support

cleanup for extended attribute stuff.  removed non-list server ops
in favor of list type and renamed list type.  cleaned up a few items in
request debug programs.

dded delete extended attribute feature

set_eattr, get_eattr, and get_eattr_list working

updates for adding extended attributes

move errant space

pcarns: initialize stub request and response structure that is used during
	encoder initialization to 0 for safety; some fields are read during max
	size calculation path

remember allocated size so we can free properly; total_size is reduced to whatever the received message size was

- fixed test program ls.c and admin program pvfs2-ls.c to not issue
  an extra call to readdir on every run by fixing a slight logic bug
- changed some readdir debugging
- added a directory version that is passed back from the server to
  the client on each successful readdir call (happens to be the
  directory's mtime encoded as an opaque uint64_t)
- updated client and server response structures and encoder to
  accomodate this
- NOTE: the versioning is not perfect since retrieving the directory
  mtime and iterating the handle list isn't an atomic operation with
  respect to the overall readdir op; i.e. there's still a window where
  a difference may not be detected.  then again, since the mtime is in
  seconds granularity, this should not present a major detectable
  problem anyway.  can use a different version number if problems
  arise often in practice

probably did not do this in a general-enough way: don't try to decode/encode
response-specific portions of a server response if something catastrophic
happened on the server end.

- added a client/server operation and mgmt method
  (PVFS_mgmt_get_dirdata_handle/PVFS_imgmt_get_dirdata_handle) for
  retrieving the dirdata handle of a specified 'parent' object (if
  any)
- fixed a crash bug (race) in trove that could return an object on
  queue addition that has already been serviced (since we've signaled
  the trove worker thread before returning the id in the first place);
  should not be a problem if we used safe_register for the ids, but
  otherwise not worth it

- error code sanity checks, better error handling, and cleanups on the
  sysint, le-encoder, and related server operations

- fixed typo in the mgmt-remove-object request fill
- added some stubs for mgmt-remove-dirent

- use the new sm and stop relying on the old remove

- applied Nathan's expandtab patch (expandtab-2.patch.gz)

  first referenced at:
  http://www.beowulf-underground.org/pipermail/pvfs2-developers/2004-July/000745.html

Updates for distribution renamings.

Update for rename of PVFS_Dist to PINT_dist.

Clean up more naming, and restore basic distribution as the "default" (tho
simple stripe is, as always, the de facto default.

Cleaning up distribution framework.

Regularizing distribution naming for simple_stripe and modifying params to be
include friendly.  Simple stripe distribution has been moved to files
include/pvfs2-dist-simple-stripe.h and io/description/dist-simple-stripe.c

Existing simple-stripe.[h|c] still exist but are deprecated.

- added a chdirent server operation useful during renames of target files
  that already exist.
- still working on cleaning up rename, as it does not properly remove some
  data that is left to be removed

add a mechanism for a server to send a generic response back to the client
when it receives a request with a bad version number or encoding type, so
that the user gets a clear error message at the application level too

fix warnings from adding new state machine

Move from bmi_addr_t to PVFS_BMI_addr_t for consistency.

removed unused variable.

- metadata storage overhaul (remove the METADATA_KEY from Trove_Common_Keys,
meaning that metadata is now properly stored in the dspace generic attributes
rather than as a separate key/val pair)
- clean up the 'find' test program to handle more than a fixed number of
  dirent entries
- misc other cleanups
- ChangeLog update

boring 64-bit printf changes

major rewrite and finish le-bfield implementation

fix typedef names to remove _s; seems to have been confusion with _s convention on struct names

started working on encoding of getattr (and therefore encoding of object
attributes), not done yet

still more cleanup- will hopefully start adding new request types next

more cleaning up

tinkering w/ macros for request encoder

Massive cruft removal.  PVFS_credentials.perms no longer exists.  Some
kernel breakage to be fixed asap.

bug fixes

modified so that strings are decoded in place, added sanity check

filled in more of the API; should be enough for a full getconfig exchange
(req and resp), but untested.  Not convinced this is a good approach- will
think about it a little more before continuing...

bug fixes (stuff I forgot); new encoder now works for getconfig request
struct test case...

started on a new protocol encoder (little endian bytefield encoding).  still
very rought and untested, but need to check in to test some other stuff.
also added a #define indicating the maximum request number in protocol.