[Federated-fs] Summary of root fileset discussion

[Ellard, Daniel]

Root Fileset Discussion
        
Revision date: 07/22/2008

This is a revision of the document that Renu sent out a few weeks ago,
bringing it up to date with respect to the discussion during the
fed-fs conference calls.

Definitions

1.  A fileset that implements the "top" of the namespace is called a
    root fileset.
    
    There may be more than one root fileset, but the number is typically
    small.
    
2.  A root fileset is named by an FSN.  Such an FSN is called a root
    FSN.  Servers that host instances of the root fileset are called
    root fileset servers.  The FSLs of instances of the root fileset 
    are called root FSLs.

    There will be a special flag or attribute for an FSN that labels
    the FSN as a root FSN.
        
    An NSDB must be able to respond to a query whose result is the set
    of all root FSNs that it knows about.  The NSDB schema will be
    extended, if necessary to efficiently support such a query.
        
3.  The root fileset consists of a hierarchy consisting entirely of
    directory nodes and junctions -- no other type of file system
    object is permitted.  The leaf nodes of the root fileset are  
    junctions or empty directories.
        
        
OPEN QUESTIONS   

1.  How is the root fileset represented?
        
    There were two proposals for how the root fileset might be
    represented and distributed among the root fileset servers.

    1a.  The NSDB for the root fileset stores all of the information
        contained in the root fileset.
 
        In order to make changes to the root fileset, an admin makes
        changes to the contents of the NSDB.  The root fileset servers
        will discover these changes (see next topic) via the NSDB
        and change their local copies of the root fileset.
 
        The NSDB schema will be extended to include records that
        represent the contents of the root fileset.  The details of
        this schema extension have not been worked out yet.
 
        (this information may be replicated across more than one NSDB,
        although the details have not be worked out)
 
        The root fileset servers get information about the root
        fileset from the NSDB and use this information to create and
        maintain their own instances of the root fileset.
 
    1b.  The master copy of the root fileset is implemented on the
        root file fileset servers and some replication protocol is
        keep all of the root fileset servers in synch.
 
        In order to make changes to the root fileset, an admin makes
        changes to its local instance of the root fileset (or perhaps
        a designated "master" instance).  The root fileset servers
        will discover these changes from each other via some
        filesystem replication protocol and transparently update their
        local copies of the root fileset.
 
        In this approach, the NSDB only stores the definition of the
        root FSN to help the servers find the root, but no additional
        information about the root filesets (other that what it stores
        about all filesets).
 
    CONSENSUS:  given the current lack of a portable (cross-platform)
        replication protocol, approach 1b is not feasible at this
        time.  Since the semantics of the root fileset are quite
        constrained, however, and updates to the root fileset are
        expected to be infrequent, approach 1a is feasible.
 
 
2.  How many NSDBs store the root fileset info?
 
    Is the root fileset information (as described above -- whichever
    approach is chosen) replicated across all of the NSDBs, or a
    subset?
 
    If more than one NSDB stores the root FSN information, how are the
    multiple NSDBs kept in sync?  Does one NSDB play the role of
    master?
 
    CONSENSUS:  this hasn't been discussed in depth.
 
        This ties in with some of the later issues.  It's clear that
        if we have a discovery protocol, a client must be able to find
        an NSDB that can tell it a root FSL.  The open question is
        whether all NSDBs have this info, or all NSDBs know how to
        find an NSDB that has this info, or the client must know how
        to find an NSDB that has this info.
 
3.  If we choose approach 1a (the current consensus), then how do the
    root fileservers learn about changes to the root fileset?
 
    There are three main approaches:
 
    - PUSH: the NSDB informs each root fileset server of changes as
        they occur.
 
    - PULL: each root fileset server regularly asks an NSDB for info
        about the contents of the root fileset (i.e., a delta from
        the last query).
 
    - HYBRID:  a combination of push and pull.  The root fileset
        servers can pull information from the NSDBs, and NSDBs can
        push info to the root fileset servers, depending on context.
 
    DISCUSSION:  PUSH is a attractive from several perspectives, but
        has some awkward cases:
 
        - If a root fileset server is unreachable when an admin tries
            to make a change, does the system have to block until the
            root fileset server comes back?
 
            This is not acceptable; it negates some of the benefit of
            having multiple root fileset servers if they all have to
            be up in order to make changes!
 
        - If the system doesn't block, how do disconnected root
            fileset server learn about changes they missed while
            disconnected?
 
        To the best of our knowledge, LDAP does not support a PUSH or
        HYBRID update approach.  A delta PULL can be accomplished by
        logically time-stamping the changes and having root fileset
        servers only asking for changes that have happened since their
        last update, but this is a bit ugly.
 
        If we strongly desire a PUSH or HYBRID model, then we may have
        problems with LDAP.  In the most extreme case, this could
        force us to review the decision to use LDAP.  (we might still
        use LDAP for the underlying directory, and to handle
        resolution queries, but admin requests would go through a
        higher-level interface)
 
4.  Are there any constraints on the layout of the root fileset?
 
    Is the root fileset one-level, like AFS?
 
    Or can the root fileset have multiple levels?
 
    Do we want to be able to limit updates to parts of the root
    fileset by ACL or some other mechanism, so that updates to the
    part of the namespace related to one federation member can be
    controlled by that federation member?
 
    DISCUSSION:
 
    - If we stick to a one-level fan-out, then life is MUCH simpler
        than the general case.  The only thing about the root fileset
        that an admin might care about are:
 
        a) The FSN bound to their name at the root.
 
        b) That nobody else binds to that FSN without their
            permission.
 
        We can enforce these constraints with very simple mechanisms:
        essentially, the root fileset will only accept bindings to an
        FSN X from the admin authenticated as the admin of the NSDB
        named in X.  In simpler language, the only thing that someone
        can add to the root fileset is a junction to machines they
        own, and the only then someone can remove from the root
        fileset is a junction that points to them.  The NSDB will
        reject any requests by anyone to add/delete any other
        junction.
 
    - If we stick to a one-level fan-out, then the root may become
        insanely huge.  Doing insane things is not recommended by the
        best practices documents, but it may be unavoidable in
        pathological cases.