It's important everyone agree on the operational transformation (OT) algorithm.
Waves consist of Wavelets
Wavelets have participants
Wavelets have XML documents and annotations
The different wavelets in a wave can have different participants (so parts of the wavelet are private to some groups of users)
changes are operations
operations can be composedCaptain
http://waveprotocol.org
Download source from:
http://code.google.com/p/wave-protocol/source/checkout checkout via mercurial
Google Wave Federation Architecture - Soren Lassen
What's in today's open source release
org.waveprotocol.wave.model
operational transformation (OT)
operations, documents
org.waveprotocol.wave.protocol
protobuf definitions for signing, hashing, internal use
org.waveprotocol.wave.examples.fedone
end-to-end prototype
wave server, wave client, federation port spec
some security and other gaps filled
XMPP mapping refinements
What's missing from today's release
not a reference impl
no persistent storage
crude indexing
no concern for performance, redundancy, scaling
simplistic client
no private replies
no client-side OT, no optimistic UI (get new state from svr )
crude ACLs
no groups
no attachments
simple expensive crypto signing
no Merkle-tree bundles
Q&A (after lunch & Soren's talk )
Is the protocol locked down?
The protocol is a moving target
What is an IQ stanza?
Soren: What you pass in XMPP, these fragments of XML called stanzas, comes from XMPP standards.
Do you have some kind of prioritized list of things that need to be done and a way for developers to indicate interest?
Dan: No. - Good idea and Dan would capture this in a wave.
Do you forsee any policy management at the protocol level?
Soren: The protocol draft will eventually be locked into standards org(?) when it reaches that stage.
Dan: Different providers will have to have different policies for rejection, etc. of operations
How will the XML be compressed and bit rate managed?
Soren: Total bit rates shouldn't be too scary. We don't think it is a present concern. Don't have answers in that area.
Is there anything in the protocol that can accept some notion of "throttling" ? I don't want to accept it at this rate or I don't want to accept a particular operation.
You could ignore messages and force a resend, and the resend might start bundling things.
Are we planning to support any proprietary authentication schemes? (clarification from audience: suppose you run your own server and want to auth through your own LDAP, etc... )
Allen: No, nothing like that exists in the planned prototype implementation. Dan: We provide Open ID. That would flow through. Allen: Help your self with the open code. Dan: bring your reqs to the protocol group.
Where is the XML specified that actually stores the document itself?
Soren: Should be easy to reverse engineer what the XML looks like. We should display this. We will provide specs for this.
Are there standards you are expecting to incorporate? Or are you looking to refine that? Something like RFC822
Allen: No. Waves have a different format than email. We want to accomidate different formats, but need your help to do so
HTML 5? Soren: the representation used has to be amenible to OT
If I do go about creating my own wave server, how can I get gadgets to work?
Dan: All the types of the APIs are very important to the wave experience. We will need to figure out a way to have the API layer be standarized across wave providers. Something like OpenSocial. Robots are a newer piece. The wire protocol for instance is not published anywhere right now. Embed is a more basic Javascript API. I hope it will be easy to stanardize this as well.
Will we open source the server-client?
Dan: Yes, the "lion's share" but will take awhile
Can you speak to Google's committment to wave? is Google comitted? How can we know?
Dan: Yes, we are rolling this out to 100K users starting Sept. 30
Gmail?
Dan: Email isn't disappearing anytime soon
Will we be publishing things about Google's DB layer?
Soren: We are hoping to do this. You will find in the most recent white paper on attachments there are some details about how things are stored. Most important thing is that we use BigTable, megastore and other google proprietary tools, but we need to make it work w/ storage tools that work outside of our instances.
There are certain operations that exist in our Wave sandbox but were removed from released code?
Soren: Not everything was working correctly so we removed some things. A lot of operations were added as we need them. Lots of questions: who generates ids for waves and wavelets? Clients are equipped w/ unique id generators. The first operation you do on a document, that creates the document if it is not already there.
You could have two wave servers one of which rejects a change from another. What happens? Isn't there a forking problem?
Dan: The wave is mastered on a particular wave provider. Not asking for an ack, but a command (from master to other server). Still some things which need to be worked out.
:: Pause to fix mikes ::
Soren: We try to maintain some user experience principles in our client. One is that you can't take a wave from someone else. We have a 'remove participant' at the protocol layer. One thing that can happen here is that someone removes and the other person did not want to lose that content, and we want that the person removed can keep their content and history and create a fork, but we will probably need a tweak in the protocol to support this.
Dan: This sort of already happens w/ private replies. In the UI this is represented as a...
What is no-op used for? Who would it take to combine add and remove participant operations into a single operation?
Soren: We've thought about this a lot. No-op is used for "Debug" (from Dan), a bit obscure, modify w/o creating new Wave. Can add/remove an arbitrary number of participations, would be nice to have one operation . We don't know?
Mutate document is complex operation, has this property. You can combine an arbitrary number of mutations into a single mutation.
Soren: yes, this has some interesting mathmeatical properties
Is there a process for distinguishing capabilites for different types of waves? Esp. for chatrooms and blogs if you can't modify them.
Allen: Two kinds of process which spring to mind. Conversational document w/ permissions in wuch a way that inserting replies into a blog is not allowed because it is not suport
Follow up question missed
There is going to be some kind of access control.
Dan: We kind of looked at that as commenter mode. You can't change things that were added by other
Soren: we should pride ourselves that there are no quesitons about the security protocol.
What sort of scalability has come up so far and do we expect to use things like BigTable?
Soren: Yes, things have come up and yest we expect to use things like BigTable. Storage is one place where you need to... We'd love to talk more about it but I guess we haven't come around to a white paper on that yet. Too big a question. Come talk to me.
Will the reference implementation have persistance?
Dan: We will need to work w/ people who have exp. with massive perstistent data storage that doesn't use BigTable.
Hadoop is out there. XSpace. Postgres...
How often do you send operations assuming that the client and server respond immediately?
Soren: As soon as the window clears, we made a couple of modifications in the OT protocol compared to what the OT supports, we can get around to some things that OT has to solve. Another thing we do that , another thing we do is that everything has to converge between each delta instance. What we do, manage outbound traffic, put it on the wire. If you have a lot ot say.
Backed up client can send bulk operations?
Soren: Yes, this is a side-effect you get. These will generally be bundled nicely together.
Is that the server does the same as client?
Soren: Might be some element because we use TCP conections, if
Dan: Def. room for optimization improvements (response to unheard question)
Apache?
Soren: Google hasn't its own server. Right now plain TCP.
Dan: FedOne prototype, completely independent of Google software, running it on some machines that aren't near data centers.
Missed
is Google going to hand out certificates for everybody or can we get our own certificates?
Lea: We don't plan to be the folks, but we need to have some standardization. What system we would use is up for debate. We are not handing them out, no.
What ports will Google Wave be using and will it be compatible w/ servers running over XMPP (Jabber, etc.) ?
Soren: Prototype uses only standard component protocol, and extensions. This means it uses the same port as XMPP for now.
But most people accept that when you are running on the same prot that you would implement more of XMPP than just the core.
We are not sure that this will be a problem but need to discuss it more. XMPP folks will be here later if they aren't here already and we have a seperate session on that.
Do we need to standarize wire protocol used by robots to the robot gateway? I think this is necessary.
Soren: This is not so clear cut necessarily. We have a Robot API.
But the API is. .. ?
Soren: We should probably standardize both protocols/APis.
When we start to bake in ACLs into the Client will this be via operation lists or pre-defined Roles? If Roles can you inject other Roles from another server?
Dan: We are still thinking about the moderation aspects. Wave is still a very open system. We touched earlier on the commenter role. In terms of injecting roles, I go back to Master-Slave, will be up to the person who owns wave what roles are available and which can be asked for.
Question about the Federation and white paper documentation, mention of 'globally agreed order'? Who controls this and makes sure it is unique? The order that operations are applied to the wavelet.
Specified in OT doc (?) + Crypto-magic fairy dust
Clarified that the globally agreed order is up to the server that started the wave and owns it.
Is it reasonable to expect that Google client will work with anybody elses server? Is it your expectaation that people who build servers will also necessairly take on teh burden of building their own clients?
Dan: Will be evolution. When first go out will talk to each other. We will need to standarize the Server-Client layer, but this will take a lot more time, when we do you can use a stanardized client.
Aleln: We would love to standarize the Client-Server ptoocol. We can't do yet, but we will decide what order to do things in based on feed back in the API/Protocol Groups.
When you converge towards the state that the server also converges towards, you may take different steps than the server takes and for the play back do you use the server states or client states?
We use the server states on playback, we deal w/ clients that are always online, therefore you would not normally perceive a big difference. You want consistant to gloss over. What comes out on the end preserves all fof the transformations. It is clear that there may be situations where.
Dan: There are some situations where this can be confusing.
It is important form a lot of perspectives to esp. in clients as it relates to when things happen. We need to think about time date stanmpign avarious operational transformation and how that gets played back? We need to play strict attention to ordering of operations
Danan: Every wave is owned by a wave server and time stamp comes from that server.
I think that will be a problem. They have time stamps, I have something, you have something, no one knows what time stamp is ? FRCP, Fed, regulation, Orgs, have to maintain copies of org. records, you need to be able to play back things, need to be auditable.
Dan; I agree.
audience member: print everything out
Dan: I am open to continuing questions
If you have a "commneter" mode where ir doesn't do character by character OT, (like a chat room_) how much extra capacity can I sqeeze out?
Allen: Complicated question. Costs come from sending things out to Robots indexing, indexing, . Yes, it looks like there are savings there, but we can'ts ay how mcuh
Soren: Operational Transformation has to be cheap. It is really fast.
Dan: Very core to the experience. If there are various levels of compliance...
Another security question: To the extent that this is an extension of email can also be an extension of abuse. Do you see anything rising up that could be the emergent abuse (such as phishing) ? Abuse use cases... Have you ever lost a night of sleep for a system which does robots or do you sleep very soundly?
we are going to have an abuse breakout session
Lea: We are thinking very careful about Robots. All kinds of ways to abuse things. Also very interesting ways in which you can limit abuse. And able to automatically rewind abuse. Really cool system. really cool possiblities.
::Justin deletes message from Lea. then adds it back to demonstrate possiblities of abuse::
dan: Our role is to help people understand. We use it as a tool internally quite a bit. And we have learned a lot about wave etiquette. Interesting to evolve models as Wave itself has evolved.d (2) 6K developers in, 20K to work in over next month. Sept. 30 we plan to get this out to real live users.
Soren: We have the Imaginative power to do these things . All about gadgets and robots and making things work on top of this platform. Pretty fascinating user engines, based exactly on OT which allows an optimistic client user interface. Every day we look at our client and come up with interesting things which it enables, things you can do in the browser that are better than you can do in non-browser applications.
Dan: The benefit to Google is that web-based collaborative applications make the Web stronger which also benefits us (Google).
Can a particular wave provider transfer ownership of that wave to another wave provider?
Dan: Not in the protocol at this point. Ob. some sort of graceful degradation needs to happen there. If you have other wave providers, you can have an election system.
Soren; Forking away is probably the simplist solution.
If you do what to run your own web server what kind of sizing, security issues will you have?
Allen: So are dealing w/ that ourselves.
Can you share some data? For instance, how many users can you support on a single server?
Allen: I'm not comfortable sharing the numbers we have, and these numbers will change.
Dan: This is also very difficult to map to non-google architecture.
I have a question for the audience: Is there a scheme for distributing a wave across multiple servers? seems that presence/state is pretty coupled in the server right now. bueller?
and I mean not in the client/server federation sense, but distributing load for a single wave.
Dan: I totally agree. We should keep things as distributed as possible.
How do we decide when to create new blips vs. new wavelet? If I hit reply looks like the client is creating new wavelet.
Soren; It is not a new wavelet. It is a blip(?). A private reply would be a new wavelet.
How are comments annotated? How do you know if something a comment?(inline reply)
Soren: This is not in the protocol. That is all things that we want to express in the HTML.
You have been using Wave internally for quite awhile now. How does it affect the work you do?
Dan: It is great that other people can fix your typos (yes! comment by typist Joel). We have an information overload problem we are working through.
Allen: Great to send something out to the team and something that will increase if there is interest, very easy to get a document/idea out there and get comments.
Google client doesn't do any transformation, everything goes to the server. At some point people will want to do editors that work offline. At that point, when the change sets come through it will look different in different places. The OT process aims to get the same result for different changes submitted in a different order. How do you handle offline Wave access?
Soren: With regard to the first question, we want as few lines as code as possible in the prototype, so we chose to make it a thin, dumb client that doesn't do OT. But the one we use when you disconnect you can keep on editing and keeping on editing, and keep convergent and it is all syncronized. The one thing we don't have a the moment we cannot persist the state on the client. If you close the browser it is gone. We think that OT is a wonderful vehicle to get that done.
Will the ability to proxy and cache for non-master wave servers be included to the protocol?
Dan: This is already built in. ::turns to Soren::
Soren: This is the expectation. The wave server will cache things. It stores it for you.
Dan: You have the option to cache them how you chose.
::Break::
Security Panel
Primary typist: Casey ( tring to keep up, Lea talks quickly)
Participants: Ben, Dirk, Lea.
Communication Model
data privacy
multi hops
No one other than the intended recipients should be able to read data.
want SSL on every hop! not always legally feasibly
Data Integrity
attribution :& integrity
Signing Strawmen
PKI every delta - public key signature on every change: VERY expensive, 1-5 ms pr delta
Symmetric key MACS on each delta. token on every delta
if you can verify you can also sign
so anyone on a wavelet can forebgv messages from everyone else
at best the hosting server cld forge messages from everyone ( MIM )
so: Bundling
we want PK sigs and efficency per delta
how? share sigs over multiple deltas
based on merkle hash trees, so we can remove parts of the tree
...
Winnowing - sign bundle and winnow it down ( pictures and hand gestures )
History hash
need to ensure that data is deleted
cumulative hashing of all operations give unique ID to each operation
also ensures that operations cannot be droped from servers view ( dectable)
Signature Data Structure ( code slide )
ProtocolSignedDelta, ProtocolSignature, ProtocolSignerInfo : http://code.google.com/p/wave-protocol/source/browse/src/org/waveprotocol/wave/protocol/common.proto
message ProtocolSignedDelta {
required ProtocolWaveletDelta delta = 1;
repeated ProtocolSignature signature = 2;
}
message ProtocolSignature {
enum SignatureAlgorithm {
SHA1_RSA = 1;
}
required bytes signatureBytes = 1;
required bytes signerId = 2;
required SignatureAlgorithm signatureAlgorithm = 3;
}
message ProtocolSignerInfo {
enum HashAlgorithm {
SHA256 = 1;
SHA512 = 2;
}
required HashAlgorithm hashAlgorithm = 1;
required string domain = 2;
repeated bytes certificate = 3;
}
Certificates
Openfire (or jabberd, etc.) uses its own certs
Wave server also uses certs for signing of deltas
FedOne implemenetation expects:
Private key PKCS#8-PEM
signer cert X509 (DER or PEM)
intermediate certs
... as files ( command line options)
you can get free certs from xmpp.net
if U use self signes certs fed peers wont eaccept your deltas unlss they run with --waveserver__disable__verification
this is different from crossfire, which will work with self signed certs through dialback
FedOne impl will by default trust JDK-provided CA;s, plus the XMPP ca
self-signed certs are in the example to make it easy. in the production federated network, self-signed certs will be turned off
QA session
[[I'll do the answer you do the questions, k?]][[OK]]. Okay, looks like you are doing the answers I do the questions :) oops! :)
[[ I htink this question was about federation trust chains>]]
Lea: so we had this discussion about doing this, in particular a lot of us are intereste din encryptino, but we;re not looking at doing this at the federation level, but at a higher level - this wavelet is encrypted here is the key. a lotof these are matters of end user trust - we've been doing a lot of thinking about it but don't have any answers yet.
Google should really think about how people can rachet it up and performance balance it.
this is an interesting topic and we're not quite at that level yet
If I have a federation for healthcaAPA?
Lea: it;s been a while since I looked at HIPAA, I actually think is interesting about using wave in the healthcare context, which we;re not at yet, is because of HIPAA auditing and you're federated in health care providing you're essentially signing al lthis "stuff." this would be an intersting application - health care, but it's really hard due to the nesting documettings
federating with yourself sounds like it would make auditing easier
Otherwise it could be a great applicatoin fore the healthcare industry, doctors could exchange images, could cinlude identifable (& private data) ?
I'm working on a summer project and building an open-source porject on top of Google-Health. How cawill the secuirty of the project work with Google health?
Lea: Google Health API and Data? so right now it does not. we're very paranoid. the two systems do not talk to each other. google health has to wait until this is more complete.
No work around for that right now?
"google is not hooking up wave and google health"
This whole problem is a hairball, almost unbounded. Can be a risk that you give people a false sense of security, where gaping holes remain unaddressed. My question is: how much of this do you expect to get done before you release the server and reference servers What security things are gating factors for releasing reference implemenation? Generally speaking, is there anything that is preventing the release of a reference implementation?
Lea: if you can think of security stuff we haven't address tell us!
Soren: things like ACL are not up to snuff, we'll be coming to you guys ( securyt group) to talk about requirements. we're in a fairly good state for the reference implementation .. [[ I didn't get that well ]]
If you put some measures of security here, like ACLs, this raises my expectations of how things will work...
Leay the point it must sound like we are saying that we're going to have security, and yet we're going to say "Buyer beware " and all that - but we're saying that because we don't believe that we can have anything useful w/out security. there are a lot of people out there with a lot of free time and who want more money. we're aware that we need to do security but tellus if you see us missing something
Is there any way to trace all our certificates, is there any way I can trace all of that information? If I have several servers, but one is known to be not secure. I will not allow a wave to display that information. is there anything in the protocol to allow the interrogation of those servers?
lea: at the security layer we are saying that we can say who is talking to us, and there has been th ought about saying ": you're a bad server and throwing out a lot of abuse" but... but that's going to be talked about in the following Abuse session.
Is there going to be any standardization of the client server interface as far as security goes? T
Lea: there are a huge number of aspects... not sure what... you are asking?
it is up to the individual wave providers to decide what they are going to do. corporations will do their own SSO, we're not going to dictate as far as other implementations are concerned.
Supportability could be, what kind of support for Single-sign on..
guy: as far as federation protocols is concerend we stop at the federated server level.
At this point your are not contemplating any way to cross-authenticate users at all?
guy: no
Basic problem is that anyone who knows my address can send me sh*t, how are you planning to address that?
they will cover that in the abuse section, but you will know who sent what due to authn, so you would be blocking the individual users andshutting them down. or you may decide you don't trust all servers. exactly how that happens will be up to the individual servers. [[?]]
Just in looking at the sort of facebook-social-network model, any message you receive, more open than this...
more like email in the exp that anyone can send you a message, but there will be something to determine say that someone has communicated with you before they will get easier access
Is there a concept of a sub-domain? If I am running multiple wave servers w/in my organizaiton for example? Can you have an internal domain?
sure the whole l point is that the originating server is always identified by its signatures.
Can you talk a little bit about user provisioning? We have a compnay w/ 8000 people we want to start waving. How do we get our users in bulk on to the system?
well if its your own server it's up to you, we're not dictating provisioning details , we're saying that once you have done that you send a message to identify the super and the message is signed identifing the server
Do you guys envision any sort of network of what is a trusted server?
lea: right back to the Abuse seminar: we will not bless your server, WE are going to say Hey youare sending us spam which makes us sad. we hope ppl will get together and decide,say, not to talk to server Alice because it's spending a lot of spam.
Is there a notion of email BCC in wave?
Lea: i don't think it's built in right now, there has been a lot of discussion about related concepts,
o you can make a copy of a wave and replace the body of the message so its kind of the same thing
Is that through an alias? Putting it in a different group?
::positive gurnt: [huh?]
When you copy a group you get a static?
when you copy a wave you get a static thing and you can put other people on it but they won't see the live updates on the originating wave
Any control where you cannot forward, print, etc. a wave?
the ACL model isn't worked out, right? there are practical limititainos to some of these suggestions like "no printing this message"
There was someone who mentioned a button where this can be circumvented, where no one new can be added? An off the record feature. This gives a measure of comfort for people who want to talk off-the-record.
lea: so yes, I think that it is an interesting issue which we haven't worked out yet, we want to see conversation about this, what we want it to look like.
The way I understand right now. I can access everybody in the directory? There is no kind of hiding right now? I change whether or not someone can see my presence?
Lea: what is the latest version of the directory?? yes we do know that seeing presence is controversial, we thinkt here will be measures about hiding that, it's not going to be magically available to everyone.the idea is that your presence will have to have that approved by somebody
[whew] [Great job Casey, you had the difficult part!] ha thanks I'm dizzy from typing. [I'm going to create a new wave for commenting on the presentation, sound good? That will help us organize potential contributors.] [ one wave for logging the Q&A, another for the presentation]?
Allen: Slides will be published, but will take a little while
FedOne demo
The rest of the demo code was pushed and it reports to build ( applause )
http://tr.im/FedOne
lightweight impl, fed gateway, text client which you may of seen at Google
we can't cover all the detail right now
demoing 3 clients, 2 in one domain, other in another
sitting on 2 machines in oregon, opeking the ports now or soon and so you should be able to run aginst them. they will be upfor the next few months. if you have patches / contrib use the email list that some of you may of heard of the wave protocol group.
OK, those XMPP servers on the backend support component extension, OpenFire,
as soon as you add a person from the other server the actions are pushed across
[[ DEMO with multiple terminal windows with really big fonts]]
presenter starts a conversation with anthony, an inbox with all the waves ( in ASCII ). All these waves are in-memory, feel free to write a persistance impl. presenter goes on to add anthony via terminal interface.
on other projection we see java lanucning client from cmd line... anthony replies in text client.
present adds sam at other wave server
[[ lots of logging flies by in another projector. lots of key exchanges fly by, etc. ]]
presenter: hmm.
presenter: er, it was working 2 minuites before , but this room knows what it's like (laughter)
they get it to work after a client restart [[ I think? ]] applause
Q: what is the point of this? people typing in text windows?
A: it's really an experiment and a place for integrators and developers to kick tires.
A2: we may check this on a regular basis to respond. A3: we could write a robot to talk back to you guys as you connect from your clients/servers.
[[ whiteboard is added to array of presentation technologies ]]
presenters go over the stack: federation host, wave server, client front end and federation remote in one process. client front end talks to text clients via a not-the-real protocol ( this protocol is intended to be an example). hosts and remote talk XMPP into OpenFire.
Q: there is a mess of XMPP talking at OSCON this week? are you guys there are what? A: yes we are ( their OSCON reps walk in, applause )
Q: will you be keeping these in sync?
A: yes that is a goal, there will be a cmd line option for setting the protocol version as thing leapfrog/get out of sync.
Q: is the protocol spec up to date
A: it's a static snapshot and it'sl not yet refresed ( as of this time ) A2: it's in HG, the most recent one.
Q: is the client server spec in here? A: no it is not
[[ missed question ]]
A: there is an apache java project for XMPP impl, but it's nowhere near working we looked at it for a week. Openfire can be just used
At this point there is a split of activities: a discussion of span and abuse in one area,
and hacking / discussing the FedOne release in the other.
(spare bulltes for chasing) (ammunition)
.jpg)