Network Advice
Submitted by jalynch on Thu, 05/27/2004 - 12:34.
Hi,
I have mentioned elsewhere that I was interested in using JXTA to perform peer operatoins within some of my projects, and specifically the component manager that I am making.
I have since attempted to use JXTA and found it rough going. The JXTA specification is stated in language independed terms, and in complying with the specification, the java implementation has had to use some techniques that are generaly considered bad form. Also there is the idea that all communications are based on XML advertiesments which are posted to rendezvous and there await interested parties to search for them.
This proccess creates a very slow, asynchronus communications flow, wich is a daunting prospect to developers, because you are unsure whether your threads are waiting for a responce, or there has been a failure.
All in all, I became dissatisfied with JXTA, and began to consider how I would implement a peer protocal.
I like to think that my ideas are unique or revelutionary, in the sense that they do not conform to old ideas, but explore new possiblities. I like to think that.
I have come up with, what I think is a good idea. It is completely untested, so I will keep it to myself for now, but I do need some advice.
The entire protocal hinges on unrestricted access to one UDP port and one TCP port. These ports must be open to the internet, by nat, dmz, or direct connection, it order of favorablity.
My question is this. Is it too much to ask that these ports be exposed? Any one who is behind a firewall should be able to configur it, and those that are not will be open. Is this acceptable?
Thanks,
Alex
- Login to post comments
Details
I have now completed the first tests of the peer protocal mentioned above. I implemented the plugin transfer function of my component manager in this new peer protocal and successfully completed a set of chained transfers. (plugin a needs b needs c).
I have not tested every part of the protocal yet, only the parts that were relevent to the component manager; however I will now outline the protocal, as I promised.
First let me repete the goals of this protocal, most of which have been made important to me by their abscience in JXTA.
1) Simple, straight forward symantics.
2) Familar, java data types (not JXTA xml documents).
3) Synchonous peer comunication (as-in not asynchronous).
The protocal starts by emiting a datagram packet on a well known udp multicast port. This datagram contains in serialized form the String "open", a String that is called the peerTypeUID, and the InetAddress of the sending peer.
The "open" String is used as a communications check to insure that the datagram was sent by someone complying to the protocal. The peerTypeUID is the String returned from abstract getPeerTypeUID(). This method is implemented by extending PeerNetObject (more later), to return a name that will identify a peer as a member of a certain 'group' of peers. Only peers that have the same peerTypeUID will be able to communicate (more later). Lastly, if open.equals("open") && peerTypeUID.equals(this.getPeerTypeUID) then the InetAddress in the datagram is added to the local cache of the peer that recieved the message.
The datagram is sent on a multicast address with a large time-to-live. This could potentially result in a large number of duplicated addresses in the local caches of peers in the near network. Because of this the cache size is limited to 7 InetAddresses. This limit results in a less circular (or more liniar) path between peers. (The ttl and cache limit may need to be changed after they have been tested for optimization).
Above is the first part of the protocal, initial contact. Next is the second part, data-transfer connectivity.
The second part of the protocal is built on RMI. PeerNetObject extends UnicastRemoteObject, and defines extra behavior specific to the protocal.
First of these behaviors are isActive() shutdown() which check thread state, and end thread execution respectivly.
Next are behaviors used to build propogate sub-protocals. getPeers() returns an InetAddress[] from the local cache. resolvePeer() accepts an InetAddress and returns an instance of Remote, which is the remote peer signified by the InetAdress. If getPeerTypeUID is unique to this type of peer then the Remote object can be safly cast into the remote interface of this PeerNetObject.
Lastly are the final implementations of basic sub-protocals: propogate, poll, and locate.
Propogate takes a String mehodName, Class[] types, Object[] args. These values are used to invoke a method dynamially. This method is first invoked localy, and then the propogate signal is sent to every peer in the contact cache. Those peers then repete the proccess, invoke localy and relay. Now, because of potential duplicates in contacts a single peer might be contacted with the propogate signal several times. To avoid this a PropogationMonitor is used.
PropogationMonitor is UnicastRemoteObject that acts as a front for a Collection. InetAddress of peers are added to the PropogationMonitor to represent peers that have been contacted, and it can be queried as to whether or not a certain peer has been contacted already. These are the add() and contains() methods of the collection. PropogaionMonitor is Remote, so the dataset is not transfered to every peer involved, only the remote interface.
In this way every peer that is alive on the peer network will recieve the method call only once. propogate() is synchronous but could easily be put on a separate thread.
Poll builds on the propogate sub-protocal. poll() takes arguments to invoke a method. The diference is that as the propogate takes place, the return value of the method is added to a runing tally of returns. The tally is maintained in a subclass of PropogateMonitor, PollMonitor. PollMonitor compaires return values with equals() and keeps only unique values. With the values is kept the number of times that value has been returned. poll() returns a List containing VoteValuePairs, in order of most votes.
Last is locate(). locate() takes the method invocation data and behaves like propogate(), except that locate() returns with the first non-null return from the specified method. This means that not every peer may be contacted.
This is the extent of my work. I am plesed to say that this protocal is very easily extensible. I used this protocal to implement the plugin transfer functionaly of my PluginManager by defining only two methods (one is getPeerTypeUID and the other is locateClass() which I call by the locate() sub-protocal).
These are my results. What do you think?
Alex
No Comment?
No comment at all? Aren't there any oppinons?
What do you think about scalablity, or performance?
What about reliablity, practicality, or general usefulness?
Awaiting your comment,
Alex
can you tell us more about what you want do implement?
What more exactly you wanna implement? I have no clue what JXTA is...I think a lot of us have no idea what you are talking about... maybe you can tell us what your system should do..
update of system like for gentoo ?
I think he is talking about an automatic update of JNode components/plugins.
A computer with JNode can discover other computers that are running JNode on a network or, at least, that implement his protocol and then download an updated plugin.
Fabien
Trust
Hi,
are you sure that I can Trust another Pc somewhere and download Java ByteCode to run on my Pc ? I see some potential Security Problems ... with Digital Signatures could be the same problem, but with a minimum level of trust.
In effect, Gentoo works having the updates NOT from other Pc, but from a list of Trusted Mirror Servers, and they are more controlled by the team (I hope ...).
The JNode Web Site is already mirrored worldwide by SourceForge.
The best couldn't be to have also the other mirrors on the SourceForge net ? With a list of main Servers to contact.
I don't know if this could work related to the SourceForge License Agreement, and to have this in automatic mode (to download a file from a SourceForge mirror I must choose manually a mirror from a web page, but we need all of this in automatic).
If this could be a problem, we could use other mirror sites ...
Bye,
Sandro
Security et. al
I am not a security expert. However this is my understanding, because the code transfer is accomplished by transfering complete jars, the jar would maintain all of its signatures. The signatures directly affect what the code in the jar is able to do. If there are no signatures then it will not be able to do much; however if the jar was signed with a fully trusted signature then it could do anything it wanted.
The signing of jars is the security that I depend on. Is it not enough?
Now, as for networking, I do not want a centralized peer sceme. Centralization always results in a bottleneck on the server side as hundress and thousands of people are trying to download at the same time. A decentralided peer net, however disperses bandwith consumption evenly across the network as a result of natural network topgraphy.
However since monochromata mentioned that ISPs dont forward multicast signals, I have been unable to think of another sollution.
Do you all have any ideas?
Alex
Security and Digital Signatures
> The signing of jars is the security that I depend on. Is it not enough?
Some year ago someone has copied the Digital Signature of Microsoft Corporation (I don't remember if the original, or if they have meke a digital signature similar to tha of Microsoft), and they have created some Viral ActiveX signed with this Certificate ... of course downloaded not from Microsoft Web Sites, but from others ...
Having all the Packages to Install Digitally Signed is a requirement, but could not be enough (but enforcing the download of all OS Critical pieces from JNode official mirrors could be another piece of security), and having all JNode OpenSource don't you think that someone could get the keystore to sign some jar with the JNode Community Sign ? OK, but I could exaggerate in this vision.
Be,
Sandro
Re: ...
Well the multicast stuff is fully ok, you will just need an additional unicast protocol used when no nodes can be reached using multicast. When you do so you might use some kind of seednodes file containing a number of remote nodes that accept unicast requests from other nodes.
Part of the unicast protocol must be mechanisms to distribute information on which nodes accept unicast requests. This means that (a) nodes need a means of telling other nodes which other (unicast) nodes they know and (b) nodes need to be able to have a look at the list of nodes they know and tell (all) of them they are there and are also able to accept unicast requests.
These network management capabilities are common to all P2P and mesh networks so there might be a number of working solutions already available that might be studied before implementing a custom one.
I don't know whether signed jars provide sufficient protection. As I understand signed jars they are used to ensure the contents of the jar file haven't been compromised while being transported over the network. This only works if you are obtaining a JAR file from a trusted source. If you can't be sure that the sending node creates a non-malicious JAR this doesn't help. Maybe I'm wrong, don't know it 100%.
Regards,
Sebastian
Jars and Stuff
I believe that the certificates attatched to a jar describe who the jar is from and the signature says that the jar has not been tampered with since the author okayed it. These two together are what make up security, (I think).
As for the networking, my best goal is a completely decentralized peer network. However, you are right, isp's donot forward multicast. I may be forced to settle for a unicast core (centrailization), but I still want to ask, does anyone know a way to make multicast(like) trnasmitions through isp's?
Not too much Security
Hi,
I can create a self-made Certificate, and then the classes inside are trusted to make anything. OK, but the User (Experienced) should be warned that the Certificate is Home-made ... and this is another kind of problem. At the moment a JNode user is NOT this, absolutely, but in the future could be.
For ex. this is a common technique needed by Java WebStart to grant some permission to a Client Application, and I can do all of this without using a public Certificate.
What you are saying works OK, but I think we must find a way for the System to protect itself by some dangerous operation (inspecting the code to install ??), for ex. a full delete of all files on te hard disk (without considering the user grants in the FileSystem, which could only minimize the damage).
I suggest the creation of a Work Group inside JNode dedicated to Crash Tests, to try to force the System in many ways.
Bye,
Sandro
Re : Not too much Security
I agree that the system should warn of self-signed certificates. Self-signed certificates still assure that they belong to a certain entity, but there is noone telling you that this entity exists and can be trusted (which is the why certificates are normally signed by a Certificate Authority).
Can you please elaborate more on how you think code inspection will work better than denying permissions?
Crash tests are always fun, but I don't know if Jnode is currently in a stage where they make much sense.
Regards,
Sebastian
maybe JGroups can help
Maybe the JGroups project can be usefull (licence is LGPL).
It can do multicast or unicast with protocols like : UDP (IP Multicast), TCP, JMS.
Fabien
Great
The home page looks great, Thanks! I will read up and report here soon.
Alex
Excellent
I have read a good part of the user guide and api for JGroups, and it will be an excellent sollution to my multicast problem. It would also be an excellent sollution for any other kind of multi-node networking, and has impressive benifits for normal, client/server situations.
I will not recite the information that I have read, it is too extensive. If any one is interested, go to the JGroups home page, then the Overview, then the User's Guide.
Thanks to Fabien for a good find,
Alex
Info
The initials JXTA are generally taken to mean Java XML Transaction Architecture. However the JXTA specification is written in language independent terms, so it is not specifically for java. JXTA is a set of protocals for defining peer networks. Not just file sharing, but any type of decentralized, multi node, destributed computing.
Creating peer networks might not seem quite pertinant at this point in development, but allow me to share my thinking. Many of the developers have stated a desire to have an intuitive and flexible jnode interact with humans. We want the computer to recieve a signal from the user, interpret its meaning, and find ways to help the user accomplish his goal.
Artifical Inteligence being somewhere beyond my means, I began searching for another way to interpret user signals. Anyone who has been programing long has imagined ways of interpreting signals. The problem with any of the ways devised so far are that they are unreliable (inacurate), and require the construction of a large number of rules, or precidents to opperate.
I decided to tackle the first problem first. After thinking about it, I decided that the most acurate way to interpret a natural human signal is to ask other humans what it means. The more humans you ask the more confident you can be of your answere.
Examples of this include:
* Opening a Object (or file) of a type that does not have an accociated viewer. Ask the peers!
* To accomplish an abstract interface to plugins that may or may not be installed on the local machine, require the user to type a command like "Edit Document" then, Ask the peers! to find out what it means and what to do.
* Modify the device-to-driver mapper to, Ask the peers! and see what they did.
These ideas do not rely on AI but instead rely on Jnode power-users customizing their computers effectively, and other peers asking them how they did it.
For these reasons and others I began looking for an established peer protocal to build our network on. I found JXTA, tried it and gave up for reasons outlined above. I then considered how I would make a peer protocal. I came up with an idea, wrote it and have tested it successfully.
The code I wrote comprises a tool to efortlessly communicate with every instance of an implementing class that is on the network. That communication includes several types of data searches and transfers.
I described above in detal how the system works, and I cannot better that description. Please read it again with the understanding that it is a communications protocal, and let me know if you have questions or comments.
Thanks,
Alex
Re: Info
Hey Alex, good work. I didn't quite understand it after reading your 'Details' post, but the 'Info' post provided the clarifying higher-level description.
Just a note on the implementation: you're using multicast packets. That's a cool idea (that Jini is using too
) but AFAIK it doesn't work over normal internet connections (I think providers will neither transport broadcast nor multicast packets). I.e. you will also need a unicast protocol to discover (well-known) remote peers.
I'm currently lagging posting of my Jini-based proposal because I do some other Jini stuff. I originally wanted to provide some type of prototype too, but I think this will take too much time. I hope to get the article out until end of june to describe my point. I still think there will be many similarities in our implementations.
Regards,
Sebastian
Perfectly OK
Hi Alex,
I think it's perfectly ok require two ports to be open. Ok there are P2P protocols that are able to bypass firewalls, but besides the fact that I didn't understood why someone uses firewalls if they can be bypassed by some clever programs, it's in the nature of network applications that they require publicly accessible ports.
Webservers and Mailservers do so, so why shouldn't all other applications do so too.
Regards,
Sebastian
Good, Thanks
Thank you, that is good news. I will report here with a description of my idea, and my results, in a few days.
Thanks
Alex
I have looked at the readme.t
I have looked at the readme.txt file in the bundle file you mailed me, I guess your intension to replace the raw networking with JXTA is out.
I'm not that sure that it could have replaced the basic networking, as far as I could see on the jxta homepage it's build on top of an existing network.
Anyway I'll look more into your work.
Regards,
Martin
JXTA
That is correct, I have given up on JXTA. The idea that I have mentioned is showing early promose, but I have not yet worked out enough bugs to make a full test. I will report here as soon as I have solid results.
Alex