Referee 1 ******************************************* E: Referee Comments (For Author and Editor) ------------------------------ The paper provides a nice story of developing interoperable resource brokers. Do not you think we still need to have some more advancement in P2P and Grid computing ? Otherwise, it leads to one project dominating the whole scene, which is not good in long run. In Table 1, collective/Resource Broker layer, scheduling part is missing. I recommend authors improve Section 7 by including a Case Study of Executing Real Application. Since that provides better confidence on the system rather that performing typical Computer Scientist Results. F: Presentation Changes 1. Given that your paper really focuses on the use of P2P techniques much more than Grid, a better title for your work is: ------ Engineering Interoperable Computational Collaboratories for P2P Computing - Advances in the DISCOVER Project --------------------------------------- 2. References: Can you please change URL references to full references by citing a specific paper by authors of those projects. Then you can cite URL. Instead of : PROJECT ABC : www.abc.org Use: Manish Prashar and Vijaya ..., THE P2P Computing!, In Proceedings of ..., April 2002. URL: www.abc.org 3. You may cite some P2P papers. Referee 2 ******************************************* General comments: The paper is well organized and made interesting reading. I however have some issues that I have put forth in the technical issues part of my report. Some typo suggestions Page 2,6: 3rd party ' third party Page 5: 4th line from the bottom ' The requirements of ".. as customized user level specific to each ".. Should this be : as customized user level services specific to ". Page 11: sec 6.1, 2nd paragraph, Page 12 section 6.2, 1st paragraph:' 3 could be three Page 14 section 6.3.3 2nd paragraph ' "do not" instead of "don't" Page 16:There's a typo in the heading for section 7 ' Should be An experimental evaluation. Page 16, section 7.1 ' 2 could be two. Figures: After I printed the PDF document out, most of the diagrams seem smudged probably an artifact of conversion from the original document format to PDF. Though it didn't hamper me at all from grasping the concepts, the authors should probably look into this issue. Technical issues: The authors referred (page 7, line 11) to servers in their model being light-weight, manageable. This does not seem to be entirely true, as from what I read I believe they have an ORB running at each server. On page 10: Clients using CORBA/IIOP to connect to servers may actually reduce latencies This seems to be open to debate. Since clients need to have an ORB on their side to communicate in IIOP in which case the clients are not really lightweight. Plus, in the case of ORBlets that are setup inside browsers there is quite some time (usually a few seconds) involved in downloading the ORB inside the browsers Java VM On page 11: section 6.1 2nd paragraph: There are 3 communication channels that are set up. Does this imply that there socket connections are initiated with the server? Or is this simply an abstraction. Also are any of these channels encrypted since some applications may be really sensitive to tempering of requests. Further, there's a reference to clients differentiating between messages employing Java's reflection mechanism. This would, IMHO, increase communication latencies since the reflection process is a very slow process. Is there a reason why reflection needs to be used? Wouldn't a simple investigation of message header would do. Or are we constrained to do so, because all communication involves CORBA objects? General question: Could a client connect to multiple servers? If so, is there a limit on preventing clients from doing so? Page 12, section 6.2.1 : There exists methods for a client to query a server and obtain a list of users logged in? Do they also form a collaboration group. Is there a way to allow clients anonymity and prevent invasive communications from other clients. Page 13, section 6.3.1 2nd paragraph: I was under the impression that an application is connected to a local server, access control etc. are much easier to implement in such settings. However, I see in this paragraph that it possible for the same application to be connected to multiple servers. What is the security strategy that would need to be in place to support such a scheme. It seems to me, that a distributed token/locking mechanism should also be present. Page 14, section 6.3.3: All clients connected an application form a collaboration group. I believe an application could have multiple instances, similarly an application could be connected to multiple servers. I assume, the authors are referring to clients connected to an application instance forming a group. Since the same application instance could be connected to multiple servers would all clients accessing the same application instance from potentially different servers form a collaboration group. Results: Section 7.3 (Evaluation of Server Memory requirements) This was one area where I had some rather serious reservations. Values returned by freeMemory(), totalMemory() as rightfully suggested are very approximate. Compounding the issue is the fact that the system thread which is responsible for returning these values are not always scheduled for running when the most intensive operations are performed. They at best return values after the CPU intensive operations are performed. Also, this value is ever changing and continuously decreases after an operation is performed. If the authors feel this is a crucial element of their test results what would be more appropriate is using the NT Task Manager like utility that would available on their system. Native system calls do help in getting this number down to a great degree. Also, I believe the contention that actual values would be lower than plotted in the paper is open to debate. First if it's a Java client one has to account for the JVM's utilization too. Intuitively it seems the values should be significantly higher than the ones plotted in the paper, Referee 3 ******************************************* E: Referee Comments to Author and Editor Technically, this paper poses an interesting problem in how to combine focused collaboratories and allow them to inter-operate. It provides some approaches on how this could be performed and also postulates that a middleware substrate has been built that supports this interoperation. As written, the first part of the paper (through Section 4) is not supported by the remaining sections of the paper. The paper does not describe the process of combining different collaboratories, but only how the distributed operation of the DISCOVER collaboratory is being supported. The paper also does not motivate the concept of tying collaboratories together from a domain scientists perspective, it really don't establish the need for this work. The implied message is "If we build it, they will come". What would make this an interesting paper is an example of two of the different collaboratories mentioned, being modified to use this middleware substrate and reporting what was required to make this happen, what problems were encountered and overcome, and what was learned from the process. The paper doesn't outline or analyze what would be required for an existing collaboratory to utilize this middleware substrate. Does it require modifying client-side software, server-side software or both to use the CORBA IDL specification? Furthermore, while this is technically possible, experience shows isn't a practical or feasible solution because of organizational considerations, unless there is almost universal acceptance of the solution and a real impetus to change existing software. CORBA has not been able to attain this status. The paper also seems to equate focused Collaboratories and PSEs. In our opinion this isn't accurate, and not all of the collaboratories mentioned in the introduction are focused, therefore they definitely do not qualify as a PSE. Several of the collaboratories are also not grid-enabled, unless a very broad definition of the grid is being used that essentially equates to distributed computing. Although, there are technical topics of interest in this paper, the different concepts brought forth (interoperability between collaboratories, middleware substrate, DISCOVER computational collaboratory) were not tied together well and caused confusion about what was trying to be conveyed. Section 3.1 states that true interoperability can be achieved with Shared Protocols, but the implementation that has been followed by the project is Shared Interfaces and APIs. A path towards the Shared Protocol approach is mentioned by integrating the services into the CORBA ORB as standard CORBA services. Is this feasible? Does this require involvement in the standards process? Section 4 states that a server that provides a single instance of an application or a service is only required to provide the second level interface. A justification for why this was done was not provided. It seems, like this service would still need to advertise its capabilities. Also, in this section the paper states CORBA IDL was chosen instead of XML to describe interfaces, but no justification was given. The experimental evaluation is also not very extensive and should undergo further evaluation. More than ten simulations should be used for the values used in the graphs. Not all of the diagrams said they were mean averages, were they? This is important for Figure 9, since the explanation is not sufficient if it is a mean response time. Regarding server memory consumption in Figure 10, given the total memory use of the server is less than 10MB for 25 clients then this doesn't seem like a very important evaluation criteria. F: Presentation Changes The test describing the hourglass model in Section 3.2 should have a diagram, the text alone was confusing. The conclusion mainly summarizes the paper contents, what are the real conclusions? The information regarding future work is interesting and could be further expanded upon. The appendices did not add to the paper and should be removed. I recommend the use of a technical editor, there were several instances throughout the document where sentences were missing words, were poorly structured or were otherwise incomplete.