We're having difficulties implementing our code but we have an idea of what we need to do which we've explained in our report. Does that still constitute an F for the term project if our code is not finished? And can we still pass the class if our final grade is above a C even with an F on the term project?
I of course won't require you to have working code to get a passing grade on the Term Project. The story I told in class involved someone who was quite disappointed that he had gotten a C (maybe B-, I don't remember) on the project because the code did not work, which in turn was due to waiting too long to start the project.
Once again...all the information regarding the computation of course grades -- e.g. relative weights of quizzes and homework -- is in the class syllabus.
The last time I taught this course, Winter 2017, the distribution of course grades was
A A- A+ B B- B+ C+ 20 19 5 9 4 24 1
For the Machine Repair 2 example, when we try to implement it we are getting resource counts that shouldn't even be possible, such as -1. We think we are decrementing and incrementing the resource count at the correct places. Does this mean that we would need locks? The bigmemory documentation doesn't include anything about locks, so we aren't sure.
Though your Rposim package will only be similar in spirit to SimPy, it probably would be worth drawing some insight from SimPy.
SimPy uses Python generators, which you may recall can be used to implement non-preemptive threads, as they do not need locks. Thus SimPy does not have locks, either at the application or "OS" levels.
Thus you probably should not need locks in Rposim. The fact that all actions go through the "OS," together with the fact that each application thread has its own "signal/wait" variables, will probably obviate the need for locks.
If you do wish to use locks, there is a companion package to bigmemory called synchronicity. Both are available on CSIF in ~matloff/Pub/Rlib/. Or you can use the "quick and dirty" lock method used in Rdsm based on mkdir.
We are having difficulties in implementing simmer with example 1. From my understanding the trajectory will be using timeout to set the uptimes and repair times, but I am confused in how the generators work here...
There are tons of examples, and the full text of a published paper, on the simmer Web page. (Click on 'Articles'.) One of the examples is actually quite close to the one you are implementing for our project.
The syllabus mentions that the will be weighted based on the fact that it's supposed to be approximately 2x as much work as a normal homework. Does that mean that it will be worth approximately 2x a normal homework within the homework category?
Yes, here is what the syllabus says:
It will involve work approximately 2 times that of a regular assignment, and will be weighted accordingly, especially given the dual role of the Project as both Homework and a replacement for a final exam.
This is in addition to the fact that a good project will result in an extra bump to your course grade, which can be quite substantial.
I was just wondering if you could possibly elaborate a bit on what you're looking for in the simmer section of the report. Did you want a full simmer implementation of MachRep1?
Yes.
Our group has been trying to write the code for activate(), which requires calling Run() in a background thread. However, there doesn't really seem to be an easy way to do this, because run is called on an object and if we change some attributes in that object during the creation of the object, we can't just call the default run object. Is there a way to pass the entire process object between threads so we can call Run() on them?
The specs merely say,
[You will write a package] similar to SimPy...Your Rposim package must have functionality comparable to that of the aspects of SimPy in the three machine repair examples in my tutorial.
This is NOT a one-to-conversion of SimPy to R, which would be impossible anyway, due to R's lack of generators. Your package simply must be feature-rich enough so that you can simulate the three machine repair models.
Should we be implementing the ability to handle an indeterminate amount of user-defined threads with our project?
By this, I mean, should our package give the user the capability to define multiple threads when constructing their simulation? For example, if they are simulating three different types of machines being repaired, they could have all of that happening in a single thread/instance of the user code, just with different event codes specifying which type of machine was triggered. Or, they could design the project so that each machine type is a seperate thread.
While the latter example would be a bit more work to code the handling of the threads with the library defined function, it would also lead to more-clear distinction of the event creators.
Should we specifically be implementing either one of the approaches, or is that just a design descision for this project?
I'm not sure I fully understand your question. Here are some comments that should help; feel free to ask further questions.
First and foremost, your library code MUST be general, able to accommmodate any number of threads. How you do that is up to you, many possible routes here.
Second, note that the number of threads in an application could be different for different programmers. For instance, consider the machine repair model. In addition to the two threads for the machines, one could have a thread for the repairperson. That third thread would watch the job queue, and when a job came in, would start processing it.
I am still a bit confused about what constitutes a "thread" in R? I believe that each thread (supplied by the application program) will have an invocation of the R DES library which will be shared through big memory and another thread for managing these application threads. Does this mean we have 1(OS) + n threads where n is the application specific number?
R is not threaded. One can get the effect of threading by running k instances of R for k "threads," and using bigmemory to provide the shared memory that is the hallmark of threading.
In the machine repair example in our Python book, there are two app threads, so counting the "OS," you'd have three threads. If you launch them manually, then you'd have three terminal windows, and would type 'R' into each of them.
I don't know what you mean by the "R DES library." If you mean the package named des, then that is wrong. You should not be making use of the code in that package at all, for as pointed out here earlier, that is a different DES paradigm, not directly relevant; the des package is useful only for rough guidance as to how an "OS" might work.
The educational value of the project in large part stems from the fact that you are deciding not only how to implement the operations, but also what operations to implement. Thus for instance you are deciding upon the APIs, e.g. the syntax of the "hold" operation, hence I cannot (and should not) specify a "Step A, Step B,..." formulation. But in any case the threading will operate as above.
From the problem specs:
Each "thread" will be implemented in a separate invocation of R. For instance, consider the machine repair example in the *SimPy* example in our Python text. There will be two invocations of R for the machine processes, one for the repairperson, and one for the "operating system" (OS). The latter manages the event list, continually looping around, removing the earliest event from the event list, updating the current simulated time, "waking" the proper "thread" and so on.
Are there four or two threads when you are taking about the two invocations of R. As in are there two total of two per application process and OS process. Also does the invocation of the OS is confusing are you saying that the DES library that has the OS as in the main while loop that manages thread in terms of which is woken and such?
There is one invocation of R per thread.
The "OS" will certainly not be the same code as in the des package -- how could it? One package is process-oriented, based on threads, and one is event-oriented, non-threaded. But any DES package needs an "OS " to do things like it states above, e.g. "removing the earliest event from the event list, updating the current simulated time," and so on.
In your description of the requirements for the project, you talk about having multiple "invocations" of R. Our group was wondering what exactly you mean by this. Are we supposed to open multipe rscripts manually, that each alter the shared bigmemory matrix? Or is there some way to start an rscript from another r script, in parallel?
You could start multiple invocations of Rscript by hand, or in a shell loop, or from an R loop paired with system(). Or you could use my Rdsm library to save some work.
For the purposes of debugging your code, it is best to set up invocations of R rather than Rscript, so that you can run debug() and interact.
For HwkII, can we assume that the HwkII.py file is in the main directory of the host machines?
Yes.
We have been stuck on how the client code is set up. Does the client send the function calls to the server or does the client have a line in the code that calls the function without sending? For example: Would it be: s.send('dread()') or would the code just have dread()? In the instructions, it seems like the client will just call the function themselves, but how will the server receives this information?
Remember, there is no little man inside the machine. :-) No magical powers either. It just does what you tell it to do. And you decide what you will tell it to do, based on your goals. When you ask above, "how will the server receives this information?", the answer is that YOU must write client code to send it. And YOU must decide in what form the client will send it, and YOU must write the server code to receive it and act on it. There is no little man to act on it.
So, reason it out. Consider the case of dread(). Ultimately, you need the server program to call read(), right? So, how can you make that happen? Can you simply have the client send a string to the server, "Server, please call read()"? :-) You could do that if there were a little man at the server, but since there isn't, you have to do is write code ON BOTH ENDS, client and server, coordinating with each other, to achieve what you want. You'll write client code that sends -- in a form of your choice -- the file name, read action, number of bytes etc., and write the server code to make the proper read() call based on what it receives from the server.
Where does the socket number come into play here if shell commands can be executed using os.system()? For example, in the client dread() function, can't we just execute something like
os.system( ssh username@host head -c nBytes /tmp/fileName)
First of all, this probably wouldn't work for binary files.
But much worse, you would have a serious problem with file pointers. Say a user wants to read 3 bytes then 2 more. Under your approach, you would have to do head -c 3 THEN head -c 5 | tail -2. That would mean reading the same bytes TWICE!
On NotPiazza, you said "HwkII.py will NOT be running". Does this mean [we] do not have to use threads?
I'm not sure what the connection is between that first sentence and the second. Clearly the servers either have to either be threaded or use asynchronous I/O.
I have read your NotPiazza.html, and I am still confused as to the role of "manager." Is HwkII.py the manager?
HwkII.py is simply a collection of library functions. Among those functions, one will be manager code. The manager -- the person, e.g. John in my example -- will run the manager function, which will start the servers at the various hosts.
My guess is that you [the person who asked this question] would find this assignment much easier if you make sure you understand the real-world goals, as outlined [here] in NotPiazza.html. The importance of understanding the goals is enormous; if you don't understand this, not only will you have trouble completing this assignment, but also you would likely get a very poor grade during the interactive grading of the assignment. If you have trouble understanding the goals, contact me and/or the TA.
Also, if you have not already done so, make sure you try the Omsi example in the file -- actually RUN it.
Me and my group members were running into a problem with the sysStart function. We understand how to ssh into one of the servers, however we are having issues with SSHing into multiple servers at the same time. We eventually set up threads to launch each server, but aren't sure how to handle password requests sent by the server. Could you offer any pointers to help us get started in the right direction?
You'd best set up passwordless login (from one CSIF machine to another). This is an ssh issue, and it is described by CSIF in one of their FAQs. I don't have the URL handy right now, but let me know if you can't find it or if it is unclear.
I think a short example of what proper input/output might look like from the perspective of the person using the program (like the examples for __add__ and __mul__ in the HwkI prompt) would go a long way in clarifying much of the confusion that people have regarding HwkII. Would that be possible?
Before I answer, let me state that it's vital that everyone understand we are working on a real world problem here (albeit simplified). If you don't understand the motivation, it will be difficult if not impossible to understand what you are expected to produce.
The term distributed database is an important one in Computerland. (You may wish to Google it.) The idea is to get better file system performance (for reads) by having multiple disk drives, possibly attached to different machines, maybe even widely geographically dispersed machines. By enabling multiple reads to be performed in parallel, we hope to speed things up, as well as attain some fault tolerance. Typically the data is replicated across servers, but for the sake of simplicity we aren't doing that here.
Now, in this assignment there will be no universal test script that the TA will be able to devise to check your code. That follows from the lattitude I gave you in my blog post of Wednesday, January 31, 9:30 am, and even without that lattitude it would be difficult to write such a script, given that multiple actors (manager, servers, clients).
Well then, what must your code do in order to be correct? Basically it must satisfy the following:
Your README file will need to tell the TA what to do to demonstrate that your code works as above. (Keep it simple!)
Should there be two instances of HwkII.py running in interactive mode when we are testing the distributed file system, one for system management and one for a client to access the system?
HwkII.py will NOT be running.
Consider the C Library. Suppose the source code for the library were all in one file, cl.c. That file will NOT be compiled into an executable file that can run on its own. But a huge variety of programs will make calls to FUNCTIONS within cl.c.I think most of my confusion comes from not understanding what "having them start their file servers" means. When you say "ssh(x)" I assume you mean run the program through ssh, like in the ls example?
I think the following will help your thinking. From your laptop or desktop, do the following (with your own username instead of mine):
ssh matloff@pc4.cs.ucdavis.edu python OmsiServer.py 2500 xyz
Then test it, by using the client from your laptop/desktop to connect to that server. And recall that ssh or any other shell command can be run through Python's system() function.
I guess I'm still kind of confused by what exactly the prompt means when it's talking about the "manager", and how that manager relates to the server, client, and Python's interactive mode.
Say John will run the servers (as our TA runs the OMSI server), and Jim and Jane wish to run clients. Jim and Jane have written programs that make use of, e.g. dread(). John will go into Python's interactive mode, and start the servers, say at pc4, pc5 and pc6. He will verbally inform Jim and Jane as to the port number (and hosts). Jim and Jane will run their code, which includes calls to dread() etc.