Before really putting any comment on "Parallel or Distributed Computing" I was curious to know why we really need it ? I remember famous quote of Moore
"CPUs are double faster in every 18 months ...."
I believe this law still some how hold the truth but than big question is "why we need any thing called parallel or Distributed Computing when I know I have 2 times faster CPUs in next 18 months ? "
[1] The data in the web is more than doubling in few months only ....In last few years we are adding more and more devices which put more and more data on the web ...I Phone, I Pad , Kindle devices all of these are new source of data generation and access on the web.
[2] Moore's law might not be keep holding truth with time go : As we are moving to world where every one prefer devices with small and small in size and weight ... We already reaching to a point where chip size will reach to it's minimum size and getting more instructions execution per second out of it will not be that easy
[3] Even with a super Computer it might not be easy and fast to create 50 thousands animated soldiers in the war of "The Mummy " movie. Where each animated solider composed of millions of frames and each frame have it's own effect and attributes.
There are hundred more reasons which were the reason why Computer Scientist started looking on "Parallel and Distributed Systems around 20-30 years back" and today we reached to a stage where no Software Company across the world can breath with out Distributed Computing.
I have to first understand what exactly Distributed Computing is ?
I start with a very small example. Any software programmer start his programming life with a For loop code so let's see how it work in different scenarios
for(int i=0; i< 100 ; i++ )
{
c[i] = a[i] + b[i];
}
A machine which have a single CPU core to execute this program :
For each iteration of i , value of a[i] and b[i] will be loaded in memory and than addition operation will happen . Suppose time time taken in loading this data in memory is T than some how around it will take 100 * T time to complete this execution ...
Parallel Programming:
Suppose you have a N Core CPU and your compiler is brilliant enough to work in Parallel programming paradigm so In Parallel it will keep uploading data for N addition operation and executing the plus operation and some how your the same for loop execution it will take time around ( 100 * T / N )
In laymn's term we can say in Parallel programming CPU instructions run in parallel which share same In Memory data. There is no role of network or data on wire in it.
Now we move to Distributed Programming where we have M number of machines , each of them have them have N CPU Core . These machines are connected by a Network . We again have same task to run in
In this case each machine will be busy in loading data in memory for N addition operation and than doing the plus operation. It will reduce the total time taken for our task to ( 100 * T / N * M ) + X ...
What is this X ? I will try to cover in next post of this blog.
"CPUs are double faster in every 18 months ...."
I believe this law still some how hold the truth but than big question is "why we need any thing called parallel or Distributed Computing when I know I have 2 times faster CPUs in next 18 months ? "
[1] The data in the web is more than doubling in few months only ....In last few years we are adding more and more devices which put more and more data on the web ...I Phone, I Pad , Kindle devices all of these are new source of data generation and access on the web.
[2] Moore's law might not be keep holding truth with time go : As we are moving to world where every one prefer devices with small and small in size and weight ... We already reaching to a point where chip size will reach to it's minimum size and getting more instructions execution per second out of it will not be that easy
[3] Even with a super Computer it might not be easy and fast to create 50 thousands animated soldiers in the war of "The Mummy " movie. Where each animated solider composed of millions of frames and each frame have it's own effect and attributes.
There are hundred more reasons which were the reason why Computer Scientist started looking on "Parallel and Distributed Systems around 20-30 years back" and today we reached to a stage where no Software Company across the world can breath with out Distributed Computing.
I have to first understand what exactly Distributed Computing is ?
I start with a very small example. Any software programmer start his programming life with a For loop code so let's see how it work in different scenarios
for(int i=0; i< 100 ; i++ )
{
c[i] = a[i] + b[i];
}
A machine which have a single CPU core to execute this program :
For each iteration of i , value of a[i] and b[i] will be loaded in memory and than addition operation will happen . Suppose time time taken in loading this data in memory is T than some how around it will take 100 * T time to complete this execution ...
Parallel Programming:
Suppose you have a N Core CPU and your compiler is brilliant enough to work in Parallel programming paradigm so In Parallel it will keep uploading data for N addition operation and executing the plus operation and some how your the same for loop execution it will take time around ( 100 * T / N )
In laymn's term we can say in Parallel programming CPU instructions run in parallel which share same In Memory data. There is no role of network or data on wire in it.
Now we move to Distributed Programming where we have M number of machines , each of them have them have N CPU Core . These machines are connected by a Network . We again have same task to run in
In this case each machine will be busy in loading data in memory for N addition operation and than doing the plus operation. It will reduce the total time taken for our task to ( 100 * T / N * M ) + X ...
What is this X ? I will try to cover in next post of this blog.
Upar se nikal gaya yeh mere :) ..phir se padna padega :)
ReplyDeleteSo what I understand from these lines is that if 9 people do a task they will do it faster. Of course computer terminology will stands with it. on the contrary are you considering the capacity of that M. That must have a limit. Hope to see that sooner.
ReplyDeleteHi Gaurav, Thanks for raising this valid point...Of course factors like capacity of M , network latency by which these M machines are attached to each other will play a role .... Other than that there should be some mechanism running on one of the machines which will be diving the tasks in between M machines... Collecting the result back from these M machines ..combining these results and return back to main application will also be contributing in time ... I will try to grab more data on this and will put it sooner ..
ReplyDeleteGood post Rohit.But dont you think that whole of the world is moving now towards cloud computing rather than distributed ?
ReplyDeleteAnkit I agree with you ... but how I see it "Behind any cloud there is is big set of distributed systems running" .... As for example If some one use Amazon EC2 cloud ..in the end he / she get a set of machines on which applications will run ...the benefit is EC2 cloud customer don't need to worry how these machines will talk to each other ..and if something goes wrong with one machine, EC2 takes care of it and user don't need to worry about it.
ReplyDelete@Ankit I am not completly agree with you. there are still possibility for distributed computing in many many field. moreover not everyone can have so much money to move towards that.
ReplyDeleteAnyways Rohit ...... good post ..... keep it up....... ALL the best !!!!
ReplyDeleteThanks Gaurav ...nd Ankit for putting your thought on this...!
ReplyDeletethough the concept of having parallel computations is attractive, the biggest challenge is how do u decide what sections should run parallel. Does the application programmer decide what sections should run parallelly or does the distributed system do it on its own? If the programmer should decide, can you trust him to come up with an optimal division?
ReplyDeleteJust my 2 cents on the topic... nicely written Rohit bhai!
Thanks Ashwin :)
ReplyDeleteYes I agree it take a lot from the programmer to write a code which can divide a single tasks and get executed on multiple hosts in parallel. As this is difficult job for a programmer to able to write such codes so there are some computing system where programmer don't need to think about this division of task at granular level. I am reading about Hadoop these days which I believe is such system !!