Diploma project with the Linux kernel

Diploma project with the Linux kernel

[Boian Karatotev]

Hello,

I am a Computer Science student and for my last year I need to make and
present a 'diploma project' at the end of June. So far I want to make a
kernel module, whose description is in the following paragraph. I feel
comfortable with C and my OS knowledge is maybe slightly better than my OS
course. My question is: Would it possible to pull this off? I have no
experience with the kernel and I want to get into kernel development, so
this would be a perfect opportunity for that. My only issue is that this
may be too complex for my experience.

My idea: Something along the lines of checkpoint-restart as a kernel
module. I want to ultimately be able to migrate a running process to a
different machine (assuming same at least some basic similarity). I know of
BLCR <http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/>
and I am planning on using it as a guide, although I am unsure about
working on it directly. As far as I know, the grading process does not
require this to be 100% complete, so I am aiming at transferring at least
all the memory, restoring file descriptors and maybe child
processes/threads.

I will really appreciate any feedback!
Cheers,
Boyan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20181004/fc890bad/attachment.html>

Diploma project with the Linux kernel

[Ruben Safir]

On 10/4/18 2:44 PM, Boian Karatotev wrote:
> Hello,
> 
> I am a Computer Science student and for my last year I need to make and
> present a 'diploma project' at the end of June. So far I want to make a
> kernel module, whose description is in the following paragraph. I feel
> comfortable with C and my OS knowledge is maybe slightly better than my OS
> course. My question is: Would it possible to pull this off? I have no
> experience with the kernel and I want to get into kernel development, so
> this would be a perfect opportunity for that. My only issue is that this
> may be too complex for my experience.
> 
> My idea: Something along the lines of checkpoint-restart as a kernel
> module. I want to ultimately be able to migrate a running process to a
> different machine (assuming same at least some basic similarity). I know of
> BLCR <http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/>
> and I am planning on using it as a guide, although I am unsure about
> working on it directly. As far as I know, the grading process does not
> require this to be 100% complete, so I am aiming at transferring at least
> all the memory, restoring file descriptors and maybe child
> processes/threads.
> 
> I will really appreciate any feedback!
> Cheers,
> Boyan
> 

Interesting.  I wouldn't mind helping.

Ruben

> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 


-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002

http://www.nylxs.com - Leadership Development in Free Software
http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013

Diploma project with the Linux kernel

[valdis.kletnieks at vt.edu]

On Thu, 04 Oct 2018 21:44:14 +0300, Boian Karatotev said:

> I am a Computer Science student and for my last year I need to make and
> present a 'diploma project' at the end of June. So far I want to make a
> kernel module, whose description is in the following paragraph. I feel
> comfortable with C and my OS knowledge is maybe slightly better than my OS
> course. My question is: Would it possible to pull this off? I have no
> experience with the kernel and I want to get into kernel development, so
> this would be a perfect opportunity for that. My only issue is that this
> may be too complex for my experience.

> My idea: Something along the lines of checkpoint-restart as a kernel
> module. I want to ultimately be able to migrate a running process to a
> different machine (assuming same at least some basic similarity). I know of
> BLCR <http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/>
> and I am planning on using it as a guide, although I am unsure about
> working on it directly. As far as I know, the grading process does not
> require this to be 100% complete, so I am aiming at transferring at least
> all the memory, restoring file descriptors and maybe child
> processes/threads.

You mean you want to re-invent the current checkpoint-restart code that's been
in the kernel since v3.10 back in June 2013? (see kernel/kcmp.c for the gory
details).

Note that migrating a running process to a different machine is a *lot*
trickier, especially if it has things like open files or network connections.
"Assume at least some basic similarity" isn't anywhere *near* good enough - if
the process has /home/fred/wombats/my_terabyte_database open, you're going to
need to have it at the same place in the filesystem and data synced across to
the new target (particularly fun if the process scribbles some more on the file
while you're busy migrating it, or if it hasn't done an fsync). Similarly, if
it has a TCP connection open to someplace else, you're going to have to figure
out what to do with the IP 4-tuple and sequence numbers to avoid breaking the
connection. And if it's HPC software using MPI configured to do RDMA over
Infiniband, that's even uglier....

In fact, migrating an entire virtual machine is easier than migrating one
process, because you don't have to worry about recovering the process state,
that's all in kernel memory that you migrate with the VM.  Move the VM, take
down the IP on the old hypervisor, set up the IP on the new one, toss out a
gratuitous ARP packet so other machines on the subnet notice, and you're ready
to go...

There's a *reason* why VMWare gets away with charging lots of money for their
enterprise-class software that supports migrating a live VM across hypervisors.
It's a lot harder to do than you think.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20181004/b87b40d3/attachment.sig>

Diploma project with the Linux kernel

[Mike Krinkin]

This might be of interest to you: https://www.criu.org

On Thu, Oct 4, 2018 at 8:23 PM <valdis.kletnieks@vt.edu> wrote:

> On Thu, 04 Oct 2018 21:44:14 +0300, Boian Karatotev said:
>
> > I am a Computer Science student and for my last year I need to make and
> > present a 'diploma project' at the end of June. So far I want to make a
> > kernel module, whose description is in the following paragraph. I feel
> > comfortable with C and my OS knowledge is maybe slightly better than my
> OS
> > course. My question is: Would it possible to pull this off? I have no
> > experience with the kernel and I want to get into kernel development, so
> > this would be a perfect opportunity for that. My only issue is that this
> > may be too complex for my experience.
>
> > My idea: Something along the lines of checkpoint-restart as a kernel
> > module. I want to ultimately be able to migrate a running process to a
> > different machine (assuming same at least some basic similarity). I know
> of
> > BLCR <
> http://crd.lbl.gov/departments/computer-science/CLaSS/research/BLCR/>
> > and I am planning on using it as a guide, although I am unsure about
> > working on it directly. As far as I know, the grading process does not
> > require this to be 100% complete, so I am aiming at transferring at least
> > all the memory, restoring file descriptors and maybe child
> > processes/threads.
>
> You mean you want to re-invent the current checkpoint-restart code that's
> been
> in the kernel since v3.10 back in June 2013? (see kernel/kcmp.c for the
> gory
> details).
>
> Note that migrating a running process to a different machine is a *lot*
> trickier, especially if it has things like open files or network
> connections.
> "Assume at least some basic similarity" isn't anywhere *near* good enough
> - if
> the process has /home/fred/wombats/my_terabyte_database open, you're going
> to
> need to have it at the same place in the filesystem and data synced across
> to
> the new target (particularly fun if the process scribbles some more on the
> file
> while you're busy migrating it, or if it hasn't done an fsync). Similarly,
> if
> it has a TCP connection open to someplace else, you're going to have to
> figure
> out what to do with the IP 4-tuple and sequence numbers to avoid breaking
> the
> connection. And if it's HPC software using MPI configured to do RDMA over
> Infiniband, that's even uglier....
>
> In fact, migrating an entire virtual machine is easier than migrating one
> process, because you don't have to worry about recovering the process
> state,
> that's all in kernel memory that you migrate with the VM.  Move the VM,
> take
> down the IP on the old hypervisor, set up the IP on the new one, toss out a
> gratuitous ARP packet so other machines on the subnet notice, and you're
> ready
> to go...
>
> There's a *reason* why VMWare gets away with charging lots of money for
> their
> enterprise-class software that supports migrating a live VM across
> hypervisors.
> It's a lot harder to do than you think.
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20181005/f4abe57d/attachment-0001.html>

Diploma project with the Linux kernel

[valdis.kletnieks at vt.edu]

On Fri, 05 Oct 2018 02:58:23 +0100, Mike Krinkin said:

> This might be of interest to you: https://www.criu.org

That's got two problems - first, it's userspace. And second, it's fairly
mature software, which means it's not suitable for a student project
by itself, and all the low-hanging fruit for improvements has probably
already been done (meaning that further extensions will be technically
challenging...)

A better source for project ideas is to do a literature search and find
proof-of-concept projects that *didn't* turn into mature software, and
need work to turn them into actual running code....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20181004/fe7c6c8f/attachment.sig>

Diploma project with the Linux kernel

[Mike Krinkin]

On Fri, Oct 5, 2018 at 3:09 AM <valdis.kletnieks@vt.edu> wrote:

> On Fri, 05 Oct 2018 02:58:23 +0100, Mike Krinkin said:
>
> > This might be of interest to you: https://www.criu.org
>
> That's got two problems - first, it's userspace.


Well, it's called userspace, but it does require a significant kernel
support.


> And second, it's fairly
> mature software, which means it's not suitable for a student project
> by itself, and all the low-hanging fruit for improvements has probably
> already been done (meaning that further extensions will be technically
> challenging...)
>

I guess, that someone from the CRIU team might be a better person to tell
whether it's the case. It doesn't take a lot of
effort to ask whether they have tasks that might become a student project,
especially considering that the team doing
CRIU did in the past at least collaborate with univeristies in Russia where
students actually worked on CRIU.


>
> A better source for project ideas is to do a literature search and find
> proof-of-concept projects that *didn't* turn into mature software, and
> need work to turn them into actual running code....
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20181005/c3ff1c86/attachment.html>

Diploma project with the Linux kernel

[Boian Karatotev]

>
> You mean you want to re-invent the current checkpoint-restart code that's
> been
> in the kernel since v3.10 back in June 2013? (see kernel/kcmp.c for the
> gory
> details).
>
Too bad I hadn't found this out earlier. I had hopes this wasn't really
done in kernels, so I had hopes to at least try as a lurning opportunity as
an official project. I guess not

It doesn't take a lot of effort to ask whether they have tasks that might
> become a student project,
>
I'll do that, thanks!

I guess this is pointless doing, so I'll look around elsewhere. Does anyone
know anything else that might be suitable for a school project?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20181005/b687589d/attachment.html>

Diploma project with the Linux kernel

[Cindy-Sue Causey]

On 10/5/18, Boian Karatotev <boian4o1@gmail.com> wrote:
>>
>> You mean you want to re-invent the current checkpoint-restart code that's
>> been
>> in the kernel since v3.10 back in June 2013? (see kernel/kcmp.c for the
>> gory
>> details).
>>
> Too bad I hadn't found this out earlier. I had hopes this wasn't really
> done in kernels, so I had hopes to at least try as a lurning opportunity as
> an official project. I guess not
>
> It doesn't take a lot of effort to ask whether they have tasks that might
>> become a student project,
>>
> I'll do that, thanks!
>
> I guess this is pointless doing, so I'll look around elsewhere. Does anyone
> know anything else that might be suitable for a school project?


Hi.. The first thing that always comes to mind for me is to track down
bug reports. For the kind of project you're talking about here...

Maybe track down *wishlist* bugs where users are asking for features
that don't exist yet?

Always feels like there's half a bazillion bug reports spread across
all the various distributions on the Internet so starting with a
keyword topic helps. If nothing friendly sounding pops up, maybe what
you do find will still spawn an idea that's attractive to your
talents.

Good luck! :)

Cindy :)
-- 
Cindy-Sue Causey
Talking Rock, Pickens County, Georgia, USA

* Woof!! ~My Dog *