Kernel thread scheduling

Kernel thread scheduling

[Vincenzo Scotti]

Hello,
I am actually studying kernel threads, and I have some doubts about them.
Let's take for example this snippet of code

static int thread_function(void *data)
{
	while (!kthread_should_stop()) {
		schedule();
	}

	pr_err("Stopped");
	return 0;
}

This way it works just fine, and waits until I call kthread_stop on it.
But if I comment out that schedule() call, it just hangs my system when I load
it (it is part of a module). I see that the loop-schedule-wakeup pattern is used
among all the others kernel threads. But I don't get why I need to call the
scheduler explicitly.
I know that the kernel is fully preemptible, and in my interpretation I thought
that it could stop every running thread, even in kernel space, using a
timer-based interrupt handler, to give cpu to other threads. Doesn't this
pattern resemble a voluntary preemption model?

Where am I wrong?

Kernel thread scheduling

[Jeff Haran]

-----Original Message-----
From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces at kernelnewbies.org] On Behalf Of Vincenzo Scotti
Sent: Friday, March 20, 2015 4:20 PM
To: kernelnewbies at kernelnewbies.org
Subject: Kernel thread scheduling

Hello,
I am actually studying kernel threads, and I have some doubts about them.
Let's take for example this snippet of code

static int thread_function(void *data)
{
	while (!kthread_should_stop()) {
		schedule();
	}

	pr_err("Stopped");
	return 0;
}

This way it works just fine, and waits until I call kthread_stop on it.
But if I comment out that schedule() call, it just hangs my system when I load it (it is part of a module). I see that the loop-schedule-wakeup pattern is used among all the others kernel threads. But I don't get why I need to call the scheduler explicitly.
I know that the kernel is fully preemptible, and in my interpretation I thought that it could stop every running thread, even in kernel space, using a timer-based interrupt handler, to give cpu to other threads. Doesn't this pattern resemble a voluntary preemption model?

Where am I wrong?

Are you sure your kernel is configured with kernel preemption on? It is a configurable option. Grep for PREEMPT in your .config file.

Jeff

Kernel thread scheduling

[Anand Moon]

Hi Jeff,

kernel_thread just like daemon process. These are self monitoring process.


These process get scheduled by the kernel when their is any state change
from TASK_INTERRUPTIBLE to TASK_RUNNING or their is some condition that get full field.

If you don't schedule() the kernel thread state is not get updated by the scheduler.


Below example can help clear the concept.


http://lxr.free-electrons.com/source/drivers/vhost/vhost.c#L225


-Anand Moon


On Saturday, March 21, 2015 4:58 AM, Jeff Haran <Jeff.Haran@citrix.com> wrote:
-----Original Message-----
From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces at kernelnewbies.org] On Behalf Of Vincenzo Scotti
Sent: Friday, March 20, 2015 4:20 PM
To: kernelnewbies at kernelnewbies.org
Subject: Kernel thread scheduling

Hello,
I am actually studying kernel threads, and I have some doubts about them.
Let's take for example this snippet of code

static int thread_function(void *data)
{
    while (!kthread_should_stop()) {
        schedule();
    }

    pr_err("Stopped");
    return 0;
}

This way it works just fine, and waits until I call kthread_stop on it.
But if I comment out that schedule() call, it just hangs my system when I load it (it is part of a module). I see that the loop-schedule-wakeup pattern is used among all the others kernel threads. But I don't get why I need to call the scheduler explicitly.
I know that the kernel is fully preemptible, and in my interpretation I thought that it could stop every running thread, even in kernel space, using a timer-based interrupt handler, to give cpu to other threads. Doesn't this pattern resemble a voluntary preemption model?

Where am I wrong?

Are you sure your kernel is configured with kernel preemption on? It is a configurable option. Grep for PREEMPT in your .config file.

Jeff


_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel thread scheduling

[Vincenzo Scotti]

Thank you for the example.

I understand what are the scheduling mechanics depending on task->state.
But suppose another situation. 
Let's say I change current state to TASK_INTERRUPTIBLE. If I start now some long
operation, then shouldn't the scheduler kick in as soon as he can and put my thread
asleep? Or should I call necessarily schedule()? And if so, why?

Kernel thread scheduling

[nick]

On 2015-03-22 07:14 PM, Vincenzo Scotti wrote:
> Thank you for the example.
> 
> I understand what are the scheduling mechanics depending on task->state.
> But suppose another situation. 
> Let's say I change current state to TASK_INTERRUPTIBLE. If I start now some long
> operation, then shouldn't the scheduler kick in as soon as he can and put my thread
> asleep? Or should I call necessarily schedule()? And if so, why?
> 
> 
> 
Greetings Vincenzo.
I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
as when I was learning the scheduler and process management for the kernel,I found these
chapters to be very useful.Further more for user space you are correct to my knowledge
schedule is called for internal tasks that are taking too long in kernel land or to 
reschedule to another task,not for user space based tasks are TASK_INTERRUPTIBLE works
fine there.
Hope this Helps,
Nick
_______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 

Kernel thread scheduling

[Ruben Safir]

On 03/22/2015 07:30 PM, nick wrote:
> I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
> as when I was learning the scheduler and process management


how much has the scheduler changed since then.  It was completely
overhauled when the CFS was created

Kernel thread scheduling

[nick]

On 2015-03-22 08:05 PM, Ruben Safir wrote:
> On 03/22/2015 07:30 PM, nick wrote:
>> I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
>> as when I was learning the scheduler and process management
> 
> 
> how much has the scheduler changed since then.  It was completely
> overhauled when the CFS was created
> 
> 
> 
The 3rd edition of this book was written after CFS was in the kernel so the chapters
are pretty up to date.
Nick
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 

Kernel thread scheduling

[Ruben Safir]

It is passover so I've read over much of this text, but I have to say
that in general, I'm way ahead of this book.  Although I have limited
knowledge of Kernel technology in the specific, the C code, data
structs, and programming concepts are spoon feed in this text and its
wasting too much time with words that are more easily explained with
coding examples and UML charts.  I don't need a chapter explaining how
to use ps and the basis of Unix architecture.  This text is targeted to
a different audience, and FWIW, I'm not certain it does a good job of
that either.  The guys who write these texts fall in love with their own
voices.  I know, I've suffered this disease myself when I've written
tech articles and books.

I can''t recommend this book to anyone.  Anyone who doesn't understand
the basics of I/O processer blocks is not going to understand

static void update_curr(struct cfs_rq *cfs_rq)


and OTOH void update_curr(struct cfs_rq *cfs_rq) is not explained well
enough for coders unfamiliar with the kernel data structs of which BTW
struct cfs_rq is not one defined in the text.

:(

I'm looking for something more like this, but flushed out more as a textbook

http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/index.html,
and some mentoring, I hope.


Ruben

On 03/22/2015 08:35 PM, nick wrote:
> 
> 
> On 2015-03-22 08:05 PM, Ruben Safir wrote:
>> On 03/22/2015 07:30 PM, nick wrote:
>>> I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
>>> as when I was learning the scheduler and process management
>>
>>
>> how much has the scheduler changed since then.  It was completely
>> overhauled when the CFS was created
>>
>>
>>
> The 3rd edition of this book was written after CFS was in the kernel so the chapters
> are pretty up to date.
> Nick
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> .
> 

Kernel thread scheduling

[Ruben Safir]

On 04/09/2015 10:00 PM, nick wrote:
> 
> 
> On 2015-04-09 09:51 PM, Ruben Safir wrote:
>>
>> It is passover so I've read over much of this text, but I have to say
>> that in general, I'm way ahead of this book.  Although I have limited
>> knowledge of Kernel technology in the specific, the C code, data
>> structs, and programming concepts are spoon feed in this text and its
>> wasting too much time with words that are more easily explained with
>> coding examples and UML charts.  I don't need a chapter explaining how
>> to use ps and the basis of Unix architecture.  This text is targeted to
>> a different audience, and FWIW, I'm not certain it does a good job of
>> that either.  The guys who write these texts fall in love with their own
>> voices.  I know, I've suffered this disease myself when I've written
>> tech articles and books.
>>
>> I can''t recommend this book to anyone.  Anyone who doesn't understand
>> the basics of I/O processer blocks is not going to understand
>>
>> static void update_curr(struct cfs_rq *cfs_rq)
>>
>>
>> and OTOH void update_curr(struct cfs_rq *cfs_rq) is not explained well
>> enough for coders unfamiliar with the kernel data structs of which BTW
>> struct cfs_rq is not one defined in the text.
>>
>> :(
>>
>> I'm looking for something more like this, but flushed out more as a textbook
>>
>> http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/index.html,
>> and some mentoring, I hope.
>>
>>
>> Ruben
>>
> Ruben,
> The book is for an intro to the kernel not a complete walk through of each subsystem.
> If that is the case,why not read the subsystem code and docs in the kernel. I am a
> novice myself in terms of patch and coding experience but will be glad to explain the
> code as I have read lots of it.


Thank you nick.  Yes, I downloaded the entire source from Kernel.org and
it is sitting on both my laptop and in virtual machines where I have
already compiled some stuff and not broken my VMs yets :)

I'm looking over  /home/ruben/linux-3.19.3/Documentation/scheduler
[ruben at stat13 scheduler]$ ls

00-INDEX   sched-deadline.txt     sched-rt-group.txt
media=legal -o sides=two-sided-long-edg  sched-design-CFS.txt
sched-stats.txt sched-arch.txt sched-domains.txt
sched-bwc.txt   sched-nice-design.txt


I also see in the code there is significant documentation.

Right now I am trying to figure out what is the relationship between
struct sched_entity and
struct cfs_rq and
struct rq_of

why do we have both??

there is a cast in update_curr
u64 now = rq_of(cfs_rq)->clock;

or is rq_of a function that returns a pointer is a struct that I missed?





> Nick
>> On 03/22/2015 08:35 PM, nick wrote:
>>>
>>>
>>> On 2015-03-22 08:05 PM, Ruben Safir wrote:
>>>> On 03/22/2015 07:30 PM, nick wrote:
>>>>> I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
>>>>> as when I was learning the scheduler and process management
>>>>
>>>>
>>>> how much has the scheduler changed since then.  It was completely
>>>> overhauled when the CFS was created
>>>>
>>>>
>>>>
>>> The 3rd edition of this book was written after CFS was in the kernel so the chapters
>>> are pretty up to date.
>>> Nick
>>>> _______________________________________________
>>>> Kernelnewbies mailing list
>>>> Kernelnewbies at kernelnewbies.org
>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>>
>>>
>>> _______________________________________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>> .
>>>
>>
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
> 
> 

Kernel thread scheduling

[nick]

On 2015-04-09 10:12 PM, Ruben Safir wrote:
> On 04/09/2015 10:00 PM, nick wrote:
>>
>>
>> On 2015-04-09 09:51 PM, Ruben Safir wrote:
>>>
>>> It is passover so I've read over much of this text, but I have to say
>>> that in general, I'm way ahead of this book.  Although I have limited
>>> knowledge of Kernel technology in the specific, the C code, data
>>> structs, and programming concepts are spoon feed in this text and its
>>> wasting too much time with words that are more easily explained with
>>> coding examples and UML charts.  I don't need a chapter explaining how
>>> to use ps and the basis of Unix architecture.  This text is targeted to
>>> a different audience, and FWIW, I'm not certain it does a good job of
>>> that either.  The guys who write these texts fall in love with their own
>>> voices.  I know, I've suffered this disease myself when I've written
>>> tech articles and books.
>>>
>>> I can''t recommend this book to anyone.  Anyone who doesn't understand
>>> the basics of I/O processer blocks is not going to understand
>>>
>>> static void update_curr(struct cfs_rq *cfs_rq)
>>>
>>>
>>> and OTOH void update_curr(struct cfs_rq *cfs_rq) is not explained well
>>> enough for coders unfamiliar with the kernel data structs of which BTW
>>> struct cfs_rq is not one defined in the text.
>>>
>>> :(
>>>
>>> I'm looking for something more like this, but flushed out more as a textbook
>>>
>>> http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/index.html,
>>> and some mentoring, I hope.
>>>
>>>
>>> Ruben
>>>

>> Ruben,
>> The book is for an intro to the kernel not a complete walk through of each subsystem.
>> If that is the case,why not read the subsystem code and docs in the kernel. I am a
>> novice myself in terms of patch and coding experience but will be glad to explain the
>> code as I have read lots of it.
> 
> 
> Thank you nick.  Yes, I downloaded the entire source from Kernel.org and
> it is sitting on both my laptop and in virtual machines where I have
> already compiled some stuff and not broken my VMs yets :)
> 
> I'm looking over  /home/ruben/linux-3.19.3/Documentation/scheduler
> [ruben at stat13 scheduler]$ ls
> 
> 00-INDEX   sched-deadline.txt     sched-rt-group.txt
> media=legal -o sides=two-sided-long-edg  sched-design-CFS.txt
> sched-stats.txt sched-arch.txt sched-domains.txt
> sched-bwc.txt   sched-nice-design.txt
> 
> 
> I also see in the code there is significant documentation.
> 
> Right now I am trying to figure out what is the relationship between
> struct sched_entity and
> struct cfs_rq and
> struct rq_of
> 
> why do we have both??
> 
The way that these structures  are related is sched_enity is the entity for the task's scheduling
information or the task_struct of the a excuting process to be exact. Furthermore  cfs_rq is the
runquenue on which the task is running or selected to run on by schedule,the main scheduling function
of the kernel. Finally here is the function definition for rq_of to answer your question,
static inline struct rq *rq_of(struct cfs_rq *cfs_rq)
{
 	return cfs_rq->rq;
}
Before asking questions again like this please look into either using lxr or ctags
to navigate the kernel tree for answers as can be faster then waiting for me or 
someone else to respond.
Thanks,
Nick
> there is a cast in update_curr
> u64 now = rq_of(cfs_rq)->clock;
> 
> or is rq_of a function that returns a pointer is a struct that I missed?
> 
> 
> 
> 
> 
>> Nick
>>> On 03/22/2015 08:35 PM, nick wrote:
>>>>
>>>>
>>>> On 2015-03-22 08:05 PM, Ruben Safir wrote:
>>>>> On 03/22/2015 07:30 PM, nick wrote:
>>>>>> I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
>>>>>> as when I was learning the scheduler and process management
>>>>>
>>>>>
>>>>> how much has the scheduler changed since then.  It was completely
>>>>> overhauled when the CFS was created
>>>>>
>>>>>
>>>>>
>>>> The 3rd edition of this book was written after CFS was in the kernel so the chapters
>>>> are pretty up to date.
>>>> Nick
>>>>> _______________________________________________
>>>>> Kernelnewbies mailing list
>>>>> Kernelnewbies at kernelnewbies.org
>>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>>>
>>>>
>>>> _______________________________________________
>>>> Kernelnewbies mailing list
>>>> Kernelnewbies at kernelnewbies.org
>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>>
>>>> .
>>>>
>>>
>>>
>>> _______________________________________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>
>>
> 

Kernel thread scheduling

[Ruben Safir]

On 04/09/2015 10:52 PM, nick wrote:
> Before asking questions again like this please look into either using lxr or ctags
> to navigate the kernel tree for answers as can be faster then waiting for me or 
> someone else to respond.


well, I reading the text is ctags aren't much value there.

Kernel thread scheduling

[Ruben Safir]

On 04/10/2015 09:09 AM, nick wrote:
> 
> 
> On 2015-04-09 11:37 PM, Ruben Safir wrote:
>> On 04/09/2015 10:52 PM, nick wrote:
>>> Before asking questions again like this please look into either using lxr or ctags
>>> to navigate the kernel tree for answers as can be faster then waiting for me or 
>>> someone else to respond.
>>
>>
>> well, I reading the text is ctags aren't much value there.
>>
> Ctags is useful for searching the code, which is why I am recommending it.
> Nick

I have it built into gvim, but you can't use it from a textbook.  I'm
finding it is not as useful as it could be for the kernel code.  There
are stacks of tags to get around.  Another 2 days to learn to get around
tags in vi is not in the agenda right now.  It is the tool I have so
I'll have to live with it right now.

I also have a question that is not obvious from the code I'm looking at.
 I'm not sure how these structs are attached together.  Or more
specifically, I'm not sure how pulling the correct sched_entity gets one
the coresponding task_entity

You have
struct task_struct with a
	struct sched_entity

struct sched_enitities are nodes in the RB tree
	which are a "container" for "struct rb_node run_node".

So a look at sched_entity ... is in ../linux/sched.h

1161 struct sched_entity {
1162    struct load_weight   load;    /* for load-balancing */
1163    struct rb_node    run_node;
1164    struct list_head  group_node;
1165    unsigned int      on_rq;
1166
1167    u64         exec_start;
1168    u64         sum_exec_runtime;
1169    u64         vruntime;
1170    u64         prev_sum_exec_runtime;
1171
1172    u64         nr_migrations;
1173
1174 #ifdef CONFIG_SCHEDSTATS
1175    struct sched_statistics statistics;
1176 #endif
1177
1178 #ifdef CONFIG_FAIR_GROUP_SCHED
1179    int         depth;
1180    struct sched_entity  *parent;
1181    /* rq on which this entity is (to be) queued: */
1182    struct cfs_rq     *cfs_rq;
1183    /* rq "owned" by this entity/group: */
1184    struct cfs_rq     *my_q;
1185 #endif
1186
1187 #ifdef CONFIG_SMP
1188    /* Per-entity load-tracking */
1189    struct sched_avg  avg;
1190 #endif
1191 };

I see no means of referencing a specific task from this struct that
forms the node.  So when you pull the node with the smallest vruntime
from the left most postion of the RB tree, by calling pick_next_task(),


static struct sched_entity *__pick_next_entity(struct sched_entity *se)
{
	struct rb_node *next = rb_next(&se->run_node);

	if (!next)
		return NULL;

	return rb_entry(next, struct sched_entity, run_node);
}


how do we know what task we are attached to?

Ruben




>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
> 
> 

Kernel thread scheduling

[Ruben Safir]

On 04/11/2015 10:21 PM, Ruben Safir wrote:
> On 04/10/2015 09:09 AM, nick wrote:
>>
>>
>> On 2015-04-09 11:37 PM, Ruben Safir wrote:
>>> On 04/09/2015 10:52 PM, nick wrote:
>>>> Before asking questions again like this please look into either using lxr or ctags
>>>> to navigate the kernel tree for answers as can be faster then waiting for me or 
>>>> someone else to respond.
>>>
>>>
>>> well, I reading the text is ctags aren't much value there.
>>>
>> Ctags is useful for searching the code, which is why I am recommending it.
>> Nick
> 
> I have it built into gvim, but you can't use it from a textbook.  I'm
> finding it is not as useful as it could be for the kernel code.  There
> are stacks of tags to get around.  Another 2 days to learn to get around
> tags in vi is not in the agenda right now.  It is the tool I have so
> I'll have to live with it right now.
> 
> I also have a question that is not obvious from the code I'm looking at.
>  I'm not sure how these structs are attached together.  Or more
> specifically, I'm not sure how pulling the correct sched_entity gets one
> the coresponding task_entity
> 
> You have
> struct task_struct with a
> 	struct sched_entity
> 
> struct sched_enitities are nodes in the RB tree
> 	which are a "container" for "struct rb_node run_node".
> 
> So a look at sched_entity ... is in ../linux/sched.h
> 
> 1161 struct sched_entity {
> 1162    struct load_weight   load;    /* for load-balancing */
> 1163    struct rb_node    run_node;
> 1164    struct list_head  group_node;
> 1165    unsigned int      on_rq;
> 1166
> 1167    u64         exec_start;
> 1168    u64         sum_exec_runtime;
> 1169    u64         vruntime;
> 1170    u64         prev_sum_exec_runtime;
> 1171
> 1172    u64         nr_migrations;
> 1173
> 1174 #ifdef CONFIG_SCHEDSTATS
> 1175    struct sched_statistics statistics;
> 1176 #endif
> 1177
> 1178 #ifdef CONFIG_FAIR_GROUP_SCHED
> 1179    int         depth;
> 1180    struct sched_entity  *parent;
> 1181    /* rq on which this entity is (to be) queued: */
> 1182    struct cfs_rq     *cfs_rq;
> 1183    /* rq "owned" by this entity/group: */
> 1184    struct cfs_rq     *my_q;
> 1185 #endif
> 1186
> 1187 #ifdef CONFIG_SMP
> 1188    /* Per-entity load-tracking */
> 1189    struct sched_avg  avg;
> 1190 #endif
> 1191 };
> 
> I see no means of referencing a specific task from this struct that
> forms the node.  So when you pull the node with the smallest vruntime
> from the left most postion of the RB tree, by calling pick_next_task(),
> 
> 
> static struct sched_entity *__pick_next_entity(struct sched_entity *se)
> {
> 	struct rb_node *next = rb_next(&se->run_node);
> 
> 	if (!next)
> 		return NULL;
> 
> 	return rb_entry(next, struct sched_entity, run_node);
> }
> 
> 
> how do we know what task we are attached to?
> 
> Ruben
> 
> 

I'm still loss on how we know which taks_struct is being used but as a
side note, I found this also very puzzling

return rb_entry(next, struct sched_entity, run_node);
With help I ran it down to this:

http://lxr.free-electrons.com/source/include/linux/rbtree.h#L50

#define rb_entry(ptr, type, member) container_of(ptr, type, member)

which leads me to yet another macro

798 #define container_of(ptr, type, member) ({                      \
799         const typeof( ((type *)0)->member ) *__mptr = (ptr);    \
800         (type *)( (char *)__mptr - offsetof(type,member) );})


This is a use of macros I'd never seen before up close.  If anyone could
help me understand it, I'd appreciate it.

Ruben
> 
> 
>>> _______________________________________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>
>>
> 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> 

Kernel thread scheduling

[nick]

On 2015-04-11 11:02 PM, Ruben Safir wrote:
> On 04/11/2015 10:21 PM, Ruben Safir wrote:
>> On 04/10/2015 09:09 AM, nick wrote:
>>>
>>>
>>> On 2015-04-09 11:37 PM, Ruben Safir wrote:
>>>> On 04/09/2015 10:52 PM, nick wrote:
>>>>> Before asking questions again like this please look into either using lxr or ctags
>>>>> to navigate the kernel tree for answers as can be faster then waiting for me or 
>>>>> someone else to respond.
>>>>
>>>>
>>>> well, I reading the text is ctags aren't much value there.
>>>>
>>> Ctags is useful for searching the code, which is why I am recommending it.
>>> Nick
>>
>> I have it built into gvim, but you can't use it from a textbook.  I'm
>> finding it is not as useful as it could be for the kernel code.  There
>> are stacks of tags to get around.  Another 2 days to learn to get around
>> tags in vi is not in the agenda right now.  It is the tool I have so
>> I'll have to live with it right now.
>>
>> I also have a question that is not obvious from the code I'm looking at.
>>  I'm not sure how these structs are attached together.  Or more
>> specifically, I'm not sure how pulling the correct sched_entity gets one
>> the coresponding task_entity
>>
>> You have
>> struct task_struct with a
>> 	struct sched_entity
>>
>> struct sched_enitities are nodes in the RB tree
>> 	which are a "container" for "struct rb_node run_node".
>>
>> So a look at sched_entity ... is in ../linux/sched.h
>>
>> 1161 struct sched_entity {
>> 1162    struct load_weight   load;    /* for load-balancing */
>> 1163    struct rb_node    run_node;
>> 1164    struct list_head  group_node;
>> 1165    unsigned int      on_rq;
>> 1166
>> 1167    u64         exec_start;
>> 1168    u64         sum_exec_runtime;
>> 1169    u64         vruntime;
>> 1170    u64         prev_sum_exec_runtime;
>> 1171
>> 1172    u64         nr_migrations;
>> 1173
>> 1174 #ifdef CONFIG_SCHEDSTATS
>> 1175    struct sched_statistics statistics;
>> 1176 #endif
>> 1177
>> 1178 #ifdef CONFIG_FAIR_GROUP_SCHED
>> 1179    int         depth;
>> 1180    struct sched_entity  *parent;
>> 1181    /* rq on which this entity is (to be) queued: */
>> 1182    struct cfs_rq     *cfs_rq;
>> 1183    /* rq "owned" by this entity/group: */
>> 1184    struct cfs_rq     *my_q;
>> 1185 #endif
>> 1186
>> 1187 #ifdef CONFIG_SMP
>> 1188    /* Per-entity load-tracking */
>> 1189    struct sched_avg  avg;
>> 1190 #endif
>> 1191 };
>>
>> I see no means of referencing a specific task from this struct that
>> forms the node.  So when you pull the node with the smallest vruntime
>> from the left most postion of the RB tree, by calling pick_next_task(),
>>
>>
>> static struct sched_entity *__pick_next_entity(struct sched_entity *se)
>> {
>> 	struct rb_node *next = rb_next(&se->run_node);
This finds the next node in the red black tree for  sched_enities.
Basically rb_next finds the next node in the tree. The argument is 
the rb_node structure embedded in the structure using a red black
tree.
>>
>> 	if (!next)
>> 		return NULL;
>>
If there is no runnable task return NULL and pick_next_task will run the 
idle_task for this cpu.
>> 	return rb_entry(next, struct sched_entity, run_node);
>> }
>>
>>
>> how do we know what task we are attached to?
>>
Also try to read Chapter 6 of Linux Kernel Development as if have read that
chapter understanding how red black trees and other data structures work in
kernel code would make more sense.
Nick
>> Ruben
>>
>>
> 
> I'm still loss on how we know which taks_struct is being used but as a
> side note, I found this also very puzzling
> 
> return rb_entry(next, struct sched_entity, run_node);
> With help I ran it down to this:
> 
> http://lxr.free-electrons.com/source/include/linux/rbtree.h#L50
> 
> #define rb_entry(ptr, type, member) container_of(ptr, type, member)
> 
> which leads me to yet another macro
> 
> 798 #define container_of(ptr, type, member) ({                      \
> 799         const typeof( ((type *)0)->member ) *__mptr = (ptr);    \
> 800         (type *)( (char *)__mptr - offsetof(type,member) );})
> 
> 
> This is a use of macros I'd never seen before up close.  If anyone could
> help me understand it, I'd appreciate it.
> 
> Ruben
>>
>>
>>>> _______________________________________________
>>>> Kernelnewbies mailing list
>>>> Kernelnewbies at kernelnewbies.org
>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>>
> 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 

Kernel thread scheduling

[Ruben Safir]

On 04/12/2015 12:16 AM, nick wrote:
> This finds the next node in the red black tree for  sched_enities.
> Basically rb_next finds the next node in the tree. The argument is 
> the rb_node structure embedded in the structure using a red black
> tree.


The problem is this


struct mystruc {
	int a;
	char * string;
} wonderful;

char * string2 = wonderful.string;



There is no way to know that string2 comes from wonderful.
That is the problem I have with the nodes

The node is an instance of sched_entity.  But unless there is a field
that says what task_entity it is embedded in, in of itself, there is no
way to know.  There is no 'this' C.

Obviously if does know.  I'll look at chapter 6

I'm also looking at this



static struct sched_entity *__pick_next_entity(struct sched_entity *se)
{
	struct rb_node *next = rb_next(&se->run_node);

	if (!next)
		return NULL;

	return rb_entry(next, struct sched_entity, run_node); <<==macro
}



The macro is:  I think:
http://lxr.free-electrons.com/source/include/linux/rbtree.h#L50

50 #define rb_entry(ptr, type, member) container_of(ptr, type, member)

container_of is
http://lxr.free-electrons.com/source/include/linux/kernel.h#L798


791 /**
792  * container_of - cast a member of a structure out to the containing
structure
793  * @ptr:        the pointer to the member.
794  * @type:       the type of the container struct this is embedded in.
795  * @member:     the name of the member within the struct.
796  *
797  */
798 #define container_of(ptr, type, member) ({                      \
799         const typeof( ((type *)0)->member ) *__mptr = (ptr);    \
800         (type *)( (char *)__mptr - offsetof(type,member) );})


Now I have embedded macros which I thought you couldn't even do...

In trying to understand this I stumbled on

http://linuxkernel51.blogspot.com/2011/02/how-containerof-macro-works-example.html
 How "container_of" macro works, & an Example
Here iam giving a small code snippet that gives and idea about working
of "container_of", this posed me little difficulty in understanding,
after google-ing i got some examples and after working on that i wrote a
simple C application that depicts its working. here i have defined two
macros "offsetof" and "container_of" which i have extracted from
"kernel.h" header.
       Please interpret this code and try some trick to understand
"container_of".

container_of macro is defined in linux/kernel.h

syntax: container_of( pointer, container_type, container_field );

This macro takes a pointer to a filed name container_field, within a
structure of type container_type, and returns a pointer to the
containing structure .

simply this is a convenience macro that may be used to obtain a pointer
to a structure from a pointer to some other structure contained with in it.


Code :

#include <stdio.h>
#include <stdlib.h>

#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)

#define container_of(ptr, type, member) ({            \
 const typeof( ((type *)0)->member ) *__mptr = (ptr);    \
 (type *)( (char *)__mptr - offsetof(type,member) );})

struct test1 {
 int a;
};

struct test2 {
 int b;
 struct test1 z;
 int c;
};

int main()
{
 /* existing structure */
 struct test2 *obj;
 obj = malloc(sizeof(struct test2));
 if(obj == NULL){
       printf("Error: Memory not allocated...!\n");
 }
 obj->z.a = 51;
 obj->b = 43;
 obj->c = 53;

 /* pointer to existing entry */
 struct test1 *obj1 = &obj->z; //both of type test1

 struct test2 *obj2 = container_of(obj1, struct test2, z);

 printf("obj2->b = %d\n", obj2->b); ///notsure what this does,need a nap

 return EXIT_SUCCESS;
}

Kernel thread scheduling

[Ruben Safir]

On 04/12/2015 12:36 AM, Mohammed Ghriga wrote:
> In addition to the suggestions that were offered, I recommend you try reading Chapter 16: https://www.google.com/search?tbo=p&tbm=bks&q=isbn:0596554672 (pages: 267-272). 
> 

http://it-ebooks.info/book/481/

FWIW I found this lovely wait example in the other kernel text under the
scheduling section

Sleeping and Waking Up page 58 in Robert Love, Linux Kernel Development

http://www.it-ebooks.info/book/819/

Kernel thread scheduling

[nick]

On April 12, 2015 1:06:57 AM EDT, Ruben Safir <ruben@mrbrklyn.com> wrote:
>On 04/12/2015 12:36 AM, Mohammed Ghriga wrote:
>> In addition to the suggestions that were offered, I recommend you try
>reading Chapter 16:
>https://www.google.com/search?tbo=p&tbm=bks&q=isbn:0596554672 (pages:
>267-272). 
>> 
>
>http://it-ebooks.info/book/481/
>
>FWIW I found this lovely wait example in the other kernel text under
>the
>scheduling section
>
>Sleeping and Waking Up page 58 in Robert Love, Linux Kernel Development
>
>http://www.it-ebooks.info/book/819/
>
I recommended this book as that was how I learn the basics of kernel development.  Again if your interested in certain areas I will be glad to link/explain it to you to the best of my knowledge. 
Nick
>Kernelnewbies mailing list
>Kernelnewbies at kernelnewbies.org
>http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel thread scheduling

[Ruben Safir]

I'm trying to find rb_node's structure and I can't find it with ctags or
in the http://lxr.linux.no website.


How do you search these things out?


Ruben

On Thu, Apr 09, 2015 at 10:52:03PM -0400, nick wrote:
> 
> 
> On 2015-04-09 10:12 PM, Ruben Safir wrote:
> > On 04/09/2015 10:00 PM, nick wrote:
> >>
> >>
> >> On 2015-04-09 09:51 PM, Ruben Safir wrote:
> >>>
> >>> It is passover so I've read over much of this text, but I have to say
> >>> that in general, I'm way ahead of this book.  Although I have limited
> >>> knowledge of Kernel technology in the specific, the C code, data
> >>> structs, and programming concepts are spoon feed in this text and its
> >>> wasting too much time with words that are more easily explained with
> >>> coding examples and UML charts.  I don't need a chapter explaining how
> >>> to use ps and the basis of Unix architecture.  This text is targeted to
> >>> a different audience, and FWIW, I'm not certain it does a good job of
> >>> that either.  The guys who write these texts fall in love with their own
> >>> voices.  I know, I've suffered this disease myself when I've written
> >>> tech articles and books.
> >>>
> >>> I can''t recommend this book to anyone.  Anyone who doesn't understand
> >>> the basics of I/O processer blocks is not going to understand
> >>>
> >>> static void update_curr(struct cfs_rq *cfs_rq)
> >>>
> >>>
> >>> and OTOH void update_curr(struct cfs_rq *cfs_rq) is not explained well
> >>> enough for coders unfamiliar with the kernel data structs of which BTW
> >>> struct cfs_rq is not one defined in the text.
> >>>
> >>> :(
> >>>
> >>> I'm looking for something more like this, but flushed out more as a textbook
> >>>
> >>> http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/index.html,
> >>> and some mentoring, I hope.
> >>>
> >>>
> >>> Ruben
> >>>
> 
> >> Ruben,
> >> The book is for an intro to the kernel not a complete walk through of each subsystem.
> >> If that is the case,why not read the subsystem code and docs in the kernel. I am a
> >> novice myself in terms of patch and coding experience but will be glad to explain the
> >> code as I have read lots of it.
> > 
> > 
> > Thank you nick.  Yes, I downloaded the entire source from Kernel.org and
> > it is sitting on both my laptop and in virtual machines where I have
> > already compiled some stuff and not broken my VMs yets :)
> > 
> > I'm looking over  /home/ruben/linux-3.19.3/Documentation/scheduler
> > [ruben at stat13 scheduler]$ ls
> > 
> > 00-INDEX   sched-deadline.txt     sched-rt-group.txt
> > media=legal -o sides=two-sided-long-edg  sched-design-CFS.txt
> > sched-stats.txt sched-arch.txt sched-domains.txt
> > sched-bwc.txt   sched-nice-design.txt
> > 
> > 
> > I also see in the code there is significant documentation.
> > 
> > Right now I am trying to figure out what is the relationship between
> > struct sched_entity and
> > struct cfs_rq and
> > struct rq_of
> > 
> > why do we have both??
> > 
> The way that these structures  are related is sched_enity is the entity for the task's scheduling
> information or the task_struct of the a excuting process to be exact. Furthermore  cfs_rq is the
> runquenue on which the task is running or selected to run on by schedule,the main scheduling function
> of the kernel. Finally here is the function definition for rq_of to answer your question,
> static inline struct rq *rq_of(struct cfs_rq *cfs_rq)
> {
>  	return cfs_rq->rq;
> }
> Before asking questions again like this please look into either using lxr or ctags
> to navigate the kernel tree for answers as can be faster then waiting for me or 
> someone else to respond.
> Thanks,
> Nick
> > there is a cast in update_curr
> > u64 now = rq_of(cfs_rq)->clock;
> > 
> > or is rq_of a function that returns a pointer is a struct that I missed?
> > 
> > 
> > 
> > 
> > 
> >> Nick
> >>> On 03/22/2015 08:35 PM, nick wrote:
> >>>>
> >>>>
> >>>> On 2015-03-22 08:05 PM, Ruben Safir wrote:
> >>>>> On 03/22/2015 07:30 PM, nick wrote:
> >>>>>> I would recommend reading Chapters 3 and  4 of Linux Kernel Development by Robert Love
> >>>>>> as when I was learning the scheduler and process management
> >>>>>
> >>>>>
> >>>>> how much has the scheduler changed since then.  It was completely
> >>>>> overhauled when the CFS was created
> >>>>>
> >>>>>
> >>>>>
> >>>> The 3rd edition of this book was written after CFS was in the kernel so the chapters
> >>>> are pretty up to date.
> >>>> Nick
> >>>>> _______________________________________________
> >>>>> Kernelnewbies mailing list
> >>>>> Kernelnewbies at kernelnewbies.org
> >>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> Kernelnewbies mailing list
> >>>> Kernelnewbies at kernelnewbies.org
> >>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>>
> >>>> .
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Kernelnewbies mailing list
> >>> Kernelnewbies at kernelnewbies.org
> >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>
> >>
> >>
> > 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com 

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive 
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com 

Being so tracked is for FARM ANIMALS and and extermination camps, 
but incompatible with living as a free human being. -RI Safir 2013

Kernel thread scheduling

[Ricardo Ribalda Delgado]

On Thu, Apr 16, 2015 at 4:56 PM, Ruben Safir <ruben@mrbrklyn.com> wrote:
> I'm trying to find rb_node's structure and I can't find it with ctags or
> in the http://lxr.linux.no website.
>
>
> How do you search these things out?


When everything else fails, I use grep

ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$ git grep " rb_node
{" include/
include/linux/rbtree.h:struct rb_node {
ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$



-- 
Ricardo Ribalda

Kernel thread scheduling

[Aruna Hewapathirane]

>> I'm trying to find rb_node's structure and I can't find it with ctags or
>> in the http://lxr.linux.no website.

Please use this: Linux 2.6.11 and later <http://lxr.linux.no/linux>

At the top right there is a text box please type what your searching for,
in this rb_node. It will show you on the right the search results.

The Structure is defined here:  include/linux/rbtree.h, line 35
<http://lxr.linux.no/linux+*/include/linux/rbtree.h#L35>

and here it is in case you have difficulties:

struct rb_node <http://lxr.linux.no/linux+*/+code=rb_node> {  36
<http://lxr.linux.no/linux+*/include/linux/rbtree.h#L36>
unsigned long  __rb_parent_color
<http://lxr.linux.no/linux+*/+code=__rb_parent_color>;  37
<http://lxr.linux.no/linux+*/include/linux/rbtree.h#L37>        struct
rb_node <http://lxr.linux.no/linux+*/+code=rb_node> *rb_right
<http://lxr.linux.no/linux+*/+code=rb_right>;  38
<http://lxr.linux.no/linux+*/include/linux/rbtree.h#L38>        struct
rb_node <http://lxr.linux.no/linux+*/+code=rb_node> *rb_left
<http://lxr.linux.no/linux+*/+code=rb_left>;  39
<http://lxr.linux.no/linux+*/include/linux/rbtree.h#L39>}
__attribute__ <http://lxr.linux.no/linux+*/+code=__attribute__>((aligned
<http://lxr.linux.no/linux+*/+code=aligned>(sizeof(long))));  40
<http://lxr.linux.no/linux+*/include/linux/rbtree.h#L40>    /* The
alignment might seem pointless, but allegedly CRIS needs it */


Thank's - Aruna






>
> Ruben
>
> On Thu, Apr 09, 2015 at 10:52:03PM -0400, nick wrote:
> >
> >
> > On 2015-04-09 10:12 PM, Ruben Safir wrote:
> > > On 04/09/2015 10:00 PM, nick wrote:
> > >>
> > >>
> > >> On 2015-04-09 09:51 PM, Ruben Safir wrote:
> > >>>
> > >>> It is passover so I've read over much of this text, but I have to say
> > >>> that in general, I'm way ahead of this book.  Although I have limited
> > >>> knowledge of Kernel technology in the specific, the C code, data
> > >>> structs, and programming concepts are spoon feed in this text and its
> > >>> wasting too much time with words that are more easily explained with
> > >>> coding examples and UML charts.  I don't need a chapter explaining
> how
> > >>> to use ps and the basis of Unix architecture.  This text is targeted
> to
> > >>> a different audience, and FWIW, I'm not certain it does a good job of
> > >>> that either.  The guys who write these texts fall in love with their
> own
> > >>> voices.  I know, I've suffered this disease myself when I've written
> > >>> tech articles and books.
> > >>>
> > >>> I can''t recommend this book to anyone.  Anyone who doesn't
> understand
> > >>> the basics of I/O processer blocks is not going to understand
> > >>>
> > >>> static void update_curr(struct cfs_rq *cfs_rq)
> > >>>
> > >>>
> > >>> and OTOH void update_curr(struct cfs_rq *cfs_rq) is not explained
> well
> > >>> enough for coders unfamiliar with the kernel data structs of which
> BTW
> > >>> struct cfs_rq is not one defined in the text.
> > >>>
> > >>> :(
> > >>>
> > >>> I'm looking for something more like this, but flushed out more as a
> textbook
> > >>>
> > >>>
> http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/index.html
> ,
> > >>> and some mentoring, I hope.
> > >>>
> > >>>
> > >>> Ruben
> > >>>
> >
> > >> Ruben,
> > >> The book is for an intro to the kernel not a complete walk through of
> each subsystem.
> > >> If that is the case,why not read the subsystem code and docs in the
> kernel. I am a
> > >> novice myself in terms of patch and coding experience but will be
> glad to explain the
> > >> code as I have read lots of it.
> > >
> > >
> > > Thank you nick.  Yes, I downloaded the entire source from Kernel.org
> and
> > > it is sitting on both my laptop and in virtual machines where I have
> > > already compiled some stuff and not broken my VMs yets :)
> > >
> > > I'm looking over  /home/ruben/linux-3.19.3/Documentation/scheduler
> > > [ruben at stat13 scheduler]$ ls
> > >
> > > 00-INDEX   sched-deadline.txt     sched-rt-group.txt
> > > media=legal -o sides=two-sided-long-edg  sched-design-CFS.txt
> > > sched-stats.txt sched-arch.txt sched-domains.txt
> > > sched-bwc.txt   sched-nice-design.txt
> > >
> > >
> > > I also see in the code there is significant documentation.
> > >
> > > Right now I am trying to figure out what is the relationship between
> > > struct sched_entity and
> > > struct cfs_rq and
> > > struct rq_of
> > >
> > > why do we have both??
> > >
> > The way that these structures  are related is sched_enity is the entity
> for the task's scheduling
> > information or the task_struct of the a excuting process to be exact.
> Furthermore  cfs_rq is the
> > runquenue on which the task is running or selected to run on by
> schedule,the main scheduling function
> > of the kernel. Finally here is the function definition for rq_of to
> answer your question,
> > static inline struct rq *rq_of(struct cfs_rq *cfs_rq)
> > {
> >       return cfs_rq->rq;
> > }
> > Before asking questions again like this please look into either using
> lxr or ctags
> > to navigate the kernel tree for answers as can be faster then waiting
> for me or
> > someone else to respond.
> > Thanks,
> > Nick
> > > there is a cast in update_curr
> > > u64 now = rq_of(cfs_rq)->clock;
> > >
> > > or is rq_of a function that returns a pointer is a struct that I
> missed?
> > >
> > >
> > >
> > >
> > >
> > >> Nick
> > >>> On 03/22/2015 08:35 PM, nick wrote:
> > >>>>
> > >>>>
> > >>>> On 2015-03-22 08:05 PM, Ruben Safir wrote:
> > >>>>> On 03/22/2015 07:30 PM, nick wrote:
> > >>>>>> I would recommend reading Chapters 3 and  4 of Linux Kernel
> Development by Robert Love
> > >>>>>> as when I was learning the scheduler and process management
> > >>>>>
> > >>>>>
> > >>>>> how much has the scheduler changed since then.  It was completely
> > >>>>> overhauled when the CFS was created
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>> The 3rd edition of this book was written after CFS was in the
> kernel so the chapters
> > >>>> are pretty up to date.
> > >>>> Nick
> > >>>>> _______________________________________________
> > >>>>> Kernelnewbies mailing list
> > >>>>> Kernelnewbies at kernelnewbies.org
> > >>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > >>>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> Kernelnewbies mailing list
> > >>>> Kernelnewbies at kernelnewbies.org
> > >>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > >>>>
> > >>>> .
> > >>>>
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Kernelnewbies mailing list
> > >>> Kernelnewbies at kernelnewbies.org
> > >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > >>>
> > >>
> > >>
> > >
> >
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
> --
> So many immigrant groups have swept through our town
> that Brooklyn, like Atlantis, reaches mythological
> proportions in the mind of the world - RI Safir 1998
> http://www.mrbrklyn.com
>
> DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
> http://www.nylxs.com - Leadership Development in Free Software
> http://www2.mrbrklyn.com/resources - Unpublished Archive
> http://www.coinhangout.com - coins!
> http://www.brooklyn-living.com
>
> Being so tracked is for FARM ANIMALS and and extermination camps,
> but incompatible with living as a free human being. -RI Safir 2013
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150416/da5e692b/attachment-0001.html 

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 11:07 AM, Ricardo Ribalda Delgado wrote:
> On Thu, Apr 16, 2015 at 4:56 PM, Ruben Safir <ruben@mrbrklyn.com> wrote:
>> I'm trying to find rb_node's structure and I can't find it with ctags or
>> in the http://lxr.linux.no website.
>>
>>
>> How do you search these things out?
> 
> 
> When everything else fails, I use grep
> 
> ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$ git grep " rb_node
> {" include/
> include/linux/rbtree.h:struct rb_node {
> ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$
> 
> 
> 


yeah ...

Kernel thread scheduling

[Mark P]

I find that the free electrons LXR has the best search capabilities:

http://lxr.free-electrons.com/source/include/linux/rbtree.h#L35

-M

On Thu, Apr 16, 2015 at 10:56 AM, Ruben Safir <ruben@mrbrklyn.com> wrote:

> I'm trying to find rb_node's structure and I can't find it with ctags or
> in the http://lxr.linux.no website.
>
>
> How do you search these things out?
>
>
> Ruben
>
> On Thu, Apr 09, 2015 at 10:52:03PM -0400, nick wrote:
> >
> >
> > On 2015-04-09 10:12 PM, Ruben Safir wrote:
> > > On 04/09/2015 10:00 PM, nick wrote:
> > >>
> > >>
> > >> On 2015-04-09 09:51 PM, Ruben Safir wrote:
> > >>>
> > >>> It is passover so I've read over much of this text, but I have to say
> > >>> that in general, I'm way ahead of this book.  Although I have limited
> > >>> knowledge of Kernel technology in the specific, the C code, data
> > >>> structs, and programming concepts are spoon feed in this text and its
> > >>> wasting too much time with words that are more easily explained with
> > >>> coding examples and UML charts.  I don't need a chapter explaining
> how
> > >>> to use ps and the basis of Unix architecture.  This text is targeted
> to
> > >>> a different audience, and FWIW, I'm not certain it does a good job of
> > >>> that either.  The guys who write these texts fall in love with their
> own
> > >>> voices.  I know, I've suffered this disease myself when I've written
> > >>> tech articles and books.
> > >>>
> > >>> I can''t recommend this book to anyone.  Anyone who doesn't
> understand
> > >>> the basics of I/O processer blocks is not going to understand
> > >>>
> > >>> static void update_curr(struct cfs_rq *cfs_rq)
> > >>>
> > >>>
> > >>> and OTOH void update_curr(struct cfs_rq *cfs_rq) is not explained
> well
> > >>> enough for coders unfamiliar with the kernel data structs of which
> BTW
> > >>> struct cfs_rq is not one defined in the text.
> > >>>
> > >>> :(
> > >>>
> > >>> I'm looking for something more like this, but flushed out more as a
> textbook
> > >>>
> > >>>
> http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/index.html
> ,
> > >>> and some mentoring, I hope.
> > >>>
> > >>>
> > >>> Ruben
> > >>>
> >
> > >> Ruben,
> > >> The book is for an intro to the kernel not a complete walk through of
> each subsystem.
> > >> If that is the case,why not read the subsystem code and docs in the
> kernel. I am a
> > >> novice myself in terms of patch and coding experience but will be
> glad to explain the
> > >> code as I have read lots of it.
> > >
> > >
> > > Thank you nick.  Yes, I downloaded the entire source from Kernel.org
> and
> > > it is sitting on both my laptop and in virtual machines where I have
> > > already compiled some stuff and not broken my VMs yets :)
> > >
> > > I'm looking over  /home/ruben/linux-3.19.3/Documentation/scheduler
> > > [ruben at stat13 scheduler]$ ls
> > >
> > > 00-INDEX   sched-deadline.txt     sched-rt-group.txt
> > > media=legal -o sides=two-sided-long-edg  sched-design-CFS.txt
> > > sched-stats.txt sched-arch.txt sched-domains.txt
> > > sched-bwc.txt   sched-nice-design.txt
> > >
> > >
> > > I also see in the code there is significant documentation.
> > >
> > > Right now I am trying to figure out what is the relationship between
> > > struct sched_entity and
> > > struct cfs_rq and
> > > struct rq_of
> > >
> > > why do we have both??
> > >
> > The way that these structures  are related is sched_enity is the entity
> for the task's scheduling
> > information or the task_struct of the a excuting process to be exact.
> Furthermore  cfs_rq is the
> > runquenue on which the task is running or selected to run on by
> schedule,the main scheduling function
> > of the kernel. Finally here is the function definition for rq_of to
> answer your question,
> > static inline struct rq *rq_of(struct cfs_rq *cfs_rq)
> > {
> >       return cfs_rq->rq;
> > }
> > Before asking questions again like this please look into either using
> lxr or ctags
> > to navigate the kernel tree for answers as can be faster then waiting
> for me or
> > someone else to respond.
> > Thanks,
> > Nick
> > > there is a cast in update_curr
> > > u64 now = rq_of(cfs_rq)->clock;
> > >
> > > or is rq_of a function that returns a pointer is a struct that I
> missed?
> > >
> > >
> > >
> > >
> > >
> > >> Nick
> > >>> On 03/22/2015 08:35 PM, nick wrote:
> > >>>>
> > >>>>
> > >>>> On 2015-03-22 08:05 PM, Ruben Safir wrote:
> > >>>>> On 03/22/2015 07:30 PM, nick wrote:
> > >>>>>> I would recommend reading Chapters 3 and  4 of Linux Kernel
> Development by Robert Love
> > >>>>>> as when I was learning the scheduler and process management
> > >>>>>
> > >>>>>
> > >>>>> how much has the scheduler changed since then.  It was completely
> > >>>>> overhauled when the CFS was created
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>> The 3rd edition of this book was written after CFS was in the
> kernel so the chapters
> > >>>> are pretty up to date.
> > >>>> Nick
> > >>>>> _______________________________________________
> > >>>>> Kernelnewbies mailing list
> > >>>>> Kernelnewbies at kernelnewbies.org
> > >>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > >>>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> Kernelnewbies mailing list
> > >>>> Kernelnewbies at kernelnewbies.org
> > >>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > >>>>
> > >>>> .
> > >>>>
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Kernelnewbies mailing list
> > >>> Kernelnewbies at kernelnewbies.org
> > >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > >>>
> > >>
> > >>
> > >
> >
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
> --
> So many immigrant groups have swept through our town
> that Brooklyn, like Atlantis, reaches mythological
> proportions in the mind of the world - RI Safir 1998
> http://www.mrbrklyn.com
>
> DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
> http://www.nylxs.com - Leadership Development in Free Software
> http://www2.mrbrklyn.com/resources - Unpublished Archive
> http://www.coinhangout.com - coins!
> http://www.brooklyn-living.com
>
> Being so tracked is for FARM ANIMALS and and extermination camps,
> but incompatible with living as a free human being. -RI Safir 2013
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150416/7ed46b04/attachment.html 

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 11:07 AM, Ricardo Ribalda Delgado wrote:
> On Thu, Apr 16, 2015 at 4:56 PM, Ruben Safir <ruben@mrbrklyn.com> wrote:
>> I'm trying to find rb_node's structure and I can't find it with ctags or
>> in the http://lxr.linux.no website.
>>
>>
>> How do you search these things out?
> 
> 
> When everything else fails, I use grep
> 
> ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$ git grep " rb_node
> {" include/
> include/linux/rbtree.h:struct rb_node {
> ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$
> 
> 
> 


although in theory that only works if all your defined files are in your
current subtree

I wish I understood what the tags file says.  It is like autoconf.  It
can take a lifetime to master the syntax.

Ruben

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 11:10 AM, Aruna Hewapathirane wrote:
>>> I'm trying to find rb_node's structure and I can't find it with ctags or
>>> in the http://lxr.linux.no website.
> 
> Please use this: Linux 2.6.11 and later <http://lxr.linux.no/linux>
> 


91 responses is not a useful search...

Kernel thread scheduling

[Ricardo Ribalda Delgado]

On Thu, Apr 16, 2015 at 5:12 PM, Ruben Safir <ruben@mrbrklyn.com> wrote:
> On 04/16/2015 11:07 AM, Ricardo Ribalda Delgado wrote:

>>
>> When everything else fails, I use grep
>>
>> ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$ git grep " rb_node
>> {" include/
>> include/linux/rbtree.h:struct rb_node {
>> ricardo at neopili:~/curro/qtec/qt5022/kernel-cesium$
>
> although in theory that only works if all your defined files are in your
> current subtree

"git grep" works recursively

"git -r" also

Kernel thread scheduling

[Jeff Haran]

From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces at kernelnewbies.org] On Behalf Of Mark P
>Sent: Thursday, April 16, 2015 8:12 AM
>To: Ruben Safir
>Cc: nick; kernelnewbies at kernelnewbies.org
>Subject: Re: Kernel thread scheduling
>
>I find that the free electrons LXR has the best search capabilities:
>
>http://lxr.free-electrons.com/source/include/linux/rbtree.h#L35
>-M

Those interested in kernel source browsers might want to check out the code browser available at https://scan.coverity.com.

Coverity does static code analysis and sells a product to do so, but they do regular scans of popular open source projects. Most of their focus is on finding and reporting defects, but the code browser they have created to do so is far beyond anything else I've found out there. Getting to it is a little awkward, you first need to sign up for an account (I got mine for free), then browse to the linux kernel project and select a defect to get into the browser. But once there, click the folder icon at the top left of the code window and select a source file. All of function names, variable names, structure names and structure field names are hyperlinks. Left click on one of them, click the little down arrow and select from the menu to list definitions, references, etc.

You do need to find a reference to the token in question using some other browser like LXR, Coverity's doesn't seem to have a search button for that, but once located the cross-referencing provided is better than what I've seen in other text matching browsers like LXR, cscope, etc. It's particular good when you are trying to understand how a given field of a structure is used. Say you want to find out how a structure field named "lock" in some structure named "foo" is referenced. Find the definition of struct foo, click on the lock field and list references. It will show all the references to struct foo's lock but NOT show the thousands of references to all of the other fields named "lock" in other structures. That is something no other browser does, at least none that I am aware of.

Jeff Haran

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 12:31 PM, Jeff Haran wrote:
>> From: kernelnewbies-bounces at kernelnewbies.org [mailto:kernelnewbies-bounces at kernelnewbies.org] On Behalf Of Mark P
>> Sent: Thursday, April 16, 2015 8:12 AM
>> To: Ruben Safir
>> Cc: nick; kernelnewbies at kernelnewbies.org
>> Subject: Re: Kernel thread scheduling
>>
>> I find that the free electrons LXR has the best search capabilities:
>>
>> http://lxr.free-electrons.com/source/include/linux/rbtree.h#L35
>> -M
> 
> Those interested in kernel source browsers might want to check out the code browser available at https://scan.coverity.com.
> 
> Coverity does static code analysis and sells a product to do so, but they do regular scans of popular open source projects. Most of their focus is on finding and reporting defects, but the code browser they have created to do so is far beyond anything else I've found out there. Getting to it is a little awkward, you first need to sign up for an account (I got mine for free), then browse to the linux kernel project and select a defect to get into the browser. But once there, click the folder icon at the top left of the code window and select a source file. All of function names, variable names, structure names and structure field names are hyperlinks. Left click on one of them, click the little down arrow and select from the menu to list definitions, references, etc.
> 
> You do need to find a reference to the token in question using some other browser like LXR, Coverity's doesn't seem to have a search button for that, but once located the cross-referencing provided is better than what I've seen in other text matching browsers like LXR, cscope, etc. It's particular good when you are trying to understand how a given field of a structure is used. Say you want to find out how a structure field named "lock" in some structure named "foo" is referenced. Find the definition of struct foo, click on the lock field and list references. It will show all the references to struct foo's lock but NOT show the thousands of references to all of the other fields named "lock" in other structures. That is something no other browser does, at least none that I am aware of.
> 
> Jeff Haran
> 



Is it free software?  Can I download it?
How is it better than grep and sed?
Does it work better than ctags?
How can I get it to ingrate with vim?

First off, if it is not available as free Software, then I won't use it.


> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> 

Kernel thread scheduling

[Jeff Haran]

> -----Original Message-----
> From: kernelnewbies-bounces at kernelnewbies.org [mailto:kernelnewbies-
> bounces at kernelnewbies.org] On Behalf Of Ruben Safir
> Sent: Thursday, April 16, 2015 10:09 AM
> To: kernelnewbies at kernelnewbies.org
> Subject: Re: Kernel thread scheduling
> 
> On 04/16/2015 12:31 PM, Jeff Haran wrote:
> >> From: kernelnewbies-bounces at kernelnewbies.org
> [mailto:kernelnewbies-bounces at kernelnewbies.org] On Behalf Of Mark P
> >> Sent: Thursday, April 16, 2015 8:12 AM
> >> To: Ruben Safir
> >> Cc: nick; kernelnewbies at kernelnewbies.org
> >> Subject: Re: Kernel thread scheduling
> >>
> >> I find that the free electrons LXR has the best search capabilities:
> >>
> >> http://lxr.free-electrons.com/source/include/linux/rbtree.h#L35
> >> -M
> >
> > Those interested in kernel source browsers might want to check out the
> code browser available at https://scan.coverity.com.
> >
> > Coverity does static code analysis and sells a product to do so, but they do
> regular scans of popular open source projects. Most of their focus is on
> finding and reporting defects, but the code browser they have created to do
> so is far beyond anything else I've found out there. Getting to it is a little
> awkward, you first need to sign up for an account (I got mine for free), then
> browse to the linux kernel project and select a defect to get into the
> browser. But once there, click the folder icon at the top left of the code
> window and select a source file. All of function names, variable names,
> structure names and structure field names are hyperlinks. Left click on one of
> them, click the little down arrow and select from the menu to list definitions,
> references, etc.
> >
> > You do need to find a reference to the token in question using some other
> browser like LXR, Coverity's doesn't seem to have a search button for that,
> but once located the cross-referencing provided is better than what I've
> seen in other text matching browsers like LXR, cscope, etc. It's particular
> good when you are trying to understand how a given field of a structure is
> used. Say you want to find out how a structure field named "lock" in some
> structure named "foo" is referenced. Find the definition of struct foo, click on
> the lock field and list references. It will show all the references to struct foo's
> lock but NOT show the thousands of references to all of the other fields
> named "lock" in other structures. That is something no other browser does,
> at least none that I am aware of.
> >
> > Jeff Haran
> >
> 
> 
> 
> Is it free software?  Can I download it?

The web site is free to use. The code that runs it is proprietary.

> How is it better than grep and sed?
> Does it work better than ctags?

Grep, sed, ctags, cscope, etc. can only find references based on text matches. C allows duplicate field names across structure definitions. When using text based browsers to look for references to C structure field names on a large project like linux such searches yield lots of false positives, thousands of them in some cases.  Coverity's browser on the other hand seems to understand C syntax. When looking for references so say struct foo's lock field, it won't show you references to struct bar's lock field. Other tools can't tell the difference.

> How can I get it to ingrate with vim?

You can't so far as I know.
 
> First off, if it is not available as free Software, then I won't use it.

That is of course your choice to make. Others may come to different conclusions. I use www.google.com on a regular basis even though I can't download it.

Jeff Haran

Kernel thread scheduling

[Ruben Safir]

>  
> > First off, if it is not available as free Software, then I won't use it.
> 
> That is of course your choice to make. 


And it is the correct decision to make.  People who make another
decision would not care about the damage they do to themselves and
others by promoting the use of a proprietary system to access and learn
about a free software project devised to intentionally do just the
opposite and to give people access to code and ownership of their
digital system.

Why would someone sell that freedom for a minor convenience. 

This has been hashed.  If everyone depended on this Software as a
Service then you would have a nice walled garden to the Kernel Code.
That would be a nice for you __maybe__, but it would be bad for the rest
of us.

https://www.gnu.org/philosophy/who-does-that-server-really-serve.html 

>> Others may come to different conclusions. I use www.google.com on a regular basis even though I can't download it.
> 

That is not even a good analogy.  But for what it is worth, SOS is
killing free software on the google platform altogether and it is a
tragedy...  it is killing Free Software and choking the future of
development.





> Jeff Haran
> 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com 

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive 
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com 

Being so tracked is for FARM ANIMALS and and extermination camps, 
but incompatible with living as a free human being. -RI Safir 2013

Kernel thread scheduling

[John de la Garza]

On Thu, Apr 16, 2015 at 10:56:46AM -0400, Ruben Safir wrote:
> I'm trying to find rb_node's structure and I can't find it with ctags or
> in the http://lxr.linux.no website.
> 
> 
> How do you search these things out?

I run:
	make ctags

run vim

type:
	:ts rb_node

then I scanned the list for things that are from include/linux

and found this
 20 F   s    rb_node           include/linux/rbtree.h
               struct rb_node {

I select 20 and it takes me to include/linux/rbtree.h

and puts me at the line containing this:

	struct rb_node {
		unsigned long  __rb_parent_color;
		struct rb_node *rb_right;
		struct rb_node *rb_left;
	} __attribute__((aligned(sizeof(long))));
	  /* The alignment might seem pointless, but allegedly CRIS needs it */

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 02:32 PM, John de la Garza wrote:
> On Thu, Apr 16, 2015 at 10:56:46AM -0400, Ruben Safir wrote:
>> I'm trying to find rb_node's structure and I can't find it with ctags or
>> in the http://lxr.linux.no website.
>>

THANK YOU - This is for my archive of useful Linux Hints!!
>>
>> How do you search these things out?
> 
> I run:
> 	make ctags
> 
> run vim
> 
> type:
> 	:ts rb_node
> 
> then I scanned the list for things that are from include/linux
> 
> and found this
>  20 F   s    rb_node           include/linux/rbtree.h
>                struct rb_node {
> 
> I select 20 and it takes me to include/linux/rbtree.h
> 
> and puts me at the line containing this:
> 
> 	struct rb_node {
> 		unsigned long  __rb_parent_color;
> 		struct rb_node *rb_right;
> 		struct rb_node *rb_left;
> 	} __attribute__((aligned(sizeof(long))));
> 	  /* The alignment might seem pointless, but allegedly CRIS needs it */
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> 

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 02:32 PM, John de la Garza wrote:
> On Thu, Apr 16, 2015 at 10:56:46AM -0400, Ruben Safir wrote:
>> I'm trying to find rb_node's structure and I can't find it with ctags or
>> in the http://lxr.linux.no website.
>>
>>
>> How do you search these things out?
> 


That is truly beautiful.  Is that documented anywhere in plain english?

Ruben

> I run:
> 	make ctags
> 
> run vim
> 
> type:
> 	:ts rb_node
> 
> then I scanned the list for things that are from include/linux
> 
> and found this
>  20 F   s    rb_node           include/linux/rbtree.h
>                struct rb_node {
> 
> I select 20 and it takes me to include/linux/rbtree.h
> 
> and puts me at the line containing this:
> 
> 	struct rb_node {
> 		unsigned long  __rb_parent_color;
> 		struct rb_node *rb_right;
> 		struct rb_node *rb_left;
> 	} __attribute__((aligned(sizeof(long))));
> 	  /* The alignment might seem pointless, but allegedly CRIS needs it */
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> 

Kernel thread scheduling

[Valdis.Kletnieks at vt.edu]

On Thu, 16 Apr 2015 14:28:59 -0400, Ruben Safir said:

> >> Others may come to different conclusions. I use www.google.com on a regular basis even though I can't download it.

> That is not even a good analogy.

We're comparing closed Coverty-as-a-service with closed Google-as-a-service.
Seems like a good analogy to me.

I'm failing to see the moral difference between using Google SaaS to look
to see if anybody else has reported a particular kernel error and using Coverty
SaaS to examine kernel code.

As you yourself said:

> If everyone depended on this Software as a Service then you would have a
> nice walled garden to the Kernel Code.

Why do you object to Coverty when Google is almost certainly more depended
on by kernel developers?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150416/a7454bf1/attachment-0001.bin 

Kernel thread scheduling

[Silvan Jegen]

On Thu, Apr 16, 2015 at 02:42:07PM -0400, Ruben Safir wrote:
> On 04/16/2015 02:32 PM, John de la Garza wrote:
> > On Thu, Apr 16, 2015 at 10:56:46AM -0400, Ruben Safir wrote:
> >> I'm trying to find rb_node's structure and I can't find it with ctags or
> >> in the http://lxr.linux.no website.
> >>
> >>
> >> How do you search these things out?
> > 
> 
> 
> That is truly beautiful.  Is that documented anywhere in plain english?

If you mean the tag search functionality, yes. See below.


> > I run:
> > 	make ctags

Documentation for this you can find by running "make help" at the root
of the Kernel source tree.


> > run vim
> > 
> > type:
> > 	:ts rb_node

This functionality is documented in Vim itself. To access the relevant
documentation just run

:help tags

There is also the "cscope" program that can give you information about
which functions call and are being called by which other functions. It
has its own man page.


Cheers,

Silvan

> > 
> > then I scanned the list for things that are from include/linux
> > 
> > and found this
> >  20 F   s    rb_node           include/linux/rbtree.h
> >                struct rb_node {
> > 
> > I select 20 and it takes me to include/linux/rbtree.h
> > 
> > and puts me at the line containing this:
> > 
> > 	struct rb_node {
> > 		unsigned long  __rb_parent_color;
> > 		struct rb_node *rb_right;
> > 		struct rb_node *rb_left;
> > 	} __attribute__((aligned(sizeof(long))));
> > 	  /* The alignment might seem pointless, but allegedly CRIS needs it */
> > 
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > 
> > 
> 
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Kernel thread scheduling

[Jeff Haran]

> -----Original Message-----
> From: Valdis.Kletnieks at vt.edu [mailto:Valdis.Kletnieks at vt.edu]
> Sent: Thursday, April 16, 2015 11:48 AM
> To: Ruben Safir
> Cc: Jeff Haran; kernelnewbies at kernelnewbies.org
> Subject: Re: Kernel thread scheduling
> 
> On Thu, 16 Apr 2015 14:28:59 -0400, Ruben Safir said:
> 
> > >> Others may come to different conclusions. I use www.google.com on a
> regular basis even though I can't download it.
> 
> > That is not even a good analogy.
> 
> We're comparing closed Coverty-as-a-service with closed Google-as-a-
> service.
> Seems like a good analogy to me.
> 
> I'm failing to see the moral difference between using Google SaaS to look to
> see if anybody else has reported a particular kernel error and using Coverty
> SaaS to examine kernel code.
> 
> As you yourself said:
> 
> > If everyone depended on this Software as a Service then you would have
> > a nice walled garden to the Kernel Code.
> 
> Why do you object to Coverty when Google is almost certainly more
> depended on by kernel developers?

Not only that, but demonizing Coverity over this is quite unfair. They provide for free to the world the results of their static code analysis of many open source projects, Linux included. Granted, Coverity's system isn't perfect, it does generate false positives but I know from personally quite embarrassing experience that it does find real bugs in code that I've written and looked at hundreds of times that I didn't see and its pointing out to anybody who would care to look thousands of potential kernel bugs. They also have this posterior-kicking code browser that's better than all of the free competition that I know of. If somebody knows of any better, please let me know.

I often read emails on this list from people looking to help and get started in kernel development. Well, here's a good place to start. Submit patches to fix some of those Coverity identified kernel bugs. Some of them will be false positives, but some of them will be the real thing.

And just for the record, I have no financial interest in Coverity or its parent company. I've been a user of their system at a couple of companies I've worked for now, but that and my usage of their free service is the only connection I have with them.

Jeff Haran

Kernel thread scheduling

[Ruben Safir]

On 04/16/2015 02:47 PM, Valdis.Kletnieks at vt.edu wrote:
> We're comparing closed Coverty-as-a-service with closed Google-as-a-service.
> Seems like a good analogy to me.


well it isn't.  It is not demonizing them to state the facts.  First of
all, search engines do something you can't do with software, which is
crawl the internet.  The correct analogy is what google is doing with
android, which indeed is repulsive because it is inhibiting access and
development on the platform, and then worse than that, claiming that it
is all "open" when really it is not.



http://arstechnica.com/gadgets/2013/10/googles-iron-grip-on-android-controlling-open-source-by-any-means-necessary/


I have the source code for the kernel here.  If Coventry would be
interested in selling me a free software of their application, I might
be interested.  But I'm not interested in putting kernel tools behind
slavewalls.  That is exactly the opposite of what I want to do and
defies my very Raison E'Stat for studying the Linux Kernel....

nono.  If you find this Demonizing, it is not I who does this, it is
facts stated forthrightly and dispassionately.  Someone who uses closed
proprietary  software as a service to access the Linux Kernel Source is
cutting off the very air to free software development by supporting
non-free tools that compete with free tools in learning about the source
code.  Using free software tools is not a technological decision.  It is
a political decision (as all decisions ultimately are).  It is a
decision to exercise freedom over a __lack__ of freedom.

Aside from that, the question of google is irrelevant to this
conversation and is just a distraction.  Tow wrongs don't make a right.
 A company that has a search engine that uses published free software
would be much more desirable than one that doesn't do that.  No one
wants to penalize anyone for selling software services.  But that is not
reason to abandon free software tools for the studying of the kernel in
order to adopt a proprietary scheme of secret software sold as a
service.  That would just be irrational.

Ruben

Kernel thread scheduling

[Silvan Jegen]

Am 2015-04-16 23:41, schrieb Jeff Haran:
> I often read emails on this list from people looking to help and get
> started in kernel development. Well, here's a good place to start.
> Submit patches to fix some of those Coverity identified kernel bugs.
> Some of them will be false positives, but some of them will be the
> real thing.

I totally agree.

If there are people that do not want to use non-free software for static 
code analysis (even as a service) they can instead go for smatch[0]. 
Smatch is part of the 0-day Kernel test infrastructure[1] of Intel and 
is open source software.

I've written a short blog post on how to use smatch here[2]. Any 
feedback and/or criticism would be appreciated.


Cheers,

Silvan

[0] http://smatch.sourceforge.net/
[1] https://lists.01.org/mailman/listinfo/kbuild
[2] http://sillymon.ch/posts/smatchusage.html

Kernel thread scheduling

[Ruben Safir]

> 
> And just for the record, I have no financial interest in Coverity or its parent company. I've been a user of their system at a couple of companies I've worked for now, but that and my usage of their free service is the only connection I have with them.
> 
> Jeff Haran


Jeff, FWIW, I have no objection to a financial interest.  I want you to
make as much many as you can

This is not an issue of money.

Ruben

-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com 

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive 
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com 

Being so tracked is for FARM ANIMALS and and extermination camps, 
but incompatible with living as a free human being. -RI Safir 2013

Kernel thread scheduling

[Ruben Safir]

looking at it know because I have no clue what that macro is.

It is not listed as possible in kings


On Sun, Apr 12, 2015 at 04:36:53AM +0000, Mohammed Ghriga wrote:
> In addition to the suggestions that were offered, I recommend you try reading Chapter 16: https://www.google.com/search?tbo=p&tbm=bks&q=isbn:0596554672 (pages: 267-272). 
> 
> 
> -----Original Message-----
> From: nick [mailto:xerofoify at gmail.com] 
> Sent: Sunday, April 12, 2015 12:16 AM
> To: Ruben Safir; kernelnewbies at kernelnewbies.org; Mohammed Ghriga
> Subject: Re: Kernel thread scheduling
> 
> 
> 
> On 2015-04-11 11:02 PM, Ruben Safir wrote:
> > On 04/11/2015 10:21 PM, Ruben Safir wrote:
> >> On 04/10/2015 09:09 AM, nick wrote:
> >>>
> >>>
> >>> On 2015-04-09 11:37 PM, Ruben Safir wrote:
> >>>> On 04/09/2015 10:52 PM, nick wrote:
> >>>>> Before asking questions again like this please look into either 
> >>>>> using lxr or ctags to navigate the kernel tree for answers as can 
> >>>>> be faster then waiting for me or someone else to respond.
> >>>>
> >>>>
> >>>> well, I reading the text is ctags aren't much value there.
> >>>>
> >>> Ctags is useful for searching the code, which is why I am recommending it.
> >>> Nick
> >>
> >> I have it built into gvim, but you can't use it from a textbook.  I'm 
> >> finding it is not as useful as it could be for the kernel code.  
> >> There are stacks of tags to get around.  Another 2 days to learn to 
> >> get around tags in vi is not in the agenda right now.  It is the tool 
> >> I have so I'll have to live with it right now.
> >>
> >> I also have a question that is not obvious from the code I'm looking at.
> >>  I'm not sure how these structs are attached together.  Or more 
> >> specifically, I'm not sure how pulling the correct sched_entity gets 
> >> one the coresponding task_entity
> >>
> >> You have
> >> struct task_struct with a
> >> 	struct sched_entity
> >>
> >> struct sched_enitities are nodes in the RB tree
> >> 	which are a "container" for "struct rb_node run_node".
> >>
> >> So a look at sched_entity ... is in ../linux/sched.h
> >>
> >> 1161 struct sched_entity {
> >> 1162    struct load_weight   load;    /* for load-balancing */
> >> 1163    struct rb_node    run_node;
> >> 1164    struct list_head  group_node;
> >> 1165    unsigned int      on_rq;
> >> 1166
> >> 1167    u64         exec_start;
> >> 1168    u64         sum_exec_runtime;
> >> 1169    u64         vruntime;
> >> 1170    u64         prev_sum_exec_runtime;
> >> 1171
> >> 1172    u64         nr_migrations;
> >> 1173
> >> 1174 #ifdef CONFIG_SCHEDSTATS
> >> 1175    struct sched_statistics statistics;
> >> 1176 #endif
> >> 1177
> >> 1178 #ifdef CONFIG_FAIR_GROUP_SCHED
> >> 1179    int         depth;
> >> 1180    struct sched_entity  *parent;
> >> 1181    /* rq on which this entity is (to be) queued: */
> >> 1182    struct cfs_rq     *cfs_rq;
> >> 1183    /* rq "owned" by this entity/group: */
> >> 1184    struct cfs_rq     *my_q;
> >> 1185 #endif
> >> 1186
> >> 1187 #ifdef CONFIG_SMP
> >> 1188    /* Per-entity load-tracking */
> >> 1189    struct sched_avg  avg;
> >> 1190 #endif
> >> 1191 };
> >>
> >> I see no means of referencing a specific task from this struct that 
> >> forms the node.  So when you pull the node with the smallest vruntime 
> >> from the left most postion of the RB tree, by calling 
> >> pick_next_task(),
> >>
> >>
> >> static struct sched_entity *__pick_next_entity(struct sched_entity 
> >> *se) {
> >> 	struct rb_node *next = rb_next(&se->run_node);
> This finds the next node in the red black tree for  sched_enities.
> Basically rb_next finds the next node in the tree. The argument is the rb_node structure embedded in the structure using a red black tree.
> >>
> >> 	if (!next)
> >> 		return NULL;
> >>
> If there is no runnable task return NULL and pick_next_task will run the idle_task for this cpu.
> >> 	return rb_entry(next, struct sched_entity, run_node); }
> >>
> >>
> >> how do we know what task we are attached to?
> >>
> Also try to read Chapter 6 of Linux Kernel Development as if have read that
> chapter understanding how red black trees and other data structures work in
> kernel code would make more sense.
> Nick
> >> Ruben
> >>
> >>
> > 
> > I'm still loss on how we know which taks_struct is being used but as a
> > side note, I found this also very puzzling
> > 
> > return rb_entry(next, struct sched_entity, run_node);
> > With help I ran it down to this:
> > 
> > http://lxr.free-electrons.com/source/include/linux/rbtree.h#L50
> > 
> > #define rb_entry(ptr, type, member) container_of(ptr, type, member)
> > 
> > which leads me to yet another macro
> > 
> > 798 #define container_of(ptr, type, member) ({                      \
> > 799         const typeof( ((type *)0)->member ) *__mptr = (ptr);    \
> > 800         (type *)( (char *)__mptr - offsetof(type,member) );})
> > 
> > 
> > This is a use of macros I'd never seen before up close.  If anyone could
> > help me understand it, I'd appreciate it.
> > 
> > Ruben
> >>
> >>
> >>>> _______________________________________________
> >>>> Kernelnewbies mailing list
> >>>> Kernelnewbies at kernelnewbies.org
> >>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>>
> >>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> Kernelnewbies mailing list
> >> Kernelnewbies at kernelnewbies.org
> >> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>
> >>
> > 
> > 
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > 

-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com 

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive 
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com 

Being so tracked is for FARM ANIMALS and and extermination camps, 
but incompatible with living as a free human being. -RI Safir 2013

Kernel thread scheduling

[Ruben Safir]

That is written by my friend GREG!!

I know Greg.  I've had many beers with Greg.

Ruben


On Sun, Apr 12, 2015 at 04:36:53AM +0000, Mohammed Ghriga wrote:
> In addition to the suggestions that were offered, I recommend you try reading Chapter 16: https://www.google.com/search?tbo=p&tbm=bks&q=isbn:0596554672 (pages: 267-272). 
> 
> 
> -----Original Message-----
> From: nick [mailto:xerofoify at gmail.com] 
> Sent: Sunday, April 12, 2015 12:16 AM
> To: Ruben Safir; kernelnewbies at kernelnewbies.org; Mohammed Ghriga
> Subject: Re: Kernel thread scheduling
> 
> 
> 
> On 2015-04-11 11:02 PM, Ruben Safir wrote:
> > On 04/11/2015 10:21 PM, Ruben Safir wrote:
> >> On 04/10/2015 09:09 AM, nick wrote:
> >>>
> >>>
> >>> On 2015-04-09 11:37 PM, Ruben Safir wrote:
> >>>> On 04/09/2015 10:52 PM, nick wrote:
> >>>>> Before asking questions again like this please look into either 
> >>>>> using lxr or ctags to navigate the kernel tree for answers as can 
> >>>>> be faster then waiting for me or someone else to respond.
> >>>>
> >>>>
> >>>> well, I reading the text is ctags aren't much value there.
> >>>>
> >>> Ctags is useful for searching the code, which is why I am recommending it.
> >>> Nick
> >>
> >> I have it built into gvim, but you can't use it from a textbook.  I'm 
> >> finding it is not as useful as it could be for the kernel code.  
> >> There are stacks of tags to get around.  Another 2 days to learn to 
> >> get around tags in vi is not in the agenda right now.  It is the tool 
> >> I have so I'll have to live with it right now.
> >>
> >> I also have a question that is not obvious from the code I'm looking at.
> >>  I'm not sure how these structs are attached together.  Or more 
> >> specifically, I'm not sure how pulling the correct sched_entity gets 
> >> one the coresponding task_entity
> >>
> >> You have
> >> struct task_struct with a
> >> 	struct sched_entity
> >>
> >> struct sched_enitities are nodes in the RB tree
> >> 	which are a "container" for "struct rb_node run_node".
> >>
> >> So a look at sched_entity ... is in ../linux/sched.h
> >>
> >> 1161 struct sched_entity {
> >> 1162    struct load_weight   load;    /* for load-balancing */
> >> 1163    struct rb_node    run_node;
> >> 1164    struct list_head  group_node;
> >> 1165    unsigned int      on_rq;
> >> 1166
> >> 1167    u64         exec_start;
> >> 1168    u64         sum_exec_runtime;
> >> 1169    u64         vruntime;
> >> 1170    u64         prev_sum_exec_runtime;
> >> 1171
> >> 1172    u64         nr_migrations;
> >> 1173
> >> 1174 #ifdef CONFIG_SCHEDSTATS
> >> 1175    struct sched_statistics statistics;
> >> 1176 #endif
> >> 1177
> >> 1178 #ifdef CONFIG_FAIR_GROUP_SCHED
> >> 1179    int         depth;
> >> 1180    struct sched_entity  *parent;
> >> 1181    /* rq on which this entity is (to be) queued: */
> >> 1182    struct cfs_rq     *cfs_rq;
> >> 1183    /* rq "owned" by this entity/group: */
> >> 1184    struct cfs_rq     *my_q;
> >> 1185 #endif
> >> 1186
> >> 1187 #ifdef CONFIG_SMP
> >> 1188    /* Per-entity load-tracking */
> >> 1189    struct sched_avg  avg;
> >> 1190 #endif
> >> 1191 };
> >>
> >> I see no means of referencing a specific task from this struct that 
> >> forms the node.  So when you pull the node with the smallest vruntime 
> >> from the left most postion of the RB tree, by calling 
> >> pick_next_task(),
> >>
> >>
> >> static struct sched_entity *__pick_next_entity(struct sched_entity 
> >> *se) {
> >> 	struct rb_node *next = rb_next(&se->run_node);
> This finds the next node in the red black tree for  sched_enities.
> Basically rb_next finds the next node in the tree. The argument is the rb_node structure embedded in the structure using a red black tree.
> >>
> >> 	if (!next)
> >> 		return NULL;
> >>
> If there is no runnable task return NULL and pick_next_task will run the idle_task for this cpu.
> >> 	return rb_entry(next, struct sched_entity, run_node); }
> >>
> >>
> >> how do we know what task we are attached to?
> >>
> Also try to read Chapter 6 of Linux Kernel Development as if have read that
> chapter understanding how red black trees and other data structures work in
> kernel code would make more sense.
> Nick
> >> Ruben
> >>
> >>
> > 
> > I'm still loss on how we know which taks_struct is being used but as a
> > side note, I found this also very puzzling
> > 
> > return rb_entry(next, struct sched_entity, run_node);
> > With help I ran it down to this:
> > 
> > http://lxr.free-electrons.com/source/include/linux/rbtree.h#L50
> > 
> > #define rb_entry(ptr, type, member) container_of(ptr, type, member)
> > 
> > which leads me to yet another macro
> > 
> > 798 #define container_of(ptr, type, member) ({                      \
> > 799         const typeof( ((type *)0)->member ) *__mptr = (ptr);    \
> > 800         (type *)( (char *)__mptr - offsetof(type,member) );})
> > 
> > 
> > This is a use of macros I'd never seen before up close.  If anyone could
> > help me understand it, I'd appreciate it.
> > 
> > Ruben
> >>
> >>
> >>>> _______________________________________________
> >>>> Kernelnewbies mailing list
> >>>> Kernelnewbies at kernelnewbies.org
> >>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>>
> >>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> Kernelnewbies mailing list
> >> Kernelnewbies at kernelnewbies.org
> >> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>
> >>
> > 
> > 
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > 

-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com 

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive 
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com 

Being so tracked is for FARM ANIMALS and and extermination camps, 
but incompatible with living as a free human being. -RI Safir 2013