kernelci.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* can't access kernelci results
@ 2021-04-05 19:02 Todd Kjos
  2021-04-05 20:39 ` Guenter Roeck
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-04-05 19:02 UTC (permalink / raw)
  To: kernelci, Kevin Hilman

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

I get errors when attempting to access
https://linux.kernelci.org/job/android/. Is this a known issue? Any ETA for
a fix?

-Todd

[-- Attachment #2: Type: text/html, Size: 231 bytes --]

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-04-05 19:02 can't access kernelci results Todd Kjos
@ 2021-04-05 20:39 ` Guenter Roeck
  2021-04-05 20:58   ` Guillaume Tucker
  0 siblings, 1 reply; 25+ messages in thread
From: Guenter Roeck @ 2021-04-05 20:39 UTC (permalink / raw)
  To: kernelci, Todd Kjos; +Cc: Kevin Hilman

On Mon, Apr 5, 2021 at 12:30 PM Todd Kjos via groups.io
<tkjos=google.com@groups.io> wrote:
>
> I get errors when attempting to access https://linux.kernelci.org/job/android/. Is this a known issue? Any ETA for a fix?
>
Looks like the problem is not limited to android. I get error 500 for
jobs and build results.

Guenter

> -Todd
> 

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-04-05 20:39 ` Guenter Roeck
@ 2021-04-05 20:58   ` Guillaume Tucker
  2021-04-05 23:14     ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Guillaume Tucker @ 2021-04-05 20:58 UTC (permalink / raw)
  To: kernelci, groeck, Todd Kjos; +Cc: Kevin Hilman

On 05/04/2021 21:39, Guenter Roeck via groups.io wrote:
> On Mon, Apr 5, 2021 at 12:30 PM Todd Kjos via groups.io
> <tkjos=google.com@groups.io> wrote:
>>
>> I get errors when attempting to access https://linux.kernelci.org/job/android/. Is this a known issue? Any ETA for a fix?
>>
> Looks like the problem is not limited to android. I get error 500 for
> jobs and build results.

That was another case of the mongod service stopping, it's pretty
rare but has happened a couple of times before.  Things are now
back to normal, but we've missed a few results while it was down.
Sorry for the disruption.

We've deployed a monitoring tool on some servers but not yet on
this one, it should happen soon so this kind of issues would be
detected much quicker.  We've also started looking into using a
Mongo database in Azure rather than a manual install to reduce
maintenance work.

Best wishes,
Guillaume

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-04-05 20:58   ` Guillaume Tucker
@ 2021-04-05 23:14     ` Todd Kjos
  2021-06-21 19:59       ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-04-05 23:14 UTC (permalink / raw)
  To: Guillaume Tucker; +Cc: kernelci, Guenter Roeck, Kevin Hilman

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

Working again. Thanks Guillaume.

On Mon, Apr 5, 2021 at 1:58 PM Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:

> On 05/04/2021 21:39, Guenter Roeck via groups.io wrote:
> > On Mon, Apr 5, 2021 at 12:30 PM Todd Kjos via groups.io
> > <tkjos=google.com@groups.io> wrote:
> >>
> >> I get errors when attempting to access
> https://linux.kernelci.org/job/android/. Is this a known issue? Any ETA
> for a fix?
> >>
> > Looks like the problem is not limited to android. I get error 500 for
> > jobs and build results.
>
> That was another case of the mongod service stopping, it's pretty
> rare but has happened a couple of times before.  Things are now
> back to normal, but we've missed a few results while it was down.
> Sorry for the disruption.
>
> We've deployed a monitoring tool on some servers but not yet on
> this one, it should happen soon so this kind of issues would be
> detected much quicker.  We've also started looking into using a
> Mongo database in Azure rather than a manual install to reduce
> maintenance work.
>
> Best wishes,
> Guillaume
>

[-- Attachment #2: Type: text/html, Size: 1763 bytes --]

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-04-05 23:14     ` Todd Kjos
@ 2021-06-21 19:59       ` Todd Kjos
  2021-06-22  9:43         ` Michał Gałka
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-06-21 19:59 UTC (permalink / raw)
  To: Guillaume Tucker; +Cc: kernelci, Guenter Roeck, Kevin Hilman

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

I'm having this issue again (can't access kernelci results at
https://linux.kernelci.org/job/android/). Is the mongod service borked
again?

On Mon, Apr 5, 2021 at 4:14 PM Todd Kjos <tkjos@google.com> wrote:

> Working again. Thanks Guillaume.
>
> On Mon, Apr 5, 2021 at 1:58 PM Guillaume Tucker <
> guillaume.tucker@collabora.com> wrote:
>
>> On 05/04/2021 21:39, Guenter Roeck via groups.io wrote:
>> > On Mon, Apr 5, 2021 at 12:30 PM Todd Kjos via groups.io
>> > <tkjos=google.com@groups.io> wrote:
>> >>
>> >> I get errors when attempting to access
>> https://linux.kernelci.org/job/android/. Is this a known issue? Any ETA
>> for a fix?
>> >>
>> > Looks like the problem is not limited to android. I get error 500 for
>> > jobs and build results.
>>
>> That was another case of the mongod service stopping, it's pretty
>> rare but has happened a couple of times before.  Things are now
>> back to normal, but we've missed a few results while it was down.
>> Sorry for the disruption.
>>
>> We've deployed a monitoring tool on some servers but not yet on
>> this one, it should happen soon so this kind of issues would be
>> detected much quicker.  We've also started looking into using a
>> Mongo database in Azure rather than a manual install to reduce
>> maintenance work.
>>
>> Best wishes,
>> Guillaume
>>
>

[-- Attachment #2: Type: text/html, Size: 2327 bytes --]

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-06-21 19:59       ` Todd Kjos
@ 2021-06-22  9:43         ` Michał Gałka
  2021-06-22 15:07           ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Michał Gałka @ 2021-06-22  9:43 UTC (permalink / raw)
  To: kernelci, tkjos, Guillaume Tucker; +Cc: Guenter Roeck, Kevin Hilman

On 21.06.2021 21:59, Todd Kjos via groups.io wrote:
> I'm having this issue again (can't access kernelci results
> at https://linux.kernelci.org/job/android/
> <https://linux.kernelci.org/job/android/>). Is the mongod service borked
> again?
> 

Yes, there was an issue with mongod service again. It's working now,
celery processed pending tasks and everything seems responsive.

Thanks,
Michał

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-06-22  9:43         ` Michał Gałka
@ 2021-06-22 15:07           ` Todd Kjos
  2021-06-22 15:16             ` Guillaume Tucker
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-06-22 15:07 UTC (permalink / raw)
  To: Michał Gałka
  Cc: kernelci, Guillaume Tucker, Guenter Roeck, Kevin Hilman

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

Hmm, it's still not working for me -- did it break again?

On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com>
wrote:

> On 21.06.2021 21:59, Todd Kjos via groups.io wrote:
> > I'm having this issue again (can't access kernelci results
> > at https://linux.kernelci.org/job/android/
> > <https://linux.kernelci.org/job/android/>). Is the mongod service borked
> > again?
> >
>
> Yes, there was an issue with mongod service again. It's working now,
> celery processed pending tasks and everything seems responsive.
>
> Thanks,
> Michał
>

[-- Attachment #2: Type: text/html, Size: 1162 bytes --]

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-06-22 15:07           ` Todd Kjos
@ 2021-06-22 15:16             ` Guillaume Tucker
  2021-09-02 15:58               ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Guillaume Tucker @ 2021-06-22 15:16 UTC (permalink / raw)
  To: Todd Kjos, Michał Gałka; +Cc: kernelci, Guenter Roeck, Kevin Hilman

On 22/06/2021 16:07, Todd Kjos wrote:
> Hmm, it's still not working for me -- did it break again?

Yes there was an OOM issue:

[4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
[4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

We're looking into moving the Mongo DB instance to the cloud in
Azure.  Hopefully that will save us fixing this kind of
maintenance issues.  Meanwhile, we should take a look and see if
there's some config option to limit the amount of RAM used by
Mongo DB to work around the problem.

Best wishes,
Guillaume


> On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
> 
>     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> wrote:
>     > I'm having this issue again (can't access kernelci results
>     > at https://linux.kernelci.org/job/android/
>     > <https://linux.kernelci.org/job/android/>). Is the mongod service borked
>     > again?
>     >
> 
>     Yes, there was an issue with mongod service again. It's working now,
>     celery processed pending tasks and everything seems responsive.
> 
>     Thanks,
>     Michał
> 


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-06-22 15:16             ` Guillaume Tucker
@ 2021-09-02 15:58               ` Todd Kjos
  2021-09-02 19:28                 ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-09-02 15:58 UTC (permalink / raw)
  To: Guillaume Tucker
  Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

I'm getting errors again when attempting to access
https://linux.kernelci.org/job/android/. mongod issues again?

"Error while loading data from the server (error code: 500). Please
contact the website administrator"

On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:
>
> On 22/06/2021 16:07, Todd Kjos wrote:
> > Hmm, it's still not working for me -- did it break again?
>
> Yes there was an OOM issue:
>
> [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>
> We're looking into moving the Mongo DB instance to the cloud in
> Azure.  Hopefully that will save us fixing this kind of
> maintenance issues.  Meanwhile, we should take a look and see if
> there's some config option to limit the amount of RAM used by
> Mongo DB to work around the problem.
>
> Best wishes,
> Guillaume
>
>
> > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
> >
> >     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> wrote:
> >     > I'm having this issue again (can't access kernelci results
> >     > at https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>
> >     > <https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>>). Is the mongod service borked
> >     > again?
> >     >
> >
> >     Yes, there was an issue with mongod service again. It's working now,
> >     celery processed pending tasks and everything seems responsive.
> >
> >     Thanks,
> >     Michał
> >
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-09-02 15:58               ` Todd Kjos
@ 2021-09-02 19:28                 ` Todd Kjos
  2021-11-30 16:35                   ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-09-02 19:28 UTC (permalink / raw)
  To: Guillaume Tucker
  Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

It's working again. Thanks.

On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
>
> I'm getting errors again when attempting to access
> https://linux.kernelci.org/job/android/. mongod issues again?
>
> "Error while loading data from the server (error code: 500). Please
> contact the website administrator"
>
> On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> <guillaume.tucker@collabora.com> wrote:
> >
> > On 22/06/2021 16:07, Todd Kjos wrote:
> > > Hmm, it's still not working for me -- did it break again?
> >
> > Yes there was an OOM issue:
> >
> > [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> >
> > We're looking into moving the Mongo DB instance to the cloud in
> > Azure.  Hopefully that will save us fixing this kind of
> > maintenance issues.  Meanwhile, we should take a look and see if
> > there's some config option to limit the amount of RAM used by
> > Mongo DB to work around the problem.
> >
> > Best wishes,
> > Guillaume
> >
> >
> > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
> > >
> > >     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> wrote:
> > >     > I'm having this issue again (can't access kernelci results
> > >     > at https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>
> > >     > <https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>>). Is the mongod service borked
> > >     > again?
> > >     >
> > >
> > >     Yes, there was an issue with mongod service again. It's working now,
> > >     celery processed pending tasks and everything seems responsive.
> > >
> > >     Thanks,
> > >     Michał
> > >
> >

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-09-02 19:28                 ` Todd Kjos
@ 2021-11-30 16:35                   ` Todd Kjos
  2021-11-30 16:54                     ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-11-30 16:35 UTC (permalink / raw)
  To: Guillaume Tucker
  Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

I can't load the data from https://linux.kernelci.org/job/android/
("Error loading build data."). Is mongod "borked" again?

On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
>
> It's working again. Thanks.
>
> On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
> >
> > I'm getting errors again when attempting to access
> > https://linux.kernelci.org/job/android/. mongod issues again?
> >
> > "Error while loading data from the server (error code: 500). Please
> > contact the website administrator"
> >
> > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > <guillaume.tucker@collabora.com> wrote:
> > >
> > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > Hmm, it's still not working for me -- did it break again?
> > >
> > > Yes there was an OOM issue:
> > >
> > > [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > >
> > > We're looking into moving the Mongo DB instance to the cloud in
> > > Azure.  Hopefully that will save us fixing this kind of
> > > maintenance issues.  Meanwhile, we should take a look and see if
> > > there's some config option to limit the amount of RAM used by
> > > Mongo DB to work around the problem.
> > >
> > > Best wishes,
> > > Guillaume
> > >
> > >
> > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
> > > >
> > > >     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> wrote:
> > > >     > I'm having this issue again (can't access kernelci results
> > > >     > at https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>
> > > >     > <https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>>). Is the mongod service borked
> > > >     > again?
> > > >     >
> > > >
> > > >     Yes, there was an issue with mongod service again. It's working now,
> > > >     celery processed pending tasks and everything seems responsive.
> > > >
> > > >     Thanks,
> > > >     Michał
> > > >
> > >

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-11-30 16:35                   ` Todd Kjos
@ 2021-11-30 16:54                     ` Todd Kjos
  2021-12-01  9:48                       ` Guillaume Tucker
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2021-11-30 16:54 UTC (permalink / raw)
  To: Guillaume Tucker
  Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

...and now it's working again. Sorry about the noise if the issue was
on my side.

On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com> wrote:
>
> I can't load the data from https://linux.kernelci.org/job/android/
> ("Error loading build data."). Is mongod "borked" again?
>
> On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
> >
> > It's working again. Thanks.
> >
> > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
> > >
> > > I'm getting errors again when attempting to access
> > > https://linux.kernelci.org/job/android/. mongod issues again?
> > >
> > > "Error while loading data from the server (error code: 500). Please
> > > contact the website administrator"
> > >
> > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > <guillaume.tucker@collabora.com> wrote:
> > > >
> > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > Hmm, it's still not working for me -- did it break again?
> > > >
> > > > Yes there was an OOM issue:
> > > >
> > > > [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > >
> > > > We're looking into moving the Mongo DB instance to the cloud in
> > > > Azure.  Hopefully that will save us fixing this kind of
> > > > maintenance issues.  Meanwhile, we should take a look and see if
> > > > there's some config option to limit the amount of RAM used by
> > > > Mongo DB to work around the problem.
> > > >
> > > > Best wishes,
> > > > Guillaume
> > > >
> > > >
> > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
> > > > >
> > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> wrote:
> > > > >     > I'm having this issue again (can't access kernelci results
> > > > >     > at https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>
> > > > >     > <https://linux.kernelci.org/job/android/ <https://linux.kernelci.org/job/android/>>). Is the mongod service borked
> > > > >     > again?
> > > > >     >
> > > > >
> > > > >     Yes, there was an issue with mongod service again. It's working now,
> > > > >     celery processed pending tasks and everything seems responsive.
> > > > >
> > > > >     Thanks,
> > > > >     Michał
> > > > >
> > > >

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-11-30 16:54                     ` Todd Kjos
@ 2021-12-01  9:48                       ` Guillaume Tucker
  2022-03-18 15:47                         ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Guillaume Tucker @ 2021-12-01  9:48 UTC (permalink / raw)
  To: Todd Kjos; +Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

Hi Todd,

On 30/11/2021 16:54, Todd Kjos wrote:
> ...and now it's working again. Sorry about the noise if the issue was
> on my side.

The MongoDB server did crash but it was restarted a few minutes
later.  There's some kind of watchdog to do that automatically,
maybe you were out of luck and checked the website during that
window.  Thanks for reporting the issue in any case.

Best wishes,
Guillaume

> On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com> wrote:
>>
>> I can't load the data from https://linux.kernelci.org/job/android/
>> ("Error loading build data."). Is mongod "borked" again?
>>
>> On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
>>>
>>> It's working again. Thanks.
>>>
>>> On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
>>>>
>>>> I'm getting errors again when attempting to access
>>>> https://linux.kernelci.org/job/android/. mongod issues again?
>>>>
>>>> "Error while loading data from the server (error code: 500). Please
>>>> contact the website administrator"
>>>>
>>>> On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
>>>> <guillaume.tucker@collabora.com> wrote:
>>>>>
>>>>> On 22/06/2021 16:07, Todd Kjos wrote:
>>>>>> Hmm, it's still not working for me -- did it break again?
>>>>>
>>>>> Yes there was an OOM issue:
>>>>>
>>>>> [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
>>>>> [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>>>>>
>>>>> We're looking into moving the Mongo DB instance to the cloud in
>>>>> Azure.  Hopefully that will save us fixing this kind of
>>>>> maintenance issues.  Meanwhile, we should take a look and see if
>>>>> there's some config option to limit the amount of RAM used by
>>>>> Mongo DB to work around the problem.
>>>>>
>>>>> Best wishes,
>>>>> Guillaume
>>>>>
>>>>>
>>>>>> On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
>>>>>>
>>>>>>     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> wrote:
>>>>>>     > I'm having this issue again (can't access kernelci results
>>>>>>     > at https://linux.kernelci.org/job/android/
>>>>>>     > <https://linux.kernelci.org/job/android/>). Is the mongod service borked
>>>>>>     > again?
>>>>>>     >
>>>>>>
>>>>>>     Yes, there was an issue with mongod service again. It's working now,
>>>>>>     celery processed pending tasks and everything seems responsive.
>>>>>>
>>>>>>     Thanks,
>>>>>>     Michał
>>>>>>
>>>>>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2021-12-01  9:48                       ` Guillaume Tucker
@ 2022-03-18 15:47                         ` Todd Kjos
  2022-03-18 17:19                           ` Guillaume Tucker
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2022-03-18 15:47 UTC (permalink / raw)
  To: Guillaume Tucker
  Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

[-- Attachment #1: Type: text/plain, Size: 3102 bytes --]

Is the MongoDB server down again? I don't get results at
https://linux.kernelci.org/job/android/.

On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:

> Hi Todd,
>
> On 30/11/2021 16:54, Todd Kjos wrote:
> > ...and now it's working again. Sorry about the noise if the issue was
> > on my side.
>
> The MongoDB server did crash but it was restarted a few minutes
> later.  There's some kind of watchdog to do that automatically,
> maybe you were out of luck and checked the website during that
> window.  Thanks for reporting the issue in any case.
>
> Best wishes,
> Guillaume
>
> > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com> wrote:
> >>
> >> I can't load the data from https://linux.kernelci.org/job/android/
> >> ("Error loading build data."). Is mongod "borked" again?
> >>
> >> On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
> >>>
> >>> It's working again. Thanks.
> >>>
> >>> On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
> >>>>
> >>>> I'm getting errors again when attempting to access
> >>>> https://linux.kernelci.org/job/android/. mongod issues again?
> >>>>
> >>>> "Error while loading data from the server (error code: 500). Please
> >>>> contact the website administrator"
> >>>>
> >>>> On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> >>>> <guillaume.tucker@collabora.com> wrote:
> >>>>>
> >>>>> On 22/06/2021 16:07, Todd Kjos wrote:
> >>>>>> Hmm, it's still not working for me -- did it break again?
> >>>>>
> >>>>> Yes there was an OOM issue:
> >>>>>
> >>>>> [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB,
> anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> >>>>> [4316412.060511] oom_reaper: reaped process 20620 (mongod), now
> anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> >>>>>
> >>>>> We're looking into moving the Mongo DB instance to the cloud in
> >>>>> Azure.  Hopefully that will save us fixing this kind of
> >>>>> maintenance issues.  Meanwhile, we should take a look and see if
> >>>>> there's some config option to limit the amount of RAM used by
> >>>>> Mongo DB to work around the problem.
> >>>>>
> >>>>> Best wishes,
> >>>>> Guillaume
> >>>>>
> >>>>>
> >>>>>> On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
> michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
> >>>>>>
> >>>>>>     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io>
> wrote:
> >>>>>>     > I'm having this issue again (can't access kernelci results
> >>>>>>     > at https://linux.kernelci.org/job/android/ <
> https://linux.kernelci.org/job/android/>
> >>>>>>     > <https://linux.kernelci.org/job/android/ <
> https://linux.kernelci.org/job/android/>>). Is the mongod service borked
> >>>>>>     > again?
> >>>>>>     >
> >>>>>>
> >>>>>>     Yes, there was an issue with mongod service again. It's working
> now,
> >>>>>>     celery processed pending tasks and everything seems responsive.
> >>>>>>
> >>>>>>     Thanks,
> >>>>>>     Michał
> >>>>>>
> >>>>>
>
>

[-- Attachment #2: Type: text/html, Size: 5507 bytes --]

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-03-18 15:47                         ` Todd Kjos
@ 2022-03-18 17:19                           ` Guillaume Tucker
  2022-03-18 17:26                             ` Guillaume Tucker
  0 siblings, 1 reply; 25+ messages in thread
From: Guillaume Tucker @ 2022-03-18 17:19 UTC (permalink / raw)
  To: Todd Kjos; +Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

On 18/03/2022 15:47, Todd Kjos wrote:
> Is the MongoDB server down again? I don't get results at https://linux.kernelci.org/job/android/.

It's not that, unfortunately there's some persisting issue which seems to have started yesterday. We're investigating now, please bear with us. Thank you for reporting the problem.

Best wishes,
Guillaume

>
> On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <guillaume.tucker@collabora.com> wrote:
>
>     Hi Todd,
>
>     On 30/11/2021 16:54, Todd Kjos wrote:
>     > ...and now it's working again. Sorry about the noise if the issue was
>     > on my side.
>
>     The MongoDB server did crash but it was restarted a few minutes
>     later.  There's some kind of watchdog to do that automatically,
>     maybe you were out of luck and checked the website during that
>     window.  Thanks for reporting the issue in any case.
>
>     Best wishes,
>     Guillaume
>
>     > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com> wrote:
>     >>
>     >> I can't load the data from https://linux.kernelci.org/job/android/
>     >> ("Error loading build data."). Is mongod "borked" again?
>     >>
>     >> On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
>     >>>
>     >>> It's working again. Thanks.
>     >>>
>     >>> On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
>     >>>>
>     >>>> I'm getting errors again when attempting to access
>     >>>> https://linux.kernelci.org/job/android/. mongod issues again?
>     >>>>
>     >>>> "Error while loading data from the server (error code: 500). Please
>     >>>> contact the website administrator"
>     >>>>
>     >>>> On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
>     >>>> <guillaume.tucker@collabora.com> wrote:
>     >>>>>
>     >>>>> On 22/06/2021 16:07, Todd Kjos wrote:
>     >>>>>> Hmm, it's still not working for me -- did it break again?
>     >>>>>
>     >>>>> Yes there was an OOM issue:
>     >>>>>
>     >>>>> [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
>     >>>>> [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>     >>>>>
>     >>>>> We're looking into moving the Mongo DB instance to the cloud in
>     >>>>> Azure.  Hopefully that will save us fixing this kind of
>     >>>>> maintenance issues.  Meanwhile, we should take a look and see if
>     >>>>> there's some config option to limit the amount of RAM used by
>     >>>>> Mongo DB to work around the problem.
>     >>>>>
>     >>>>> Best wishes,
>     >>>>> Guillaume
>     >>>>>
>     >>>>>
>     >>>>>> On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
>     >>>>>>
>     >>>>>>     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> <http://groups.io> wrote:
>     >>>>>>     > I'm having this issue again (can't access kernelci results
>     >>>>>>     > at https://linux.kernelci.org/job/android/
>     >>>>>>     > <https://linux.kernelci.org/job/android/>). Is the mongod service borked
>     >>>>>>     > again?
>     >>>>>>     >
>     >>>>>>
>     >>>>>>     Yes, there was an issue with mongod service again. It's working now,
>     >>>>>>     celery processed pending tasks and everything seems responsive.
>     >>>>>>
>     >>>>>>     Thanks,
>     >>>>>>     Michał
>     >>>>>>
>     >>>>>
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-03-18 17:19                           ` Guillaume Tucker
@ 2022-03-18 17:26                             ` Guillaume Tucker
  2022-09-26 15:29                               ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Guillaume Tucker @ 2022-03-18 17:26 UTC (permalink / raw)
  To: Todd Kjos; +Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

On 18/03/2022 17:19, Guillaume Tucker wrote:
> On 18/03/2022 15:47, Todd Kjos wrote:
>> Is the MongoDB server down again? I don't get results at https://linux.kernelci.org/job/android/.
>
> It's not that, unfortunately there's some persisting issue which seems to have started yesterday. We're investigating now, please bear with us. Thank you for reporting the problem.
>
Alright, there was a problem with background processing tasks on the server. That's resolved now, although a number of kernel builds need to be restarted which is why so many trees appear to be "in progress" on the web dashboard. Things will start picking up again gradually, it might take a few hours to get back to normal. Sorry for the inconvenience.

Best wishes,
Guillaume

>>
>> On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <guillaume.tucker@collabora.com> wrote:
>>
>>     Hi Todd,
>>
>>     On 30/11/2021 16:54, Todd Kjos wrote:
>>     > ...and now it's working again. Sorry about the noise if the issue was
>>     > on my side.
>>
>>     The MongoDB server did crash but it was restarted a few minutes
>>     later.  There's some kind of watchdog to do that automatically,
>>     maybe you were out of luck and checked the website during that
>>     window.  Thanks for reporting the issue in any case.
>>
>>     Best wishes,
>>     Guillaume
>>
>>     > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com> wrote:
>>     >>
>>     >> I can't load the data from https://linux.kernelci.org/job/android/
>>     >> ("Error loading build data."). Is mongod "borked" again?
>>     >>
>>     >> On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
>>     >>>
>>     >>> It's working again. Thanks.
>>     >>>
>>     >>> On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
>>     >>>>
>>     >>>> I'm getting errors again when attempting to access
>>     >>>> https://linux.kernelci.org/job/android/. mongod issues again?
>>     >>>>
>>     >>>> "Error while loading data from the server (error code: 500). Please
>>     >>>> contact the website administrator"
>>     >>>>
>>     >>>> On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
>>     >>>> <guillaume.tucker@collabora.com> wrote:
>>     >>>>>
>>     >>>>> On 22/06/2021 16:07, Todd Kjos wrote:
>>     >>>>>> Hmm, it's still not working for me -- did it break again?
>>     >>>>>
>>     >>>>> Yes there was an OOM issue:
>>     >>>>>
>>     >>>>> [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB, anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
>>     >>>>> [4316412.060511] oom_reaper: reaped process 20620 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>>     >>>>>
>>     >>>>> We're looking into moving the Mongo DB instance to the cloud in
>>     >>>>> Azure.  Hopefully that will save us fixing this kind of
>>     >>>>> maintenance issues.  Meanwhile, we should take a look and see if
>>     >>>>> there's some config option to limit the amount of RAM used by
>>     >>>>> Mongo DB to work around the problem.
>>     >>>>>
>>     >>>>> Best wishes,
>>     >>>>> Guillaume
>>     >>>>>
>>     >>>>>
>>     >>>>>> On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
>>     >>>>>>
>>     >>>>>>     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io> <http://groups.io> wrote:
>>     >>>>>>     > I'm having this issue again (can't access kernelci results
>>     >>>>>>     > at https://linux.kernelci.org/job/android/
>>     >>>>>>     > <https://linux.kernelci.org/job/android/>). Is the mongod service borked
>>     >>>>>>     > again?
>>     >>>>>>     >
>>     >>>>>>
>>     >>>>>>     Yes, there was an issue with mongod service again. It's working now,
>>     >>>>>>     celery processed pending tasks and everything seems responsive.
>>     >>>>>>
>>     >>>>>>     Thanks,
>>     >>>>>>     Michał
>>     >>>>>>
>>     >>>>>
>>
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-03-18 17:26                             ` Guillaume Tucker
@ 2022-09-26 15:29                               ` Todd Kjos
  2022-09-26 15:35                                 ` Denys Fedoryshchenko
       [not found]                                 ` <17187305E642C450.18151@groups.io>
  0 siblings, 2 replies; 25+ messages in thread
From: Todd Kjos @ 2022-09-26 15:29 UTC (permalink / raw)
  To: Guillaume Tucker
  Cc: Michał Gałka, kernelci, Guenter Roeck, Kevin Hilman

Having trouble accessing https://linux.kernelci.org/job/android/ today -
server issues?

On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:

> On 18/03/2022 17:19, Guillaume Tucker wrote:
>
> On 18/03/2022 15:47, Todd Kjos wrote:
>
> Is the MongoDB server down again? I don't get results at
> https://linux.kernelci.org/job/android/.
>
> It's not that, unfortunately there's some persisting issue which seems to
> have started yesterday. We're investigating now, please bear with us. Thank
> you for reporting the problem.
>
> Alright, there was a problem with background processing tasks on the
> server. That's resolved now, although a number of kernel builds need to be
> restarted which is why so many trees appear to be "in progress" on the web
> dashboard. Things will start picking up again gradually, it might take a
> few hours to get back to normal. Sorry for the inconvenience.
>
> Best wishes,
> Guillaume
>
>
> On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> guillaume.tucker@collabora.com> wrote:
>
>> Hi Todd,
>>
>> On 30/11/2021 16:54, Todd Kjos wrote:
>> > ...and now it's working again. Sorry about the noise if the issue was
>> > on my side.
>>
>> The MongoDB server did crash but it was restarted a few minutes
>> later.  There's some kind of watchdog to do that automatically,
>> maybe you were out of luck and checked the website during that
>> window.  Thanks for reporting the issue in any case.
>>
>> Best wishes,
>> Guillaume
>>
>> > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com> wrote:
>> >>
>> >> I can't load the data from https://linux.kernelci.org/job/android/
>> >> ("Error loading build data."). Is mongod "borked" again?
>> >>
>> >> On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com> wrote:
>> >>>
>> >>> It's working again. Thanks.
>> >>>
>> >>> On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com> wrote:
>> >>>>
>> >>>> I'm getting errors again when attempting to access
>> >>>> https://linux.kernelci.org/job/android/. mongod issues again?
>> >>>>
>> >>>> "Error while loading data from the server (error code: 500). Please
>> >>>> contact the website administrator"
>> >>>>
>> >>>> On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
>> >>>> <guillaume.tucker@collabora.com> wrote:
>> >>>>>
>> >>>>> On 22/06/2021 16:07, Todd Kjos wrote:
>> >>>>>> Hmm, it's still not working for me -- did it break again?
>> >>>>>
>> >>>>> Yes there was an OOM issue:
>> >>>>>
>> >>>>> [4316410.416514] Killed process 20620 (mongod) total-vm:20171340kB,
>> anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
>> >>>>> [4316412.060511] oom_reaper: reaped process 20620 (mongod), now
>> anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>> >>>>>
>> >>>>> We're looking into moving the Mongo DB instance to the cloud in
>> >>>>> Azure.  Hopefully that will save us fixing this kind of
>> >>>>> maintenance issues.  Meanwhile, we should take a look and see if
>> >>>>> there's some config option to limit the amount of RAM used by
>> >>>>> Mongo DB to work around the problem.
>> >>>>>
>> >>>>> Best wishes,
>> >>>>> Guillaume
>> >>>>>
>> >>>>>
>> >>>>>> On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
>> michal.galka@collabora.com <mailto:michal.galka@collabora.com>> wrote:
>> >>>>>>
>> >>>>>>     On 21.06.2021 21:59, Todd Kjos via groups.io <http://groups.io>
>> wrote:
>> >>>>>>     > I'm having this issue again (can't access kernelci results
>> >>>>>>     > at https://linux.kernelci.org/job/android/ <
>> https://linux.kernelci.org/job/android/>
>> >>>>>>     > <https://linux.kernelci.org/job/android/ <
>> https://linux.kernelci.org/job/android/>>). Is the mongod service borked
>> >>>>>>     > again?
>> >>>>>>     >
>> >>>>>>
>> >>>>>>     Yes, there was an issue with mongod service again. It's
>> working now,
>> >>>>>>     celery processed pending tasks and everything seems responsive.
>> >>>>>>
>> >>>>>>     Thanks,
>> >>>>>>     Michał
>> >>>>>>
>> >>>>>
>>
>>
>
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-09-26 15:29                               ` Todd Kjos
@ 2022-09-26 15:35                                 ` Denys Fedoryshchenko
       [not found]                                 ` <17187305E642C450.18151@groups.io>
  1 sibling, 0 replies; 25+ messages in thread
From: Denys Fedoryshchenko @ 2022-09-26 15:35 UTC (permalink / raw)
  To: kernelci, tkjos, Guillaume Tucker
  Cc: Michał Gałka, Guenter Roeck, Kevin Hilman

We are investingating right now and trying to solve the issue.

On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io wrote:
> Having trouble accessing
> https://linux.kernelci.org/job/android/ today -
> server issues?
> 
> On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> guillaume.tucker@collabora.com> wrote:
> 
> > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > 
> > On 18/03/2022 15:47, Todd Kjos wrote:
> > 
> > Is the MongoDB server down again? I don't get results at
> > https://linux.kernelci.org/job/android/.
> > 
> > It's not that, unfortunately there's some persisting issue which
> > seems to
> > have started yesterday. We're investigating now, please bear with
> > us. Thank
> > you for reporting the problem.
> > 
> > Alright, there was a problem with background processing tasks on
> > the
> > server. That's resolved now, although a number of kernel builds
> > need to be
> > restarted which is why so many trees appear to be "in progress" on
> > the web
> > dashboard. Things will start picking up again gradually, it might
> > take a
> > few hours to get back to normal. Sorry for the inconvenience.
> > 
> > Best wishes,
> > Guillaume
> > 
> > 
> > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > guillaume.tucker@collabora.com> wrote:
> > 
> > > Hi Todd,
> > > 
> > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > ...and now it's working again. Sorry about the noise if the
> > > > issue was
> > > > on my side.
> > > 
> > > The MongoDB server did crash but it was restarted a few minutes
> > > later.  There's some kind of watchdog to do that automatically,
> > > maybe you were out of luck and checked the website during that
> > > window.  Thanks for reporting the issue in any case.
> > > 
> > > Best wishes,
> > > Guillaume
> > > 
> > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com>
> > > > wrote:
> > > > > 
> > > > > I can't load the data from
> > > > > https://linux.kernelci.org/job/android/
> > > > > ("Error loading build data."). Is mongod "borked" again?
> > > > > 
> > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos <tkjos@google.com>
> > > > > wrote:
> > > > > > 
> > > > > > It's working again. Thanks.
> > > > > > 
> > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos <tkjos@google.com>
> > > > > > wrote:
> > > > > > > 
> > > > > > > I'm getting errors again when attempting to access
> > > > > > > https://linux.kernelci.org/job/android/. mongod issues
> > > > > > > again?
> > > > > > > 
> > > > > > > "Error while loading data from the server (error code:
> > > > > > > 500). Please
> > > > > > > contact the website administrator"
> > > > > > > 
> > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > 
> > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > Hmm, it's still not working for me -- did it break
> > > > > > > > > again?
> > > > > > > > 
> > > > > > > > Yes there was an OOM issue:
> > > > > > > > 
> > > > > > > > [4316410.416514] Killed process 20620 (mongod) total-
> > > > > > > > vm:20171340kB,
> > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > [4316412.060511] oom_reaper: reaped process 20620
> > > > > > > > (mongod), now
> > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > 
> > > > > > > > We're looking into moving the Mongo DB instance to the
> > > > > > > > cloud in
> > > > > > > > Azure.  Hopefully that will save us fixing this kind of
> > > > > > > > maintenance issues.  Meanwhile, we should take a look
> > > > > > > > and see if
> > > > > > > > there's some config option to limit the amount of RAM
> > > > > > > > used by
> > > > > > > > Mongo DB to work around the problem.
> > > > > > > > 
> > > > > > > > Best wishes,
> > > > > > > > Guillaume
> > > > > > > > 
> > > > > > > > 
> > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
> > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
> > > wrote:
> > > > > > > > > 
> > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io
> > > > > > > > > <http://groups.io>
> > > wrote:
> > > > > > > > >     > I'm having this issue again (can't access
> > > > > > > > > kernelci results
> > > > > > > > >     > at https://linux.kernelci.org/job/android/ <
> > > https://linux.kernelci.org/job/android/>
> > > > > > > > >     > <https://linux.kernelci.org/job/android/ <
> > > https://linux.kernelci.org/job/android/>>). Is the mongod service
> > > borked
> > > > > > > > >     > again?
> > > > > > > > >     >
> > > > > > > > > 
> > > > > > > > >     Yes, there was an issue with mongod service
> > > > > > > > > again. It's
> > > working now,
> > > > > > > > >     celery processed pending tasks and everything
> > > > > > > > > seems responsive.
> > > > > > > > > 
> > > > > > > > >     Thanks,
> > > > > > > > >     Michał
> > > > > > > > > 
> > > > > > > > 
> > > 
> > > 
> > 
> > 
> 
> 
> 
> 
> 



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
       [not found]                                 ` <17187305E642C450.18151@groups.io>
@ 2022-09-26 16:06                                   ` Denys Fedoryshchenko
  2022-09-26 16:13                                     ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Denys Fedoryshchenko @ 2022-09-26 16:06 UTC (permalink / raw)
  To: kernelci, tkjos, Guillaume Tucker
  Cc: Michał Gałka, Guenter Roeck, Kevin Hilman

Upd: dashboard should be fine now, but we are running update right now,
so some glitches might happen.

On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
> We are investingating right now and trying to solve the issue.
> 
> On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io wrote:
> > Having trouble accessing
> > https://linux.kernelci.org/job/android/ today -
> > server issues?
> > 
> > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> > guillaume.tucker@collabora.com> wrote:
> > 
> > > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > > 
> > > On 18/03/2022 15:47, Todd Kjos wrote:
> > > 
> > > Is the MongoDB server down again? I don't get results at
> > > https://linux.kernelci.org/job/android/.
> > > 
> > > It's not that, unfortunately there's some persisting issue which
> > > seems to
> > > have started yesterday. We're investigating now, please bear with
> > > us. Thank
> > > you for reporting the problem.
> > > 
> > > Alright, there was a problem with background processing tasks on
> > > the
> > > server. That's resolved now, although a number of kernel builds
> > > need to be
> > > restarted which is why so many trees appear to be "in progress"
> > > on
> > > the web
> > > dashboard. Things will start picking up again gradually, it might
> > > take a
> > > few hours to get back to normal. Sorry for the inconvenience.
> > > 
> > > Best wishes,
> > > Guillaume
> > > 
> > > 
> > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > > guillaume.tucker@collabora.com> wrote:
> > > 
> > > > Hi Todd,
> > > > 
> > > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > > ...and now it's working again. Sorry about the noise if the
> > > > > issue was
> > > > > on my side.
> > > > 
> > > > The MongoDB server did crash but it was restarted a few minutes
> > > > later.  There's some kind of watchdog to do that automatically,
> > > > maybe you were out of luck and checked the website during that
> > > > window.  Thanks for reporting the issue in any case.
> > > > 
> > > > Best wishes,
> > > > Guillaume
> > > > 
> > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com>
> > > > > wrote:
> > > > > > 
> > > > > > I can't load the data from
> > > > > > https://linux.kernelci.org/job/android/
> > > > > > ("Error loading build data."). Is mongod "borked" again?
> > > > > > 
> > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
> > > > > > <tkjos@google.com>
> > > > > > wrote:
> > > > > > > 
> > > > > > > It's working again. Thanks.
> > > > > > > 
> > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
> > > > > > > <tkjos@google.com>
> > > > > > > wrote:
> > > > > > > > 
> > > > > > > > I'm getting errors again when attempting to access
> > > > > > > > https://linux.kernelci.org/job/android/. mongod issues
> > > > > > > > again?
> > > > > > > > 
> > > > > > > > "Error while loading data from the server (error code:
> > > > > > > > 500). Please
> > > > > > > > contact the website administrator"
> > > > > > > > 
> > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > > 
> > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > > Hmm, it's still not working for me -- did it break
> > > > > > > > > > again?
> > > > > > > > > 
> > > > > > > > > Yes there was an OOM issue:
> > > > > > > > > 
> > > > > > > > > [4316410.416514] Killed process 20620 (mongod) total-
> > > > > > > > > vm:20171340kB,
> > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > [4316412.060511] oom_reaper: reaped process 20620
> > > > > > > > > (mongod), now
> > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > 
> > > > > > > > > We're looking into moving the Mongo DB instance to
> > > > > > > > > the
> > > > > > > > > cloud in
> > > > > > > > > Azure.  Hopefully that will save us fixing this kind
> > > > > > > > > of
> > > > > > > > > maintenance issues.  Meanwhile, we should take a look
> > > > > > > > > and see if
> > > > > > > > > there's some config option to limit the amount of RAM
> > > > > > > > > used by
> > > > > > > > > Mongo DB to work around the problem.
> > > > > > > > > 
> > > > > > > > > Best wishes,
> > > > > > > > > Guillaume
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
> > > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
> > > > wrote:
> > > > > > > > > > 
> > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io
> > > > > > > > > > <http://groups.io>
> > > > wrote:
> > > > > > > > > >     > I'm having this issue again (can't access
> > > > > > > > > > kernelci results
> > > > > > > > > >     > at https://linux.kernelci.org/job/android/ <
> > > > https://linux.kernelci.org/job/android/>
> > > > > > > > > >     > <https://linux.kernelci.org/job/android/ <
> > > > https://linux.kernelci.org/job/android/>>). Is the mongod
> > > > service
> > > > borked
> > > > > > > > > >     > again?
> > > > > > > > > >     >
> > > > > > > > > > 
> > > > > > > > > >     Yes, there was an issue with mongod service
> > > > > > > > > > again. It's
> > > > working now,
> > > > > > > > > >     celery processed pending tasks and everything
> > > > > > > > > > seems responsive.
> > > > > > > > > > 
> > > > > > > > > >     Thanks,
> > > > > > > > > >     Michał
> > > > > > > > > > 
> > > > > > > > > 
> > > > 
> > > > 
> > > 
> > > 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 
> 
> 
> 
> 



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-09-26 16:06                                   ` Denys Fedoryshchenko
@ 2022-09-26 16:13                                     ` Todd Kjos
  2022-10-14 18:12                                       ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2022-09-26 16:13 UTC (permalink / raw)
  To: Denys Fedoryshchenko
  Cc: kernelci, Guillaume Tucker, Michał Gałka,
	Guenter Roeck, Kevin Hilman

Thanks for the quick response! It's working for me now.

On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> Upd: dashboard should be fine now, but we are running update right now,
> so some glitches might happen.
>
> On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
> > We are investingating right now and trying to solve the issue.
> >
> > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io wrote:
> > > Having trouble accessing
> > > https://linux.kernelci.org/job/android/ today -
> > > server issues?
> > >
> > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> > > guillaume.tucker@collabora.com> wrote:
> > >
> > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > > >
> > > > On 18/03/2022 15:47, Todd Kjos wrote:
> > > >
> > > > Is the MongoDB server down again? I don't get results at
> > > > https://linux.kernelci.org/job/android/.
> > > >
> > > > It's not that, unfortunately there's some persisting issue which
> > > > seems to
> > > > have started yesterday. We're investigating now, please bear with
> > > > us. Thank
> > > > you for reporting the problem.
> > > >
> > > > Alright, there was a problem with background processing tasks on
> > > > the
> > > > server. That's resolved now, although a number of kernel builds
> > > > need to be
> > > > restarted which is why so many trees appear to be "in progress"
> > > > on
> > > > the web
> > > > dashboard. Things will start picking up again gradually, it might
> > > > take a
> > > > few hours to get back to normal. Sorry for the inconvenience.
> > > >
> > > > Best wishes,
> > > > Guillaume
> > > >
> > > >
> > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > > > guillaume.tucker@collabora.com> wrote:
> > > >
> > > > > Hi Todd,
> > > > >
> > > > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > > > ...and now it's working again. Sorry about the noise if the
> > > > > > issue was
> > > > > > on my side.
> > > > >
> > > > > The MongoDB server did crash but it was restarted a few minutes
> > > > > later.  There's some kind of watchdog to do that automatically,
> > > > > maybe you were out of luck and checked the website during that
> > > > > window.  Thanks for reporting the issue in any case.
> > > > >
> > > > > Best wishes,
> > > > > Guillaume
> > > > >
> > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > I can't load the data from
> > > > > > > https://linux.kernelci.org/job/android/
> > > > > > > ("Error loading build data."). Is mongod "borked" again?
> > > > > > >
> > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
> > > > > > > <tkjos@google.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > It's working again. Thanks.
> > > > > > > >
> > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
> > > > > > > > <tkjos@google.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I'm getting errors again when attempting to access
> > > > > > > > > https://linux.kernelci.org/job/android/. mongod issues
> > > > > > > > > again?
> > > > > > > > >
> > > > > > > > > "Error while loading data from the server (error code:
> > > > > > > > > 500). Please
> > > > > > > > > contact the website administrator"
> > > > > > > > >
> > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > > > Hmm, it's still not working for me -- did it break
> > > > > > > > > > > again?
> > > > > > > > > >
> > > > > > > > > > Yes there was an OOM issue:
> > > > > > > > > >
> > > > > > > > > > [4316410.416514] Killed process 20620 (mongod) total-
> > > > > > > > > > vm:20171340kB,
> > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > [4316412.060511] oom_reaper: reaped process 20620
> > > > > > > > > > (mongod), now
> > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > >
> > > > > > > > > > We're looking into moving the Mongo DB instance to
> > > > > > > > > > the
> > > > > > > > > > cloud in
> > > > > > > > > > Azure.  Hopefully that will save us fixing this kind
> > > > > > > > > > of
> > > > > > > > > > maintenance issues.  Meanwhile, we should take a look
> > > > > > > > > > and see if
> > > > > > > > > > there's some config option to limit the amount of RAM
> > > > > > > > > > used by
> > > > > > > > > > Mongo DB to work around the problem.
> > > > > > > > > >
> > > > > > > > > > Best wishes,
> > > > > > > > > > Guillaume
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
> > > > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io
> > > > > > > > > > > <http://groups.io>
> > > > > wrote:
> > > > > > > > > > >     > I'm having this issue again (can't access
> > > > > > > > > > > kernelci results
> > > > > > > > > > >     > at https://linux.kernelci.org/job/android/ <
> > > > > https://linux.kernelci.org/job/android/>
> > > > > > > > > > >     > <https://linux.kernelci.org/job/android/ <
> > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
> > > > > service
> > > > > borked
> > > > > > > > > > >     > again?
> > > > > > > > > > >     >
> > > > > > > > > > >
> > > > > > > > > > >     Yes, there was an issue with mongod service
> > > > > > > > > > > again. It's
> > > > > working now,
> > > > > > > > > > >     celery processed pending tasks and everything
> > > > > > > > > > > seems responsive.
> > > > > > > > > > >
> > > > > > > > > > >     Thanks,
> > > > > > > > > > >     Michał
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> > 
> >
> >
>
>
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-09-26 16:13                                     ` Todd Kjos
@ 2022-10-14 18:12                                       ` Todd Kjos
  2022-10-14 18:16                                         ` Denys Fedoryshchenko
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2022-10-14 18:12 UTC (permalink / raw)
  To: Denys Fedoryshchenko
  Cc: kernelci, Guillaume Tucker, Michał Gałka,
	Guenter Roeck, Kevin Hilman

The dashboard is broken again. Known issue?

"Error while loading data from the server (error code: 500). Please contact
the website administrator"

On Mon, Sep 26, 2022 at 9:13 AM Todd Kjos <tkjos@google.com> wrote:

> Thanks for the quick response! It's working for me now.
>
> On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko <
> denys.f@collabora.com> wrote:
>
>> Upd: dashboard should be fine now, but we are running update right now,
>> so some glitches might happen.
>>
>> On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
>> > We are investingating right now and trying to solve the issue.
>> >
>> > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io wrote:
>> > > Having trouble accessing
>> > > https://linux.kernelci.org/job/android/ today -
>> > > server issues?
>> > >
>> > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
>> > > guillaume.tucker@collabora.com> wrote:
>> > >
>> > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
>> > > >
>> > > > On 18/03/2022 15:47, Todd Kjos wrote:
>> > > >
>> > > > Is the MongoDB server down again? I don't get results at
>> > > > https://linux.kernelci.org/job/android/.
>> > > >
>> > > > It's not that, unfortunately there's some persisting issue which
>> > > > seems to
>> > > > have started yesterday. We're investigating now, please bear with
>> > > > us. Thank
>> > > > you for reporting the problem.
>> > > >
>> > > > Alright, there was a problem with background processing tasks on
>> > > > the
>> > > > server. That's resolved now, although a number of kernel builds
>> > > > need to be
>> > > > restarted which is why so many trees appear to be "in progress"
>> > > > on
>> > > > the web
>> > > > dashboard. Things will start picking up again gradually, it might
>> > > > take a
>> > > > few hours to get back to normal. Sorry for the inconvenience.
>> > > >
>> > > > Best wishes,
>> > > > Guillaume
>> > > >
>> > > >
>> > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
>> > > > guillaume.tucker@collabora.com> wrote:
>> > > >
>> > > > > Hi Todd,
>> > > > >
>> > > > > On 30/11/2021 16:54, Todd Kjos wrote:
>> > > > > > ...and now it's working again. Sorry about the noise if the
>> > > > > > issue was
>> > > > > > on my side.
>> > > > >
>> > > > > The MongoDB server did crash but it was restarted a few minutes
>> > > > > later.  There's some kind of watchdog to do that automatically,
>> > > > > maybe you were out of luck and checked the website during that
>> > > > > window.  Thanks for reporting the issue in any case.
>> > > > >
>> > > > > Best wishes,
>> > > > > Guillaume
>> > > > >
>> > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > I can't load the data from
>> > > > > > > https://linux.kernelci.org/job/android/
>> > > > > > > ("Error loading build data."). Is mongod "borked" again?
>> > > > > > >
>> > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
>> > > > > > > <tkjos@google.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > It's working again. Thanks.
>> > > > > > > >
>> > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
>> > > > > > > > <tkjos@google.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > I'm getting errors again when attempting to access
>> > > > > > > > > https://linux.kernelci.org/job/android/. mongod issues
>> > > > > > > > > again?
>> > > > > > > > >
>> > > > > > > > > "Error while loading data from the server (error code:
>> > > > > > > > > 500). Please
>> > > > > > > > > contact the website administrator"
>> > > > > > > > >
>> > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
>> > > > > > > > > <guillaume.tucker@collabora.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
>> > > > > > > > > > > Hmm, it's still not working for me -- did it break
>> > > > > > > > > > > again?
>> > > > > > > > > >
>> > > > > > > > > > Yes there was an OOM issue:
>> > > > > > > > > >
>> > > > > > > > > > [4316410.416514] Killed process 20620 (mongod) total-
>> > > > > > > > > > vm:20171340kB,
>> > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
>> > > > > > > > > > [4316412.060511] oom_reaper: reaped process 20620
>> > > > > > > > > > (mongod), now
>> > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>> > > > > > > > > >
>> > > > > > > > > > We're looking into moving the Mongo DB instance to
>> > > > > > > > > > the
>> > > > > > > > > > cloud in
>> > > > > > > > > > Azure.  Hopefully that will save us fixing this kind
>> > > > > > > > > > of
>> > > > > > > > > > maintenance issues.  Meanwhile, we should take a look
>> > > > > > > > > > and see if
>> > > > > > > > > > there's some config option to limit the amount of RAM
>> > > > > > > > > > used by
>> > > > > > > > > > Mongo DB to work around the problem.
>> > > > > > > > > >
>> > > > > > > > > > Best wishes,
>> > > > > > > > > > Guillaume
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
>> > > > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
>> > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io
>> > > > > > > > > > > <http://groups.io>
>> > > > > wrote:
>> > > > > > > > > > >     > I'm having this issue again (can't access
>> > > > > > > > > > > kernelci results
>> > > > > > > > > > >     > at https://linux.kernelci.org/job/android/ <
>> > > > > https://linux.kernelci.org/job/android/>
>> > > > > > > > > > >     > <https://linux.kernelci.org/job/android/ <
>> > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
>> > > > > service
>> > > > > borked
>> > > > > > > > > > >     > again?
>> > > > > > > > > > >     >
>> > > > > > > > > > >
>> > > > > > > > > > >     Yes, there was an issue with mongod service
>> > > > > > > > > > > again. It's
>> > > > > working now,
>> > > > > > > > > > >     celery processed pending tasks and everything
>> > > > > > > > > > > seems responsive.
>> > > > > > > > > > >
>> > > > > > > > > > >     Thanks,
>> > > > > > > > > > >     Michał
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> > 
>> >
>> >
>>
>>
>>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-10-14 18:12                                       ` Todd Kjos
@ 2022-10-14 18:16                                         ` Denys Fedoryshchenko
  2022-10-14 18:39                                           ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Denys Fedoryshchenko @ 2022-10-14 18:16 UTC (permalink / raw)
  To: Todd Kjos
  Cc: kernelci, Guillaume Tucker, Michał Gałka,
	Guenter Roeck, Kevin Hilman

Thanks for info, i am investigating issue and will fix problem shortly.
Hopefully next week we will have monitoring system in place.

On Fri, 2022-10-14 at 11:12 -0700, Todd Kjos wrote:
> The dashboard is broken again. Known issue?
> 
> "Error while loading data from the server (error code: 500). Please
> contact the website administrator"
> 
> On Mon, Sep 26, 2022 at 9:13 AM Todd Kjos <tkjos@google.com> wrote:
> > Thanks for the quick response! It's working for me now.
> > 
> > On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko
> > <denys.f@collabora.com> wrote:
> > > Upd: dashboard should be fine now, but we are running update
> > > right now,
> > > so some glitches might happen.
> > > 
> > > On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
> > > > We are investingating right now and trying to solve the issue.
> > > > 
> > > > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io
> > > wrote:
> > > > > Having trouble accessing
> > > > > https://linux.kernelci.org/job/android/ today -
> > > > > server issues?
> > > > > 
> > > > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> > > > > guillaume.tucker@collabora.com> wrote:
> > > > > 
> > > > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > > > > > 
> > > > > > On 18/03/2022 15:47, Todd Kjos wrote:
> > > > > > 
> > > > > > Is the MongoDB server down again? I don't get results at
> > > > > > https://linux.kernelci.org/job/android/.
> > > > > > 
> > > > > > It's not that, unfortunately there's some persisting issue
> > > which
> > > > > > seems to
> > > > > > have started yesterday. We're investigating now, please
> > > bear with
> > > > > > us. Thank
> > > > > > you for reporting the problem.
> > > > > > 
> > > > > > Alright, there was a problem with background processing
> > > tasks on
> > > > > > the
> > > > > > server. That's resolved now, although a number of kernel
> > > builds
> > > > > > need to be
> > > > > > restarted which is why so many trees appear to be "in
> > > progress"
> > > > > > on
> > > > > > the web
> > > > > > dashboard. Things will start picking up again gradually, it
> > > might
> > > > > > take a
> > > > > > few hours to get back to normal. Sorry for the
> > > inconvenience.
> > > > > > 
> > > > > > Best wishes,
> > > > > > Guillaume
> > > > > > 
> > > > > > 
> > > > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > > > > > guillaume.tucker@collabora.com> wrote:
> > > > > > 
> > > > > > > Hi Todd,
> > > > > > > 
> > > > > > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > > > > > ...and now it's working again. Sorry about the noise if
> > > the
> > > > > > > > issue was
> > > > > > > > on my side.
> > > > > > > 
> > > > > > > The MongoDB server did crash but it was restarted a few
> > > minutes
> > > > > > > later.  There's some kind of watchdog to do that
> > > automatically,
> > > > > > > maybe you were out of luck and checked the website during
> > > that
> > > > > > > window.  Thanks for reporting the issue in any case.
> > > > > > > 
> > > > > > > Best wishes,
> > > > > > > Guillaume
> > > > > > > 
> > > > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos
> > > <tkjos@google.com>
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > I can't load the data from
> > > > > > > > > https://linux.kernelci.org/job/android/
> > > > > > > > > ("Error loading build data."). Is mongod "borked"
> > > again?
> > > > > > > > > 
> > > > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
> > > > > > > > > <tkjos@google.com>
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > It's working again. Thanks.
> > > > > > > > > > 
> > > > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
> > > > > > > > > > <tkjos@google.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > I'm getting errors again when attempting to
> > > access
> > > > > > > > > > > https://linux.kernelci.org/job/android/. mongod
> > > issues
> > > > > > > > > > > again?
> > > > > > > > > > > 
> > > > > > > > > > > "Error while loading data from the server (error
> > > code:
> > > > > > > > > > > 500). Please
> > > > > > > > > > > contact the website administrator"
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > > > > > Hmm, it's still not working for me -- did it
> > > break
> > > > > > > > > > > > > again?
> > > > > > > > > > > > 
> > > > > > > > > > > > Yes there was an OOM issue:
> > > > > > > > > > > > 
> > > > > > > > > > > > [4316410.416514] Killed process 20620 (mongod)
> > > total-
> > > > > > > > > > > > vm:20171340kB,
> > > > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > > > [4316412.060511] oom_reaper: reaped process
> > > 20620
> > > > > > > > > > > > (mongod), now
> > > > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > > > 
> > > > > > > > > > > > We're looking into moving the Mongo DB instance
> > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > cloud in
> > > > > > > > > > > > Azure.  Hopefully that will save us fixing this
> > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > maintenance issues.  Meanwhile, we should take
> > > a look
> > > > > > > > > > > > and see if
> > > > > > > > > > > > there's some config option to limit the amount
> > > of RAM
> > > > > > > > > > > > used by
> > > > > > > > > > > > Mongo DB to work around the problem.
> > > > > > > > > > > > 
> > > > > > > > > > > > Best wishes,
> > > > > > > > > > > > Guillaume
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka
> > > <
> > > > > > >
> > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
> > > > > > > wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via
> > > groups.io
> > > > > > > > > > > > > <http://groups.io>
> > > > > > > wrote:
> > > > > > > > > > > > >     > I'm having this issue again (can't
> > > access
> > > > > > > > > > > > > kernelci results
> > > > > > > > > > > > >     > at
> > > https://linux.kernelci.org/job/android/ <
> > > > > > > https://linux.kernelci.org/job/android/>
> > > > > > > > > > > > >     >
> > > <https://linux.kernelci.org/job/android/ <
> > > > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
> > > > > > > service
> > > > > > > borked
> > > > > > > > > > > > >     > again?
> > > > > > > > > > > > >     >
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     Yes, there was an issue with mongod
> > > service
> > > > > > > > > > > > > again. It's
> > > > > > > working now,
> > > > > > > > > > > > >     celery processed pending tasks and
> > > everything
> > > > > > > > > > > > > seems responsive.
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     Thanks,
> > > > > > > > > > > > >     Michał
> > > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-10-14 18:16                                         ` Denys Fedoryshchenko
@ 2022-10-14 18:39                                           ` Todd Kjos
  2022-10-21 16:25                                             ` Todd Kjos
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2022-10-14 18:39 UTC (permalink / raw)
  To: Denys Fedoryshchenko
  Cc: kernelci, Guillaume Tucker, Michał Gałka,
	Guenter Roeck, Kevin Hilman

Thanks. It's working again.

On Fri, Oct 14, 2022 at 11:17 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> Thanks for info, i am investigating issue and will fix problem shortly.
> Hopefully next week we will have monitoring system in place.
>
> On Fri, 2022-10-14 at 11:12 -0700, Todd Kjos wrote:
>
> The dashboard is broken again. Known issue?
>
> "Error while loading data from the server (error code: 500). Please
> contact the website administrator"
>
> On Mon, Sep 26, 2022 at 9:13 AM Todd Kjos <tkjos@google.com> wrote:
>
> Thanks for the quick response! It's working for me now.
>
> On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko <
> denys.f@collabora.com> wrote:
>
> Upd: dashboard should be fine now, but we are running update right now,
> so some glitches might happen.
>
> On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
> > We are investingating right now and trying to solve the issue.
> >
> > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io wrote:
> > > Having trouble accessing
> > > https://linux.kernelci.org/job/android/ today -
> > > server issues?
> > >
> > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> > > guillaume.tucker@collabora.com> wrote:
> > >
> > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > > >
> > > > On 18/03/2022 15:47, Todd Kjos wrote:
> > > >
> > > > Is the MongoDB server down again? I don't get results at
> > > > https://linux.kernelci.org/job/android/.
> > > >
> > > > It's not that, unfortunately there's some persisting issue which
> > > > seems to
> > > > have started yesterday. We're investigating now, please bear with
> > > > us. Thank
> > > > you for reporting the problem.
> > > >
> > > > Alright, there was a problem with background processing tasks on
> > > > the
> > > > server. That's resolved now, although a number of kernel builds
> > > > need to be
> > > > restarted which is why so many trees appear to be "in progress"
> > > > on
> > > > the web
> > > > dashboard. Things will start picking up again gradually, it might
> > > > take a
> > > > few hours to get back to normal. Sorry for the inconvenience.
> > > >
> > > > Best wishes,
> > > > Guillaume
> > > >
> > > >
> > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > > > guillaume.tucker@collabora.com> wrote:
> > > >
> > > > > Hi Todd,
> > > > >
> > > > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > > > ...and now it's working again. Sorry about the noise if the
> > > > > > issue was
> > > > > > on my side.
> > > > >
> > > > > The MongoDB server did crash but it was restarted a few minutes
> > > > > later.  There's some kind of watchdog to do that automatically,
> > > > > maybe you were out of luck and checked the website during that
> > > > > window.  Thanks for reporting the issue in any case.
> > > > >
> > > > > Best wishes,
> > > > > Guillaume
> > > > >
> > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > I can't load the data from
> > > > > > > https://linux.kernelci.org/job/android/
> > > > > > > ("Error loading build data."). Is mongod "borked" again?
> > > > > > >
> > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
> > > > > > > <tkjos@google.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > It's working again. Thanks.
> > > > > > > >
> > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
> > > > > > > > <tkjos@google.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I'm getting errors again when attempting to access
> > > > > > > > > https://linux.kernelci.org/job/android/. mongod issues
> > > > > > > > > again?
> > > > > > > > >
> > > > > > > > > "Error while loading data from the server (error code:
> > > > > > > > > 500). Please
> > > > > > > > > contact the website administrator"
> > > > > > > > >
> > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > > > Hmm, it's still not working for me -- did it break
> > > > > > > > > > > again?
> > > > > > > > > >
> > > > > > > > > > Yes there was an OOM issue:
> > > > > > > > > >
> > > > > > > > > > [4316410.416514] Killed process 20620 (mongod) total-
> > > > > > > > > > vm:20171340kB,
> > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > [4316412.060511] oom_reaper: reaped process 20620
> > > > > > > > > > (mongod), now
> > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > >
> > > > > > > > > > We're looking into moving the Mongo DB instance to
> > > > > > > > > > the
> > > > > > > > > > cloud in
> > > > > > > > > > Azure.  Hopefully that will save us fixing this kind
> > > > > > > > > > of
> > > > > > > > > > maintenance issues.  Meanwhile, we should take a look
> > > > > > > > > > and see if
> > > > > > > > > > there's some config option to limit the amount of RAM
> > > > > > > > > > used by
> > > > > > > > > > Mongo DB to work around the problem.
> > > > > > > > > >
> > > > > > > > > > Best wishes,
> > > > > > > > > > Guillaume
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
> > > > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io
> > > > > > > > > > > <http://groups.io>
> > > > > wrote:
> > > > > > > > > > >     > I'm having this issue again (can't access
> > > > > > > > > > > kernelci results
> > > > > > > > > > >     > at https://linux.kernelci.org/job/android/ <
> > > > > https://linux.kernelci.org/job/android/>
> > > > > > > > > > >     > <https://linux.kernelci.org/job/android/ <
> > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
> > > > > service
> > > > > borked
> > > > > > > > > > >     > again?
> > > > > > > > > > >     >
> > > > > > > > > > >
> > > > > > > > > > >     Yes, there was an issue with mongod service
> > > > > > > > > > > again. It's
> > > > > working now,
> > > > > > > > > > >     celery processed pending tasks and everything
> > > > > > > > > > > seems responsive.
> > > > > > > > > > >
> > > > > > > > > > >     Thanks,
> > > > > > > > > > >     Michał
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> > 
> >
> >
>
>
>
>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-10-14 18:39                                           ` Todd Kjos
@ 2022-10-21 16:25                                             ` Todd Kjos
  2022-10-22  7:00                                               ` Denys Fedoryshchenko
  0 siblings, 1 reply; 25+ messages in thread
From: Todd Kjos @ 2022-10-21 16:25 UTC (permalink / raw)
  To: Denys Fedoryshchenko
  Cc: kernelci, Guillaume Tucker, Michał Gałka,
	Guenter Roeck, Kevin Hilman

Dashboard broken again (https://linux.kernelci.org/job/android/)? (Error
while loading data from the server (error code: 500). Please contact the
website administrator)

On Fri, Oct 14, 2022 at 11:39 AM Todd Kjos <tkjos@google.com> wrote:

> Thanks. It's working again.
>
> On Fri, Oct 14, 2022 at 11:17 AM Denys Fedoryshchenko <
> denys.f@collabora.com> wrote:
>
>> Thanks for info, i am investigating issue and will fix problem shortly.
>> Hopefully next week we will have monitoring system in place.
>>
>> On Fri, 2022-10-14 at 11:12 -0700, Todd Kjos wrote:
>>
>> The dashboard is broken again. Known issue?
>>
>> "Error while loading data from the server (error code: 500). Please
>> contact the website administrator"
>>
>> On Mon, Sep 26, 2022 at 9:13 AM Todd Kjos <tkjos@google.com> wrote:
>>
>> Thanks for the quick response! It's working for me now.
>>
>> On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko <
>> denys.f@collabora.com> wrote:
>>
>> Upd: dashboard should be fine now, but we are running update right now,
>> so some glitches might happen.
>>
>> On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
>> > We are investingating right now and trying to solve the issue.
>> >
>> > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io wrote:
>> > > Having trouble accessing
>> > > https://linux.kernelci.org/job/android/ today -
>> > > server issues?
>> > >
>> > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
>> > > guillaume.tucker@collabora.com> wrote:
>> > >
>> > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
>> > > >
>> > > > On 18/03/2022 15:47, Todd Kjos wrote:
>> > > >
>> > > > Is the MongoDB server down again? I don't get results at
>> > > > https://linux.kernelci.org/job/android/.
>> > > >
>> > > > It's not that, unfortunately there's some persisting issue which
>> > > > seems to
>> > > > have started yesterday. We're investigating now, please bear with
>> > > > us. Thank
>> > > > you for reporting the problem.
>> > > >
>> > > > Alright, there was a problem with background processing tasks on
>> > > > the
>> > > > server. That's resolved now, although a number of kernel builds
>> > > > need to be
>> > > > restarted which is why so many trees appear to be "in progress"
>> > > > on
>> > > > the web
>> > > > dashboard. Things will start picking up again gradually, it might
>> > > > take a
>> > > > few hours to get back to normal. Sorry for the inconvenience.
>> > > >
>> > > > Best wishes,
>> > > > Guillaume
>> > > >
>> > > >
>> > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
>> > > > guillaume.tucker@collabora.com> wrote:
>> > > >
>> > > > > Hi Todd,
>> > > > >
>> > > > > On 30/11/2021 16:54, Todd Kjos wrote:
>> > > > > > ...and now it's working again. Sorry about the noise if the
>> > > > > > issue was
>> > > > > > on my side.
>> > > > >
>> > > > > The MongoDB server did crash but it was restarted a few minutes
>> > > > > later.  There's some kind of watchdog to do that automatically,
>> > > > > maybe you were out of luck and checked the website during that
>> > > > > window.  Thanks for reporting the issue in any case.
>> > > > >
>> > > > > Best wishes,
>> > > > > Guillaume
>> > > > >
>> > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos <tkjos@google.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > I can't load the data from
>> > > > > > > https://linux.kernelci.org/job/android/
>> > > > > > > ("Error loading build data."). Is mongod "borked" again?
>> > > > > > >
>> > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
>> > > > > > > <tkjos@google.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > It's working again. Thanks.
>> > > > > > > >
>> > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
>> > > > > > > > <tkjos@google.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > I'm getting errors again when attempting to access
>> > > > > > > > > https://linux.kernelci.org/job/android/. mongod issues
>> > > > > > > > > again?
>> > > > > > > > >
>> > > > > > > > > "Error while loading data from the server (error code:
>> > > > > > > > > 500). Please
>> > > > > > > > > contact the website administrator"
>> > > > > > > > >
>> > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
>> > > > > > > > > <guillaume.tucker@collabora.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
>> > > > > > > > > > > Hmm, it's still not working for me -- did it break
>> > > > > > > > > > > again?
>> > > > > > > > > >
>> > > > > > > > > > Yes there was an OOM issue:
>> > > > > > > > > >
>> > > > > > > > > > [4316410.416514] Killed process 20620 (mongod) total-
>> > > > > > > > > > vm:20171340kB,
>> > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
>> > > > > > > > > > [4316412.060511] oom_reaper: reaped process 20620
>> > > > > > > > > > (mongod), now
>> > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>> > > > > > > > > >
>> > > > > > > > > > We're looking into moving the Mongo DB instance to
>> > > > > > > > > > the
>> > > > > > > > > > cloud in
>> > > > > > > > > > Azure.  Hopefully that will save us fixing this kind
>> > > > > > > > > > of
>> > > > > > > > > > maintenance issues.  Meanwhile, we should take a look
>> > > > > > > > > > and see if
>> > > > > > > > > > there's some config option to limit the amount of RAM
>> > > > > > > > > > used by
>> > > > > > > > > > Mongo DB to work around the problem.
>> > > > > > > > > >
>> > > > > > > > > > Best wishes,
>> > > > > > > > > > Guillaume
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka <
>> > > > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
>> > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via groups.io
>> > > > > > > > > > > <http://groups.io>
>> > > > > wrote:
>> > > > > > > > > > >     > I'm having this issue again (can't access
>> > > > > > > > > > > kernelci results
>> > > > > > > > > > >     > at https://linux.kernelci.org/job/android/ <
>> > > > > https://linux.kernelci.org/job/android/>
>> > > > > > > > > > >     > <https://linux.kernelci.org/job/android/ <
>> > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
>> > > > > service
>> > > > > borked
>> > > > > > > > > > >     > again?
>> > > > > > > > > > >     >
>> > > > > > > > > > >
>> > > > > > > > > > >     Yes, there was an issue with mongod service
>> > > > > > > > > > > again. It's
>> > > > > working now,
>> > > > > > > > > > >     celery processed pending tasks and everything
>> > > > > > > > > > > seems responsive.
>> > > > > > > > > > >
>> > > > > > > > > > >     Thanks,
>> > > > > > > > > > >     Michał
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> > 
>> >
>> >
>>
>>
>>
>>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: can't access kernelci results
  2022-10-21 16:25                                             ` Todd Kjos
@ 2022-10-22  7:00                                               ` Denys Fedoryshchenko
  0 siblings, 0 replies; 25+ messages in thread
From: Denys Fedoryshchenko @ 2022-10-22  7:00 UTC (permalink / raw)
  To: kernelci, tkjos
  Cc: Guillaume Tucker, Michał Gałka, Guenter Roeck, Kevin Hilman

Hello,

Issue found and fixed. 
Thanks!

P.S. Also i opened github issue where i will document stability issues
and taken actions, as some of persisting problems require allocation of
additional resources and additional investigation:
https://github.com/kernelci/kernelci-project/issues/133


On Fri, 2022-10-21 at 09:25 -0700, Todd Kjos via groups.io wrote:
> Dashboard broken again (https://linux.kernelci.org/job/android/)?
> (Error
> while loading data from the server (error code: 500). Please contact
> the
> website administrator)
> 
> On Fri, Oct 14, 2022 at 11:39 AM Todd Kjos <tkjos@google.com> wrote:
> 
> > Thanks. It's working again.
> > 
> > On Fri, Oct 14, 2022 at 11:17 AM Denys Fedoryshchenko <
> > denys.f@collabora.com> wrote:
> > 
> > > Thanks for info, i am investigating issue and will fix problem
> > > shortly.
> > > Hopefully next week we will have monitoring system in place.
> > > 
> > > On Fri, 2022-10-14 at 11:12 -0700, Todd Kjos wrote:
> > > 
> > > The dashboard is broken again. Known issue?
> > > 
> > > "Error while loading data from the server (error code: 500).
> > > Please
> > > contact the website administrator"
> > > 
> > > On Mon, Sep 26, 2022 at 9:13 AM Todd Kjos <tkjos@google.com>
> > > wrote:
> > > 
> > > Thanks for the quick response! It's working for me now.
> > > 
> > > On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko <
> > > denys.f@collabora.com> wrote:
> > > 
> > > Upd: dashboard should be fine now, but we are running update
> > > right now,
> > > so some glitches might happen.
> > > 
> > > On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
> > > > We are investingating right now and trying to solve the issue.
> > > > 
> > > > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io
> > > > wrote:
> > > > > Having trouble accessing
> > > > > https://linux.kernelci.org/job/android/ today -
> > > > > server issues?
> > > > > 
> > > > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> > > > > guillaume.tucker@collabora.com> wrote:
> > > > > 
> > > > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > > > > > 
> > > > > > On 18/03/2022 15:47, Todd Kjos wrote:
> > > > > > 
> > > > > > Is the MongoDB server down again? I don't get results at
> > > > > > https://linux.kernelci.org/job/android/.
> > > > > > 
> > > > > > It's not that, unfortunately there's some persisting issue
> > > > > > which
> > > > > > seems to
> > > > > > have started yesterday. We're investigating now, please
> > > > > > bear with
> > > > > > us. Thank
> > > > > > you for reporting the problem.
> > > > > > 
> > > > > > Alright, there was a problem with background processing
> > > > > > tasks on
> > > > > > the
> > > > > > server. That's resolved now, although a number of kernel
> > > > > > builds
> > > > > > need to be
> > > > > > restarted which is why so many trees appear to be "in
> > > > > > progress"
> > > > > > on
> > > > > > the web
> > > > > > dashboard. Things will start picking up again gradually, it
> > > > > > might
> > > > > > take a
> > > > > > few hours to get back to normal. Sorry for the
> > > > > > inconvenience.
> > > > > > 
> > > > > > Best wishes,
> > > > > > Guillaume
> > > > > > 
> > > > > > 
> > > > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > > > > > guillaume.tucker@collabora.com> wrote:
> > > > > > 
> > > > > > > Hi Todd,
> > > > > > > 
> > > > > > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > > > > > ...and now it's working again. Sorry about the noise if
> > > > > > > > the
> > > > > > > > issue was
> > > > > > > > on my side.
> > > > > > > 
> > > > > > > The MongoDB server did crash but it was restarted a few
> > > > > > > minutes
> > > > > > > later.  There's some kind of watchdog to do that
> > > > > > > automatically,
> > > > > > > maybe you were out of luck and checked the website during
> > > > > > > that
> > > > > > > window.  Thanks for reporting the issue in any case.
> > > > > > > 
> > > > > > > Best wishes,
> > > > > > > Guillaume
> > > > > > > 
> > > > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos
> > > > > > > > <tkjos@google.com>
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > I can't load the data from
> > > > > > > > > https://linux.kernelci.org/job/android/
> > > > > > > > > ("Error loading build data."). Is mongod "borked"
> > > > > > > > > again?
> > > > > > > > > 
> > > > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
> > > > > > > > > <tkjos@google.com>
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > It's working again. Thanks.
> > > > > > > > > > 
> > > > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
> > > > > > > > > > <tkjos@google.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > I'm getting errors again when attempting to
> > > > > > > > > > > access
> > > > > > > > > > > https://linux.kernelci.org/job/android/. mongod
> > > > > > > > > > > issues
> > > > > > > > > > > again?
> > > > > > > > > > > 
> > > > > > > > > > > "Error while loading data from the server (error
> > > > > > > > > > > code:
> > > > > > > > > > > 500). Please
> > > > > > > > > > > contact the website administrator"
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > > > > > Hmm, it's still not working for me -- did it
> > > > > > > > > > > > > break
> > > > > > > > > > > > > again?
> > > > > > > > > > > > 
> > > > > > > > > > > > Yes there was an OOM issue:
> > > > > > > > > > > > 
> > > > > > > > > > > > [4316410.416514] Killed process 20620 (mongod)
> > > > > > > > > > > > total-
> > > > > > > > > > > > vm:20171340kB,
> > > > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > > > [4316412.060511] oom_reaper: reaped process
> > > > > > > > > > > > 20620
> > > > > > > > > > > > (mongod), now
> > > > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > > > 
> > > > > > > > > > > > We're looking into moving the Mongo DB instance
> > > > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > cloud in
> > > > > > > > > > > > Azure.  Hopefully that will save us fixing this
> > > > > > > > > > > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > maintenance issues.  Meanwhile, we should take
> > > > > > > > > > > > a look
> > > > > > > > > > > > and see if
> > > > > > > > > > > > there's some config option to limit the amount
> > > > > > > > > > > > of RAM
> > > > > > > > > > > > used by
> > > > > > > > > > > > Mongo DB to work around the problem.
> > > > > > > > > > > > 
> > > > > > > > > > > > Best wishes,
> > > > > > > > > > > > Guillaume
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka
> > > > > > > > > > > > > <
> > > > > > > michal.galka@collabora.com <mailto:michal.galka@collabora.com
> > > > > > > >>
> > > > > > > wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via
> > > > > > > > > > > > > groups.io
> > > > > > > > > > > > > <http://groups.io>
> > > > > > > wrote:
> > > > > > > > > > > > >     > I'm having this issue again (can't
> > > > > > > > > > > > > access
> > > > > > > > > > > > > kernelci results
> > > > > > > > > > > > >     > at
> > > > > > > > > > > > > https://linux.kernelci.org/job/android/ <
> > > > > > > https://linux.kernelci.org/job/android/>
> > > > > > > > > > > > >     >
> > > > > > > > > > > > > <https://linux.kernelci.org/job/android/ <
> > > > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
> > > > > > > service
> > > > > > > borked
> > > > > > > > > > > > >     > again?
> > > > > > > > > > > > >     >
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     Yes, there was an issue with mongod
> > > > > > > > > > > > > service
> > > > > > > > > > > > > again. It's
> > > > > > > working now,
> > > > > > > > > > > > >     celery processed pending tasks and
> > > > > > > > > > > > > everything
> > > > > > > > > > > > > seems responsive.
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     Thanks,
> > > > > > > > > > > > >     Michał
> > > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> 
> 
> 
> 
> 



^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2022-10-22  7:00 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-05 19:02 can't access kernelci results Todd Kjos
2021-04-05 20:39 ` Guenter Roeck
2021-04-05 20:58   ` Guillaume Tucker
2021-04-05 23:14     ` Todd Kjos
2021-06-21 19:59       ` Todd Kjos
2021-06-22  9:43         ` Michał Gałka
2021-06-22 15:07           ` Todd Kjos
2021-06-22 15:16             ` Guillaume Tucker
2021-09-02 15:58               ` Todd Kjos
2021-09-02 19:28                 ` Todd Kjos
2021-11-30 16:35                   ` Todd Kjos
2021-11-30 16:54                     ` Todd Kjos
2021-12-01  9:48                       ` Guillaume Tucker
2022-03-18 15:47                         ` Todd Kjos
2022-03-18 17:19                           ` Guillaume Tucker
2022-03-18 17:26                             ` Guillaume Tucker
2022-09-26 15:29                               ` Todd Kjos
2022-09-26 15:35                                 ` Denys Fedoryshchenko
     [not found]                                 ` <17187305E642C450.18151@groups.io>
2022-09-26 16:06                                   ` Denys Fedoryshchenko
2022-09-26 16:13                                     ` Todd Kjos
2022-10-14 18:12                                       ` Todd Kjos
2022-10-14 18:16                                         ` Denys Fedoryshchenko
2022-10-14 18:39                                           ` Todd Kjos
2022-10-21 16:25                                             ` Todd Kjos
2022-10-22  7:00                                               ` Denys Fedoryshchenko

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).