Re: can't access kernelci results

["Denys Fedoryshchenko" <denys.f@collabora.com>]

Thanks for info, i am investigating issue and will fix problem shortly.
Hopefully next week we will have monitoring system in place.

On Fri, 2022-10-14 at 11:12 -0700, Todd Kjos wrote:
> The dashboard is broken again. Known issue?
> 
> "Error while loading data from the server (error code: 500). Please
> contact the website administrator"
> 
> On Mon, Sep 26, 2022 at 9:13 AM Todd Kjos <tkjos@google.com> wrote:
> > Thanks for the quick response! It's working for me now.
> > 
> > On Mon, Sep 26, 2022 at 9:06 AM Denys Fedoryshchenko
> > <denys.f@collabora.com> wrote:
> > > Upd: dashboard should be fine now, but we are running update
> > > right now,
> > > so some glitches might happen.
> > > 
> > > On Mon, 2022-09-26 at 18:35 +0300, Denys Fedoryshchenko wrote:
> > > > We are investingating right now and trying to solve the issue.
> > > > 
> > > > On Mon, 2022-09-26 at 08:29 -0700, Todd Kjos via groups.io
> > > wrote:
> > > > > Having trouble accessing
> > > > > https://linux.kernelci.org/job/android/ today -
> > > > > server issues?
> > > > > 
> > > > > On Fri, Mar 18, 2022 at 10:26 AM Guillaume Tucker <
> > > > > guillaume.tucker@collabora.com> wrote:
> > > > > 
> > > > > > On 18/03/2022 17:19, Guillaume Tucker wrote:
> > > > > > 
> > > > > > On 18/03/2022 15:47, Todd Kjos wrote:
> > > > > > 
> > > > > > Is the MongoDB server down again? I don't get results at
> > > > > > https://linux.kernelci.org/job/android/.
> > > > > > 
> > > > > > It's not that, unfortunately there's some persisting issue
> > > which
> > > > > > seems to
> > > > > > have started yesterday. We're investigating now, please
> > > bear with
> > > > > > us. Thank
> > > > > > you for reporting the problem.
> > > > > > 
> > > > > > Alright, there was a problem with background processing
> > > tasks on
> > > > > > the
> > > > > > server. That's resolved now, although a number of kernel
> > > builds
> > > > > > need to be
> > > > > > restarted which is why so many trees appear to be "in
> > > progress"
> > > > > > on
> > > > > > the web
> > > > > > dashboard. Things will start picking up again gradually, it
> > > might
> > > > > > take a
> > > > > > few hours to get back to normal. Sorry for the
> > > inconvenience.
> > > > > > 
> > > > > > Best wishes,
> > > > > > Guillaume
> > > > > > 
> > > > > > 
> > > > > > On Wed, Dec 1, 2021 at 1:49 AM Guillaume Tucker <
> > > > > > guillaume.tucker@collabora.com> wrote:
> > > > > > 
> > > > > > > Hi Todd,
> > > > > > > 
> > > > > > > On 30/11/2021 16:54, Todd Kjos wrote:
> > > > > > > > ...and now it's working again. Sorry about the noise if
> > > the
> > > > > > > > issue was
> > > > > > > > on my side.
> > > > > > > 
> > > > > > > The MongoDB server did crash but it was restarted a few
> > > minutes
> > > > > > > later.  There's some kind of watchdog to do that
> > > automatically,
> > > > > > > maybe you were out of luck and checked the website during
> > > that
> > > > > > > window.  Thanks for reporting the issue in any case.
> > > > > > > 
> > > > > > > Best wishes,
> > > > > > > Guillaume
> > > > > > > 
> > > > > > > > On Tue, Nov 30, 2021 at 8:35 AM Todd Kjos
> > > <tkjos@google.com>
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > I can't load the data from
> > > > > > > > > https://linux.kernelci.org/job/android/
> > > > > > > > > ("Error loading build data."). Is mongod "borked"
> > > again?
> > > > > > > > > 
> > > > > > > > > On Thu, Sep 2, 2021 at 12:28 PM Todd Kjos
> > > > > > > > > <tkjos@google.com>
> > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > It's working again. Thanks.
> > > > > > > > > > 
> > > > > > > > > > On Thu, Sep 2, 2021 at 8:58 AM Todd Kjos
> > > > > > > > > > <tkjos@google.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > I'm getting errors again when attempting to
> > > access
> > > > > > > > > > > https://linux.kernelci.org/job/android/. mongod
> > > issues
> > > > > > > > > > > again?
> > > > > > > > > > > 
> > > > > > > > > > > "Error while loading data from the server (error
> > > code:
> > > > > > > > > > > 500). Please
> > > > > > > > > > > contact the website administrator"
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, Jun 22, 2021 at 8:17 AM Guillaume Tucker
> > > > > > > > > > > <guillaume.tucker@collabora.com> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > On 22/06/2021 16:07, Todd Kjos wrote:
> > > > > > > > > > > > > Hmm, it's still not working for me -- did it
> > > break
> > > > > > > > > > > > > again?
> > > > > > > > > > > > 
> > > > > > > > > > > > Yes there was an OOM issue:
> > > > > > > > > > > > 
> > > > > > > > > > > > [4316410.416514] Killed process 20620 (mongod)
> > > total-
> > > > > > > > > > > > vm:20171340kB,
> > > > > > > anon-rss:18654740kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > > > [4316412.060511] oom_reaper: reaped process
> > > 20620
> > > > > > > > > > > > (mongod), now
> > > > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> > > > > > > > > > > > 
> > > > > > > > > > > > We're looking into moving the Mongo DB instance
> > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > cloud in
> > > > > > > > > > > > Azure.  Hopefully that will save us fixing this
> > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > maintenance issues.  Meanwhile, we should take
> > > a look
> > > > > > > > > > > > and see if
> > > > > > > > > > > > there's some config option to limit the amount
> > > of RAM
> > > > > > > > > > > > used by
> > > > > > > > > > > > Mongo DB to work around the problem.
> > > > > > > > > > > > 
> > > > > > > > > > > > Best wishes,
> > > > > > > > > > > > Guillaume
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > On Tue, Jun 22, 2021 at 2:43 AM Michał Gałka
> > > <
> > > > > > >
> > > michal.galka@collabora.com <mailto:michal.galka@collabora.com>>
> > > > > > > wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     On 21.06.2021 21:59, Todd Kjos via
> > > groups.io
> > > > > > > > > > > > > <http://groups.io>
> > > > > > > wrote:
> > > > > > > > > > > > >     > I'm having this issue again (can't
> > > access
> > > > > > > > > > > > > kernelci results
> > > > > > > > > > > > >     > at
> > > https://linux.kernelci.org/job/android/ <
> > > > > > > https://linux.kernelci.org/job/android/>
> > > > > > > > > > > > >     >
> > > <https://linux.kernelci.org/job/android/ <
> > > > > > > https://linux.kernelci.org/job/android/>>). Is the mongod
> > > > > > > service
> > > > > > > borked
> > > > > > > > > > > > >     > again?
> > > > > > > > > > > > >     >
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     Yes, there was an issue with mongod
> > > service
> > > > > > > > > > > > > again. It's
> > > > > > > working now,
> > > > > > > > > > > > >     celery processed pending tasks and
> > > everything
> > > > > > > > > > > > > seems responsive.
> > > > > > > > > > > > > 
> > > > > > > > > > > > >     Thanks,
> > > > > > > > > > > > >     Michał
> > > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > >