[PATCH 1/2] npmsw: Avoid race condition with multiple npm fetchers

[PATCH 1/2] npmsw: Avoid race condition with multiple npm fetchers

[Caner Altinbasak]

If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.

Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
---
 lib/bb/fetch2/npmsw.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/fetch2/npmsw.py b/lib/bb/fetch2/npmsw.py
index 9523cece1..426a13965 100644
--- a/lib/bb/fetch2/npmsw.py
+++ b/lib/bb/fetch2/npmsw.py
@@ -30,6 +30,8 @@ from bb.fetch2.npm import npm_integrity
 from bb.fetch2.npm import npm_localfile
 from bb.fetch2.npm import npm_unpack
 from bb.utils import is_semver
+from bb.utils import lockfile
+from bb.utils import unlockfile

 def foreach_dependencies(shrinkwrap, callback=None, dev=False):
     """
@@ -195,7 +197,9 @@ class NpmShrinkWrap(FetchMethod):
             proxy_ud = ud.proxy.ud[proxy_url]
             proxy_d = ud.proxy.d
             proxy_ud.setup_localpath(proxy_d)
+            lf = lockfile(proxy_ud.lockfile)
             returns.append(handle(proxy_ud.method, proxy_ud, proxy_d))
+            unlockfile(lf)
         return returns

     def verify_donestamp(self, ud, d):
--
2.20.1

BrightSign considers your privacy to be very important. The emails you send to us will be protected and secured. Furthermore, we will only use your email and contact information for the reasons you sent them to us and for tracking how effectively we respond to your requests.

[PATCH 2/2] fetch2: Do not attempt to verify donestamp if download is not complete

[Caner Altinbasak]

npmsw(and maybe other fetchers) may try to download same files simulatenously.

if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.

In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.

Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
---
 lib/bb/fetch2/__init__.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 259b2637a..650969020 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1721,7 +1721,9 @@ class Fetch(object):
                     self.d.setVar("BB_NO_NETWORK", "1")

                 firsterr = None
-                verified_stamp = m.verify_donestamp(ud, self.d)
+                verified_stamp = False
+                if done:
+                    verified_stamp = m.verify_donestamp(ud, self.d)
                 if not done and (not verified_stamp or m.need_update(ud, self.d)):
                     try:
                         if not trusted_network(self.d, ud.url):
--
2.20.1

BrightSign considers your privacy to be very important. The emails you send to us will be protected and secured. Furthermore, we will only use your email and contact information for the reasons you sent them to us and for tracking how effectively we respond to your requests.