* [Buildroot] [PATCH 1/2] scanpypi: new utility
@ 2015-07-28 13:15 Denis THULIN
2015-07-28 13:15 ` [Buildroot] [PATCH 2/2] python-robotframework: New package Denis THULIN
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Denis THULIN @ 2015-07-28 13:15 UTC (permalink / raw)
To: buildroot
An utility for creating python package from the python package index
It fetches packages info from http://pypi.python.org and generates
corresponding packages files.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0: initial commit
python-pacakage-generator.py is an utility for automatically generating a
python package. It fetches packages info from http://pypi.python.org and
generates corresponding packages files.
v1:
- renamed python-package-generator to scanpypi
- split the huge script into a lot of functions
- fixed mistakes and small bugs
v2:
- Rewrited most of the functions into a class
- Changed the method for importing setup.py
- Created a main function to avoid use of global variable
- Now adds new dependencies to the list of packages to create
- Droppped the .py extension
v3:
- Fixed bugs on import setup (Relative import works again)
- Can handle packages as zipfile
- Now avoids bdist packages
- Changed behaviour for packages that are not hosted on PyPI
- Added various clarifications of the code
- Works with: flask django robotframework pyxml pyzmq Twisted Six
- Does not work with: setuptools
---
docs/manual/adding-packages-python.txt | 43 +++
support/scripts/scanpypi | 653 +++++++++++++++++++++++++++++++++
2 files changed, 696 insertions(+)
create mode 100755 support/scripts/scanpypi
diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
index 588dbf8..1f160ca 100644
--- a/docs/manual/adding-packages-python.txt
+++ b/docs/manual/adding-packages-python.txt
@@ -7,6 +7,49 @@ This infrastructure applies to Python packages that use the standard
Python setuptools mechanism as their build system, generally
recognizable by the usage of a +setup.py+ script.
+[[scanpypi]]
+
+==== Generating a +python-package+ from a PyPI repository
+
+You may want to use the +scanpypi+ located in +support/script+ to generate a
+package from an existing PyPI package.
+
+you can find the list of existing PyPI package https://pypi.python.org[here].
+
+Please keep in mind that you most likely need to manually check the package for
+any mistakes as there are things that cannot be guessed by the generator (e.g.
+dependencies on any of the python core modules such as BR2_PACKAGE_PYTHON_ZLIB).
+Also, please take note that the license and license files are guessed and must
+be checked. You also need to manually add the package to the +package/Config.in+
+file.
+
+When at the root of your buildroot directory just do :
+
+-----------------------
+./support/script/scanpypi foo bar -o package
+-----------------------
+
+This will generate packages +python-foo+ and +python-bar+ in the package
+folder if they exist on https://pypi.python.org.
+
+Find the +external python modules+ menu and insert your package inside.
+Keep in mind that the items inside a menu should be in alphabetical order.
+
+If your package is external, use the -o flag.
+
+-----------------------
+./support/script/scanpypi foo bar -o other_package_dir
+-----------------------
+
+This will generate packages +python-foo+ and +python-bar+ in the
++other_package_directory+ instead of +package+.
+
+Option +-h+ wil list the options available
+
+-----------------------
+./support/script/scanpypi -h
+-----------------------
+
[[python-package-tutorial]]
==== +python-package+ tutorial
diff --git a/support/scripts/scanpypi b/support/scripts/scanpypi
new file mode 100755
index 0000000..3e51bae
--- /dev/null
+++ b/support/scripts/scanpypi
@@ -0,0 +1,653 @@
+#!/usr/bin/python2
+"""
+Utility for building buildroot packages for existing pypi packages
+
+Any package built by scanpypi should be manually checked for
+errors.
+"""
+from __future__ import print_function
+import argparse
+import json
+import urllib2
+import sys
+import os
+import shutil
+import StringIO
+import tarfile
+import zipfile
+import errno
+import hashlib
+import re
+import textwrap
+import tempfile
+import imp
+from functools import wraps
+
+
+def setup_decorator(func, method):
+ """
+ Decorator for distutils.core.setup and setuptools.setup.
+ Puts the arguments with which setup is called as a dict
+ Add key 'method' which should be either 'setuptools' or 'distutils'.
+
+ Keyword arguments:
+ func -- either setuptools.setup or distutils.core.setup
+ method -- either 'setuptools' or 'distutils'
+ """
+
+ @wraps(func)
+ def closure(*args, **kwargs):
+ # Any python packages calls its setup function to be installed.
+ # Argument 'name' of this setup function is the package's name
+ BuildrootPackage.setup_args[kwargs['name']] = kwargs
+ BuildrootPackage.setup_args[kwargs['name']]['method'] = method
+ return closure
+
+
+# monkey patch
+import setuptools
+setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
+import distutils
+distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
+
+
+def find_file_upper_case(filenames, path='./'):
+ """
+ List generator:
+ Recursively find files that matches one of the specified filenames.
+ Returns a relative path starting with path argument.
+
+ Keyword arguments:
+ filenames -- List of filenames to be found
+ path -- Path to the directory to search
+ """
+ for root, dirs, files in os.walk(path):
+ for file in files:
+ if file.upper() in filenames:
+ yield (os.path.join(root, file))
+
+
+def pkg_buildroot_name(pkg_name):
+ """
+ Returns the buildroot package name for the PyPI package pkg_name.
+ Remove all non alphanumeric characters except -
+ Also lowers the name and adds 'python-' suffix
+
+ Keyword arguments:
+ pkg_name -- String to rename
+ """
+ name = re.sub('[^\w-]', '', pkg_name.lower())
+ prefix = 'python-'
+ pattern = re.compile('^(?!' + prefix + ')(.+?)$')
+ name = pattern.sub(r'python-\1', name)
+ return name
+
+
+class DownloadFailed(Exception):
+ pass
+
+
+class BuildrootPackage():
+ """
+ This class's methods are not meant to be used individually please use those
+ in the correct order:
+ __init__
+
+ download_package
+
+ extract_package
+
+ load_module
+
+ get_requirements
+
+ create_package_mk
+
+ create_hash_file
+
+ create_config_in
+ """
+ setup_args = {}
+
+ def __init__(self, real_name, pkg_folder):
+ self.real_name = real_name
+ self.buildroot_name = pkg_buildroot_name(self.real_name)
+ self.pkg_dir = os.path.join(pkg_folder, self.buildroot_name)
+ self.mk_name = self.buildroot_name.upper().replace('-', '_')
+ self.as_string = None
+ self.md5_sum = None
+ self.metadata = None
+ self.metadata_name = None
+ self.metadata_url = None
+ self.pkg_req = None
+ self.setup_metadata = None
+ self.tmp_extract = None
+ self.used_url = None
+ self.filename = None
+ self.url = None
+ self.version = None
+
+ def fetch_package_info(self):
+ """
+ Fetch a package's metadata from the python package index
+ """
+ self.metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
+ pkg=self.real_name)
+ try:
+ pkg_json = urllib2.urlopen(self.metadata_url).read().decode()
+ except urllib2.HTTPError as error:
+ print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '
+ .format(pkg=self.real_name))
+ raise
+ except urllib2.URLError:
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '
+ .format(pkg=self.real_name))
+ raise
+ self.metadata = json.loads(pkg_json)
+ self.version = self.metadata['info']['version']
+ self.metadata_name = self.metadata['info']['name']
+
+ def download_package(self):
+ """
+ Download a package using metadata from pypi
+ """
+ try:
+ self.metadata['urls'][0]['filename']
+ except IndexError:
+ print(
+ 'Non-conventional package, ',
+ 'please check carefully after creation')
+ self.metadata['urls'] = [{
+ 'packagetype': 'sdist',
+ 'url': self.metadata['info']['download_url'],
+ 'md5_digest': None}]
+ # In this case, we can't get the name of the downloaded file
+ # from the pypi api, so we need to find it, this should work
+ urlpath = urllib2.urlparse.urlparse(
+ self.metadata['info']['download_url']).path
+ # urlparse().path give something like
+ # /path/to/file-version.tar.gz
+ # We use basename to remove /path/to
+ self.metadata['urls'][0]['filename'] = os.path.basename(urlpath)
+ for download_url in self.metadata['urls']:
+ if 'bdist' in download_url['packagetype']:
+ continue
+ try:
+ print('Downloading package {pkg} from {url}...'.format(
+ pkg=self.real_name, url=download_url['url']))
+ download = urllib2.urlopen(download_url['url'])
+ except urllib2.HTTPError as http_error:
+ download = http_error
+ else:
+ self.used_url = download_url
+ self.as_string = download.read()
+ if not download_url['md5_digest']:
+ break
+ self.md5_sum = hashlib.md5(self.as_string).hexdigest()
+ if self.md5_sum == download_url['md5_digest']:
+ break
+ else:
+ if download.__class__ == urllib2.HTTPError:
+ raise download
+ raise DownloadFailed('Failed to downloas package {pkg}'
+ .format(pkg=self.real_name))
+ self.filename = self.used_url['filename']
+ self.url = self.used_url['url']
+
+ def extract_package(self, tmp_path):
+ """
+ Extract the package contents into a directrory
+
+ Keyword arguments:
+ tmp_path -- directory where you want the package to be extracted
+ """
+ as_file = StringIO.StringIO(self.as_string)
+ if self.filename[-3:] == 'zip':
+ with zipfile.open(fileobj=as_file) as as_zipfile:
+ tmp_pkg = os.path.join(tmp_path, self.buildroot_name)
+ try:
+ os.makedirs(tmp_pkg)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ return None, None
+ print('WARNING:', exception.message, file=sys.stderr)
+ print('Removing {pkg}...'.format(pkg=tmp_pkg))
+ shutil.rmtree(tmp_pkg)
+ os.makedirs(tmp_pkg)
+ as_zipfile.extractall(tmp_pkg)
+ else:
+ with tarfile.open(fileobj=as_file) as as_tarfile:
+ tmp_pkg = os.path.join(tmp_path, self.buildroot_name)
+ try:
+ os.makedirs(tmp_pkg)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ return None, None
+ print('WARNING:', exception.message, file=sys.stderr)
+ print('Removing {pkg}...'.format(pkg=tmp_pkg))
+ shutil.rmtree(tmp_pkg)
+ os.makedirs(tmp_pkg)
+ as_tarfile.extractall(tmp_pkg)
+
+ tmp_extract = '{folder}/{name}-{version}'
+ self.tmp_extract = tmp_extract.format(
+ folder=tmp_pkg,
+ name=self.metadata_name,
+ version=self.version)
+
+ def load_setup(self):
+ """
+ Loads the corresponding setup and store its metadata
+ """
+ current_dir = os.getcwd()
+ os.chdir(self.tmp_extract)
+ sys.path.append(self.tmp_extract)
+ s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract])
+ setup = imp.load_module('setup', s_file, s_path, s_desc)
+ try:
+ self.setup_metadata = self.setup_args[self.metadata_name]
+ except KeyError:
+ # This means setup was not called which most likely mean that it is
+ # called through the if __name__ == '__main__' directive.
+ # In this case, we can only pray that it is called through a
+ # function called main() in setup.py.
+ setup.main([]) # Will raise AttributeError if not found
+ self.setup_metadata = self.setup_args[self.metadata_name]
+ # Here we must remove the module the hard way.
+ # We must do this because of a very sepcific case: if a package calls
+ # setup from the __main__ but does not come with a 'main()' function,
+ # for some reason setup.main([]) will successfully call the main
+ # function of a previous package...
+ sys.modules.pop('setup',None)
+ del setup
+ os.chdir(current_dir)
+ sys.path.remove(self.tmp_extract)
+
+ def get_requirements(self, pkg_folder):
+ """
+ Retrieve dependencies from the metadata found in the setup.py script of
+ a pypi package.
+
+ Keyword Arguments:
+ pkg_folder -- location of the already created packages
+ """
+ if 'install_requires' not in self.setup_metadata:
+ self.pkg_req = None
+ return set()
+ self.pkg_req = self.setup_metadata['install_requires']
+ self.pkg_req = [re.sub('([-.\w]+).*', r'\1', req)
+ for req in self.pkg_req]
+ req_not_found = self.pkg_req
+ self.pkg_req = map(pkg_buildroot_name, self.pkg_req)
+ pkg_tuples = zip(req_not_found, self.pkg_req)
+ # pkg_tuples is a list of tuples that looks like
+ # ('werkzeug','python-werkzeug') because I need both when checking if
+ # dependencies already exist or are already in the download list
+ req_not_found = set(
+ pkg[0] for pkg in pkg_tuples
+ if not os.path.isdir(pkg[1])
+ )
+ return req_not_found
+
+ def __create_mk_header(self):
+ """
+ Create the header of the <package_name>.mk file
+ """
+ header = ['#' * 80 + '\n']
+ header.append('#\n')
+ header.append('# {name}\n'.format(name=self.buildroot_name))
+ header.append('#\n')
+ header.append('#' * 80 + '\n')
+ header.append('\n')
+ return header
+
+ def __create_mk_download_info(self):
+ """
+ Create the lines refering to the download information of the
+ <package_name>.mk file
+ """
+ lines = []
+ version_line = '{name}_VERSION = {version}\n'.format(
+ name=self.mk_name,
+ version=self.version)
+ lines.append(version_line)
+
+ targz = self.filename.replace(
+ self.version,
+ '$({name}_VERSION)'.format(name=self.mk_name))
+ targz_line = '{name}_SOURCE = {filename}\n'.format(
+ name=self.mk_name,
+ filename=targz)
+ lines.append(targz_line)
+
+ if self.filename not in self.url:
+ # Sometimes the filename is in the url, sometimes it's not
+ site_url = self.url
+ else:
+ site_url = self.url[:self.url.find(self.filename)]
+ site_line = '{name}_SITE = {url}'.format(name=self.mk_name,
+ url=site_url)
+ site_line = site_line.rstrip('/') + '\n'
+ lines.append(site_line)
+ return lines
+
+ def __create_mk_setup(self):
+ """
+ Create the line refering to the setup method of the package of the
+ <package_name>.mk file
+
+ There are two things you can use to make an installer
+ for a python package: distutils or setuptools
+ distutils comes with python but does not support dependencies.
+ distutils is mostly still there for backward support.
+ setuptools is what smart people use,
+ but it is not shipped with python :(
+ """
+ lines = []
+ setup_type_line = '{name}_SETUP_TYPE = {method}\n'.format(
+ name=self.mk_name,
+ method=self.setup_metadata['method'])
+ lines.append(setup_type_line)
+ return lines
+
+ def __create_mk_license(self):
+ """
+ Create the lines referring to the package's license informations of the
+ <package_name>.mk file
+
+ The license is found using the metadata from pypi.
+ In the metadata, the license can be found either with standard names in
+ the classifiers part or with naming from the packager in the "License"
+ part.
+
+ From the classifiers, the license is "translated" according to
+ buildroot standards if need be (i.e. from Apache Software License to
+ Apache-2.0).
+
+ From the License part, we cannot guess what formatting the packager
+ used. Hence, it is likely to be incorrect. (i.e. Apache License 2.0
+ instead of Apache-2.0).
+
+ The license's files are found by searching the package for files named
+ license or license.txt (case insensitive).
+ If more than one license file is found, the user is asked to select
+ which ones he wants to use.
+ """
+ license_dict = {
+ 'Apache Software License': 'Apache-2.0',
+ 'BSD License': 'BSD',
+ 'European Union Public Licence 1.0': 'EUPLv1.0',
+ 'European Union Public Licence 1.1': 'EUPLv1.1',
+ "GNU General Public License": "GPL",
+ "GNU General Public License v2": "GPLv2",
+ "GNU General Public License v2 or later": "GPLv2+",
+ "GNU General Public License v3": "GPLv3",
+ "GNU General Public License v3 or later": "GPLv3+",
+ "GNU Lesser General Public License v2": "LGPLv2.1",
+ "GNU Lesser General Public License v2 or later": "LGPLv2.1+",
+ "GNU Lesser General Public License v3": "LGPLv3",
+ "GNU Lesser General Public License v3 or later": "LGPLv3+",
+ "GNU Library or Lesser General Public License": "LGPLv2",
+ "ISC License": "ISC",
+ "MIT License": "MIT",
+ "Mozilla Public License 1.0": "MPL-1.0",
+ "Mozilla Public License 1.1": "MPL-1.1",
+ "Mozilla Public License 2.0": "MPL-2.0",
+ "Zope Public License": "ZPL"
+ }
+ regexp = re.compile('^License :* *.* *:+ (.*)( \(.*\))?$')
+ classifiers_licenses = [regexp.sub(r"\1", lic)
+ for lic in self.metadata['info']['classifiers']
+ if regexp.match(lic)]
+ licenses = map(lambda x: license_dict[x] if x in license_dict else x,
+ classifiers_licenses)
+ lines = []
+ if not len(licenses):
+ print('WARNING: License has been set to "{license}". It is most'
+ ' likely wrong, please change it if need be'.format(
+ license=', '.join(licenses)))
+ licenses = [self.metadata['info']['license']]
+ license_line = '{name}_LICENSE = {license}\n'.format(
+ name=self.mk_name,
+ license=', '.join(licenses))
+ lines.append(license_line)
+
+ filenames = ['LICENSE', 'LICENSE.TXT', 'COPYING', 'COPYING.TXT']
+ license_files = list(find_file_upper_case(filenames, self.tmp_extract))
+ license_files = [license.replace(self.tmp_extract, '')[1:]
+ for license in license_files]
+ if len(license_files) > 0:
+ if len(license_files) > 1:
+ print('More than one file found for license:',
+ ', '.join(license_files))
+ license_files = [filename
+ for index, filename in enumerate(license_files)]
+ license_file_line = ('{name}_LICENSE_FILES ='
+ ' {files}\n'.format(
+ name=self.mk_name,
+ files=' '.join(license_files)))
+ lines.append(license_file_line)
+ else:
+ print('WARNING: No license file found,'
+ ' please specify it manually afterwards')
+ license_file_line = '# No license file found\n'
+
+ return lines
+
+ def __create_mk_requirements(self):
+ """
+ Create the lines referring to the dependencies of the of the
+ <package_name>.mk file
+
+ Keyword Arguments:
+ pkg_name -- name of the package
+ pkg_req -- dependencies of the package
+ """
+ lines = []
+ dependencies_line = ('{name}_DEPENDENCIES ='
+ ' {reqs}\n'.format(
+ name=self.mk_name,
+ reqs=' '.join(self.pkg_req)))
+ lines.append(dependencies_line)
+ return lines
+
+ def create_package_mk(self):
+ """
+ Create the lines corresponding to the <package_name>.mk file
+ """
+ pkg_mk = '{name}.mk'.format(name=self.buildroot_name)
+ path_to_mk = os.path.join(self.pkg_dir, pkg_mk)
+ print('Creating {file}...'.format(file=path_to_mk))
+ lines = self.__create_mk_header()
+ lines += self.__create_mk_download_info()
+ lines += self.__create_mk_setup()
+ lines += self.__create_mk_license()
+ if self.pkg_req:
+ lines += self.__create_mk_requirements()
+
+ lines.append('\n')
+ lines.append('$(eval $(python-package))')
+ lines.append('\n')
+ with open(path_to_mk, 'w') as mk_file:
+ mk_file.writelines(lines)
+
+ def create_hash_file(self):
+ """
+ Create the lines corresponding to the <package_name>.hash files
+ """
+ pkg_hash = '{name}.hash'.format(name=self.buildroot_name)
+ path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
+ print('Creating {filename}...'.format(filename=path_to_hash))
+ lines = []
+ if self.used_url['md5_digest']:
+ md5_comment = '# md5 from {url}\n'.format(url=self.metadata_url)
+ lines.append(md5_comment)
+ hash_line = '{method}\t{digest} {filename}\n'.format(
+ method='md5',
+ digest=self.used_url['md5_digest'],
+ filename=self.filename)
+ lines.append(hash_line)
+ sha256_comment = '# sha256 calculated by scanpypi\n'
+ lines.append(sha256_comment)
+ digest = hashlib.sha256(self.as_string).hexdigest()
+ hash_line = '{method}\t{digest} {filename}\n'.format(
+ method='sha256',
+ digest=digest,
+ filename=self.filename)
+ lines.append(hash_line)
+
+ with open(path_to_hash, 'w') as hash_file:
+ hash_file.writelines(lines)
+
+ def create_config_in(self):
+ """
+ Creates the Config.in file of a package
+ """
+ path_to_config = os.path.join(self.pkg_dir, 'Config.in')
+ print('Creating {file}...'.format(file=path_to_config))
+ lines = []
+ config_line = 'config BR2_PACKAGE_{name}\n'.format(
+ name=self.mk_name)
+ lines.append(config_line)
+
+ bool_line = '\tbool "{name}"\n'.format(name=self.buildroot_name)
+ lines.append(bool_line)
+ if self.pkg_req:
+ for dep in self.pkg_req:
+ dep_line = '\tselect BR2_PACKAGE_{req}\n'.format(
+ req=dep.upper().replace('-', '_'))
+ lines.append(dep_line)
+
+ lines.append('\thelp\n')
+
+ help_lines = textwrap.wrap(self.metadata['info']['summary'],
+ initial_indent='\t ',
+ subsequent_indent='\t ')
+ # \t + two spaces is 3 char long
+ help_lines.append('')
+ help_lines.append('\t ' + self.metadata['info']['home_page'])
+ help_lines = map(lambda x: x + '\n', help_lines)
+ lines += help_lines
+
+ with open(path_to_config, 'w') as config_file:
+ config_file.writelines(lines)
+
+
+def main():
+ # Building the parser
+ parser = argparse.ArgumentParser(
+ description="Creates buildroot packages from the metadata of "
+ "an existing PyPI packages and include it "
+ "in menuconfig")
+ parser.add_argument("packages",
+ help="list of packages to be created",
+ nargs='+')
+ parser.add_argument("-o", "--output",
+ help="""
+ Output directory for packages.
+ Default is ./package
+ """,
+ default='./package')
+
+ args = parser.parse_args()
+ packages = list(set(args.packages))
+
+ # tmp_path is where we'll extract the files later
+ tmp_prefix = 'scanpypi-'
+ pkg_folder = args.output
+ tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
+ try:
+ for real_pkg_name in packages:
+ package = BuildrootPackage(real_pkg_name, pkg_folder)
+ print('buildroot package name for {}:'.format(package.real_name),
+ package.buildroot_name)
+ # First we download the package
+ # Most of the info we need can only be found inside the package
+ print('Package:', package.buildroot_name)
+ print('Fetching package', package.real_name)
+ try:
+ package.fetch_package_info()
+ except (urllib2.URLError, urllib2.HTTPError):
+ continue
+ if package.metadata_name.lower() == 'setuptools':
+ # setuptools imports itself, that does not work very well
+ # with the monkey path at the begining
+ print('Error: setuptools cannot be built using scanPyPI')
+ continue
+
+ try:
+ package.download_package()
+ except urllib2.HTTPError as error:
+ print('Error: {code} {reason}'.format(code=error.code,
+ reason=error.reason))
+ print('Error downloading package :', package.buildroot_name)
+ print()
+ continue
+
+ # extract the tarball
+ try:
+ package.extract_package(tmp_path)
+ except (tarfile.ReadError, zipfile.BadZipfile):
+ print('Error extracting package {}'.format(package.real_name))
+ print()
+ continue
+
+ # Loading the package install info from the package
+ try:
+ package.load_setup()
+ except ImportError as err:
+ if 'buildutils' in err.message:
+ print('This package needs buildutils')
+ else:
+ raise
+ continue
+ except AttributeError:
+ print('Error: Could not install package {pkg}'.format(
+ pkg=package.real_name))
+ continue
+
+ # Package requirement are an argument of the setup function
+ req_not_found = package.get_requirements(pkg_folder)
+ req_not_found = req_not_found.difference(packages)
+
+ packages += req_not_found
+ if req_not_found:
+ print('Added packages \'{pkgs}\' as dependencies of {pkg}'
+ .format(pkgs=", ".join(req_not_found),
+ pkg=package.buildroot_name))
+ print('Checking if package {name} already exists...'.format(
+ name=package.pkg_dir))
+ try:
+ os.makedirs(package.pkg_dir)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ continue
+ print('Error: Package {name} already exists'
+ .format(name=package.pkg_dir))
+ del_pkg = raw_input(
+ 'Do you want to delete existing package ? [y/N]')
+ if del_pkg.lower() == 'y':
+ shutil.rmtree(package.pkg_dir)
+ os.makedirs(package.pkg_dir)
+ else:
+ continue
+ package.create_package_mk()
+
+ package.create_hash_file()
+
+ package.create_config_in()
+ print()
+ # printing an empty line for visual confort
+ finally:
+ shutil.rmtree(tmp_path)
+
+if __name__ == "__main__":
+ main()
--
2.4.6
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 2/2] python-robotframework: New package
2015-07-28 13:15 [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
@ 2015-07-28 13:15 ` Denis THULIN
2015-08-31 15:58 ` [Buildroot] [PATCH 1/2] scanpypi: new utility Denis Thulin
` (2 subsequent siblings)
3 siblings, 0 replies; 14+ messages in thread
From: Denis THULIN @ 2015-07-28 13:15 UTC (permalink / raw)
To: buildroot
A generic test automation framework written in python.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0:
- Generated using python-package-generator.py
- Dependencies on python-zlib and python-pyexpat were added manually
- Modification of packages/Config.in done manually.
v1:
- fixed indentation error on dependencies (was 4 spaces instead of tab)
- Changed hash from md5 to sha256 (modification of scanpypi)
- Fixed typos (modification of scanpypi)
v2:
- Added the md5 hash from pypi
- Fixed typos
v3:
- Changed the hash file
---
package/Config.in | 1 +
package/python-robotframework/Config.in | 9 +++++++++
package/python-robotframework/python-robotframework.hash | 4 ++++
package/python-robotframework/python-robotframework.mk | 14 ++++++++++++++
4 files changed, 28 insertions(+)
create mode 100644 package/python-robotframework/Config.in
create mode 100644 package/python-robotframework/python-robotframework.hash
create mode 100644 package/python-robotframework/python-robotframework.mk
diff --git a/package/Config.in b/package/Config.in
index 1e39c74..84e7f43 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -655,6 +655,7 @@ menu "external python modules"
source "package/python-pyyaml/Config.in"
source "package/python-pyzmq/Config.in"
source "package/python-requests/Config.in"
+ source "package/python-robotframework/Config.in"
source "package/python-rtslib-fb/Config.in"
source "package/python-serial/Config.in"
source "package/python-setuptools/Config.in"
diff --git a/package/python-robotframework/Config.in b/package/python-robotframework/Config.in
new file mode 100644
index 0000000..9523893
--- /dev/null
+++ b/package/python-robotframework/Config.in
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+ bool "python-robotframework"
+ depends on BR2_PACKAGE_PYTHON
+ select BR2_PACKAGE_PYTHON_ZLIB
+ select BR2_PACKAGE_PYTHON_PYEXPAT
+ help
+ A generic test automation framework
+
+ http://robotframework.org
diff --git a/package/python-robotframework/python-robotframework.hash b/package/python-robotframework/python-robotframework.hash
new file mode 100644
index 0000000..60c21a9
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.hash
@@ -0,0 +1,4 @@
+# md5 from https://pypi.python.org/pypi/robotframework/json
+md5 85f5d20f5ef8cb7fe6f102985b209bf6 robotframework-2.9rc1.tar.gz
+# sha256 calculated by scanpypi
+sha256 4786e7d31822e9dd8427098ea2cd38068b69af46fb29c0337193cde41ef5d78f robotframework-2.9rc1.tar.gz
diff --git a/package/python-robotframework/python-robotframework.mk b/package/python-robotframework/python-robotframework.mk
new file mode 100644
index 0000000..ba948d2
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# python-robotframework
+#
+################################################################################
+
+PYTHON_ROBOTFRAMEWORK_VERSION = 2.9rc1
+PYTHON_ROBOTFRAMEWORK_SOURCE = robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
+PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework
+PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = distutils
+PYTHON_ROBOTFRAMEWORK_LICENSE = Apache-2.0
+PYTHON_ROBOTFRAMEWORK_LICENSE_FILES = LICENSE.txt
+
+$(eval $(python-package))
--
2.4.6
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2015-07-28 13:15 [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
2015-07-28 13:15 ` [Buildroot] [PATCH 2/2] python-robotframework: New package Denis THULIN
@ 2015-08-31 15:58 ` Denis Thulin
2016-01-10 10:59 ` Yann E. MORIN
2016-03-01 1:44 ` Carlos Santos
3 siblings, 0 replies; 14+ messages in thread
From: Denis Thulin @ 2015-08-31 15:58 UTC (permalink / raw)
To: buildroot
Hi all,
There was no comments on that patch, so I'm assuming it went unnoticed.
If there is anything wrong or not working, just let me know.
Have a good day.
Denis.
----- Le 28 Juil 15, ? 15:15, Denis THULIN denis.thulin at openwide.fr a ?crit :
> An utility for creating python package from the python package index
> It fetches packages info from http://pypi.python.org and generates
> corresponding packages files.
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> ---
> v0: initial commit
> python-pacakage-generator.py is an utility for automatically generating a
> python package. It fetches packages info from http://pypi.python.org and
> generates corresponding packages files.
>
> v1:
> - renamed python-package-generator to scanpypi
> - split the huge script into a lot of functions
> - fixed mistakes and small bugs
>
> v2:
> - Rewrited most of the functions into a class
> - Changed the method for importing setup.py
> - Created a main function to avoid use of global variable
> - Now adds new dependencies to the list of packages to create
> - Droppped the .py extension
>
> v3:
> - Fixed bugs on import setup (Relative import works again)
> - Can handle packages as zipfile
> - Now avoids bdist packages
> - Changed behaviour for packages that are not hosted on PyPI
> - Added various clarifications of the code
> - Works with: flask django robotframework pyxml pyzmq Twisted Six
> - Does not work with: setuptools
> ---
> docs/manual/adding-packages-python.txt | 43 +++
> support/scripts/scanpypi | 653 +++++++++++++++++++++++++++++++++
> 2 files changed, 696 insertions(+)
> create mode 100755 support/scripts/scanpypi
>
> diff --git a/docs/manual/adding-packages-python.txt
> b/docs/manual/adding-packages-python.txt
> index 588dbf8..1f160ca 100644
> --- a/docs/manual/adding-packages-python.txt
> +++ b/docs/manual/adding-packages-python.txt
> @@ -7,6 +7,49 @@ This infrastructure applies to Python packages that use the
> standard
> Python setuptools mechanism as their build system, generally
> recognizable by the usage of a +setup.py+ script.
>
> +[[scanpypi]]
> +
> +==== Generating a +python-package+ from a PyPI repository
> +
> +You may want to use the +scanpypi+ located in +support/script+ to generate a
> +package from an existing PyPI package.
> +
> +you can find the list of existing PyPI package https://pypi.python.org[here].
> +
> +Please keep in mind that you most likely need to manually check the package for
> +any mistakes as there are things that cannot be guessed by the generator (e.g.
> +dependencies on any of the python core modules such as
> BR2_PACKAGE_PYTHON_ZLIB).
> +Also, please take note that the license and license files are guessed and must
> +be checked. You also need to manually add the package to the
> +package/Config.in+
> +file.
> +
> +When at the root of your buildroot directory just do :
> +
> +-----------------------
> +./support/script/scanpypi foo bar -o package
> +-----------------------
> +
> +This will generate packages +python-foo+ and +python-bar+ in the package
> +folder if they exist on https://pypi.python.org.
> +
> +Find the +external python modules+ menu and insert your package inside.
> +Keep in mind that the items inside a menu should be in alphabetical order.
> +
> +If your package is external, use the -o flag.
> +
> +-----------------------
> +./support/script/scanpypi foo bar -o other_package_dir
> +-----------------------
> +
> +This will generate packages +python-foo+ and +python-bar+ in the
> ++other_package_directory+ instead of +package+.
> +
> +Option +-h+ wil list the options available
> +
> +-----------------------
> +./support/script/scanpypi -h
> +-----------------------
> +
> [[python-package-tutorial]]
>
> ==== +python-package+ tutorial
> diff --git a/support/scripts/scanpypi b/support/scripts/scanpypi
> new file mode 100755
> index 0000000..3e51bae
> --- /dev/null
> +++ b/support/scripts/scanpypi
> @@ -0,0 +1,653 @@
> +#!/usr/bin/python2
> +"""
> +Utility for building buildroot packages for existing pypi packages
> +
> +Any package built by scanpypi should be manually checked for
> +errors.
> +"""
> +from __future__ import print_function
> +import argparse
> +import json
> +import urllib2
> +import sys
> +import os
> +import shutil
> +import StringIO
> +import tarfile
> +import zipfile
> +import errno
> +import hashlib
> +import re
> +import textwrap
> +import tempfile
> +import imp
> +from functools import wraps
> +
> +
> +def setup_decorator(func, method):
> + """
> + Decorator for distutils.core.setup and setuptools.setup.
> + Puts the arguments with which setup is called as a dict
> + Add key 'method' which should be either 'setuptools' or 'distutils'.
> +
> + Keyword arguments:
> + func -- either setuptools.setup or distutils.core.setup
> + method -- either 'setuptools' or 'distutils'
> + """
> +
> + @wraps(func)
> + def closure(*args, **kwargs):
> + # Any python packages calls its setup function to be installed.
> + # Argument 'name' of this setup function is the package's name
> + BuildrootPackage.setup_args[kwargs['name']] = kwargs
> + BuildrootPackage.setup_args[kwargs['name']]['method'] = method
> + return closure
> +
> +
> +# monkey patch
> +import setuptools
> +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> +import distutils
> +distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
> +
> +
> +def find_file_upper_case(filenames, path='./'):
> + """
> + List generator:
> + Recursively find files that matches one of the specified filenames.
> + Returns a relative path starting with path argument.
> +
> + Keyword arguments:
> + filenames -- List of filenames to be found
> + path -- Path to the directory to search
> + """
> + for root, dirs, files in os.walk(path):
> + for file in files:
> + if file.upper() in filenames:
> + yield (os.path.join(root, file))
> +
> +
> +def pkg_buildroot_name(pkg_name):
> + """
> + Returns the buildroot package name for the PyPI package pkg_name.
> + Remove all non alphanumeric characters except -
> + Also lowers the name and adds 'python-' suffix
> +
> + Keyword arguments:
> + pkg_name -- String to rename
> + """
> + name = re.sub('[^\w-]', '', pkg_name.lower())
> + prefix = 'python-'
> + pattern = re.compile('^(?!' + prefix + ')(.+?)$')
> + name = pattern.sub(r'python-\1', name)
> + return name
> +
> +
> +class DownloadFailed(Exception):
> + pass
> +
> +
> +class BuildrootPackage():
> + """
> + This class's methods are not meant to be used individually please use those
> + in the correct order:
> + __init__
> +
> + download_package
> +
> + extract_package
> +
> + load_module
> +
> + get_requirements
> +
> + create_package_mk
> +
> + create_hash_file
> +
> + create_config_in
> + """
> + setup_args = {}
> +
> + def __init__(self, real_name, pkg_folder):
> + self.real_name = real_name
> + self.buildroot_name = pkg_buildroot_name(self.real_name)
> + self.pkg_dir = os.path.join(pkg_folder, self.buildroot_name)
> + self.mk_name = self.buildroot_name.upper().replace('-', '_')
> + self.as_string = None
> + self.md5_sum = None
> + self.metadata = None
> + self.metadata_name = None
> + self.metadata_url = None
> + self.pkg_req = None
> + self.setup_metadata = None
> + self.tmp_extract = None
> + self.used_url = None
> + self.filename = None
> + self.url = None
> + self.version = None
> +
> + def fetch_package_info(self):
> + """
> + Fetch a package's metadata from the python package index
> + """
> + self.metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> + pkg=self.real_name)
> + try:
> + pkg_json = urllib2.urlopen(self.metadata_url).read().decode()
> + except urllib2.HTTPError as error:
> + print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '
> + .format(pkg=self.real_name))
> + raise
> + except urllib2.URLError:
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '
> + .format(pkg=self.real_name))
> + raise
> + self.metadata = json.loads(pkg_json)
> + self.version = self.metadata['info']['version']
> + self.metadata_name = self.metadata['info']['name']
> +
> + def download_package(self):
> + """
> + Download a package using metadata from pypi
> + """
> + try:
> + self.metadata['urls'][0]['filename']
> + except IndexError:
> + print(
> + 'Non-conventional package, ',
> + 'please check carefully after creation')
> + self.metadata['urls'] = [{
> + 'packagetype': 'sdist',
> + 'url': self.metadata['info']['download_url'],
> + 'md5_digest': None}]
> + # In this case, we can't get the name of the downloaded file
> + # from the pypi api, so we need to find it, this should work
> + urlpath = urllib2.urlparse.urlparse(
> + self.metadata['info']['download_url']).path
> + # urlparse().path give something like
> + # /path/to/file-version.tar.gz
> + # We use basename to remove /path/to
> + self.metadata['urls'][0]['filename'] = os.path.basename(urlpath)
> + for download_url in self.metadata['urls']:
> + if 'bdist' in download_url['packagetype']:
> + continue
> + try:
> + print('Downloading package {pkg} from {url}...'.format(
> + pkg=self.real_name, url=download_url['url']))
> + download = urllib2.urlopen(download_url['url'])
> + except urllib2.HTTPError as http_error:
> + download = http_error
> + else:
> + self.used_url = download_url
> + self.as_string = download.read()
> + if not download_url['md5_digest']:
> + break
> + self.md5_sum = hashlib.md5(self.as_string).hexdigest()
> + if self.md5_sum == download_url['md5_digest']:
> + break
> + else:
> + if download.__class__ == urllib2.HTTPError:
> + raise download
> + raise DownloadFailed('Failed to downloas package {pkg}'
> + .format(pkg=self.real_name))
> + self.filename = self.used_url['filename']
> + self.url = self.used_url['url']
> +
> + def extract_package(self, tmp_path):
> + """
> + Extract the package contents into a directrory
> +
> + Keyword arguments:
> + tmp_path -- directory where you want the package to be extracted
> + """
> + as_file = StringIO.StringIO(self.as_string)
> + if self.filename[-3:] == 'zip':
> + with zipfile.open(fileobj=as_file) as as_zipfile:
> + tmp_pkg = os.path.join(tmp_path, self.buildroot_name)
> + try:
> + os.makedirs(tmp_pkg)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + return None, None
> + print('WARNING:', exception.message, file=sys.stderr)
> + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> + shutil.rmtree(tmp_pkg)
> + os.makedirs(tmp_pkg)
> + as_zipfile.extractall(tmp_pkg)
> + else:
> + with tarfile.open(fileobj=as_file) as as_tarfile:
> + tmp_pkg = os.path.join(tmp_path, self.buildroot_name)
> + try:
> + os.makedirs(tmp_pkg)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + return None, None
> + print('WARNING:', exception.message, file=sys.stderr)
> + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> + shutil.rmtree(tmp_pkg)
> + os.makedirs(tmp_pkg)
> + as_tarfile.extractall(tmp_pkg)
> +
> + tmp_extract = '{folder}/{name}-{version}'
> + self.tmp_extract = tmp_extract.format(
> + folder=tmp_pkg,
> + name=self.metadata_name,
> + version=self.version)
> +
> + def load_setup(self):
> + """
> + Loads the corresponding setup and store its metadata
> + """
> + current_dir = os.getcwd()
> + os.chdir(self.tmp_extract)
> + sys.path.append(self.tmp_extract)
> + s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract])
> + setup = imp.load_module('setup', s_file, s_path, s_desc)
> + try:
> + self.setup_metadata = self.setup_args[self.metadata_name]
> + except KeyError:
> + # This means setup was not called which most likely mean that it is
> + # called through the if __name__ == '__main__' directive.
> + # In this case, we can only pray that it is called through a
> + # function called main() in setup.py.
> + setup.main([]) # Will raise AttributeError if not found
> + self.setup_metadata = self.setup_args[self.metadata_name]
> + # Here we must remove the module the hard way.
> + # We must do this because of a very sepcific case: if a package calls
> + # setup from the __main__ but does not come with a 'main()' function,
> + # for some reason setup.main([]) will successfully call the main
> + # function of a previous package...
> + sys.modules.pop('setup',None)
> + del setup
> + os.chdir(current_dir)
> + sys.path.remove(self.tmp_extract)
> +
> + def get_requirements(self, pkg_folder):
> + """
> + Retrieve dependencies from the metadata found in the setup.py script of
> + a pypi package.
> +
> + Keyword Arguments:
> + pkg_folder -- location of the already created packages
> + """
> + if 'install_requires' not in self.setup_metadata:
> + self.pkg_req = None
> + return set()
> + self.pkg_req = self.setup_metadata['install_requires']
> + self.pkg_req = [re.sub('([-.\w]+).*', r'\1', req)
> + for req in self.pkg_req]
> + req_not_found = self.pkg_req
> + self.pkg_req = map(pkg_buildroot_name, self.pkg_req)
> + pkg_tuples = zip(req_not_found, self.pkg_req)
> + # pkg_tuples is a list of tuples that looks like
> + # ('werkzeug','python-werkzeug') because I need both when checking if
> + # dependencies already exist or are already in the download list
> + req_not_found = set(
> + pkg[0] for pkg in pkg_tuples
> + if not os.path.isdir(pkg[1])
> + )
> + return req_not_found
> +
> + def __create_mk_header(self):
> + """
> + Create the header of the <package_name>.mk file
> + """
> + header = ['#' * 80 + '\n']
> + header.append('#\n')
> + header.append('# {name}\n'.format(name=self.buildroot_name))
> + header.append('#\n')
> + header.append('#' * 80 + '\n')
> + header.append('\n')
> + return header
> +
> + def __create_mk_download_info(self):
> + """
> + Create the lines refering to the download information of the
> + <package_name>.mk file
> + """
> + lines = []
> + version_line = '{name}_VERSION = {version}\n'.format(
> + name=self.mk_name,
> + version=self.version)
> + lines.append(version_line)
> +
> + targz = self.filename.replace(
> + self.version,
> + '$({name}_VERSION)'.format(name=self.mk_name))
> + targz_line = '{name}_SOURCE = {filename}\n'.format(
> + name=self.mk_name,
> + filename=targz)
> + lines.append(targz_line)
> +
> + if self.filename not in self.url:
> + # Sometimes the filename is in the url, sometimes it's not
> + site_url = self.url
> + else:
> + site_url = self.url[:self.url.find(self.filename)]
> + site_line = '{name}_SITE = {url}'.format(name=self.mk_name,
> + url=site_url)
> + site_line = site_line.rstrip('/') + '\n'
> + lines.append(site_line)
> + return lines
> +
> + def __create_mk_setup(self):
> + """
> + Create the line refering to the setup method of the package of the
> + <package_name>.mk file
> +
> + There are two things you can use to make an installer
> + for a python package: distutils or setuptools
> + distutils comes with python but does not support dependencies.
> + distutils is mostly still there for backward support.
> + setuptools is what smart people use,
> + but it is not shipped with python :(
> + """
> + lines = []
> + setup_type_line = '{name}_SETUP_TYPE = {method}\n'.format(
> + name=self.mk_name,
> + method=self.setup_metadata['method'])
> + lines.append(setup_type_line)
> + return lines
> +
> + def __create_mk_license(self):
> + """
> + Create the lines referring to the package's license informations of the
> + <package_name>.mk file
> +
> + The license is found using the metadata from pypi.
> + In the metadata, the license can be found either with standard names in
> + the classifiers part or with naming from the packager in the "License"
> + part.
> +
> + From the classifiers, the license is "translated" according to
> + buildroot standards if need be (i.e. from Apache Software License to
> + Apache-2.0).
> +
> + From the License part, we cannot guess what formatting the packager
> + used. Hence, it is likely to be incorrect. (i.e. Apache License 2.0
> + instead of Apache-2.0).
> +
> + The license's files are found by searching the package for files named
> + license or license.txt (case insensitive).
> + If more than one license file is found, the user is asked to select
> + which ones he wants to use.
> + """
> + license_dict = {
> + 'Apache Software License': 'Apache-2.0',
> + 'BSD License': 'BSD',
> + 'European Union Public Licence 1.0': 'EUPLv1.0',
> + 'European Union Public Licence 1.1': 'EUPLv1.1',
> + "GNU General Public License": "GPL",
> + "GNU General Public License v2": "GPLv2",
> + "GNU General Public License v2 or later": "GPLv2+",
> + "GNU General Public License v3": "GPLv3",
> + "GNU General Public License v3 or later": "GPLv3+",
> + "GNU Lesser General Public License v2": "LGPLv2.1",
> + "GNU Lesser General Public License v2 or later": "LGPLv2.1+",
> + "GNU Lesser General Public License v3": "LGPLv3",
> + "GNU Lesser General Public License v3 or later": "LGPLv3+",
> + "GNU Library or Lesser General Public License": "LGPLv2",
> + "ISC License": "ISC",
> + "MIT License": "MIT",
> + "Mozilla Public License 1.0": "MPL-1.0",
> + "Mozilla Public License 1.1": "MPL-1.1",
> + "Mozilla Public License 2.0": "MPL-2.0",
> + "Zope Public License": "ZPL"
> + }
> + regexp = re.compile('^License :* *.* *:+ (.*)( \(.*\))?$')
> + classifiers_licenses = [regexp.sub(r"\1", lic)
> + for lic in self.metadata['info']['classifiers']
> + if regexp.match(lic)]
> + licenses = map(lambda x: license_dict[x] if x in license_dict else x,
> + classifiers_licenses)
> + lines = []
> + if not len(licenses):
> + print('WARNING: License has been set to "{license}". It is most'
> + ' likely wrong, please change it if need be'.format(
> + license=', '.join(licenses)))
> + licenses = [self.metadata['info']['license']]
> + license_line = '{name}_LICENSE = {license}\n'.format(
> + name=self.mk_name,
> + license=', '.join(licenses))
> + lines.append(license_line)
> +
> + filenames = ['LICENSE', 'LICENSE.TXT', 'COPYING', 'COPYING.TXT']
> + license_files = list(find_file_upper_case(filenames, self.tmp_extract))
> + license_files = [license.replace(self.tmp_extract, '')[1:]
> + for license in license_files]
> + if len(license_files) > 0:
> + if len(license_files) > 1:
> + print('More than one file found for license:',
> + ', '.join(license_files))
> + license_files = [filename
> + for index, filename in enumerate(license_files)]
> + license_file_line = ('{name}_LICENSE_FILES ='
> + ' {files}\n'.format(
> + name=self.mk_name,
> + files=' '.join(license_files)))
> + lines.append(license_file_line)
> + else:
> + print('WARNING: No license file found,'
> + ' please specify it manually afterwards')
> + license_file_line = '# No license file found\n'
> +
> + return lines
> +
> + def __create_mk_requirements(self):
> + """
> + Create the lines referring to the dependencies of the of the
> + <package_name>.mk file
> +
> + Keyword Arguments:
> + pkg_name -- name of the package
> + pkg_req -- dependencies of the package
> + """
> + lines = []
> + dependencies_line = ('{name}_DEPENDENCIES ='
> + ' {reqs}\n'.format(
> + name=self.mk_name,
> + reqs=' '.join(self.pkg_req)))
> + lines.append(dependencies_line)
> + return lines
> +
> + def create_package_mk(self):
> + """
> + Create the lines corresponding to the <package_name>.mk file
> + """
> + pkg_mk = '{name}.mk'.format(name=self.buildroot_name)
> + path_to_mk = os.path.join(self.pkg_dir, pkg_mk)
> + print('Creating {file}...'.format(file=path_to_mk))
> + lines = self.__create_mk_header()
> + lines += self.__create_mk_download_info()
> + lines += self.__create_mk_setup()
> + lines += self.__create_mk_license()
> + if self.pkg_req:
> + lines += self.__create_mk_requirements()
> +
> + lines.append('\n')
> + lines.append('$(eval $(python-package))')
> + lines.append('\n')
> + with open(path_to_mk, 'w') as mk_file:
> + mk_file.writelines(lines)
> +
> + def create_hash_file(self):
> + """
> + Create the lines corresponding to the <package_name>.hash files
> + """
> + pkg_hash = '{name}.hash'.format(name=self.buildroot_name)
> + path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
> + print('Creating {filename}...'.format(filename=path_to_hash))
> + lines = []
> + if self.used_url['md5_digest']:
> + md5_comment = '# md5 from {url}\n'.format(url=self.metadata_url)
> + lines.append(md5_comment)
> + hash_line = '{method}\t{digest} {filename}\n'.format(
> + method='md5',
> + digest=self.used_url['md5_digest'],
> + filename=self.filename)
> + lines.append(hash_line)
> + sha256_comment = '# sha256 calculated by scanpypi\n'
> + lines.append(sha256_comment)
> + digest = hashlib.sha256(self.as_string).hexdigest()
> + hash_line = '{method}\t{digest} {filename}\n'.format(
> + method='sha256',
> + digest=digest,
> + filename=self.filename)
> + lines.append(hash_line)
> +
> + with open(path_to_hash, 'w') as hash_file:
> + hash_file.writelines(lines)
> +
> + def create_config_in(self):
> + """
> + Creates the Config.in file of a package
> + """
> + path_to_config = os.path.join(self.pkg_dir, 'Config.in')
> + print('Creating {file}...'.format(file=path_to_config))
> + lines = []
> + config_line = 'config BR2_PACKAGE_{name}\n'.format(
> + name=self.mk_name)
> + lines.append(config_line)
> +
> + bool_line = '\tbool "{name}"\n'.format(name=self.buildroot_name)
> + lines.append(bool_line)
> + if self.pkg_req:
> + for dep in self.pkg_req:
> + dep_line = '\tselect BR2_PACKAGE_{req}\n'.format(
> + req=dep.upper().replace('-', '_'))
> + lines.append(dep_line)
> +
> + lines.append('\thelp\n')
> +
> + help_lines = textwrap.wrap(self.metadata['info']['summary'],
> + initial_indent='\t ',
> + subsequent_indent='\t ')
> + # \t + two spaces is 3 char long
> + help_lines.append('')
> + help_lines.append('\t ' + self.metadata['info']['home_page'])
> + help_lines = map(lambda x: x + '\n', help_lines)
> + lines += help_lines
> +
> + with open(path_to_config, 'w') as config_file:
> + config_file.writelines(lines)
> +
> +
> +def main():
> + # Building the parser
> + parser = argparse.ArgumentParser(
> + description="Creates buildroot packages from the metadata of "
> + "an existing PyPI packages and include it "
> + "in menuconfig")
> + parser.add_argument("packages",
> + help="list of packages to be created",
> + nargs='+')
> + parser.add_argument("-o", "--output",
> + help="""
> + Output directory for packages.
> + Default is ./package
> + """,
> + default='./package')
> +
> + args = parser.parse_args()
> + packages = list(set(args.packages))
> +
> + # tmp_path is where we'll extract the files later
> + tmp_prefix = 'scanpypi-'
> + pkg_folder = args.output
> + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
> + try:
> + for real_pkg_name in packages:
> + package = BuildrootPackage(real_pkg_name, pkg_folder)
> + print('buildroot package name for {}:'.format(package.real_name),
> + package.buildroot_name)
> + # First we download the package
> + # Most of the info we need can only be found inside the package
> + print('Package:', package.buildroot_name)
> + print('Fetching package', package.real_name)
> + try:
> + package.fetch_package_info()
> + except (urllib2.URLError, urllib2.HTTPError):
> + continue
> + if package.metadata_name.lower() == 'setuptools':
> + # setuptools imports itself, that does not work very well
> + # with the monkey path at the begining
> + print('Error: setuptools cannot be built using scanPyPI')
> + continue
> +
> + try:
> + package.download_package()
> + except urllib2.HTTPError as error:
> + print('Error: {code} {reason}'.format(code=error.code,
> + reason=error.reason))
> + print('Error downloading package :', package.buildroot_name)
> + print()
> + continue
> +
> + # extract the tarball
> + try:
> + package.extract_package(tmp_path)
> + except (tarfile.ReadError, zipfile.BadZipfile):
> + print('Error extracting package {}'.format(package.real_name))
> + print()
> + continue
> +
> + # Loading the package install info from the package
> + try:
> + package.load_setup()
> + except ImportError as err:
> + if 'buildutils' in err.message:
> + print('This package needs buildutils')
> + else:
> + raise
> + continue
> + except AttributeError:
> + print('Error: Could not install package {pkg}'.format(
> + pkg=package.real_name))
> + continue
> +
> + # Package requirement are an argument of the setup function
> + req_not_found = package.get_requirements(pkg_folder)
> + req_not_found = req_not_found.difference(packages)
> +
> + packages += req_not_found
> + if req_not_found:
> + print('Added packages \'{pkgs}\' as dependencies of {pkg}'
> + .format(pkgs=", ".join(req_not_found),
> + pkg=package.buildroot_name))
> + print('Checking if package {name} already exists...'.format(
> + name=package.pkg_dir))
> + try:
> + os.makedirs(package.pkg_dir)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('Error: Package {name} already exists'
> + .format(name=package.pkg_dir))
> + del_pkg = raw_input(
> + 'Do you want to delete existing package ? [y/N]')
> + if del_pkg.lower() == 'y':
> + shutil.rmtree(package.pkg_dir)
> + os.makedirs(package.pkg_dir)
> + else:
> + continue
> + package.create_package_mk()
> +
> + package.create_hash_file()
> +
> + package.create_config_in()
> + print()
> + # printing an empty line for visual confort
> + finally:
> + shutil.rmtree(tmp_path)
> +
> +if __name__ == "__main__":
> + main()
> --
> 2.4.6
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2015-07-28 13:15 [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
2015-07-28 13:15 ` [Buildroot] [PATCH 2/2] python-robotframework: New package Denis THULIN
2015-08-31 15:58 ` [Buildroot] [PATCH 1/2] scanpypi: new utility Denis Thulin
@ 2016-01-10 10:59 ` Yann E. MORIN
2016-01-10 15:36 ` Arnout Vandecappelle
2016-02-02 18:02 ` Eelco Chaudron
2016-03-01 1:44 ` Carlos Santos
3 siblings, 2 replies; 14+ messages in thread
From: Yann E. MORIN @ 2016-01-10 10:59 UTC (permalink / raw)
To: buildroot
Denis, All,
Sorry for the long delay. I'm now having a look at this patch.
On 2015-07-28 15:15 +0200, Denis THULIN spake thusly:
> An utility for creating python package from the python package index
> It fetches packages info from http://pypi.python.org and generates
> corresponding packages files.
So, we currently have scancpan to create perl packages. You are adding
scanpypi to create Python packages. There's also someone who submitted
a script to generate a 'generic' package (i.e. not perl/python) [0].
The scancpan is written in perl, yours and the generic one in Python.
Besides Perl and Python, we also have nodejs which provides a similar
"package-store" and for which it would become interesting to provide a
helper script to generate packages [1].
What I would love to see is that we have a single script to add
packages. Something like:
$ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
with TYPE being one of the package types we currently have (generic,
autotools... python, perl...) or an abstract type (nodejs...).
Then, the cpan, pypi, nodejs... script would be just 'backends' that
would provide classes called by the main script, like;
pkg = new PythonPkg("foo")
pkg.get_br_name() returns the BR2_PACKAGE_ name
pkg.get_version() returns the _VERSION string
pkg.get_source() returns the _SOURCE string
pkg.get_site() returns the _SITE string
pkg.get_method() returns the _SITE_METHOD string
pkg.get_dependencies() returns the _DEPENDENCIES list
... and so on, you get the idea. ;-)
That would also recursively generate the packages for the dependencies,
if not already present.
Of course, that would mean we'd have to standardise on a single
language. I think Python is the way to go here.
Would you be interested in pursuing this?
[0] https://patchwork.ozlabs.org/patch/523257/
[1] that could also require a nodejs-package infra, but not necessarily.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2016-01-10 10:59 ` Yann E. MORIN
@ 2016-01-10 15:36 ` Arnout Vandecappelle
2016-01-13 15:23 ` Thomas Petazzoni
2016-02-02 18:02 ` Eelco Chaudron
1 sibling, 1 reply; 14+ messages in thread
From: Arnout Vandecappelle @ 2016-01-10 15:36 UTC (permalink / raw)
To: buildroot
On 10-01-16 11:59, Yann E. MORIN wrote:
> Denis, All,
>
> Sorry for the long delay. I'm now having a look at this patch.
>
> On 2015-07-28 15:15 +0200, Denis THULIN spake thusly:
>> An utility for creating python package from the python package index
>> It fetches packages info from http://pypi.python.org and generates
>> corresponding packages files.
>
> So, we currently have scancpan to create perl packages. You are adding
> scanpypi to create Python packages. There's also someone who submitted
> a script to generate a 'generic' package (i.e. not perl/python) [0].
> The scancpan is written in perl, yours and the generic one in Python.
>
> Besides Perl and Python, we also have nodejs which provides a similar
> "package-store" and for which it would become interesting to provide a
> helper script to generate packages [1].
>
> What I would love to see is that we have a single script to add
> packages. Something like:
>
> $ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
>
> with TYPE being one of the package types we currently have (generic,
> autotools... python, perl...) or an abstract type (nodejs...).
>
> Then, the cpan, pypi, nodejs... script would be just 'backends' that
> would provide classes called by the main script, like;
>
> pkg = new PythonPkg("foo")
> pkg.get_br_name() returns the BR2_PACKAGE_ name
> pkg.get_version() returns the _VERSION string
> pkg.get_source() returns the _SOURCE string
> pkg.get_site() returns the _SITE string
> pkg.get_method() returns the _SITE_METHOD string
> pkg.get_dependencies() returns the _DEPENDENCIES list
> ... and so on, you get the idea. ;-)
However, it is more natural for a CPAN-accessor to be written in perl. So I
guess the backend scripts should be really independent scripts that report the
package metadata in a specified format. Hm, you know what, let's use Config.in
and pkg.mk as the specification!
In short, I'm not so convinced that having everything written in the same
language is such an advantage.
But of course, if someone shows me the patches, I could change my mind.
Regards,
Arnout
> That would also recursively generate the packages for the dependencies,
> if not already present.
>
> Of course, that would mean we'd have to standardise on a single
> language. I think Python is the way to go here.
>
> Would you be interested in pursuing this?
>
> [0] https://patchwork.ozlabs.org/patch/523257/
> [1] that could also require a nodejs-package infra, but not necessarily.
>
> Regards,
> Yann E. MORIN.
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2016-01-10 15:36 ` Arnout Vandecappelle
@ 2016-01-13 15:23 ` Thomas Petazzoni
2016-01-14 8:32 ` Yegor Yefremov
0 siblings, 1 reply; 14+ messages in thread
From: Thomas Petazzoni @ 2016-01-13 15:23 UTC (permalink / raw)
To: buildroot
Arnout, Yann,
On Sun, 10 Jan 2016 16:36:50 +0100, Arnout Vandecappelle wrote:
> > So, we currently have scancpan to create perl packages. You are adding
> > scanpypi to create Python packages. There's also someone who submitted
> > a script to generate a 'generic' package (i.e. not perl/python) [0].
> > The scancpan is written in perl, yours and the generic one in Python.
> >
> > Besides Perl and Python, we also have nodejs which provides a similar
> > "package-store" and for which it would become interesting to provide a
> > helper script to generate packages [1].
> >
> > What I would love to see is that we have a single script to add
> > packages. Something like:
> >
> > $ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
> >
> > with TYPE being one of the package types we currently have (generic,
> > autotools... python, perl...) or an abstract type (nodejs...).
> >
> > Then, the cpan, pypi, nodejs... script would be just 'backends' that
> > would provide classes called by the main script, like;
> >
> > pkg = new PythonPkg("foo")
> > pkg.get_br_name() returns the BR2_PACKAGE_ name
> > pkg.get_version() returns the _VERSION string
> > pkg.get_source() returns the _SOURCE string
> > pkg.get_site() returns the _SITE string
> > pkg.get_method() returns the _SITE_METHOD string
> > pkg.get_dependencies() returns the _DEPENDENCIES list
> > ... and so on, you get the idea. ;-)
>
> However, it is more natural for a CPAN-accessor to be written in perl. So I
> guess the backend scripts should be really independent scripts that report the
> package metadata in a specified format. Hm, you know what, let's use Config.in
> and pkg.mk as the specification!
>
> In short, I'm not so convinced that having everything written in the same
> language is such an advantage.
>
> But of course, if someone shows me the patches, I could change my mind.
I also initially would have preferred to have the scancpan script
written in Python. But 1/ it's not very practical to query CPAN from
Python, and more importantly 2/ it's a bit weird to ask to a Perl
fan who maintains the Perl stuff in Buildroot to write something like
scancpan in Python.
Bottom line, my opinion is that:
1/ We should keep scancpan in Perl
2/ We should keep scanpipy in Python, refine it and merge it.
3/ The script that generates just a generic package skeleton is a bit
useless IMO and is not very useful to merge.
Both scancpan and canpipy are really internal tools, for Buildroot
developers, so it's not super important if they don't look like /
behave the same.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2016-01-13 15:23 ` Thomas Petazzoni
@ 2016-01-14 8:32 ` Yegor Yefremov
2016-01-27 13:30 ` Yegor Yefremov
0 siblings, 1 reply; 14+ messages in thread
From: Yegor Yefremov @ 2016-01-14 8:32 UTC (permalink / raw)
To: buildroot
On Wed, Jan 13, 2016 at 4:23 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Arnout, Yann,
>
> On Sun, 10 Jan 2016 16:36:50 +0100, Arnout Vandecappelle wrote:
>
>> > So, we currently have scancpan to create perl packages. You are adding
>> > scanpypi to create Python packages. There's also someone who submitted
>> > a script to generate a 'generic' package (i.e. not perl/python) [0].
>> > The scancpan is written in perl, yours and the generic one in Python.
>> >
>> > Besides Perl and Python, we also have nodejs which provides a similar
>> > "package-store" and for which it would become interesting to provide a
>> > helper script to generate packages [1].
>> >
>> > What I would love to see is that we have a single script to add
>> > packages. Something like:
>> >
>> > $ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
>> >
>> > with TYPE being one of the package types we currently have (generic,
>> > autotools... python, perl...) or an abstract type (nodejs...).
>> >
>> > Then, the cpan, pypi, nodejs... script would be just 'backends' that
>> > would provide classes called by the main script, like;
>> >
>> > pkg = new PythonPkg("foo")
>> > pkg.get_br_name() returns the BR2_PACKAGE_ name
>> > pkg.get_version() returns the _VERSION string
>> > pkg.get_source() returns the _SOURCE string
>> > pkg.get_site() returns the _SITE string
>> > pkg.get_method() returns the _SITE_METHOD string
>> > pkg.get_dependencies() returns the _DEPENDENCIES list
>> > ... and so on, you get the idea. ;-)
>>
>> However, it is more natural for a CPAN-accessor to be written in perl. So I
>> guess the backend scripts should be really independent scripts that report the
>> package metadata in a specified format. Hm, you know what, let's use Config.in
>> and pkg.mk as the specification!
>>
>> In short, I'm not so convinced that having everything written in the same
>> language is such an advantage.
>>
>> But of course, if someone shows me the patches, I could change my mind.
>
> I also initially would have preferred to have the scancpan script
> written in Python. But 1/ it's not very practical to query CPAN from
> Python, and more importantly 2/ it's a bit weird to ask to a Perl
> fan who maintains the Perl stuff in Buildroot to write something like
> scancpan in Python.
>
> Bottom line, my opinion is that:
>
> 1/ We should keep scancpan in Perl
>
> 2/ We should keep scanpipy in Python, refine it and merge it.
>
> 3/ The script that generates just a generic package skeleton is a bit
> useless IMO and is not very useful to merge.
>
> Both scancpan and canpipy are really internal tools, for Buildroot
> developers, so it's not super important if they don't look like /
> behave the same.
I gave scanpypi a try: http://patchwork.ozlabs.org/patch/567288/
So far it was working as expected. Thanks for the very useful tool.
Yegor
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2016-01-14 8:32 ` Yegor Yefremov
@ 2016-01-27 13:30 ` Yegor Yefremov
0 siblings, 0 replies; 14+ messages in thread
From: Yegor Yefremov @ 2016-01-27 13:30 UTC (permalink / raw)
To: buildroot
On Thu, Jan 14, 2016 at 9:32 AM, Yegor Yefremov
<yegorslists@googlemail.com> wrote:
> On Wed, Jan 13, 2016 at 4:23 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Arnout, Yann,
>>
>> On Sun, 10 Jan 2016 16:36:50 +0100, Arnout Vandecappelle wrote:
>>
>>> > So, we currently have scancpan to create perl packages. You are adding
>>> > scanpypi to create Python packages. There's also someone who submitted
>>> > a script to generate a 'generic' package (i.e. not perl/python) [0].
>>> > The scancpan is written in perl, yours and the generic one in Python.
>>> >
>>> > Besides Perl and Python, we also have nodejs which provides a similar
>>> > "package-store" and for which it would become interesting to provide a
>>> > helper script to generate packages [1].
>>> >
>>> > What I would love to see is that we have a single script to add
>>> > packages. Something like:
>>> >
>>> > $ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
>>> >
>>> > with TYPE being one of the package types we currently have (generic,
>>> > autotools... python, perl...) or an abstract type (nodejs...).
>>> >
>>> > Then, the cpan, pypi, nodejs... script would be just 'backends' that
>>> > would provide classes called by the main script, like;
>>> >
>>> > pkg = new PythonPkg("foo")
>>> > pkg.get_br_name() returns the BR2_PACKAGE_ name
>>> > pkg.get_version() returns the _VERSION string
>>> > pkg.get_source() returns the _SOURCE string
>>> > pkg.get_site() returns the _SITE string
>>> > pkg.get_method() returns the _SITE_METHOD string
>>> > pkg.get_dependencies() returns the _DEPENDENCIES list
>>> > ... and so on, you get the idea. ;-)
>>>
>>> However, it is more natural for a CPAN-accessor to be written in perl. So I
>>> guess the backend scripts should be really independent scripts that report the
>>> package metadata in a specified format. Hm, you know what, let's use Config.in
>>> and pkg.mk as the specification!
>>>
>>> In short, I'm not so convinced that having everything written in the same
>>> language is such an advantage.
>>>
>>> But of course, if someone shows me the patches, I could change my mind.
>>
>> I also initially would have preferred to have the scancpan script
>> written in Python. But 1/ it's not very practical to query CPAN from
>> Python, and more importantly 2/ it's a bit weird to ask to a Perl
>> fan who maintains the Perl stuff in Buildroot to write something like
>> scancpan in Python.
>>
>> Bottom line, my opinion is that:
>>
>> 1/ We should keep scancpan in Perl
>>
>> 2/ We should keep scanpipy in Python, refine it and merge it.
>>
>> 3/ The script that generates just a generic package skeleton is a bit
>> useless IMO and is not very useful to merge.
>>
>> Both scancpan and canpipy are really internal tools, for Buildroot
>> developers, so it's not super important if they don't look like /
>> behave the same.
>
> I gave scanpypi a try: http://patchwork.ozlabs.org/patch/567288/
>
> So far it was working as expected. Thanks for the very useful tool.
scanpypi has an issue with packages, that have a __main__ block like
this one https://github.com/hynek/characteristic
package.load_setup()
comes with
AttributeError: 'module' object has no attribute 'main'
@Denis: could you take a loot at it?
Thanks.
Yegor
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2016-01-10 10:59 ` Yann E. MORIN
2016-01-10 15:36 ` Arnout Vandecappelle
@ 2016-02-02 18:02 ` Eelco Chaudron
2016-02-02 19:54 ` Eelco Chaudron
1 sibling, 1 reply; 14+ messages in thread
From: Eelco Chaudron @ 2016-02-02 18:02 UTC (permalink / raw)
To: buildroot
I was trying your script and I seem to run into an issue with zip?ed archives;
$ ~/scanpypi WTForms -o package
buildroot package name for WTForms: python-wtforms
Package: python-wtforms
Fetching package WTForms
Downloading package WTForms from https://pypi.python.org/packages/source/W/WTForms/WTForms-2.1.zip...
Traceback (most recent call last):
File "/home/echaudron/scanpypi", line 653, in <module>
main()
File "/home/echaudron/scanpypi", line 596, in main
package.extract_package(tmp_path)
File "/home/echaudron/scanpypi", line 211, in extract_package
with zipfile.open(fileobj=as_file) as as_zipfile:
AttributeError: 'module' object has no attribute ?open'
//Eelco
> On 10 Jan 2016, at 11:59, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Denis, All,
>
> Sorry for the long delay. I'm now having a look at this patch.
>
> On 2015-07-28 15:15 +0200, Denis THULIN spake thusly:
>> An utility for creating python package from the python package index
>> It fetches packages info from http://pypi.python.org and generates
>> corresponding packages files.
>
> So, we currently have scancpan to create perl packages. You are adding
> scanpypi to create Python packages. There's also someone who submitted
> a script to generate a 'generic' package (i.e. not perl/python) [0].
> The scancpan is written in perl, yours and the generic one in Python.
>
> Besides Perl and Python, we also have nodejs which provides a similar
> "package-store" and for which it would become interesting to provide a
> helper script to generate packages [1].
>
> What I would love to see is that we have a single script to add
> packages. Something like:
>
> $ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
>
> with TYPE being one of the package types we currently have (generic,
> autotools... python, perl...) or an abstract type (nodejs...).
>
> Then, the cpan, pypi, nodejs... script would be just 'backends' that
> would provide classes called by the main script, like;
>
> pkg = new PythonPkg("foo")
> pkg.get_br_name() returns the BR2_PACKAGE_ name
> pkg.get_version() returns the _VERSION string
> pkg.get_source() returns the _SOURCE string
> pkg.get_site() returns the _SITE string
> pkg.get_method() returns the _SITE_METHOD string
> pkg.get_dependencies() returns the _DEPENDENCIES list
> ... and so on, you get the idea. ;-)
>
> That would also recursively generate the packages for the dependencies,
> if not already present.
>
> Of course, that would mean we'd have to standardise on a single
> language. I think Python is the way to go here.
>
> Would you be interested in pursuing this?
>
> [0] https://patchwork.ozlabs.org/patch/523257/
> [1] that could also require a nodejs-package infra, but not necessarily.
>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
> | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160202/7d987657/attachment-0001.html>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2016-02-02 18:02 ` Eelco Chaudron
@ 2016-02-02 19:54 ` Eelco Chaudron
0 siblings, 0 replies; 14+ messages in thread
From: Eelco Chaudron @ 2016-02-02 19:54 UTC (permalink / raw)
To: buildroot
Guess it might be my version of python, however changing the zip part to the following fixed it;
if self.filename[-3:] == 'zip':
with zipfile.ZipFile(as_file) as as_zipfile:
tmp_pkg = os.path.join(tmp_path, self.buildroot_name)
python
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
//Eelco
> On 02 Feb 2016, at 19:02, Eelco Chaudron <echaudron@xiot.nl> wrote:
>
> I was trying your script and I seem to run into an issue with zip?ed archives;
>
> $ ~/scanpypi WTForms -o package
> buildroot package name for WTForms: python-wtforms
> Package: python-wtforms
> Fetching package WTForms
> Downloading package WTForms from https://pypi.python.org/packages/source/W/WTForms/WTForms-2.1.zip <https://pypi.python.org/packages/source/W/WTForms/WTForms-2.1.zip>...
> Traceback (most recent call last):
> File "/home/echaudron/scanpypi", line 653, in <module>
> main()
> File "/home/echaudron/scanpypi", line 596, in main
> package.extract_package(tmp_path)
> File "/home/echaudron/scanpypi", line 211, in extract_package
> with zipfile.open(fileobj=as_file) as as_zipfile:
> AttributeError: 'module' object has no attribute ?open'
>
> //Eelco
>
>> On 10 Jan 2016, at 11:59, Yann E. MORIN <yann.morin.1998 at free.fr <mailto:yann.morin.1998@free.fr>> wrote:
>>
>> Denis, All,
>>
>> Sorry for the long delay. I'm now having a look at this patch.
>>
>> On 2015-07-28 15:15 +0200, Denis THULIN spake thusly:
>>> An utility for creating python package from the python package index
>>> It fetches packages info from http://pypi.python.org <http://pypi.python.org/> and generates
>>> corresponding packages files.
>>
>> So, we currently have scancpan to create perl packages. You are adding
>> scanpypi to create Python packages. There's also someone who submitted
>> a script to generate a 'generic' package (i.e. not perl/python) [0].
>> The scancpan is written in perl, yours and the generic one in Python.
>>
>> Besides Perl and Python, we also have nodejs which provides a similar
>> "package-store" and for which it would become interesting to provide a
>> helper script to generate packages [1].
>>
>> What I would love to see is that we have a single script to add
>> packages. Something like:
>>
>> $ ./support/script/add-package -t TYPE [OPTS] PKG [PKG...]
>>
>> with TYPE being one of the package types we currently have (generic,
>> autotools... python, perl...) or an abstract type (nodejs...).
>>
>> Then, the cpan, pypi, nodejs... script would be just 'backends' that
>> would provide classes called by the main script, like;
>>
>> pkg = new PythonPkg("foo")
>> pkg.get_br_name() returns the BR2_PACKAGE_ name
>> pkg.get_version() returns the _VERSION string
>> pkg.get_source() returns the _SOURCE string
>> pkg.get_site() returns the _SITE string
>> pkg.get_method() returns the _SITE_METHOD string
>> pkg.get_dependencies() returns the _DEPENDENCIES list
>> ... and so on, you get the idea. ;-)
>>
>> That would also recursively generate the packages for the dependencies,
>> if not already present.
>>
>> Of course, that would mean we'd have to standardise on a single
>> language. I think Python is the way to go here.
>>
>> Would you be interested in pursuing this?
>>
>> [0] https://patchwork.ozlabs.org/patch/523257/ <https://patchwork.ozlabs.org/patch/523257/>
>> [1] that could also require a nodejs-package infra, but not necessarily.
>>
>> Regards,
>> Yann E. MORIN.
>>
>> --
>> .-----------------.--------------------.------------------.--------------------.
>> | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
>> | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
>> | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
>> | http://ymorin.is-a-geek.org/ <http://ymorin.is-a-geek.org/> | _/*\_ | / \ HTML MAIL | v conspiracy. |
>> '------------------------------^-------^------------------^--------------------'
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net <mailto:buildroot@busybox.net>
>> http://lists.busybox.net/mailman/listinfo/buildroot
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160202/1e833d9f/attachment.html>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2015-07-28 13:15 [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
` (2 preceding siblings ...)
2016-01-10 10:59 ` Yann E. MORIN
@ 2016-03-01 1:44 ` Carlos Santos
3 siblings, 0 replies; 14+ messages in thread
From: Carlos Santos @ 2016-03-01 1:44 UTC (permalink / raw)
To: buildroot
> An utility for creating python package from the python package index
> It fetches packages info from http://pypi.python.org and generates
> corresponding packages files.
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
Tested-by: Carlos Santos <casantos@datacom.ind.br>
When I ran the script on my machine I got an error from Python because
it requires python-setuptools, which I did not have installed.
$ uname -a Linux p7-1130br 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 00:58:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ support/scripts/scanpypi delayqueue
Traceback (most recent call last):
File "support/scripts/scanpypi", line 48, in <module>
import setuptools
ImportError: No module named setuptools
I don't think it's a big deal for a support script but it would be nice
to document the requirements in the initial comment block.
Going forward...
$ sudo apt-get install python-setuptools
[...]
Choose a random python pachage from PyPI and created the Buidroot package.
$ support/scripts/scanpypi delayqueue
buildroot package name for delayqueue: python-delayqueue
Package: python-delayqueue
Fetching package delayqueue
Downloading package delayqueue from https://pypi.python.org/packages/source/d/delayqueue/delayqueue-0.0.3.tar.gz...
Checking if package ./package/python-delayqueue already exists...
Creating ./package/python-delayqueue/python-delayqueue.mk...
WARNING: No license file found, please specify it manually afterwards
Creating ./package/python-delayqueue/python-delayqueue.hash...
Creating ./package/python-delayqueue/Config.in...
[...]
Edit package/Config.in and add to the "External python modules" menu
a line containing
source "package/python-delayqueue/Config.in"
[...]
$ make menuconfig
[ Select the python-delayqueue package ]
$ make python-delayqueue
[...]
Success!
Verdict: your script seems to be a useful tool to create Python packages
for Buildroot. Of course the package that it creates must be tested before
submitting it upstream but that's another story.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2015-07-11 12:56 ` Arnout Vandecappelle
@ 2015-07-15 14:08 ` Denis Thulin
0 siblings, 0 replies; 14+ messages in thread
From: Denis Thulin @ 2015-07-15 14:08 UTC (permalink / raw)
To: buildroot
Hi Arnout,
----- Mail original -----
> On 07/09/15 15:31, Denis THULIN wrote:
> > An utility for creating python package from the python package
> > index
>
> It would make sense to list here the packages with which you have
> tested it.
> I've tried it for about 5 packages and it worked for none of them...
> Usually the
> problem is that setup.py either has an if __name__ == '__main__', or
> it tries to
> load other modules from the package.
>
> I think you should at least test it with the packages that already
> are in
> buildroot. django, pyzmq, six, twisted are a few nice examples. This
> also allows
> you to see how the automatically generated files differ from what we
> have already.
django used to work fine, but I made some mistake and I did not test it again.
Now, It works again on my local version.
The other works too.
>
> >
> > Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> > ---
> > v0: initial commit
> > python-pacakage-generator.py is an utility for automatically
> > generating a
> > python package. It fetches packages info from
> > http://pypi.python.org and
> > generates corresponding packages files.
>
> Actually, this sentence should go into the commit log as well.
Ok.
>
> >
> > v1:
> > - renamed python-package-generator to scanpypi
> > - split the huge script into a lot of functions
> > - fixed mistakes and small bugs
> >
> > v2:
> > - Rewrited most of the functions into a class
> > - Changed the method for importing setup.py
>
> And what was the reason for that change? It turns out not to work so
> well...
Well I thougt using the proper "python" way of doing these kind of import
would be better that what I was doing previously.
>
> > - Created a main function to avoid use of global variable
> > - Now adds new dependencies to the list of packages to create
> > - Droppped the .py extension
> >
> > Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> > ---
> > docs/manual/adding-packages-python.txt | 32 ++
> > support/scripts/scanpypi | 620
> > +++++++++++++++++++++++++++++++++
> > 2 files changed, 652 insertions(+)
> > create mode 100755 support/scripts/scanpypi
> >
> > diff --git a/docs/manual/adding-packages-python.txt
> > b/docs/manual/adding-packages-python.txt
> > index f81d625..dcde08d 100644
> > --- a/docs/manual/adding-packages-python.txt
> > +++ b/docs/manual/adding-packages-python.txt
> > @@ -7,6 +7,38 @@ This infrastructure applies to Python packages
> > that use the standard
> > Python setuptools mechanism as their build system, generally
> > recognizable by the usage of a +setup.py+ script.
> >
> > +[[scanpypi]]
> > +
> > +==== Generating a +python-package+ from a pypi repository
> > +
> > +You may want to use the +scanpypi.py+ located in
>
> Drop the .py :-) and rewrap.
>
> > ++support/script+ to generate a package from an existing pypi(pip)
> > package.
>
> PyPI is capitalized differently. Rephrase:
>
> ... from an existing PyPI package (i.e. a package that can be
> installed with pip).
>
> > +
> > +you can find the list of existing pypi package here:
> > https://pypi.python.org .
>
> You can find the list of existing PyPI packages
> https://pypi.python.org[here].
>
> > +
> > +Please keep in mind that you most likely need to manually check
> > the package for
> > +any mistakes as there are things that cannot be guessed by the
> > generator (e.g.
> > +dependencies on any of the python core modules such as
> > BR2_PACKAGE_PYTHON_ZLIB)
>
> Also useful te mention: the license and license files (they're a bit
> of ad hoc
> guesswork).
>
> > +. You need no manually add the package to the +package/Config.in+
> > file.
>
> That . should still be on the preceding line, without space in front
> of it.
>
> You also need to manually add the package the the +package/Config.in+
> file.
>
> > +
> > +When at the root of your buildroot directory just do :
> > +
> > +-----------------------
> > +./support/script/scanpypi.py foo bar -o package
>
> s/.py//
>
> > +-----------------------
> > +
> > +This will generate packages +python-foo+ and +python-bar+ in the
> > package
> > +folder if they exist on https://pypi.python.org.
> > +
> > +Find the +external python modules+ menu and insert your package
> > inside.
> > +Keep in mind that the items inside a menu should be in
> > alphabetical order.
>
>
> I would also make an explicit reference here to how to use it with
> BR2_EXTERNAL. I.e., using the -o flag.
>
> > +
> > +Option +-h+ wil list the options available
> > +
> > +-----------------------
> > +./support/script/scanpypi.py -h
>
> s/.py//
>
> > +-----------------------
> > +
> > [[python-package-tutorial]]
> >
> > ==== +python-package+ tutorial
> > diff --git a/support/scripts/scanpypi b/support/scripts/scanpypi
> > new file mode 100755
> > index 0000000..e98e8f8
> > --- /dev/null
> > +++ b/support/scripts/scanpypi
> > @@ -0,0 +1,620 @@
> > +#!/usr/bin/python2
>
> It would be nice to make it python3 compliant as well, but that can
> be done later.
>
> > +"""
> > +Utility for building buildroot packages for existing pypi packages
> > +
> > +Any package built by scanpypi should be manually checked for
> > +errors.
> > +"""
> > +from __future__ import print_function
> > +import argparse
> > +import json
> > +import urllib2
> > +import sys
> > +import os
> > +import shutil
> > +import StringIO
> > +import tarfile
> > +import errno
> > +import hashlib
> > +import re
> > +import textwrap
> > +import tempfile
> > +import imp
> > +from functools import wraps
> > +
> > +
> > +# private global
> > +_calls = {}
>
> I guess this could be converted into a class member of
> BuildrootPackage.
I'll do that.
>
> Also, the name could be a bit better, e.g. setup_args.
>
> > +
> > +
> > +def setup_info(pkg_name):
> > + """Get a package info from _calls
> > +
> > + Keyword arguments:
> > + pkg_name -- the name of the package
> > + """
> > + return _calls[pkg_name]
>
> I still don't think it's worthwhile to make a function from this.
>
> > +
> > +
> > +def setup_decorator(func, method):
> > + """
> > + Decorator for distutils.core.setup and setuptools.setup.
> > + Puts the args of setup as a dict inside global private dict
> > _calls.
>
> I finally understand what this does :-) Even better:
>
> Puts the arguments with which setup is called as a dict ...
>
> > + Add key 'method' which should be either 'setuptools' or
> > 'distutils'.
> > +
> > + Keyword arguments:
> > + func -- either setuptools.setup or distutils.core.setup
> > + method -- either 'setuptools' or 'distutils'
> > + """
> > +
> > + @wraps(func)
> > + def closure(*args, **kwargs):
> > + _calls[kwargs['name']] = kwargs
>
> Perhaps document where this 'name' comes from. It's quite obvious if
> you're
> familiar with distutils, but this wrapping is not easy to understand
> so it's
> worthwhile to mention it.
>
> > + _calls[kwargs['name']]['method'] = method
> > + return closure
> > +
> > +
> > +def find_file_upper_case(filenames, path='./'):
> > + """
> > + List generator:
> > + Recursively find files that matches one of the specified
> > filenames.
> > + Returns absolute path
>
> No it doesn't, it returns a relative path starting with path.
Yes, that is what I meant to write, I don't know why I wrote that.
>
> > +
> > + Keyword arguments:
> > + filenames -- List of filenames to be found
> > + path -- Path to the directory to search
> > + """
> > + for root, dirs, files in os.walk(path):
> > + for file in files:
> > + if file.upper() in filenames:
> > + yield (os.path.join(root, file))
> > +
> > +
> > +def pkg_buildroot_name(pkg_name):
> > + """
> > + Returns name to avoid troublesome characters.
>
> No it doesn't.
>
> Returns the buildroot package name for the PyPI package pkg_name.
>
>
> In fact, the function should also prepend the python- prefix,
> because the
> buildroot name starts with python-.
Actually, I have a question here, should I change . into - or into nothing ?
I'm asking this because when looking into existing packages I noticed 'web.py'
became 'python-webpy' but 'zope.interface' became 'python-zope-interface'
>
> > + Remove all non alphanumeric characters except -
> > + Also lowers the name
> > +
> > + Keyword arguments:
> > + pkg_name -- String to rename
> > + """
> > + name = re.sub('[^\w-]', '', pkg_name.lower())
> > + name = re.sub('^python-', '', name)
> > + return name
> > +
> > +
> > +# monkey patch
> > +import setuptools
> > +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> > +import distutils
> > +distutils.core.setup = setup_decorator(setuptools.setup,
> > 'distutils')
>
> This should go just below the definition of the decorator.
Yes, that would make things easier to understand :)
>
> > +
> > +
> > +class SetupNotFound(Exception):
> > + pass
> > +
> > +
> > +class BuildrootPackage():
> > + """
> > + This class's methods are not meant to be used individually
> > please use those
> > + in the correct order:
> > + __init__
> > +
> > + download_package
> > +
> > + extract_package
> > +
> > + load_module
> > +
> > + get_requirements
> > +
> > + create_package_mk
> > +
> > + create_hash_file
> > +
> > + create_config_in
> > + """
> > + def __init__(self, real_name, pkg_folder):
> > + self.real_name = real_name
> > + self.name = pkg_buildroot_name(self.real_name)
>
> Perhaps to clarify the distinction, call it buildroot_name instead
> of just name.
Good idea, I just called it that way to reduce line lengths
>
> > + self.pkg_dir = pkg_folder + '/python-' + self.name
>
> It's good practice to set all the members you're going to create to
> None in
> __init__, as a kind of declaration of what members are available.
Ok, I will do that
>
> > +
> > + def find_setup(self, folder):
> > + """
> > + Search for setup.py file in an archive and returns the
> > path to this
> > + file
> > + if it is found
>
> Wrapping.
>
> > +
> > + Keyword arguments:
> > + folder -- path to search in
> > + """
> > + filename = 'setup.py'
> > + # Next return the first element of an iterable,
> > + # here, it returns the path to the first file named
> > "setup.py"
> > + # found in the python package or raises StopIteration if
> > not found
> > + self.setup_location = next(os.path.join(root, filename)
> > + for root, dirs, files in
> > os.walk(folder)
> > + if filename in files)
>
> Is it just me or is setup_location not used anywhere? I anyway don't
> think this
> is a good idea: if setup.py is not in the top dir, most likely other
> things in
> this script will break.
>
> So the only thing this function does, really, is checking if there
> is any
> setup.py file in the folder. Which is later done again by calling
> imp.find_module, which will anyway raise an ImportError if the module
> isn't found.
>
> IOW, remove this function.
This was needed when I wasn't using imp, now it's not
I will remove it.
>
> > +
> > + def fetch_package_info(self):
> > + """
> > + Fetch a package's metadata for the python package index
>
> for -> from
>
> > + """
> > + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
>
> Already asign to metadata_url here.
>
> > + pkg=self.real_name)
> > + print('URL:', url)
>
> This print is not needed.
That's a debug print I forgot to remove
>
> > + try:
> > + pkg_json = urllib2.urlopen(url).read().decode()
> > + except urllib2.HTTPError as error:
> > + print('ERROR:', error.getcode(), error.msg,
> > file=sys.stderr)
> > + print('ERROR: Could not find package {pkg}.\n'
> > + 'Check syntax inside the python package
> > index:\n'
> > + 'https://pypi.python.org/pypi/ '
> > + .format(pkg=self.real_name))
> > + raise
> > + except urllib2.URLError:
> > + print('ERROR: Could not find package {pkg}.\n'
> > + 'Check syntax inside the python package
> > index:\n'
> > + 'https://pypi.python.org/pypi/ '
> > + .format(pkg=self.real_name))
> > + raise
> > + else:
>
> Else is not needed here since the exceptions are raise'd.
True, I forgot to remove it when I decided to raise the exception
>
> > + self.metadata_url = url
> > + self.metadata = json.loads(pkg_json)
> > + self.metadata_name = self.metadata['info']['name']
> > +
> > + def download_package(self):
> > + """
> > + Download a package using metadata from pypi
> > + """
> > + try:
> > + self.metadata['urls'][0]['filename']
> > + except IndexError:
> > + print(
> > + 'Non conventional package, ',
>
> Non-conventional
>
> > + 'please check manually after creation')
>
> manually -> carefully
>
> > + download_url = self.metadata['info']['download_url']
>
> I think it would be simpler to, instead of repeating the download
> infra, just
> emulate the needed metadata and use the same infra as below. So:
>
> download_url = [{
> 'packagetype': 'tgz',
> 'url', self.metadata['info']['download_url'],
> 'md5_digest': None,
> }]
>
> > + try:
> > + self.download = urllib2.urlopen(download_url)
>
> self.download is not used outside this function, so make it a local
> variable.
> Or actually, drop it completely and read directly into
> self.as_string().
>
> > + except urllib2.HTTPError as http_error:
> > + self.download = http_error
> > + else:
> > + self.as_string = self.download.read()
> > + self.used_url = {'url': download_url,
> > + 'md5_digest':
> > hashlib.md5(self.as_string)
> > +
> > .hexdigest()}
>
> I don't think it makes a lot os sense to compute the md5 here.
> Better set it to
> None and not mention it in the .hash file.
True
>
> > + # In this case, we can't get the name of the
> > downloaded file
> > + # from the pypi api, so we need to find it, this
> > should work
> > + urlpath =
> > urllib2.urlparse.urlparse(self.download.url).path
> > + # urlparse().path give something like
> > + # /path/to/file-version.tar.gz
> > + # We use basename to remove /path/to
> > + self.targz = os.path.basename(urlpath)
> > + self.used_url['filename'] = self.targz
>
> This magic is still needed to set download_url['filename'], but it
> doesn't need
> to be downloaded yet to do that.
That's true, but if I download it first, I'm sure the filename is correct.
>
> > + else:
> > + for download_url in self.metadata['urls']:
> > + if 'wheel' in download_url['packagetype']:
>
> This could be zip or egg as well. Isn't there a positive match
> possible?
Yes, I'm changing that to avoid bdist packages rather than wheel.
And I'm using zipfile as well in the next version
>
> > + continue
> > + try:
>
> I think it's useful to print the download URL here, just before
> downloading. So
> move the print ('Downloading package ...') here instead, and add the
> download
> URL to it.
>
> > + self.download =
> > urllib2.urlopen(download_url['url'])
> > + except urllib2.HTTPError as http_error:
> > + self.download = http_error
>
> This is only used to raise it later, so you can just drop the entire
> try block
> and let the exception leak up.
No it's not. Some package like pyzmq have more than one url I can use,
(one with a zip and one with a tar.gz) I raise it only if all urls fail.
>
> > + else:
> > + self.used_url = download_url
> > + self.as_string = self.download.read()
> > + self.md5_sum =
> > hashlib.md5(self.as_string).hexdigest()
> > + self.targz = self.used_url['filename']
> > + if self.md5_sum == download_url['md5_digest']:
>
> This should probably raise an exception if it doesn't match.
Again, I don't raise an exception because I want to try other urls if
there are any left. I should raise one if the last on fails though.
>
> > + break
>
> The for loop needs an else: in case none of them can be downloaded.
>
> > + if self.download.__class__ == urllib2.HTTPError:
> > + raise self.download
> > + else:
> > + self.digest_method = 'sha256'
>
> This is always sha256 so no point making a variable for it.
>
> > + self.digest =
> > hashlib.sha256(self.as_string).hexdigest()
>
> Minor nit: it's used only once, so I'd calculate it at the time that
> you use it.
>
> > +
> > + def extract_package(self, tmp_path):
> > + """
> > + Create folders used for extracting a package as file
> > object and extract
> > + it
>
> A bit too detailed...
>
> Extract the package contents into a directory.
>
> > +
> > + Keyword arguments:
> > + tmp_path -- folder where you want the package to be
> > extracted
>
> s/folder/directory/g
>
> > + """
> > + as_file = StringIO.StringIO(self.as_string)
> > + with tarfile.open(fileobj=as_file) as as_tarfile:
> > + tmp_pkg = os.path.join(tmp_path, self.name)
> > + try:
> > + os.makedirs(tmp_pkg)
> > + except OSError as exception:
> > + if exception.errno != errno.EEXIST:
> > + print("ERROR: ", exception.message,
> > file=sys.stderr)
> > + return None, None
> > + print('WARNING:', exception.message,
> > file=sys.stderr)
> > + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> > + shutil.rmtree(tmp_pkg)
> > + os.makedirs(tmp_pkg)
> > + as_tarfile.extractall(tmp_pkg)
> > + try:
> > + self.find_setup(tmp_pkg)
> > + except StopIteration:
> > + raise SetupNotFound('Could not find file setup.py for
> > package {}'
> > + .format(self.real_name))
>
> As I said before, this is not needed since it is covered by
> load_setup().
>
> > + tmp_extract = '{folder}/{name}-{version}'
>
> Is this guaranteed to be the directory used by the package?
sdist and bdist create your archive, so the directory should allways
have that name.
>
> > + self.tmp_extract = tmp_extract.format(
> > + folder=tmp_pkg,
> > + name=self.metadata_name,
> > + version=self.metadata['info']['version'])
> > +
> > + def load_setup(self):
> > + """
> > + Loads the corresponding setup and store its metadata
> > + """
> > + s_file, s_path, s_desc = imp.find_module('setup',
> > [self.tmp_extract])
> > + imp.load_module('setup', s_file, s_path, s_desc)
> > + self.setup_metadata = setup_info(self.metadata_name)
>
> Is the name given in the metadata guaranteed to be the same as what
> is passed
> to setup()? Especially considering case...
Case is the reason I use the metadata. The name in the metadata from PyPi
is the same as the one from setup. As the PyPi name comes from the setup.py
>
> > +
> > + def get_requirements(self, pkg_folder, packages):
> > + """
> > + Retrieve dependencies from the metadata found in the
> > setup.py script of
> > + a pypi package.
> > +
> > + Keyword Arguments:
> > + pkg_folder -- location of the already created packages
> > + packages -- other packages to be built
> > + """
> > + if 'install_requires' not in self.setup_metadata:
> > + self.pkg_req = None
> > + self.req_not_found = []
> > + return
> > + self.pkg_req = self.setup_metadata['install_requires']
> > + self.pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1',
> > req).lower()
>
> The .lower() is handled by pkg_buildroot_name so not needed here.
Correct.
>
> > + for req in self.pkg_req]
> > + self.pkg_req = map(pkg_buildroot_name, self.pkg_req)
> > + self.req_not_found = [
> > + pkg for pkg in self.pkg_req
> > + if 'python-{name}'.format(name=pkg)
>
> pkg_buildroot_name already adds python- (or at least it should :-).
It will :).
>
> > + not in os.listdir(pkg_folder)
>
> Maybe better use "not os.path.isdir(pkg)".
Yes, that would be better
>
> > + ]
> > + self.req_not_found = [pkg for pkg in self.req_not_found
> > + if pkg not in packages]
>
> Small nit: I'd make this function just return req_not_found instead
> of adding
> it as a member, and move the set.difference logic up to the caller.
> Oh, and
> perhaps use a set :-)
I will do that as well
>
> > +
> > + def __create_mk_header(self):
> > + """
> > + Create the header of the <package_name>.mk file
> > + """
> > + header = ['#' * 80 + '\n']
> > + header.append('#\n')
> > + header.append('# python-{name}\n'.format(name=self.name))
> > + header.append('#\n')
> > + header.append('#' * 80 + '\n')
> > + header.append('\n')
> > + return header
> > +
> > + def __create_mk_download_info(self):
> > + """
> > + Create the lines refering to the download information of
> > the
> > + <package_name>.mk file
> > + """
> > + lines = []
> > + version_line = 'PYTHON_{name}_VERSION =
> > {version}\n'.format(
> > + name=self.name.upper().replace('-', '_'),
>
> Like you have pkg_buildroot_name, you should also have something for
> the
> conversion to uppercase. Becasue below, you forget the replace('-',
> '_'). Also
> it's probably best to add the uppercase name as a member.
Nice suggestion, thanks :)
>
> > + version=self.metadata['info']['version'])
>
> I think it makes sense to add these things as direct members of
> BuildrootPackage rather than going through metadata all the time:
> version, url,
> filename.
I will change that
>
> > + lines.append(version_line)
> > +
> > + targz = self.targz.replace(
> > + self.metadata['info']['version'],
> > +
> > '$(PYTHON_{name}_VERSION)'.format(name=self.name.upper()))
> > + targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
> > + name=self.name.upper(),
> > + filename=targz)
> > + lines.append(targz_line)
> > +
> > + site_line = ('PYTHON_{name}_SITE = {url}'.format(
> > + name=self.name.upper(),
> > +
> > url=self.used_url['url'].replace(self.used_url['filename'],
> > '')))
> > + if 'sourceforge' in site_line:
> > + site_line = ('PYTHON_{name}_SITE = {url}'.format(
> > + name=self.name.upper(),
> > + url=self.used_url['url']))
>
> This looks very weird... At least add an explanatory comment. And
> anyway, it's
> better to do any fixups in a new variable (e.g. 'site') and format
> the line only
> once.
Ok.
>
> > + site_line = site_line.rstrip('/') + '\n'
> > + lines.append(site_line)
> > + return lines
> > +
> > + def __create_mk_setup(self):
> > + """
> > + Create the line refering to the setup method of the
> > package of the
> > + <package_name>.mk file
> > +
> > + There are two things you can use to make an installer
> > + for a python package: distutils or setuptools
> > + distutils comes with python but does not support
> > dependencies.
> > + distutils is mostly still there for backward support.
> > + setuptools is what smart people use,
> > + but it is not shipped with python :(
>
> This comment is a bit redundant, but OK.
>
> > + """
> > + lines = []
> > + setup_type_line = 'PYTHON_{name}_SETUP_TYPE =
> > {method}\n'.format(
> > + name=self.name.upper(),
> > + method=self.setup_metadata['method'])
> > + lines.append(setup_type_line)
> > + return lines
> > +
> > + def __create_mk_license(self):
> > + """
> > + Create the lines referring to the package's license
> > informations of the
> > + <package_name>.mk file
> > +
> > + The license is found using the metadata from pypi.
> > + In the metadata, the license can be found either with
> > standard names in
> > + the classifiers part or with naming from the packager in
> > the "License"
> > + part.
> > +
> > + From the classifiers, the license is "translated"
> > according to
> > + buildroot standards if need be (i.e. from Apache Software
> > License to
> > + Apache-2.0).
> > +
> > + From the License part, we cannot guess what formatting the
> > packager
> > + used. Hence, it is likely to be incorrect. (i.e. Apache
> > License 2.0
> > + instead of Apache-2.0).
> > +
> > + The license's files are found by searching the package for
> > files named
> > + license or license.txt (case insensitive).
> > + If more than one license file is found, the user is asked
> > to select
> > + which ones he wants to use.
> > + """
> > + license_dict = {
> > + 'Apache Software License': 'Apache-2.0',
> > + 'BSD License': 'BSD',
>
> We normally put BSD-2c, BSD-3c, etc. but it may be difficult to find
> that out
> automatically.
Yes, I wanted to, but I have no way to know which version it is.
>
> > + 'European Union Public Licence 1.0': 'EUPLv1.0',
> > + 'European Union Public Licence 1.1': 'EUPLv1.1',
> > + "GNU General Public License": "GPL",
> > + "GNU General Public License v2": "GPLv2",
> > + "GNU General Public License v2 or later": "GPLv2+",
> > + "GNU General Public License v3": "GPLv3",
> > + "GNU General Public License v3 or later": "GPLv3+",
> > + "GNU Lesser General Public License v2": "LGPLv2",
> > + "GNU Lesser General Public License v2 or later":
> > "LGPLv2+",
>
> That's actually LGPVLv2.1(+). LGPLv2 is the GNU Library General
> Public License.
Ok. Thanks
>
> > + "GNU Lesser General Public License v3": "LGPLv3",
> > + "GNU Lesser General Public License v3 or later":
> > "LGPLv3+",
> > + "GNU Library or Lesser General Public License":
> > "LGPL",
>
> So this is probably LGPLv2 (there was no library/lesser version of
> v1).
Ok.
>
> > + "ISC License": "ISC",
> > + "MIT License": "MIT",
> > + "Mozilla Public License 1.0": "MPL-1.0",
> > + "Mozilla Public License 1.1": "MPL-1.1",
> > + "Mozilla Public License 2.0": "MPL-2.0",
> > + "Zope Public License": "ZPL"
> > + }
>
> Nice list!
>
> > + regexp = re.compile('^License :* *.* *:+ (.*)( \(.*\))?$')
> > + classifiers_licenses = [regexp.sub(r"\1", lic)
> > + for lic in
> > self.metadata['info']['classifiers']
> > + if regexp.match(lic)]
> > + licenses = map(lambda x: license_dict[x] if x in
> > license_dict else x,
> > + classifiers_licenses)
>
> I would give an explicit warning if the license is not in
> license_dict, since
> it most likely means it's wrong.
Ok.
>
>
> > + lines = []
> > + if not len(licenses):
> > + licenses = [self.metadata['info']['license']]
>
> This should probably go before the license_dict map.
This is a default case, as the license in the metadata is not subject
to any kind of standard naming. It is very unlikely that it matches the
classifiers syntax if the packager did not use classifiers...
But I should give a warning in that case.
>
> > + license_line = 'PYTHON_{name}_LICENSE =
> > {license}\n'.format(
> > + name=self.name.upper(),
> > + license=', '.join(licenses))
> > + lines.append(license_line)
> > + print('WARNING: License has been set to "{license}",'
> > + ' please change it manually if necessary'.format(
> > + license=', '.join(licenses)))
>
> So this warning is not necessary if it was found in the dict.
That is true.
>
> > +
> > + filenames = ['LICENSE', 'LICENSE.TXT']
>
> Shouldn't we also look for COPYING?
That would be a good idea.
>
> > + license_files = list(find_file_upper_case(filenames,
> > self.tmp_extract))
> > + license_files = [license.replace(self.tmp_extract, '')[1:]
> > + for license in license_files]
> > + if len(license_files) > 0:
> > + if len(license_files) > 1:
> > + print('More than one file found for license:',
> > + ', '.join(license_files))
> > + license_files = [filename
> > + for index, filename in
> > enumerate(license_files)]
> > + license_file_line = ('PYTHON_{name}_LICENSE_FILES ='
> > + ' {files}\n'.format(
> > + name=self.name.upper(),
> > + files='
> > '.join(license_files)))
> > + lines.append(license_file_line)
> > + elif len(license_files) == 0:
>
> Can len(license_files) be anything else than 0 at this point?
No it cannot.
Unless someone overrides len so that it returns something negative.
But that would be a curious thing to do :).
>
> > + print('WARNING: No license file found,'
> > + ' please specify it manually afterward')
>
> afterwards
>
> > + license_file_line = '# No license file found\n'
> > +
> > + return lines
> > +
> > + def __create_mk_requirements(self):
> > + """
> > + Create the lines referring to the dependencies of the of
> > the
> > + <package_name>.mk file
> > +
> > + Keyword Arguments:
> > + pkg_name -- name of the package
> > + pkg_req -- dependencies of the package
> > + """
> > + lines = []
> > + python_pkg_req = ['python-{name}'.format(name=pkg)
> > + for pkg in self.pkg_req]
> > + dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
> > + ' {reqs}\n'.format(
> > + name=self.name.upper(),
> > + reqs=' '.join(python_pkg_req)))
> > + lines.append(dependencies_line)
> > + return lines
> > +
> > + def create_package_mk(self):
> > + """
> > + Create the lines corresponding to the <package_name>.mk
> > file
> > + """
> > + pkg_mk = 'python-{name}.mk'.format(name=self.name)
> > + path_to_mk = os.path.join(self.pkg_dir, pkg_mk)
> > + print('Creating {file}...'.format(file=path_to_mk))
> > + lines = self.__create_mk_header()
> > + lines += self.__create_mk_download_info()
> > + lines += self.__create_mk_setup()
> > + lines += self.__create_mk_license()
> > + if self.pkg_req:
> > + lines += self.__create_mk_requirements()
> > +
> > + lines.append('\n')
> > + lines.append('$(eval $(python-package))')
> > + lines.append('\n')
> > + with open(path_to_mk, 'w') as mk_file:
> > + mk_file.writelines(lines)
> > +
> > + def create_hash_file(self):
> > + """
> > + Create the lines corresponding to the <package_name>.hash
> > files
> > + """
> > + pkg_hash = 'python-{name}.hash'.format(name=self.name)
> > + path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
> > + print('Creating
> > {filename}...'.format(filename=path_to_hash))
> > + lines = []
> > + commented_line = '# md5 from {url}, {method} calculated by
> > scanpypi\n'
>
> Actually, the md5 comes from the pypi/json url, so mention that one.
> So just:
Ok. Thanks
>
> # From https://pypi.python.org/pypi/{pkg}/json
>
> > + commented_line =
> > commented_line.format(url=self.used_url['url'],
> > +
> > method=self.digest_method)
> > + lines.append(commented_line)
> > + hash_line = '{method}\t{digest} {filename}\n'.format(
> > + method='md5',
> > + digest=self.used_url['md5_digest'],
> > + filename=self.used_url['filename'])
>
> The md5 should only be added if there really was an md5.
Yes, I will test if it exists
>
> > + lines.append(hash_line)
> > + hash_line = '{method}\t{digest} {filename}\n'.format(
> > + method=self.digest_method,
> > + digest=self.digest,
> > + filename=self.used_url['filename'])
> > + lines.append(hash_line)
> > +
> > + with open(path_to_hash, 'w') as hash_file:
> > + hash_file.writelines(lines)
> > +
> > + def create_config_in(self):
> > + """
> > + Creates the Config.in file of a package
> > + """
> > + path_to_config = os.path.join(self.pkg_dir, 'Config.in')
> > + print('Creating {file}...'.format(file=path_to_config))
> > + lines = []
> > + config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
> > + name=self.name.upper())
> > + lines.append(config_line)
> > +
> > + bool_line = '\tbool
> > "python-{name}"\n'.format(name=self.name)
> > + lines.append(bool_line)
> > + if self.pkg_req:
> > + for dep in self.pkg_req:
> > + dep_line = '\tselect
> > BR2_PACKAGE_PYTHON_{req}\n'.format(
> > + req=dep.upper())
> > + lines.append(dep_line)
> > +
> > + lines.append('\thelp\n')
> > +
> > + help_lines =
> > textwrap.wrap(self.metadata['info']['summary'], 67)
>
> I'm not really sure if we should use the summary or the description
> here... The
> summary is _really_ short.
Well the description can be very long and it's full of \n
i.e.: https://pypi.python.org/pypi/requests/json
>
> Also, use textwrap.fill and add the initial_indent='\t ' argument,
> then all
> the mangling below should no longer be necessary.
Oh, I did not see that argument. Thanks :)
>
> > + # \t + two spaces is 3 char long
> > + help_lines.append('')
> > + help_lines.append(self.metadata['info']['home_page'])
> > + help_lines = map(lambda l: '\t {}\n'.format(l)
> > + if not l == '' else '\n',
> > + help_lines)
> > + # help_lines = ['\t
> > {line}\n'.format(line=line).replace('\t \n','\n')
> > + # for line in help_lines]
> > + lines += help_lines
> > +
> > + with open(path_to_config, 'w') as config_file:
> > + config_file.writelines(lines)
> > +
> > +
> > +def main():
> > + # Building the parser
> > + parser = argparse.ArgumentParser(
> > + description="Creates buildroot packages from the metadata
> > of "
> > + "an existing pypi(pip) packages and include it
> > "
>
> PyPI
>
> > + "in menuconfig")
> > + parser.add_argument("packages",
> > + help="list of packages to be made",
>
> made -> created
>
> > + nargs='+')
> > + parser.add_argument("-o", "--output",
> > + help="""
> > + Output directory for packages
>
> Please explicitly mention the default in the help text.
>
> > + """,
> > + default='./package')
> > +
> > + args = parser.parse_args()
> > + packages = list(set(args.packages))
> > +
> > + # tmp_path is where we'll extract the files later
> > + tmp_prefix = 'scanpypi-'
> > + pkg_folder = args.output
> > + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
> > + try:
> > + for real_pkg_name in packages:
> > + package = BuildrootPackage(real_pkg_name, pkg_folder)
> > + print('buildroot package name for
> > {}:'.format(package.real_name),
> > + package.name)
> > + # First we download the package
> > + # Most of the info we need can only be found inside
> > the package
> > + print('Package:', package.name)
> > + print('Fetching package', package.real_name)
> > + try:
> > + package.fetch_package_info()
> > + except (urllib2.URLError, urllib2.HTTPError):
> > + continue
> > +
> > + print('Downloading package {pkg}...'.format(
> > + pkg=package.metadata['info']['name']))
> > + try:
> > + package.download_package()
> > + except urllib2.HTTPError as error:
> > + print('Error: {code}
> > {reason}'.format(code=error.code,
> > +
> > reason=error.reason))
> > + print('Error downloading package :', package.name)
> > + continue
> > +
> > + # extract the tarball
> > + package.extract_package(tmp_path)
> > + print(package.metadata_name)
> > + print(package.tmp_extract)
>
> These two prints are redundant.
debug prints again.
>
> > +
> > + # Loading the package install info from the package
> > + package.load_setup()
> > +
> > + # Package requirement are an argument of the setup
> > function
> > + package.get_requirements(pkg_folder, packages)
> > + packages += package.req_not_found
>
> Python doesn't allow updating a list while iterating over it. But I
> believe
> there is something in itertools that supports that.
I thought so too. But strangely, it works.
Try flask with option -o on an empty directory, scanpypi will
create Werkzeug, jinja2, itsdangerous and markupsafe as well.
It is probably not the best way to do it, but it works fine.
>
> > + if package.req_not_found:
> > + print('Added packages \'{pkgs}\' as dependencies
> > of {pkg}'
> > + .format(pkgs=",
> > ".join(package.req_not_found),
> > + pkg=package.name))
> > + print('Checking if package {name} already
> > exists...'.format(
> > + name=package.pkg_dir))
> > + try:
> > + os.makedirs(package.pkg_dir)
> > + except OSError as exception:
> > + if exception.errno != errno.EEXIST:
> > + print("ERROR: ", exception.message,
> > file=sys.stderr)
> > + continue
> > + print('Error: Package {name} already exists'
> > + .format(name=package.pkg_dir))
> > + del_pkg = raw_input(
> > + 'Do you want to delete existing package ?
> > [y/N]')
> > + if del_pkg.lower() == 'y':
> > + shutil.rmtree(package.pkg_dir)
> > + os.makedirs(package.pkg_dir)
> > + else:
> > + continue
> > + package.create_package_mk()
> > +
> > + package.create_hash_file()
> > +
> > + package.create_config_in()
> > + print()
> > + # printing an empty line for visual confort
> > + except:
> > + shutil.rmtree(tmp_path)
> > + raise
> > + else:
> > + shutil.rmtree(tmp_path)
>
> Isn't finally: supposed to do exactly what you're trying to do here?
Yes it is
>
> Regards,
> Arnout
Regards,
Denis
>
> > +
> > +if __name__ == "__main__":
> > + main()
> >
>
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR
> Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2015-07-09 13:31 ` [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
@ 2015-07-11 12:56 ` Arnout Vandecappelle
2015-07-15 14:08 ` Denis Thulin
0 siblings, 1 reply; 14+ messages in thread
From: Arnout Vandecappelle @ 2015-07-11 12:56 UTC (permalink / raw)
To: buildroot
On 07/09/15 15:31, Denis THULIN wrote:
> An utility for creating python package from the python package index
It would make sense to list here the packages with which you have tested it.
I've tried it for about 5 packages and it worked for none of them... Usually the
problem is that setup.py either has an if __name__ == '__main__', or it tries to
load other modules from the package.
I think you should at least test it with the packages that already are in
buildroot. django, pyzmq, six, twisted are a few nice examples. This also allows
you to see how the automatically generated files differ from what we have already.
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> ---
> v0: initial commit
> python-pacakage-generator.py is an utility for automatically generating a
> python package. It fetches packages info from http://pypi.python.org and
> generates corresponding packages files.
Actually, this sentence should go into the commit log as well.
>
> v1:
> - renamed python-package-generator to scanpypi
> - split the huge script into a lot of functions
> - fixed mistakes and small bugs
>
> v2:
> - Rewrited most of the functions into a class
> - Changed the method for importing setup.py
And what was the reason for that change? It turns out not to work so well...
> - Created a main function to avoid use of global variable
> - Now adds new dependencies to the list of packages to create
> - Droppped the .py extension
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> ---
> docs/manual/adding-packages-python.txt | 32 ++
> support/scripts/scanpypi | 620 +++++++++++++++++++++++++++++++++
> 2 files changed, 652 insertions(+)
> create mode 100755 support/scripts/scanpypi
>
> diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
> index f81d625..dcde08d 100644
> --- a/docs/manual/adding-packages-python.txt
> +++ b/docs/manual/adding-packages-python.txt
> @@ -7,6 +7,38 @@ This infrastructure applies to Python packages that use the standard
> Python setuptools mechanism as their build system, generally
> recognizable by the usage of a +setup.py+ script.
>
> +[[scanpypi]]
> +
> +==== Generating a +python-package+ from a pypi repository
> +
> +You may want to use the +scanpypi.py+ located in
Drop the .py :-) and rewrap.
> ++support/script+ to generate a package from an existing pypi(pip) package.
PyPI is capitalized differently. Rephrase:
... from an existing PyPI package (i.e. a package that can be installed with pip).
> +
> +you can find the list of existing pypi package here: https://pypi.python.org .
You can find the list of existing PyPI packages https://pypi.python.org[here].
> +
> +Please keep in mind that you most likely need to manually check the package for
> +any mistakes as there are things that cannot be guessed by the generator (e.g.
> +dependencies on any of the python core modules such as BR2_PACKAGE_PYTHON_ZLIB)
Also useful te mention: the license and license files (they're a bit of ad hoc
guesswork).
> +. You need no manually add the package to the +package/Config.in+ file.
That . should still be on the preceding line, without space in front of it.
You also need to manually add the package the the +package/Config.in+ file.
> +
> +When at the root of your buildroot directory just do :
> +
> +-----------------------
> +./support/script/scanpypi.py foo bar -o package
s/.py//
> +-----------------------
> +
> +This will generate packages +python-foo+ and +python-bar+ in the package
> +folder if they exist on https://pypi.python.org.
> +
> +Find the +external python modules+ menu and insert your package inside.
> +Keep in mind that the items inside a menu should be in alphabetical order.
I would also make an explicit reference here to how to use it with
BR2_EXTERNAL. I.e., using the -o flag.
> +
> +Option +-h+ wil list the options available
> +
> +-----------------------
> +./support/script/scanpypi.py -h
s/.py//
> +-----------------------
> +
> [[python-package-tutorial]]
>
> ==== +python-package+ tutorial
> diff --git a/support/scripts/scanpypi b/support/scripts/scanpypi
> new file mode 100755
> index 0000000..e98e8f8
> --- /dev/null
> +++ b/support/scripts/scanpypi
> @@ -0,0 +1,620 @@
> +#!/usr/bin/python2
It would be nice to make it python3 compliant as well, but that can be done later.
> +"""
> +Utility for building buildroot packages for existing pypi packages
> +
> +Any package built by scanpypi should be manually checked for
> +errors.
> +"""
> +from __future__ import print_function
> +import argparse
> +import json
> +import urllib2
> +import sys
> +import os
> +import shutil
> +import StringIO
> +import tarfile
> +import errno
> +import hashlib
> +import re
> +import textwrap
> +import tempfile
> +import imp
> +from functools import wraps
> +
> +
> +# private global
> +_calls = {}
I guess this could be converted into a class member of BuildrootPackage.
Also, the name could be a bit better, e.g. setup_args.
> +
> +
> +def setup_info(pkg_name):
> + """Get a package info from _calls
> +
> + Keyword arguments:
> + pkg_name -- the name of the package
> + """
> + return _calls[pkg_name]
I still don't think it's worthwhile to make a function from this.
> +
> +
> +def setup_decorator(func, method):
> + """
> + Decorator for distutils.core.setup and setuptools.setup.
> + Puts the args of setup as a dict inside global private dict _calls.
I finally understand what this does :-) Even better:
Puts the arguments with which setup is called as a dict ...
> + Add key 'method' which should be either 'setuptools' or 'distutils'.
> +
> + Keyword arguments:
> + func -- either setuptools.setup or distutils.core.setup
> + method -- either 'setuptools' or 'distutils'
> + """
> +
> + @wraps(func)
> + def closure(*args, **kwargs):
> + _calls[kwargs['name']] = kwargs
Perhaps document where this 'name' comes from. It's quite obvious if you're
familiar with distutils, but this wrapping is not easy to understand so it's
worthwhile to mention it.
> + _calls[kwargs['name']]['method'] = method
> + return closure
> +
> +
> +def find_file_upper_case(filenames, path='./'):
> + """
> + List generator:
> + Recursively find files that matches one of the specified filenames.
> + Returns absolute path
No it doesn't, it returns a relative path starting with path.
> +
> + Keyword arguments:
> + filenames -- List of filenames to be found
> + path -- Path to the directory to search
> + """
> + for root, dirs, files in os.walk(path):
> + for file in files:
> + if file.upper() in filenames:
> + yield (os.path.join(root, file))
> +
> +
> +def pkg_buildroot_name(pkg_name):
> + """
> + Returns name to avoid troublesome characters.
No it doesn't.
Returns the buildroot package name for the PyPI package pkg_name.
In fact, the function should also prepend the python- prefix, because the
buildroot name starts with python-.
> + Remove all non alphanumeric characters except -
> + Also lowers the name
> +
> + Keyword arguments:
> + pkg_name -- String to rename
> + """
> + name = re.sub('[^\w-]', '', pkg_name.lower())
> + name = re.sub('^python-', '', name)
> + return name
> +
> +
> +# monkey patch
> +import setuptools
> +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> +import distutils
> +distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
This should go just below the definition of the decorator.
> +
> +
> +class SetupNotFound(Exception):
> + pass
> +
> +
> +class BuildrootPackage():
> + """
> + This class's methods are not meant to be used individually please use those
> + in the correct order:
> + __init__
> +
> + download_package
> +
> + extract_package
> +
> + load_module
> +
> + get_requirements
> +
> + create_package_mk
> +
> + create_hash_file
> +
> + create_config_in
> + """
> + def __init__(self, real_name, pkg_folder):
> + self.real_name = real_name
> + self.name = pkg_buildroot_name(self.real_name)
Perhaps to clarify the distinction, call it buildroot_name instead of just name.
> + self.pkg_dir = pkg_folder + '/python-' + self.name
It's good practice to set all the members you're going to create to None in
__init__, as a kind of declaration of what members are available.
> +
> + def find_setup(self, folder):
> + """
> + Search for setup.py file in an archive and returns the path to this
> + file
> + if it is found
Wrapping.
> +
> + Keyword arguments:
> + folder -- path to search in
> + """
> + filename = 'setup.py'
> + # Next return the first element of an iterable,
> + # here, it returns the path to the first file named "setup.py"
> + # found in the python package or raises StopIteration if not found
> + self.setup_location = next(os.path.join(root, filename)
> + for root, dirs, files in os.walk(folder)
> + if filename in files)
Is it just me or is setup_location not used anywhere? I anyway don't think this
is a good idea: if setup.py is not in the top dir, most likely other things in
this script will break.
So the only thing this function does, really, is checking if there is any
setup.py file in the folder. Which is later done again by calling
imp.find_module, which will anyway raise an ImportError if the module isn't found.
IOW, remove this function.
> +
> + def fetch_package_info(self):
> + """
> + Fetch a package's metadata for the python package index
for -> from
> + """
> + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
Already asign to metadata_url here.
> + pkg=self.real_name)
> + print('URL:', url)
This print is not needed.
> + try:
> + pkg_json = urllib2.urlopen(url).read().decode()
> + except urllib2.HTTPError as error:
> + print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '
> + .format(pkg=self.real_name))
> + raise
> + except urllib2.URLError:
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '
> + .format(pkg=self.real_name))
> + raise
> + else:
Else is not needed here since the exceptions are raise'd.
> + self.metadata_url = url
> + self.metadata = json.loads(pkg_json)
> + self.metadata_name = self.metadata['info']['name']
> +
> + def download_package(self):
> + """
> + Download a package using metadata from pypi
> + """
> + try:
> + self.metadata['urls'][0]['filename']
> + except IndexError:
> + print(
> + 'Non conventional package, ',
Non-conventional
> + 'please check manually after creation')
manually -> carefully
> + download_url = self.metadata['info']['download_url']
I think it would be simpler to, instead of repeating the download infra, just
emulate the needed metadata and use the same infra as below. So:
download_url = [{
'packagetype': 'tgz',
'url', self.metadata['info']['download_url'],
'md5_digest': None,
}]
> + try:
> + self.download = urllib2.urlopen(download_url)
self.download is not used outside this function, so make it a local variable.
Or actually, drop it completely and read directly into self.as_string().
> + except urllib2.HTTPError as http_error:
> + self.download = http_error
> + else:
> + self.as_string = self.download.read()
> + self.used_url = {'url': download_url,
> + 'md5_digest': hashlib.md5(self.as_string)
> + .hexdigest()}
I don't think it makes a lot os sense to compute the md5 here. Better set it to
None and not mention it in the .hash file.
> + # In this case, we can't get the name of the downloaded file
> + # from the pypi api, so we need to find it, this should work
> + urlpath = urllib2.urlparse.urlparse(self.download.url).path
> + # urlparse().path give something like
> + # /path/to/file-version.tar.gz
> + # We use basename to remove /path/to
> + self.targz = os.path.basename(urlpath)
> + self.used_url['filename'] = self.targz
This magic is still needed to set download_url['filename'], but it doesn't need
to be downloaded yet to do that.
> + else:
> + for download_url in self.metadata['urls']:
> + if 'wheel' in download_url['packagetype']:
This could be zip or egg as well. Isn't there a positive match possible?
> + continue
> + try:
I think it's useful to print the download URL here, just before downloading. So
move the print ('Downloading package ...') here instead, and add the download
URL to it.
> + self.download = urllib2.urlopen(download_url['url'])
> + except urllib2.HTTPError as http_error:
> + self.download = http_error
This is only used to raise it later, so you can just drop the entire try block
and let the exception leak up.
> + else:
> + self.used_url = download_url
> + self.as_string = self.download.read()
> + self.md5_sum = hashlib.md5(self.as_string).hexdigest()
> + self.targz = self.used_url['filename']
> + if self.md5_sum == download_url['md5_digest']:
This should probably raise an exception if it doesn't match.
> + break
The for loop needs an else: in case none of them can be downloaded.
> + if self.download.__class__ == urllib2.HTTPError:
> + raise self.download
> + else:
> + self.digest_method = 'sha256'
This is always sha256 so no point making a variable for it.
> + self.digest = hashlib.sha256(self.as_string).hexdigest()
Minor nit: it's used only once, so I'd calculate it at the time that you use it.
> +
> + def extract_package(self, tmp_path):
> + """
> + Create folders used for extracting a package as file object and extract
> + it
A bit too detailed...
Extract the package contents into a directory.
> +
> + Keyword arguments:
> + tmp_path -- folder where you want the package to be extracted
s/folder/directory/g
> + """
> + as_file = StringIO.StringIO(self.as_string)
> + with tarfile.open(fileobj=as_file) as as_tarfile:
> + tmp_pkg = os.path.join(tmp_path, self.name)
> + try:
> + os.makedirs(tmp_pkg)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + return None, None
> + print('WARNING:', exception.message, file=sys.stderr)
> + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> + shutil.rmtree(tmp_pkg)
> + os.makedirs(tmp_pkg)
> + as_tarfile.extractall(tmp_pkg)
> + try:
> + self.find_setup(tmp_pkg)
> + except StopIteration:
> + raise SetupNotFound('Could not find file setup.py for package {}'
> + .format(self.real_name))
As I said before, this is not needed since it is covered by load_setup().
> + tmp_extract = '{folder}/{name}-{version}'
Is this guaranteed to be the directory used by the package?
> + self.tmp_extract = tmp_extract.format(
> + folder=tmp_pkg,
> + name=self.metadata_name,
> + version=self.metadata['info']['version'])
> +
> + def load_setup(self):
> + """
> + Loads the corresponding setup and store its metadata
> + """
> + s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract])
> + imp.load_module('setup', s_file, s_path, s_desc)
> + self.setup_metadata = setup_info(self.metadata_name)
Is the name given in the metadata guaranteed to be the same as what is passed
to setup()? Especially considering case...
> +
> + def get_requirements(self, pkg_folder, packages):
> + """
> + Retrieve dependencies from the metadata found in the setup.py script of
> + a pypi package.
> +
> + Keyword Arguments:
> + pkg_folder -- location of the already created packages
> + packages -- other packages to be built
> + """
> + if 'install_requires' not in self.setup_metadata:
> + self.pkg_req = None
> + self.req_not_found = []
> + return
> + self.pkg_req = self.setup_metadata['install_requires']
> + self.pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
The .lower() is handled by pkg_buildroot_name so not needed here.
> + for req in self.pkg_req]
> + self.pkg_req = map(pkg_buildroot_name, self.pkg_req)
> + self.req_not_found = [
> + pkg for pkg in self.pkg_req
> + if 'python-{name}'.format(name=pkg)
pkg_buildroot_name already adds python- (or at least it should :-).
> + not in os.listdir(pkg_folder)
Maybe better use "not os.path.isdir(pkg)".
> + ]
> + self.req_not_found = [pkg for pkg in self.req_not_found
> + if pkg not in packages]
Small nit: I'd make this function just return req_not_found instead of adding
it as a member, and move the set.difference logic up to the caller. Oh, and
perhaps use a set :-)
> +
> + def __create_mk_header(self):
> + """
> + Create the header of the <package_name>.mk file
> + """
> + header = ['#' * 80 + '\n']
> + header.append('#\n')
> + header.append('# python-{name}\n'.format(name=self.name))
> + header.append('#\n')
> + header.append('#' * 80 + '\n')
> + header.append('\n')
> + return header
> +
> + def __create_mk_download_info(self):
> + """
> + Create the lines refering to the download information of the
> + <package_name>.mk file
> + """
> + lines = []
> + version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
> + name=self.name.upper().replace('-', '_'),
Like you have pkg_buildroot_name, you should also have something for the
conversion to uppercase. Becasue below, you forget the replace('-', '_'). Also
it's probably best to add the uppercase name as a member.
> + version=self.metadata['info']['version'])
I think it makes sense to add these things as direct members of
BuildrootPackage rather than going through metadata all the time: version, url,
filename.
> + lines.append(version_line)
> +
> + targz = self.targz.replace(
> + self.metadata['info']['version'],
> + '$(PYTHON_{name}_VERSION)'.format(name=self.name.upper()))
> + targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
> + name=self.name.upper(),
> + filename=targz)
> + lines.append(targz_line)
> +
> + site_line = ('PYTHON_{name}_SITE = {url}'.format(
> + name=self.name.upper(),
> + url=self.used_url['url'].replace(self.used_url['filename'], '')))
> + if 'sourceforge' in site_line:
> + site_line = ('PYTHON_{name}_SITE = {url}'.format(
> + name=self.name.upper(),
> + url=self.used_url['url']))
This looks very weird... At least add an explanatory comment. And anyway, it's
better to do any fixups in a new variable (e.g. 'site') and format the line only
once.
> + site_line = site_line.rstrip('/') + '\n'
> + lines.append(site_line)
> + return lines
> +
> + def __create_mk_setup(self):
> + """
> + Create the line refering to the setup method of the package of the
> + <package_name>.mk file
> +
> + There are two things you can use to make an installer
> + for a python package: distutils or setuptools
> + distutils comes with python but does not support dependencies.
> + distutils is mostly still there for backward support.
> + setuptools is what smart people use,
> + but it is not shipped with python :(
This comment is a bit redundant, but OK.
> + """
> + lines = []
> + setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
> + name=self.name.upper(),
> + method=self.setup_metadata['method'])
> + lines.append(setup_type_line)
> + return lines
> +
> + def __create_mk_license(self):
> + """
> + Create the lines referring to the package's license informations of the
> + <package_name>.mk file
> +
> + The license is found using the metadata from pypi.
> + In the metadata, the license can be found either with standard names in
> + the classifiers part or with naming from the packager in the "License"
> + part.
> +
> + From the classifiers, the license is "translated" according to
> + buildroot standards if need be (i.e. from Apache Software License to
> + Apache-2.0).
> +
> + From the License part, we cannot guess what formatting the packager
> + used. Hence, it is likely to be incorrect. (i.e. Apache License 2.0
> + instead of Apache-2.0).
> +
> + The license's files are found by searching the package for files named
> + license or license.txt (case insensitive).
> + If more than one license file is found, the user is asked to select
> + which ones he wants to use.
> + """
> + license_dict = {
> + 'Apache Software License': 'Apache-2.0',
> + 'BSD License': 'BSD',
We normally put BSD-2c, BSD-3c, etc. but it may be difficult to find that out
automatically.
> + 'European Union Public Licence 1.0': 'EUPLv1.0',
> + 'European Union Public Licence 1.1': 'EUPLv1.1',
> + "GNU General Public License": "GPL",
> + "GNU General Public License v2": "GPLv2",
> + "GNU General Public License v2 or later": "GPLv2+",
> + "GNU General Public License v3": "GPLv3",
> + "GNU General Public License v3 or later": "GPLv3+",
> + "GNU Lesser General Public License v2": "LGPLv2",
> + "GNU Lesser General Public License v2 or later": "LGPLv2+",
That's actually LGPVLv2.1(+). LGPLv2 is the GNU Library General Public License.
> + "GNU Lesser General Public License v3": "LGPLv3",
> + "GNU Lesser General Public License v3 or later": "LGPLv3+",
> + "GNU Library or Lesser General Public License": "LGPL",
So this is probably LGPLv2 (there was no library/lesser version of v1).
> + "ISC License": "ISC",
> + "MIT License": "MIT",
> + "Mozilla Public License 1.0": "MPL-1.0",
> + "Mozilla Public License 1.1": "MPL-1.1",
> + "Mozilla Public License 2.0": "MPL-2.0",
> + "Zope Public License": "ZPL"
> + }
Nice list!
> + regexp = re.compile('^License :* *.* *:+ (.*)( \(.*\))?$')
> + classifiers_licenses = [regexp.sub(r"\1", lic)
> + for lic in self.metadata['info']['classifiers']
> + if regexp.match(lic)]
> + licenses = map(lambda x: license_dict[x] if x in license_dict else x,
> + classifiers_licenses)
I would give an explicit warning if the license is not in license_dict, since
it most likely means it's wrong.
> + lines = []
> + if not len(licenses):
> + licenses = [self.metadata['info']['license']]
This should probably go before the license_dict map.
> + license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
> + name=self.name.upper(),
> + license=', '.join(licenses))
> + lines.append(license_line)
> + print('WARNING: License has been set to "{license}",'
> + ' please change it manually if necessary'.format(
> + license=', '.join(licenses)))
So this warning is not necessary if it was found in the dict.
> +
> + filenames = ['LICENSE', 'LICENSE.TXT']
Shouldn't we also look for COPYING?
> + license_files = list(find_file_upper_case(filenames, self.tmp_extract))
> + license_files = [license.replace(self.tmp_extract, '')[1:]
> + for license in license_files]
> + if len(license_files) > 0:
> + if len(license_files) > 1:
> + print('More than one file found for license:',
> + ', '.join(license_files))
> + license_files = [filename
> + for index, filename in enumerate(license_files)]
> + license_file_line = ('PYTHON_{name}_LICENSE_FILES ='
> + ' {files}\n'.format(
> + name=self.name.upper(),
> + files=' '.join(license_files)))
> + lines.append(license_file_line)
> + elif len(license_files) == 0:
Can len(license_files) be anything else than 0 at this point?
> + print('WARNING: No license file found,'
> + ' please specify it manually afterward')
afterwards
> + license_file_line = '# No license file found\n'
> +
> + return lines
> +
> + def __create_mk_requirements(self):
> + """
> + Create the lines referring to the dependencies of the of the
> + <package_name>.mk file
> +
> + Keyword Arguments:
> + pkg_name -- name of the package
> + pkg_req -- dependencies of the package
> + """
> + lines = []
> + python_pkg_req = ['python-{name}'.format(name=pkg)
> + for pkg in self.pkg_req]
> + dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
> + ' {reqs}\n'.format(
> + name=self.name.upper(),
> + reqs=' '.join(python_pkg_req)))
> + lines.append(dependencies_line)
> + return lines
> +
> + def create_package_mk(self):
> + """
> + Create the lines corresponding to the <package_name>.mk file
> + """
> + pkg_mk = 'python-{name}.mk'.format(name=self.name)
> + path_to_mk = os.path.join(self.pkg_dir, pkg_mk)
> + print('Creating {file}...'.format(file=path_to_mk))
> + lines = self.__create_mk_header()
> + lines += self.__create_mk_download_info()
> + lines += self.__create_mk_setup()
> + lines += self.__create_mk_license()
> + if self.pkg_req:
> + lines += self.__create_mk_requirements()
> +
> + lines.append('\n')
> + lines.append('$(eval $(python-package))')
> + lines.append('\n')
> + with open(path_to_mk, 'w') as mk_file:
> + mk_file.writelines(lines)
> +
> + def create_hash_file(self):
> + """
> + Create the lines corresponding to the <package_name>.hash files
> + """
> + pkg_hash = 'python-{name}.hash'.format(name=self.name)
> + path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
> + print('Creating {filename}...'.format(filename=path_to_hash))
> + lines = []
> + commented_line = '# md5 from {url}, {method} calculated by scanpypi\n'
Actually, the md5 comes from the pypi/json url, so mention that one. So just:
# From https://pypi.python.org/pypi/{pkg}/json
> + commented_line = commented_line.format(url=self.used_url['url'],
> + method=self.digest_method)
> + lines.append(commented_line)
> + hash_line = '{method}\t{digest} {filename}\n'.format(
> + method='md5',
> + digest=self.used_url['md5_digest'],
> + filename=self.used_url['filename'])
The md5 should only be added if there really was an md5.
> + lines.append(hash_line)
> + hash_line = '{method}\t{digest} {filename}\n'.format(
> + method=self.digest_method,
> + digest=self.digest,
> + filename=self.used_url['filename'])
> + lines.append(hash_line)
> +
> + with open(path_to_hash, 'w') as hash_file:
> + hash_file.writelines(lines)
> +
> + def create_config_in(self):
> + """
> + Creates the Config.in file of a package
> + """
> + path_to_config = os.path.join(self.pkg_dir, 'Config.in')
> + print('Creating {file}...'.format(file=path_to_config))
> + lines = []
> + config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
> + name=self.name.upper())
> + lines.append(config_line)
> +
> + bool_line = '\tbool "python-{name}"\n'.format(name=self.name)
> + lines.append(bool_line)
> + if self.pkg_req:
> + for dep in self.pkg_req:
> + dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
> + req=dep.upper())
> + lines.append(dep_line)
> +
> + lines.append('\thelp\n')
> +
> + help_lines = textwrap.wrap(self.metadata['info']['summary'], 67)
I'm not really sure if we should use the summary or the description here... The
summary is _really_ short.
Also, use textwrap.fill and add the initial_indent='\t ' argument, then all
the mangling below should no longer be necessary.
> + # \t + two spaces is 3 char long
> + help_lines.append('')
> + help_lines.append(self.metadata['info']['home_page'])
> + help_lines = map(lambda l: '\t {}\n'.format(l)
> + if not l == '' else '\n',
> + help_lines)
> + # help_lines = ['\t {line}\n'.format(line=line).replace('\t \n','\n')
> + # for line in help_lines]
> + lines += help_lines
> +
> + with open(path_to_config, 'w') as config_file:
> + config_file.writelines(lines)
> +
> +
> +def main():
> + # Building the parser
> + parser = argparse.ArgumentParser(
> + description="Creates buildroot packages from the metadata of "
> + "an existing pypi(pip) packages and include it "
PyPI
> + "in menuconfig")
> + parser.add_argument("packages",
> + help="list of packages to be made",
made -> created
> + nargs='+')
> + parser.add_argument("-o", "--output",
> + help="""
> + Output directory for packages
Please explicitly mention the default in the help text.
> + """,
> + default='./package')
> +
> + args = parser.parse_args()
> + packages = list(set(args.packages))
> +
> + # tmp_path is where we'll extract the files later
> + tmp_prefix = 'scanpypi-'
> + pkg_folder = args.output
> + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
> + try:
> + for real_pkg_name in packages:
> + package = BuildrootPackage(real_pkg_name, pkg_folder)
> + print('buildroot package name for {}:'.format(package.real_name),
> + package.name)
> + # First we download the package
> + # Most of the info we need can only be found inside the package
> + print('Package:', package.name)
> + print('Fetching package', package.real_name)
> + try:
> + package.fetch_package_info()
> + except (urllib2.URLError, urllib2.HTTPError):
> + continue
> +
> + print('Downloading package {pkg}...'.format(
> + pkg=package.metadata['info']['name']))
> + try:
> + package.download_package()
> + except urllib2.HTTPError as error:
> + print('Error: {code} {reason}'.format(code=error.code,
> + reason=error.reason))
> + print('Error downloading package :', package.name)
> + continue
> +
> + # extract the tarball
> + package.extract_package(tmp_path)
> + print(package.metadata_name)
> + print(package.tmp_extract)
These two prints are redundant.
> +
> + # Loading the package install info from the package
> + package.load_setup()
> +
> + # Package requirement are an argument of the setup function
> + package.get_requirements(pkg_folder, packages)
> + packages += package.req_not_found
Python doesn't allow updating a list while iterating over it. But I believe
there is something in itertools that supports that.
> + if package.req_not_found:
> + print('Added packages \'{pkgs}\' as dependencies of {pkg}'
> + .format(pkgs=", ".join(package.req_not_found),
> + pkg=package.name))
> + print('Checking if package {name} already exists...'.format(
> + name=package.pkg_dir))
> + try:
> + os.makedirs(package.pkg_dir)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('Error: Package {name} already exists'
> + .format(name=package.pkg_dir))
> + del_pkg = raw_input(
> + 'Do you want to delete existing package ? [y/N]')
> + if del_pkg.lower() == 'y':
> + shutil.rmtree(package.pkg_dir)
> + os.makedirs(package.pkg_dir)
> + else:
> + continue
> + package.create_package_mk()
> +
> + package.create_hash_file()
> +
> + package.create_config_in()
> + print()
> + # printing an empty line for visual confort
> + except:
> + shutil.rmtree(tmp_path)
> + raise
> + else:
> + shutil.rmtree(tmp_path)
Isn't finally: supposed to do exactly what you're trying to do here?
Regards,
Arnout
> +
> +if __name__ == "__main__":
> + main()
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Buildroot] [PATCH 1/2] scanpypi: new utility
2015-07-09 13:31 [Buildroot] [PATCH 0/2] python-package-generator Denis THULIN
@ 2015-07-09 13:31 ` Denis THULIN
2015-07-11 12:56 ` Arnout Vandecappelle
0 siblings, 1 reply; 14+ messages in thread
From: Denis THULIN @ 2015-07-09 13:31 UTC (permalink / raw)
To: buildroot
An utility for creating python package from the python package index
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0: initial commit
python-pacakage-generator.py is an utility for automatically generating a
python package. It fetches packages info from http://pypi.python.org and
generates corresponding packages files.
v1:
- renamed python-package-generator to scanpypi
- split the huge script into a lot of functions
- fixed mistakes and small bugs
v2:
- Rewrited most of the functions into a class
- Changed the method for importing setup.py
- Created a main function to avoid use of global variable
- Now adds new dependencies to the list of packages to create
- Droppped the .py extension
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
docs/manual/adding-packages-python.txt | 32 ++
support/scripts/scanpypi | 620 +++++++++++++++++++++++++++++++++
2 files changed, 652 insertions(+)
create mode 100755 support/scripts/scanpypi
diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
index f81d625..dcde08d 100644
--- a/docs/manual/adding-packages-python.txt
+++ b/docs/manual/adding-packages-python.txt
@@ -7,6 +7,38 @@ This infrastructure applies to Python packages that use the standard
Python setuptools mechanism as their build system, generally
recognizable by the usage of a +setup.py+ script.
+[[scanpypi]]
+
+==== Generating a +python-package+ from a pypi repository
+
+You may want to use the +scanpypi.py+ located in
++support/script+ to generate a package from an existing pypi(pip) package.
+
+you can find the list of existing pypi package here: https://pypi.python.org .
+
+Please keep in mind that you most likely need to manually check the package for
+any mistakes as there are things that cannot be guessed by the generator (e.g.
+dependencies on any of the python core modules such as BR2_PACKAGE_PYTHON_ZLIB)
+. You need no manually add the package to the +package/Config.in+ file.
+
+When at the root of your buildroot directory just do :
+
+-----------------------
+./support/script/scanpypi.py foo bar -o package
+-----------------------
+
+This will generate packages +python-foo+ and +python-bar+ in the package
+folder if they exist on https://pypi.python.org.
+
+Find the +external python modules+ menu and insert your package inside.
+Keep in mind that the items inside a menu should be in alphabetical order.
+
+Option +-h+ wil list the options available
+
+-----------------------
+./support/script/scanpypi.py -h
+-----------------------
+
[[python-package-tutorial]]
==== +python-package+ tutorial
diff --git a/support/scripts/scanpypi b/support/scripts/scanpypi
new file mode 100755
index 0000000..e98e8f8
--- /dev/null
+++ b/support/scripts/scanpypi
@@ -0,0 +1,620 @@
+#!/usr/bin/python2
+"""
+Utility for building buildroot packages for existing pypi packages
+
+Any package built by scanpypi should be manually checked for
+errors.
+"""
+from __future__ import print_function
+import argparse
+import json
+import urllib2
+import sys
+import os
+import shutil
+import StringIO
+import tarfile
+import errno
+import hashlib
+import re
+import textwrap
+import tempfile
+import imp
+from functools import wraps
+
+
+# private global
+_calls = {}
+
+
+def setup_info(pkg_name):
+ """Get a package info from _calls
+
+ Keyword arguments:
+ pkg_name -- the name of the package
+ """
+ return _calls[pkg_name]
+
+
+def setup_decorator(func, method):
+ """
+ Decorator for distutils.core.setup and setuptools.setup.
+ Puts the args of setup as a dict inside global private dict _calls.
+ Add key 'method' which should be either 'setuptools' or 'distutils'.
+
+ Keyword arguments:
+ func -- either setuptools.setup or distutils.core.setup
+ method -- either 'setuptools' or 'distutils'
+ """
+
+ @wraps(func)
+ def closure(*args, **kwargs):
+ _calls[kwargs['name']] = kwargs
+ _calls[kwargs['name']]['method'] = method
+ return closure
+
+
+def find_file_upper_case(filenames, path='./'):
+ """
+ List generator:
+ Recursively find files that matches one of the specified filenames.
+ Returns absolute path
+
+ Keyword arguments:
+ filenames -- List of filenames to be found
+ path -- Path to the directory to search
+ """
+ for root, dirs, files in os.walk(path):
+ for file in files:
+ if file.upper() in filenames:
+ yield (os.path.join(root, file))
+
+
+def pkg_buildroot_name(pkg_name):
+ """
+ Returns name to avoid troublesome characters.
+ Remove all non alphanumeric characters except -
+ Also lowers the name
+
+ Keyword arguments:
+ pkg_name -- String to rename
+ """
+ name = re.sub('[^\w-]', '', pkg_name.lower())
+ name = re.sub('^python-', '', name)
+ return name
+
+
+# monkey patch
+import setuptools
+setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
+import distutils
+distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
+
+
+class SetupNotFound(Exception):
+ pass
+
+
+class BuildrootPackage():
+ """
+ This class's methods are not meant to be used individually please use those
+ in the correct order:
+ __init__
+
+ download_package
+
+ extract_package
+
+ load_module
+
+ get_requirements
+
+ create_package_mk
+
+ create_hash_file
+
+ create_config_in
+ """
+ def __init__(self, real_name, pkg_folder):
+ self.real_name = real_name
+ self.name = pkg_buildroot_name(self.real_name)
+ self.pkg_dir = pkg_folder + '/python-' + self.name
+
+ def find_setup(self, folder):
+ """
+ Search for setup.py file in an archive and returns the path to this
+ file
+ if it is found
+
+ Keyword arguments:
+ folder -- path to search in
+ """
+ filename = 'setup.py'
+ # Next return the first element of an iterable,
+ # here, it returns the path to the first file named "setup.py"
+ # found in the python package or raises StopIteration if not found
+ self.setup_location = next(os.path.join(root, filename)
+ for root, dirs, files in os.walk(folder)
+ if filename in files)
+
+ def fetch_package_info(self):
+ """
+ Fetch a package's metadata for the python package index
+ """
+ url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
+ pkg=self.real_name)
+ print('URL:', url)
+ try:
+ pkg_json = urllib2.urlopen(url).read().decode()
+ except urllib2.HTTPError as error:
+ print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '
+ .format(pkg=self.real_name))
+ raise
+ except urllib2.URLError:
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '
+ .format(pkg=self.real_name))
+ raise
+ else:
+ self.metadata_url = url
+ self.metadata = json.loads(pkg_json)
+ self.metadata_name = self.metadata['info']['name']
+
+ def download_package(self):
+ """
+ Download a package using metadata from pypi
+ """
+ try:
+ self.metadata['urls'][0]['filename']
+ except IndexError:
+ print(
+ 'Non conventional package, ',
+ 'please check manually after creation')
+ download_url = self.metadata['info']['download_url']
+ try:
+ self.download = urllib2.urlopen(download_url)
+ except urllib2.HTTPError as http_error:
+ self.download = http_error
+ else:
+ self.as_string = self.download.read()
+ self.used_url = {'url': download_url,
+ 'md5_digest': hashlib.md5(self.as_string)
+ .hexdigest()}
+ # In this case, we can't get the name of the downloaded file
+ # from the pypi api, so we need to find it, this should work
+ urlpath = urllib2.urlparse.urlparse(self.download.url).path
+ # urlparse().path give something like
+ # /path/to/file-version.tar.gz
+ # We use basename to remove /path/to
+ self.targz = os.path.basename(urlpath)
+ self.used_url['filename'] = self.targz
+ else:
+ for download_url in self.metadata['urls']:
+ if 'wheel' in download_url['packagetype']:
+ continue
+ try:
+ self.download = urllib2.urlopen(download_url['url'])
+ except urllib2.HTTPError as http_error:
+ self.download = http_error
+ else:
+ self.used_url = download_url
+ self.as_string = self.download.read()
+ self.md5_sum = hashlib.md5(self.as_string).hexdigest()
+ self.targz = self.used_url['filename']
+ if self.md5_sum == download_url['md5_digest']:
+ break
+ if self.download.__class__ == urllib2.HTTPError:
+ raise self.download
+ else:
+ self.digest_method = 'sha256'
+ self.digest = hashlib.sha256(self.as_string).hexdigest()
+
+ def extract_package(self, tmp_path):
+ """
+ Create folders used for extracting a package as file object and extract
+ it
+
+ Keyword arguments:
+ tmp_path -- folder where you want the package to be extracted
+ """
+ as_file = StringIO.StringIO(self.as_string)
+ with tarfile.open(fileobj=as_file) as as_tarfile:
+ tmp_pkg = os.path.join(tmp_path, self.name)
+ try:
+ os.makedirs(tmp_pkg)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ return None, None
+ print('WARNING:', exception.message, file=sys.stderr)
+ print('Removing {pkg}...'.format(pkg=tmp_pkg))
+ shutil.rmtree(tmp_pkg)
+ os.makedirs(tmp_pkg)
+ as_tarfile.extractall(tmp_pkg)
+ try:
+ self.find_setup(tmp_pkg)
+ except StopIteration:
+ raise SetupNotFound('Could not find file setup.py for package {}'
+ .format(self.real_name))
+ tmp_extract = '{folder}/{name}-{version}'
+ self.tmp_extract = tmp_extract.format(
+ folder=tmp_pkg,
+ name=self.metadata_name,
+ version=self.metadata['info']['version'])
+
+ def load_setup(self):
+ """
+ Loads the corresponding setup and store its metadata
+ """
+ s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract])
+ imp.load_module('setup', s_file, s_path, s_desc)
+ self.setup_metadata = setup_info(self.metadata_name)
+
+ def get_requirements(self, pkg_folder, packages):
+ """
+ Retrieve dependencies from the metadata found in the setup.py script of
+ a pypi package.
+
+ Keyword Arguments:
+ pkg_folder -- location of the already created packages
+ packages -- other packages to be built
+ """
+ if 'install_requires' not in self.setup_metadata:
+ self.pkg_req = None
+ self.req_not_found = []
+ return
+ self.pkg_req = self.setup_metadata['install_requires']
+ self.pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
+ for req in self.pkg_req]
+ self.pkg_req = map(pkg_buildroot_name, self.pkg_req)
+ self.req_not_found = [
+ pkg for pkg in self.pkg_req
+ if 'python-{name}'.format(name=pkg)
+ not in os.listdir(pkg_folder)
+ ]
+ self.req_not_found = [pkg for pkg in self.req_not_found
+ if pkg not in packages]
+
+ def __create_mk_header(self):
+ """
+ Create the header of the <package_name>.mk file
+ """
+ header = ['#' * 80 + '\n']
+ header.append('#\n')
+ header.append('# python-{name}\n'.format(name=self.name))
+ header.append('#\n')
+ header.append('#' * 80 + '\n')
+ header.append('\n')
+ return header
+
+ def __create_mk_download_info(self):
+ """
+ Create the lines refering to the download information of the
+ <package_name>.mk file
+ """
+ lines = []
+ version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
+ name=self.name.upper().replace('-', '_'),
+ version=self.metadata['info']['version'])
+ lines.append(version_line)
+
+ targz = self.targz.replace(
+ self.metadata['info']['version'],
+ '$(PYTHON_{name}_VERSION)'.format(name=self.name.upper()))
+ targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
+ name=self.name.upper(),
+ filename=targz)
+ lines.append(targz_line)
+
+ site_line = ('PYTHON_{name}_SITE = {url}'.format(
+ name=self.name.upper(),
+ url=self.used_url['url'].replace(self.used_url['filename'], '')))
+ if 'sourceforge' in site_line:
+ site_line = ('PYTHON_{name}_SITE = {url}'.format(
+ name=self.name.upper(),
+ url=self.used_url['url']))
+ site_line = site_line.rstrip('/') + '\n'
+ lines.append(site_line)
+ return lines
+
+ def __create_mk_setup(self):
+ """
+ Create the line refering to the setup method of the package of the
+ <package_name>.mk file
+
+ There are two things you can use to make an installer
+ for a python package: distutils or setuptools
+ distutils comes with python but does not support dependencies.
+ distutils is mostly still there for backward support.
+ setuptools is what smart people use,
+ but it is not shipped with python :(
+ """
+ lines = []
+ setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
+ name=self.name.upper(),
+ method=self.setup_metadata['method'])
+ lines.append(setup_type_line)
+ return lines
+
+ def __create_mk_license(self):
+ """
+ Create the lines referring to the package's license informations of the
+ <package_name>.mk file
+
+ The license is found using the metadata from pypi.
+ In the metadata, the license can be found either with standard names in
+ the classifiers part or with naming from the packager in the "License"
+ part.
+
+ From the classifiers, the license is "translated" according to
+ buildroot standards if need be (i.e. from Apache Software License to
+ Apache-2.0).
+
+ From the License part, we cannot guess what formatting the packager
+ used. Hence, it is likely to be incorrect. (i.e. Apache License 2.0
+ instead of Apache-2.0).
+
+ The license's files are found by searching the package for files named
+ license or license.txt (case insensitive).
+ If more than one license file is found, the user is asked to select
+ which ones he wants to use.
+ """
+ license_dict = {
+ 'Apache Software License': 'Apache-2.0',
+ 'BSD License': 'BSD',
+ 'European Union Public Licence 1.0': 'EUPLv1.0',
+ 'European Union Public Licence 1.1': 'EUPLv1.1',
+ "GNU General Public License": "GPL",
+ "GNU General Public License v2": "GPLv2",
+ "GNU General Public License v2 or later": "GPLv2+",
+ "GNU General Public License v3": "GPLv3",
+ "GNU General Public License v3 or later": "GPLv3+",
+ "GNU Lesser General Public License v2": "LGPLv2",
+ "GNU Lesser General Public License v2 or later": "LGPLv2+",
+ "GNU Lesser General Public License v3": "LGPLv3",
+ "GNU Lesser General Public License v3 or later": "LGPLv3+",
+ "GNU Library or Lesser General Public License": "LGPL",
+ "ISC License": "ISC",
+ "MIT License": "MIT",
+ "Mozilla Public License 1.0": "MPL-1.0",
+ "Mozilla Public License 1.1": "MPL-1.1",
+ "Mozilla Public License 2.0": "MPL-2.0",
+ "Zope Public License": "ZPL"
+ }
+ regexp = re.compile('^License :* *.* *:+ (.*)( \(.*\))?$')
+ classifiers_licenses = [regexp.sub(r"\1", lic)
+ for lic in self.metadata['info']['classifiers']
+ if regexp.match(lic)]
+ licenses = map(lambda x: license_dict[x] if x in license_dict else x,
+ classifiers_licenses)
+ lines = []
+ if not len(licenses):
+ licenses = [self.metadata['info']['license']]
+ license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
+ name=self.name.upper(),
+ license=', '.join(licenses))
+ lines.append(license_line)
+ print('WARNING: License has been set to "{license}",'
+ ' please change it manually if necessary'.format(
+ license=', '.join(licenses)))
+
+ filenames = ['LICENSE', 'LICENSE.TXT']
+ license_files = list(find_file_upper_case(filenames, self.tmp_extract))
+ license_files = [license.replace(self.tmp_extract, '')[1:]
+ for license in license_files]
+ if len(license_files) > 0:
+ if len(license_files) > 1:
+ print('More than one file found for license:',
+ ', '.join(license_files))
+ license_files = [filename
+ for index, filename in enumerate(license_files)]
+ license_file_line = ('PYTHON_{name}_LICENSE_FILES ='
+ ' {files}\n'.format(
+ name=self.name.upper(),
+ files=' '.join(license_files)))
+ lines.append(license_file_line)
+ elif len(license_files) == 0:
+ print('WARNING: No license file found,'
+ ' please specify it manually afterward')
+ license_file_line = '# No license file found\n'
+
+ return lines
+
+ def __create_mk_requirements(self):
+ """
+ Create the lines referring to the dependencies of the of the
+ <package_name>.mk file
+
+ Keyword Arguments:
+ pkg_name -- name of the package
+ pkg_req -- dependencies of the package
+ """
+ lines = []
+ python_pkg_req = ['python-{name}'.format(name=pkg)
+ for pkg in self.pkg_req]
+ dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
+ ' {reqs}\n'.format(
+ name=self.name.upper(),
+ reqs=' '.join(python_pkg_req)))
+ lines.append(dependencies_line)
+ return lines
+
+ def create_package_mk(self):
+ """
+ Create the lines corresponding to the <package_name>.mk file
+ """
+ pkg_mk = 'python-{name}.mk'.format(name=self.name)
+ path_to_mk = os.path.join(self.pkg_dir, pkg_mk)
+ print('Creating {file}...'.format(file=path_to_mk))
+ lines = self.__create_mk_header()
+ lines += self.__create_mk_download_info()
+ lines += self.__create_mk_setup()
+ lines += self.__create_mk_license()
+ if self.pkg_req:
+ lines += self.__create_mk_requirements()
+
+ lines.append('\n')
+ lines.append('$(eval $(python-package))')
+ lines.append('\n')
+ with open(path_to_mk, 'w') as mk_file:
+ mk_file.writelines(lines)
+
+ def create_hash_file(self):
+ """
+ Create the lines corresponding to the <package_name>.hash files
+ """
+ pkg_hash = 'python-{name}.hash'.format(name=self.name)
+ path_to_hash = os.path.join(self.pkg_dir, pkg_hash)
+ print('Creating {filename}...'.format(filename=path_to_hash))
+ lines = []
+ commented_line = '# md5 from {url}, {method} calculated by scanpypi\n'
+ commented_line = commented_line.format(url=self.used_url['url'],
+ method=self.digest_method)
+ lines.append(commented_line)
+ hash_line = '{method}\t{digest} {filename}\n'.format(
+ method='md5',
+ digest=self.used_url['md5_digest'],
+ filename=self.used_url['filename'])
+ lines.append(hash_line)
+ hash_line = '{method}\t{digest} {filename}\n'.format(
+ method=self.digest_method,
+ digest=self.digest,
+ filename=self.used_url['filename'])
+ lines.append(hash_line)
+
+ with open(path_to_hash, 'w') as hash_file:
+ hash_file.writelines(lines)
+
+ def create_config_in(self):
+ """
+ Creates the Config.in file of a package
+ """
+ path_to_config = os.path.join(self.pkg_dir, 'Config.in')
+ print('Creating {file}...'.format(file=path_to_config))
+ lines = []
+ config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
+ name=self.name.upper())
+ lines.append(config_line)
+
+ bool_line = '\tbool "python-{name}"\n'.format(name=self.name)
+ lines.append(bool_line)
+ if self.pkg_req:
+ for dep in self.pkg_req:
+ dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
+ req=dep.upper())
+ lines.append(dep_line)
+
+ lines.append('\thelp\n')
+
+ help_lines = textwrap.wrap(self.metadata['info']['summary'], 67)
+ # \t + two spaces is 3 char long
+ help_lines.append('')
+ help_lines.append(self.metadata['info']['home_page'])
+ help_lines = map(lambda l: '\t {}\n'.format(l)
+ if not l == '' else '\n',
+ help_lines)
+ # help_lines = ['\t {line}\n'.format(line=line).replace('\t \n','\n')
+ # for line in help_lines]
+ lines += help_lines
+
+ with open(path_to_config, 'w') as config_file:
+ config_file.writelines(lines)
+
+
+def main():
+ # Building the parser
+ parser = argparse.ArgumentParser(
+ description="Creates buildroot packages from the metadata of "
+ "an existing pypi(pip) packages and include it "
+ "in menuconfig")
+ parser.add_argument("packages",
+ help="list of packages to be made",
+ nargs='+')
+ parser.add_argument("-o", "--output",
+ help="""
+ Output directory for packages
+ """,
+ default='./package')
+
+ args = parser.parse_args()
+ packages = list(set(args.packages))
+
+ # tmp_path is where we'll extract the files later
+ tmp_prefix = 'scanpypi-'
+ pkg_folder = args.output
+ tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
+ try:
+ for real_pkg_name in packages:
+ package = BuildrootPackage(real_pkg_name, pkg_folder)
+ print('buildroot package name for {}:'.format(package.real_name),
+ package.name)
+ # First we download the package
+ # Most of the info we need can only be found inside the package
+ print('Package:', package.name)
+ print('Fetching package', package.real_name)
+ try:
+ package.fetch_package_info()
+ except (urllib2.URLError, urllib2.HTTPError):
+ continue
+
+ print('Downloading package {pkg}...'.format(
+ pkg=package.metadata['info']['name']))
+ try:
+ package.download_package()
+ except urllib2.HTTPError as error:
+ print('Error: {code} {reason}'.format(code=error.code,
+ reason=error.reason))
+ print('Error downloading package :', package.name)
+ continue
+
+ # extract the tarball
+ package.extract_package(tmp_path)
+ print(package.metadata_name)
+ print(package.tmp_extract)
+
+ # Loading the package install info from the package
+ package.load_setup()
+
+ # Package requirement are an argument of the setup function
+ package.get_requirements(pkg_folder, packages)
+ packages += package.req_not_found
+ if package.req_not_found:
+ print('Added packages \'{pkgs}\' as dependencies of {pkg}'
+ .format(pkgs=", ".join(package.req_not_found),
+ pkg=package.name))
+ print('Checking if package {name} already exists...'.format(
+ name=package.pkg_dir))
+ try:
+ os.makedirs(package.pkg_dir)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ continue
+ print('Error: Package {name} already exists'
+ .format(name=package.pkg_dir))
+ del_pkg = raw_input(
+ 'Do you want to delete existing package ? [y/N]')
+ if del_pkg.lower() == 'y':
+ shutil.rmtree(package.pkg_dir)
+ os.makedirs(package.pkg_dir)
+ else:
+ continue
+ package.create_package_mk()
+
+ package.create_hash_file()
+
+ package.create_config_in()
+ print()
+ # printing an empty line for visual confort
+ except:
+ shutil.rmtree(tmp_path)
+ raise
+ else:
+ shutil.rmtree(tmp_path)
+
+if __name__ == "__main__":
+ main()
--
2.4.4
^ permalink raw reply related [flat|nested] 14+ messages in thread
end of thread, other threads:[~2016-03-01 1:44 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-28 13:15 [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
2015-07-28 13:15 ` [Buildroot] [PATCH 2/2] python-robotframework: New package Denis THULIN
2015-08-31 15:58 ` [Buildroot] [PATCH 1/2] scanpypi: new utility Denis Thulin
2016-01-10 10:59 ` Yann E. MORIN
2016-01-10 15:36 ` Arnout Vandecappelle
2016-01-13 15:23 ` Thomas Petazzoni
2016-01-14 8:32 ` Yegor Yefremov
2016-01-27 13:30 ` Yegor Yefremov
2016-02-02 18:02 ` Eelco Chaudron
2016-02-02 19:54 ` Eelco Chaudron
2016-03-01 1:44 ` Carlos Santos
-- strict thread matches above, loose matches on Subject: below --
2015-07-09 13:31 [Buildroot] [PATCH 0/2] python-package-generator Denis THULIN
2015-07-09 13:31 ` [Buildroot] [PATCH 1/2] scanpypi: new utility Denis THULIN
2015-07-11 12:56 ` Arnout Vandecappelle
2015-07-15 14:08 ` Denis Thulin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.