* [Buildroot] [PATCH 0/3] [RFC] python-package-generator
@ 2015-06-15 10:06 Denis THULIN
2015-06-15 10:06 ` [Buildroot] [PATCH 1/3] scanpypi.py: new utility Denis THULIN
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Denis THULIN @ 2015-06-15 10:06 UTC (permalink / raw)
To: buildroot
This patch series:
* Add utility scanpypi.py
* Add package python-robotframework
* Add package python-magic
scanpypi.py is an utility intended to automatically generatepython packages
for buildroot using metadata provided by the python package index
(https://pypi.python.org).
python-robotframework and python magic are packages generated using
scanpypi.py
v0: original patch
v0 -> v1: modification done according to Thomas Petazzoni and
Arnout Vandecappelle's comments
Denis THULIN (3):
scanpypi.py: new utility
python-robotframework: New package
python-magic: new package
docs/manual/adding-packages-python.txt | 36 ++
package/Config.in | 2 +
package/python-magic/Config.in | 7 +
package/python-magic/python-magic.hash | 2 +
package/python-magic/python-magic.mk | 14 +
package/python-robotframework/Config.in | 9 +
.../python-robotframework.hash | 2 +
.../python-robotframework/python-robotframework.mk | 14 +
support/scripts/scanpypi.py | 607 +++++++++++++++++++++
9 files changed, 693 insertions(+)
create mode 100644 package/python-magic/Config.in
create mode 100644 package/python-magic/python-magic.hash
create mode 100644 package/python-magic/python-magic.mk
create mode 100644 package/python-robotframework/Config.in
create mode 100644 package/python-robotframework/python-robotframework.hash
create mode 100644 package/python-robotframework/python-robotframework.mk
create mode 100755 support/scripts/scanpypi.py
--
2.4.3
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] scanpypi.py: new utility
2015-06-15 10:06 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
@ 2015-06-15 10:06 ` Denis THULIN
2015-06-21 12:29 ` Arnout Vandecappelle
2015-06-29 12:42 ` Thomas Petazzoni
2015-06-15 10:06 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
2015-06-15 10:06 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
2 siblings, 2 replies; 15+ messages in thread
From: Denis THULIN @ 2015-06-15 10:06 UTC (permalink / raw)
To: buildroot
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0: initial commit
python-pacakage-generator.py is an utility for automatically generating a
python package. It fetches packages info from http://pypi.python.org and
generates corresponding packages files.
v1:
- renamed python-package-generator to scanpypi
- split the huge script into a lot of functions
- fixed mistakes and small bugs
I did not know where to put the script so I put it in support/scripts.
I have updated the python-package section of the manual as well.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
docs/manual/adding-packages-python.txt | 36 ++
support/scripts/scanpypi.py | 607 +++++++++++++++++++++++++++++++++
2 files changed, 643 insertions(+)
create mode 100755 support/scripts/scanpypi.py
diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
index f81d625..647cb67 100644
--- a/docs/manual/adding-packages-python.txt
+++ b/docs/manual/adding-packages-python.txt
@@ -7,6 +7,42 @@ This infrastructure applies to Python packages that use the standard
Python setuptools mechanism as their build system, generally
recognizable by the usage of a +setup.py+ script.
+[[scanpypi]]
+
+==== generating a +python-package+ from a pypi repository
+
+You may want to use the +scanpypi.py+ located in
++support/script+ to generate a package from an existing pypi(pip) package.
+
+you can find the list of existing pypi package here: (https://pypi.python.org).
+
+Please keep in mind that you most likely need
+to manually check the package for any mistakes
+as there are things that cannot be guessed by the generator (e.g.
+dependencies on any of the python core modules
+such as BR2_PACKAGE_PYTHON_ZLIB).
+
+When at the root of your buildroot directory just do :
+
+-----------------------
+./support/script/scanpypi.py foo bar -o package
+-----------------------
+
+This will generate packages +python-foo+ and +python-bar+ in the package
+folder if they exist on https://pypi.python.org.
+
+You will need to manually write the path to the package inside
+the +package/Config.in+ file:
+
+Find the +external python modules+ menu and insert your package inside.
+Keep in mind that the items inside a menu should be in alphabetical order.
+
+Option +-h+ wil list the options available
+
+-----------------------
+./support/script/scanpypi.py -h
+-----------------------
+
[[python-package-tutorial]]
==== +python-package+ tutorial
diff --git a/support/scripts/scanpypi.py b/support/scripts/scanpypi.py
new file mode 100755
index 0000000..953f8d2
--- /dev/null
+++ b/support/scripts/scanpypi.py
@@ -0,0 +1,607 @@
+#!/usr/bin/python2
+"""
+Utility for building buildroot packages for existing pypi packages
+
+Any package built by scanpypi should be manually checked for
+errors.
+"""
+from __future__ import print_function
+import argparse
+import json
+import urllib2
+import sys
+import os
+import shutil
+import StringIO
+import tarfile
+import errno
+import hashlib
+import re
+import magic
+import tempfile
+from functools import wraps
+
+
+# private global
+_calls = {}
+
+
+def setup_info(pkg_name):
+ """Get a package info from _calls
+
+ Keyword arguments:
+ pkg_name -- the name of the package
+ """
+ return _calls[pkg_name]
+
+
+def setup_decorator(func, method):
+ """
+ Decorator for distutils.core.setup and setuptools.setup.
+ Puts the args of setup as a dict inside global private dict _calls.
+ Add key 'method' which should be either 'setuptools' or 'distutils'.
+
+ Keyword arguments:
+ func -- either setuptools.setup or distutils.core.setup
+ method -- either 'setuptools' or 'distutils'
+ """
+
+ @wraps(func)
+ def closure(*args, **kwargs):
+ _calls[kwargs['name']] = kwargs
+ _calls[kwargs['name']]['method'] = method
+ return closure
+
+
+# monkey patch
+import setuptools
+setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
+import distutils
+distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
+
+
+def find_file_upper_case(filenames, path='./'):
+ """
+ List generator:
+ Recursively find files that matches one of the specified filenames.
+ Returns absolute path
+
+ Keyword arguments:
+ filenames -- List of filenames to be found
+ path -- Path to the directory to search
+ """
+ for root, dirs, files in os.walk(path):
+ for file in files:
+ if file.upper() in filenames:
+ yield (os.path.join(root, file))
+
+
+def pkg_buildroot_name(pkg_name):
+ """
+ Returns name to avoid troublesome characters.
+ Remove all non alphanumeric characters except -
+ Also lowers the name
+
+ Keyword arguments:
+ pkg_name -- String to rename
+ """
+ name = re.sub('[^\w-]', '', pkg_name.lower())
+ name = re.sub('^python-', '', name)
+ return name
+
+
+def find_setup(package_name, version, archive):
+ """
+ Search for setup.py file in an archive and returns True if found
+ Used for finding the correct path to the setup.py
+
+ Keyword arguments:
+ package_name -- base name of the package to search (e.g. Flask)
+ version -- version of the package to search (e.g. 0.8.1)
+ archive -- tar archive to search in
+ """
+ try:
+ archive.getmember('{name}-{version}/setup.py'.format(
+ name=package_name,
+ version=version))
+ except KeyError:
+ return False
+ else:
+ return True
+
+
+def fetch_package_info(pkg_name):
+ """
+ Fetch a package's metadata for the python package index
+
+ Keyword arguments:
+ pkg_name -- the name of the package
+ """
+ url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
+ pkg=pkg_name)
+ print('URL:', url)
+ try:
+ pkg_json = urllib2.urlopen(url).read().decode()
+ except (urllib2.HTTPError) as error:
+ print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '.format(pkg=pkg_name))
+ return None, None
+ except urllib2.URLError:
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '.format(pkg=pkg_name))
+ return None, None
+
+ else:
+ return pkg_json, url
+
+
+def download_package(package):
+ """
+ Download a package using metadata from pypi
+
+ Keyword arguments:
+ package -- a dictionary containing info from the pypi json api
+ """
+ try:
+ targz = package['urls'][0]['filename']
+ except IndexError:
+ print(
+ 'Non conventional package, ',
+ 'please check manually after creation')
+ download_url = package['info']['download_url']
+ try:
+ download = urllib2.urlopen(download_url)
+ except urllib2.HTTPError:
+ targz = None
+ download = None
+ as_file = None
+ used_url = None
+ else:
+ used_url = {'url': download_url}
+ as_file = StringIO.StringIO(download.read())
+ as_file.seek(0)
+ extension = 'tar.gz'
+ if 'gzip' not in magic.from_buffer(as_file.read()):
+ extension = 'tar.bz2'
+ targz = '{name}-{version}.{extension}'.format(
+ name=package['info']['name'],
+ version=package['info']['version'], extension=extension)
+ as_file.seek(0)
+ used_url['filename'] = targz
+ else:
+ for download_url in package['urls']:
+ try:
+ download = urllib2.urlopen(download_url['url'])
+ except urllib2.HTTPError:
+ targz = None
+ download = None
+ as_file = None
+ used_url = None
+ else:
+ used_url = download_url
+ as_file = StringIO.StringIO(download.read())
+ md5_sum = hashlib.md5(as_file.read()).hexdigest()
+ if md5_sum == download_url['md5_digest']:
+ break
+ targz = used_url['filename']
+ return download, targz, used_url, as_file
+
+
+def extract_package(pkg_name, as_file, tmp_path):
+ """
+ Create folders used for extracting a package as file object and extract it
+
+ pkg_name -- name of the package to be extracted
+ as_file -- file object to extract
+ tmp_path -- folder where you want the package to be extracted
+ """
+ as_file.seek(0)
+ as_tarfile = tarfile.open(fileobj=as_file)
+ tmp_pkg = '/'.join([tmp_path, pkg_name])
+ try:
+ os.makedirs(tmp_pkg)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ return None, None
+ print('WARNING:', exception.message, file=sys.stderr)
+ print('Removing {pkg}...'.format(pkg=tmp_pkg))
+ shutil.rmtree(tmp_pkg)
+ os.makedirs(tmp_pkg)
+ version = package['info']['version']
+ tar_folder = package['info']['name']
+ if not find_setup(tar_folder, version, as_tarfile):
+ return None, None
+ as_tarfile.extractall(tmp_pkg)
+ as_tarfile.close()
+ as_file.close()
+ tmp_extract = '{folder}/{name}-{version}'.format(
+ folder=tmp_pkg,
+ name=tar_folder,
+ version=package['info']['version'])
+ return tar_folder, tmp_extract
+
+
+def get_requirements(package_name):
+ """
+ Retrieve dependencies of from a metadata found in the setup.py script of
+ a pypi package.
+
+ Keyword Arguments:
+ package_name -- name of the package found in the pypi metadata of the
+ package.
+ """
+ pkg_req = setup_info(package_name)['install_requires']
+ pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
+ for req in pkg_req]
+ pkg_req = map(pkg_buildroot_name, pkg_req)
+ req_not_found = [
+ pkg for pkg in pkg_req
+ if 'python-{name}'.format(name=pkg)
+ not in os.listdir(pkg_folder)
+ ]
+ req_not_found = [pkg for pkg in req_not_found
+ if pkg not in packages]
+ if (req_not_found) != 0:
+ print(
+ 'Error: could not find packages \'{packages}\''
+ 'required by {current_package}'.format(
+ packages=", ".join(req_not_found),
+ current_package=pkg_name))
+ return pkg_req
+
+
+def create_mk_header(pkg_name):
+ """
+ Create the header of the <package_name>.mk file
+
+ Keyword arguments:
+ pkg_name -- name of the package
+ """
+ header = ['#' * 80 + '\n']
+ header.append('#\n')
+ header.append('# python-{name}\n'.format(name=pkg_name))
+ header.append('#\n')
+ header.append('#' * 80 + '\n')
+ header.append('\n')
+ return header
+
+
+def create_mk_download_info(pkg_name, version, targz, url):
+ """
+ Create the lines refering to the download information of the
+ <package_name>.mk file
+
+ Keyword arguments:
+ pkg_name -- name of the package
+ version -- version of the package
+ targz -- name of the archive corresponding to the package
+ url -- url to be used for downloading the package
+ """
+ lines = []
+ version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
+ name=pkg_name.upper(),
+ version=version)
+ lines.append(version_line)
+
+ targz = targz.replace(
+ version,
+ '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
+ targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
+ name=pkg_name.upper(),
+ filename=targz)
+ lines.append(targz_line)
+
+ site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
+ name=pkg_name.upper(),
+ url=url['url'].replace(url['filename'], '')))
+ if 'sourceforge' in site_line:
+ site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
+ name=pkg_name.upper(),
+ url=url['url']))
+ lines.append(site_line)
+ return lines
+
+
+def create_mk_setup(pkg_name, tar_folder):
+ """
+ Create the line refering to the setup method of the package of the
+ <package_name>.mk file
+
+ There are two things you can use to make an installer
+ for a python package: distutils or setuptools
+ distutils comes with python but does not support dependancies.
+ distutils is mostly still there for backward support.
+ setuptools is what smart people use,
+ but it is not shipped with python :(
+
+ Keyword Arguments:
+ pkg_name -- name of the package
+ tar_folder -- name of the folder where the setup.py can be found
+ """
+ lines = []
+ setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
+ name=pkg_name.upper(),
+ method=setup_info(tar_folder)['method'])
+ lines.append(setup_type_line)
+ return lines
+
+
+def create_mk_license(pkg_name, license_name, package_location):
+ """
+ Create the lines referring to the package's license informations of the
+ <package_name>.mk file
+
+ The license's files are found by searching the package for files named
+ license or license.txt (case insensitive).
+ If more than one license file is found, the user is asked to select which
+ ones he wants to use.
+
+ Keyword Arguments:
+ pkg_name -- name of the package
+ license_name -- name of the license
+ package_location -- where to look for the licenses
+ """
+ lines = []
+ license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
+ name=pkg_name.upper(),
+ license=license_name)
+ lines.append(license_line)
+ print('WARNING: License has been set to "{license}",'
+ ' please change it manually if necessary'.format(
+ license=license_name))
+ filenames = ['LICENSE', 'LICENSE.TXT']
+ license_files = list(find_file_upper_case(filenames, package_location))
+ license_files = [license.replace(package_location, '')[1:]
+ for license in license_files]
+ if len(license_files) > 1:
+ print('More than one file found for license: ')
+ for index, item in enumerate(license_files):
+ print('\t{index})'.format(index=index), item)
+ license_choices = raw_input(
+ 'specify file numbers separated by spaces(default 0): ')
+ license_choices = [int(choice)
+ for choice in license_choices.split(' ')
+ if choice.isdigit() and int(choice) in
+ range(len(license_files))]
+ if len(license_choices) == 0:
+ license_choices = [0]
+ license_files = [file
+ for index, file in enumerate(license_files)
+ if index in license_choices]
+ elif len(license_files) == 0:
+ print('WARNING: No license file found,'
+ ' please specify it manually afterward')
+
+ license_file_line = ('PYTHON_{name}_LICENSE_FILES ='
+ ' {files}\n'.format(
+ name=pkg_name.upper(),
+ files=' '.join(license_files)))
+ license_file_line = license_file_line.replace(' \n', '\n')
+ lines.append(license_file_line)
+ return lines
+
+
+def create_mk_requirements(pkg_name, pkg_req):
+ """
+ Create the lines referring to the dependencies of the of the
+ <package_name>.mk file
+
+ Keyword Arguments:
+ pkg_name -- name of the package
+ pkg_req -- dependencies of the package
+ """
+ lines = []
+ python_pkg_req = ['python-{name}'.format(name=pkg)
+ for pkg in pkg_req]
+ dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
+ ' {reqs}\n'.format(
+ name=pkg_name.upper(),
+ reqs=' '.join(python_pkg_req)))
+ lines.append(dependencies_line)
+ return lines
+
+
+def create_config_mk(pkg_name, version, license, url, targz,
+ tar_folder, pkg_req, package_location):
+ """
+ Create the lines corresponding to the <package_name>.mk file
+
+ Keyword Arguments:
+ pkg_name -- name of the package
+ version -- version of the package
+ license -- name of the package's license
+ url -- where to download the package
+ targz -- name of the archive when downloaded
+ tar_folder -- name of the folder where the setup.py can be found
+ pkg_req -- dependencies of the package
+ package_location -- path to the extracted package
+ """
+ lines = create_mk_header(pkg_name)
+ lines += create_mk_download_info(pkg_name, version, targz, url)
+ lines += create_mk_setup(pkg_name, tar_folder)
+ lines += create_mk_license(pkg_name, license, package_location)
+ if pkg_req:
+ lines += create_mk_requirements(pkg_name, pkg_req)
+
+ lines.append('\n')
+ lines.append('$(eval $(python-package))')
+ lines.append('\n')
+
+ return lines
+
+
+def create_hash_file(url, digest, hash_function='sha356'):
+ """
+ Create the lines corresponding to the <package_name>.hash files
+
+ Keyword Arguments:
+ url -- metadata 'url' from the pypi json api
+ digest -- digest made from the downladed archive
+ hash_function -- algorythm used for hashing
+ """
+ lines = []
+ commented_line = '# {method} calculated by scanpypi\n'.format(
+ method=hash_function)
+ lines.append(commented_line)
+ hash_line = '{method}\t{digest} {filename}\n'.format(
+ method=hash_function,
+ digest=digest,
+ filename=url['filename'])
+ lines.append(hash_line)
+ return lines
+
+
+def create_config_in(pkg_name, pkg_req, package):
+ """
+ Creates the Config.in file of a package
+
+ pkg_name -- name of the package
+ pkg_req -- dependencies of the package
+ package -- metadata of the package from pypi
+ """
+ lines = []
+ config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
+ name=pkg_name.upper())
+ lines.append(config_line)
+ python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
+ lines.append(python_line)
+
+ bool_line = '\tbool "python-{name}"\n'.format(name=pkg_name)
+ lines.append(bool_line)
+ if pkg_req:
+ for dep in pkg_req:
+ dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
+ req=dep.upper())
+ lines.append(dep_line)
+
+ lines.append('\thelp\n')
+
+ help_lines = package['info']['summary'].split('\n')
+ help_lines.append('')
+ help_lines.append(package['info']['home_page'])
+ help_lines = ['\t {line}\n'.format(line=line)
+ for line in help_lines]
+ lines += help_lines
+ return lines
+
+
+if __name__ == "__main__":
+
+ # Building the parser
+ parser = argparse.ArgumentParser(
+ description="Creates buildroot packages from the metadata of "
+ "an existing pypi(pip) packages and include it "
+ "in menuconfig")
+ parser.add_argument("packages",
+ help="list of packages to be made",
+ nargs='+')
+ parser.add_argument("-o", "--output",
+ help="""
+ Output directory for packages
+ """,
+ default='.')
+
+ args = parser.parse_args()
+ packages = list(set(args.packages))
+
+ # tmp_path is where we'll extract the files later
+ tmp_prefix = 'scanpypi-'
+ pkg_folder = args.output
+ tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
+
+ for real_pkg_name in packages:
+ pkg_name = pkg_buildroot_name(real_pkg_name)
+ print('buildroot package name for {}:'.format(real_pkg_name),
+ pkg_name)
+ # First we download the package
+ # Most of the info we need can only be found inside the package
+ print('Package:', pkg_name)
+ print('Fetching package', real_pkg_name)
+ pkg_json, url = fetch_package_info(real_pkg_name)
+ if not pkg_json:
+ continue
+
+ pkg_dir = pkg_folder + '/python-' + pkg_name
+ package = json.loads(pkg_json)
+ used_url = ''
+ print('Downloading package {pkg}...'.format(
+ pkg=package['info']['name']))
+ download, targz, used_url, as_file = download_package(package)
+ version = package['info']['version']
+
+ if not download:
+ print('Error downloading package :', pkg_name)
+ continue
+
+ sha256_digest = hashlib.sha256(as_file.read()).hexdigest()
+
+ # extract the tarball
+ tar_folder, tmp_extract = extract_package(pkg_name, as_file, tmp_path)
+
+ # Loading the package install info from the package
+ sys.path.append(tmp_extract)
+ print(tmp_extract)
+ import setup
+ setup = reload(setup)
+ sys.path.remove(tmp_extract)
+
+ pkg_req = None
+ # Package requierement are an argument of the setup function
+ if 'install_requires' in setup_info(tar_folder):
+ pkg_req = get_requirements(tar_folder)
+ # We could stop here
+ # or ask the user if he still wants to continue
+
+ # Buildroot python packages require 3 files
+ # The first is the mk file
+ # See:
+ # http://buildroot.uclibc.org/downloads/manual/manual.html
+ print('Checking if package {name} already exists...'.format(
+ name=pkg_dir))
+ try:
+ os.makedirs(pkg_dir)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ continue
+ print('Error: Package {name} already exists'.format(name=pkg_dir))
+ del_pkg = raw_input(
+ 'Do you want to delete existing package ? [y/N]')
+ if del_pkg.lower() == 'y':
+ shutil.rmtree(pkg_dir)
+ os.makedirs(pkg_dir)
+ else:
+ continue
+ pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
+ path_to_mk = '/'.join([pkg_dir, pkg_mk])
+ print('Creating {file}...'.format(file=path_to_mk))
+ config_mk_lines = create_config_mk(pkg_name, version,
+ package['info']['license'],
+ used_url, targz, tar_folder,
+ pkg_req, tmp_extract)
+ with open(path_to_mk, 'w') as mk_file:
+ mk_file.writelines(config_mk_lines)
+
+ # The second file we make is the hash file
+ # It consists of hashes of the package tarball
+ # http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
+ pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
+ path_to_hash = '/'.join([pkg_dir, pkg_hash])
+ print('Creating {filename}...'.format(filename=path_to_hash))
+ hash_lines = create_hash_file(used_url, sha256_digest)
+ with open(path_to_hash, 'w') as hash_file:
+ hash_file.writelines(hash_lines)
+
+ # The Config.in is the last file we create
+ # It is used by buildroot's menuconfig, gconfig, xconfig or nconfig
+ # it is used to displayspackage info and to select requirements
+ # http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
+ path_to_config = '/'.join([pkg_dir, 'Config.in'])
+ print('Creating {file}...'.format(file=path_to_config))
+ config_in_lines = create_config_in(pkg_name, pkg_req, package)
+ with open(path_to_config, 'w') as config_file:
+ config_file.writelines(config_in_lines)
--
2.4.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-15 10:06 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
2015-06-15 10:06 ` [Buildroot] [PATCH 1/3] scanpypi.py: new utility Denis THULIN
@ 2015-06-15 10:06 ` Denis THULIN
2015-06-21 12:44 ` Arnout Vandecappelle
` (2 more replies)
2015-06-15 10:06 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
2 siblings, 3 replies; 15+ messages in thread
From: Denis THULIN @ 2015-06-15 10:06 UTC (permalink / raw)
To: buildroot
A generic test automation framework written in python.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0:
- Generated using python-package-generator.py
- Dependencies on python-zlib and python-pyexpat were added manually
- Modification of packages/Config.in done manually.
v1:
- fixed indentation error on dependencies (was 4 spaces instead of tab)
- Changed hash from md5 to sha256 (modification of scanpypi)
- Fixed typos (modification of scanpypi)
---
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
package/Config.in | 1 +
package/python-robotframework/Config.in | 9 +++++++++
package/python-robotframework/python-robotframework.hash | 2 ++
package/python-robotframework/python-robotframework.mk | 14 ++++++++++++++
4 files changed, 26 insertions(+)
create mode 100644 package/python-robotframework/Config.in
create mode 100644 package/python-robotframework/python-robotframework.hash
create mode 100644 package/python-robotframework/python-robotframework.mk
diff --git a/package/Config.in b/package/Config.in
index 6dbc32d..766b055 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -637,6 +637,7 @@ menu "external python modules"
source "package/python-pyxml/Config.in"
source "package/python-pyzmq/Config.in"
source "package/python-requests/Config.in"
+ source "package/python-robotframework/Config.in"
source "package/python-rtslib-fb/Config.in"
source "package/python-serial/Config.in"
source "package/python-setuptools/Config.in"
diff --git a/package/python-robotframework/Config.in b/package/python-robotframework/Config.in
new file mode 100644
index 0000000..fe95eea
--- /dev/null
+++ b/package/python-robotframework/Config.in
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+ depends on BR2_PACKAGE_PYTHON
+ bool "python-robotframework"
+ select BR2_PACKAGE_PYTHON_ZLIB
+ select BR2_PACKAGE_PYTHON_PYEXPAT
+ help
+ A generic test automation framework
+
+ http://robotframework.org
diff --git a/package/python-robotframework/python-robotframework.hash b/package/python-robotframework/python-robotframework.hash
new file mode 100644
index 0000000..7d9ec46
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.hash
@@ -0,0 +1,2 @@
+# sha356 calculated by scanpypi
+sha356 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 robotframework-2.9a2.tar.gz
diff --git a/package/python-robotframework/python-robotframework.mk b/package/python-robotframework/python-robotframework.mk
new file mode 100644
index 0000000..437d5f4
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# python-robotframework
+#
+################################################################################
+
+PYTHON_ROBOTFRAMEWORK_VERSION = 2.9a2
+PYTHON_ROBOTFRAMEWORK_SOURCE = robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
+PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework/
+PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = distutils
+PYTHON_ROBOTFRAMEWORK_LICENSE = Apache License 2.0
+PYTHON_ROBOTFRAMEWORK_LICENSE_FILES = LICENSE.txt
+
+$(eval $(python-package))
--
2.4.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] python-magic: new package
2015-06-15 10:06 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
2015-06-15 10:06 ` [Buildroot] [PATCH 1/3] scanpypi.py: new utility Denis THULIN
2015-06-15 10:06 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
@ 2015-06-15 10:06 ` Denis THULIN
2015-06-21 12:52 ` Arnout Vandecappelle
2015-06-28 20:39 ` Thomas Petazzoni
2 siblings, 2 replies; 15+ messages in thread
From: Denis THULIN @ 2015-06-15 10:06 UTC (permalink / raw)
To: buildroot
A file type identification for python using libmagic
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0:
- Generated using python-package-generator.py
- Modification on package/Config.in was done manually
v1:
- Fixed typos (modification of scanpypi)
- Changed hash from md5 to sha256 (modification of scanpypi)
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
package/Config.in | 1 +
package/python-magic/Config.in | 7 +++++++
package/python-magic/python-magic.hash | 2 ++
package/python-magic/python-magic.mk | 14 ++++++++++++++
4 files changed, 24 insertions(+)
create mode 100644 package/python-magic/Config.in
create mode 100644 package/python-magic/python-magic.hash
create mode 100644 package/python-magic/python-magic.mk
diff --git a/package/Config.in b/package/Config.in
index 766b055..b435fa5 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -605,6 +605,7 @@ menu "external python modules"
source "package/python-keyring/Config.in"
source "package/python-libconfig/Config.in"
source "package/python-lxml/Config.in"
+ source "package/python-magic/Config.in"
source "package/python-mako/Config.in"
source "package/python-mad/Config.in"
source "package/python-markdown/Config.in"
diff --git a/package/python-magic/Config.in b/package/python-magic/Config.in
new file mode 100644
index 0000000..e6c06b9
--- /dev/null
+++ b/package/python-magic/Config.in
@@ -0,0 +1,7 @@
+config BR2_PACKAGE_PYTHON_MAGIC
+ depends on BR2_PACKAGE_PYTHON
+ bool "python-magic"
+ help
+ File type identification using libmagic
+
+ http://github.com/ahupp/python-magic
diff --git a/package/python-magic/python-magic.hash b/package/python-magic/python-magic.hash
new file mode 100644
index 0000000..36ffd02
--- /dev/null
+++ b/package/python-magic/python-magic.hash
@@ -0,0 +1,2 @@
+# sha356 calculated by scanpypi
+sha356 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 python-magic-0.4.6.tar.gz
diff --git a/package/python-magic/python-magic.mk b/package/python-magic/python-magic.mk
new file mode 100644
index 0000000..fad3056
--- /dev/null
+++ b/package/python-magic/python-magic.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# python-magic
+#
+################################################################################
+
+PYTHON_MAGIC_VERSION = 0.4.6
+PYTHON_MAGIC_SOURCE = python-magic-$(PYTHON_MAGIC_VERSION).tar.gz
+PYTHON_MAGIC_SITE = https://pypi.python.org/packages/source/p/python-magic/
+PYTHON_MAGIC_SETUP_TYPE = setuptools
+PYTHON_MAGIC_LICENSE = PSF
+PYTHON_MAGIC_LICENSE_FILES =
+
+$(eval $(python-package))
--
2.4.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] scanpypi.py: new utility
2015-06-15 10:06 ` [Buildroot] [PATCH 1/3] scanpypi.py: new utility Denis THULIN
@ 2015-06-21 12:29 ` Arnout Vandecappelle
2015-06-29 11:49 ` Denis Thulin
2015-06-29 12:42 ` Thomas Petazzoni
1 sibling, 1 reply; 15+ messages in thread
From: Arnout Vandecappelle @ 2015-06-21 12:29 UTC (permalink / raw)
To: buildroot
On 06/15/15 12:06, Denis THULIN wrote:
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> ---
> v0: initial commit
> python-pacakage-generator.py is an utility for automatically generating a
> python package. It fetches packages info from http://pypi.python.org and
> generates corresponding packages files.
This should go above your Sob so there's an actual commit message.
>
> v1:
> - renamed python-package-generator to scanpypi
> - split the huge script into a lot of functions
> - fixed mistakes and small bugs
>
> I did not know where to put the script so I put it in support/scripts.
> I have updated the python-package section of the manual as well.
We generally make the update to the manual a separate patch, but this is OK for
me as well.
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> ---
> docs/manual/adding-packages-python.txt | 36 ++
> support/scripts/scanpypi.py | 607 +++++++++++++++++++++++++++++++++
> 2 files changed, 643 insertions(+)
> create mode 100755 support/scripts/scanpypi.py
>
> diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
> index f81d625..647cb67 100644
> --- a/docs/manual/adding-packages-python.txt
> +++ b/docs/manual/adding-packages-python.txt
> @@ -7,6 +7,42 @@ This infrastructure applies to Python packages that use the standard
> Python setuptools mechanism as their build system, generally
> recognizable by the usage of a +setup.py+ script.
>
> +[[scanpypi]]
> +
> +==== generating a +python-package+ from a pypi repository
Capitalization: Generating
> +
> +You may want to use the +scanpypi.py+ located in
> ++support/script+ to generate a package from an existing pypi(pip) package.
> +
> +you can find the list of existing pypi package here: (https://pypi.python.org).
You
The parenthesis are redundant.
> +
> +Please keep in mind that you most likely need
> +to manually check the package for any mistakes
> +as there are things that cannot be guessed by the generator (e.g.
> +dependencies on any of the python core modules
> +such as BR2_PACKAGE_PYTHON_ZLIB).
Could you rewrap this paragraph? And remove the end-of-line spaces.
> +
> +When at the root of your buildroot directory just do :
> +
> +-----------------------
> +./support/script/scanpypi.py foo bar -o package
> +-----------------------
> +
> +This will generate packages +python-foo+ and +python-bar+ in the package
> +folder if they exist on https://pypi.python.org.
> +
> +You will need to manually write the path to the package inside
> +the +package/Config.in+ file:
: -> .
But perhaps reformulate:
You need to manually add the package to the +package/Config.in+ file.
It would also be better if this sentence was part of the paragraph a couple of
lines higher, were you say that the package has to be checked manually. And the
following sentence should be part of the same paragraph.
> +
> +Find the +external python modules+ menu and insert your package inside.
> +Keep in mind that the items inside a menu should be in alphabetical order.
> +
> +Option +-h+ wil list the options available
> +
> +-----------------------
> +./support/script/scanpypi.py -h
> +-----------------------
> +
> [[python-package-tutorial]]
>
> ==== +python-package+ tutorial
> diff --git a/support/scripts/scanpypi.py b/support/scripts/scanpypi.py
> new file mode 100755
> index 0000000..953f8d2
> --- /dev/null
> +++ b/support/scripts/scanpypi.py
> @@ -0,0 +1,607 @@
> +#!/usr/bin/python2
> +"""
> +Utility for building buildroot packages for existing pypi packages
> +
> +Any package built by scanpypi should be manually checked for
> +errors.
> +"""
> +from __future__ import print_function
> +import argparse
> +import json
> +import urllib2
> +import sys
> +import os
> +import shutil
> +import StringIO
> +import tarfile
> +import errno
> +import hashlib
> +import re
> +import magic
> +import tempfile
> +from functools import wraps
> +
> +
> +# private global
> +_calls = {}
> +
> +
> +def setup_info(pkg_name):
> + """Get a package info from _calls
> +
> + Keyword arguments:
> + pkg_name -- the name of the package
> + """
> + return _calls[pkg_name]
> +
> +
> +def setup_decorator(func, method):
> + """
> + Decorator for distutils.core.setup and setuptools.setup.
> + Puts the args of setup as a dict inside global private dict _calls.
> + Add key 'method' which should be either 'setuptools' or 'distutils'.
> +
> + Keyword arguments:
> + func -- either setuptools.setup or distutils.core.setup
> + method -- either 'setuptools' or 'distutils'
> + """
> +
> + @wraps(func)
> + def closure(*args, **kwargs):
> + _calls[kwargs['name']] = kwargs
> + _calls[kwargs['name']]['method'] = method
> + return closure
> +
> +
> +# monkey patch
> +import setuptools
> +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> +import distutils
> +distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
> +
> +
> +def find_file_upper_case(filenames, path='./'):
> + """
> + List generator:
> + Recursively find files that matches one of the specified filenames.
> + Returns absolute path
> +
> + Keyword arguments:
> + filenames -- List of filenames to be found
> + path -- Path to the directory to search
> + """
> + for root, dirs, files in os.walk(path):
> + for file in files:
> + if file.upper() in filenames:
> + yield (os.path.join(root, file))
> +
> +
> +def pkg_buildroot_name(pkg_name):
> + """
> + Returns name to avoid troublesome characters.
> + Remove all non alphanumeric characters except -
> + Also lowers the name
> +
> + Keyword arguments:
> + pkg_name -- String to rename
> + """
> + name = re.sub('[^\w-]', '', pkg_name.lower())
> + name = re.sub('^python-', '', name)
> + return name
> +
> +
> +def find_setup(package_name, version, archive):
> + """
> + Search for setup.py file in an archive and returns True if found
> + Used for finding the correct path to the setup.py
> +
> + Keyword arguments:
> + package_name -- base name of the package to search (e.g. Flask)
> + version -- version of the package to search (e.g. 0.8.1)
> + archive -- tar archive to search in
> + """
> + try:
> + archive.getmember('{name}-{version}/setup.py'.format(
> + name=package_name,
> + version=version))
> + except KeyError:
> + return False
> + else:
> + return True
> +
> +
> +def fetch_package_info(pkg_name):
> + """
> + Fetch a package's metadata for the python package index
> +
> + Keyword arguments:
> + pkg_name -- the name of the package
> + """
> + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> + pkg=pkg_name)
> + print('URL:', url)
> + try:
> + pkg_json = urllib2.urlopen(url).read().decode()
> + except (urllib2.HTTPError) as error:
I don't think these parenthesis are needed?
> + print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '.format(pkg=pkg_name))
> + return None, None
> + except urllib2.URLError:
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '.format(pkg=pkg_name))
> + return None, None
> +
> + else:
> + return pkg_json, url
> +
> +
> +def download_package(package):
> + """
> + Download a package using metadata from pypi
> +
> + Keyword arguments:
> + package -- a dictionary containing info from the pypi json api
> + """
> + try:
> + targz = package['urls'][0]['filename']
> + except IndexError:
> + print(
> + 'Non conventional package, ',
> + 'please check manually after creation')
Is it even worthwhile to support this case?
> + download_url = package['info']['download_url']
> + try:
> + download = urllib2.urlopen(download_url)
> + except urllib2.HTTPError:
Shouldn't we print an error message here? Or does urrlib already do that?
> + targz = None
> + download = None
> + as_file = None
> + used_url = None
> + else:
> + used_url = {'url': download_url}
> + as_file = StringIO.StringIO(download.read())
Actually, you use the download mostly as a string, it's only used as a file in
tarfile.open(). So I'd put the StringIO call there, and keep it as a string
everywhere else.
Also, perhaps it's worthwhile to make a class for the package information
instead of passing around tuples? I'm not entirely sure if that really makes
things simpler, but it's something to consider.
> + as_file.seek(0)
> + extension = 'tar.gz'
> + if 'gzip' not in magic.from_buffer(as_file.read()):
> + extension = 'tar.bz2'
This part I really don't like. Can't we just get the extension from the URL?
> + targz = '{name}-{version}.{extension}'.format(
> + name=package['info']['name'],
> + version=package['info']['version'], extension=extension)
> + as_file.seek(0)
> + used_url['filename'] = targz
> + else:
> + for download_url in package['urls']:
> + try:
> + download = urllib2.urlopen(download_url['url'])
> + except urllib2.HTTPError:
> + targz = None
> + download = None
> + as_file = None
> + used_url = None
> + else:
> + used_url = download_url
> + as_file = StringIO.StringIO(download.read())
> + md5_sum = hashlib.md5(as_file.read()).hexdigest()
> + if md5_sum == download_url['md5_digest']:
> + break
> + targz = used_url['filename']
> + return download, targz, used_url, as_file
> +
> +
> +def extract_package(pkg_name, as_file, tmp_path):
> + """
> + Create folders used for extracting a package as file object and extract it
> +
> + pkg_name -- name of the package to be extracted
> + as_file -- file object to extract
> + tmp_path -- folder where you want the package to be extracted
> + """
> + as_file.seek(0)
> + as_tarfile = tarfile.open(fileobj=as_file)
Better use a with clause here.
> + tmp_pkg = '/'.join([tmp_path, pkg_name])
os.path.join is better I think.
> + try:
> + os.makedirs(tmp_pkg)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + return None, None
> + print('WARNING:', exception.message, file=sys.stderr)
> + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> + shutil.rmtree(tmp_pkg)
> + os.makedirs(tmp_pkg)
> + version = package['info']['version']
This accidentally works because the package variable is set in the global
scope, but I think it's better to pass this as an argument to the function. Or
better yet, make a class :-)
> + tar_folder = package['info']['name']
> + if not find_setup(tar_folder, version, as_tarfile):
> + return None, None
Since the tarball will anyway be extracted below, perhaps it's easier to do
this find_setup on the extracted directory.
> + as_tarfile.extractall(tmp_pkg)
> + as_tarfile.close()
> + as_file.close()
> + tmp_extract = '{folder}/{name}-{version}'.format(
> + folder=tmp_pkg,
> + name=tar_folder,
> + version=package['info']['version'])
> + return tar_folder, tmp_extract
> +
> +
> +def get_requirements(package_name):
> + """
> + Retrieve dependencies of from a metadata found in the setup.py script of
^^^^^^^^^
dependencies from the metadata
> + a pypi package.
> +
> + Keyword Arguments:
> + package_name -- name of the package found in the pypi metadata of the
> + package.
> + """
> + pkg_req = setup_info(package_name)['install_requires']
> + pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
> + for req in pkg_req]
> + pkg_req = map(pkg_buildroot_name, pkg_req)
> + req_not_found = [
> + pkg for pkg in pkg_req
> + if 'python-{name}'.format(name=pkg)
> + not in os.listdir(pkg_folder)
> + ]
> + req_not_found = [pkg for pkg in req_not_found
> + if pkg not in packages]
> + if (req_not_found) != 0:
Just "if req_not_found:" - it's a list, so comparing to 0 is weird.
> + print(
> + 'Error: could not find packages \'{packages}\''
> + 'required by {current_package}'.format(
> + packages=", ".join(req_not_found),
> + current_package=pkg_name))
scancpan instead adds the dependencies to the list of packages to create. We
could do something like this here as well. But that can be done in a follow-up
patch.
> + return pkg_req
> +
> +
> +def create_mk_header(pkg_name):
> + """
> + Create the header of the <package_name>.mk file
> +
> + Keyword arguments:
> + pkg_name -- name of the package
> + """
> + header = ['#' * 80 + '\n']
> + header.append('#\n')
> + header.append('# python-{name}\n'.format(name=pkg_name))
> + header.append('#\n')
> + header.append('#' * 80 + '\n')
> + header.append('\n')
> + return header
I would return a string here instead of a list, so it can all be concatenated
easily in the calling function.
> +
> +
> +def create_mk_download_info(pkg_name, version, targz, url):
> + """
> + Create the lines refering to the download information of the
> + <package_name>.mk file
> +
> + Keyword arguments:
> + pkg_name -- name of the package
> + version -- version of the package
> + targz -- name of the archive corresponding to the package
> + url -- url to be used for downloading the package
> + """
> + lines = []
> + version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
> + name=pkg_name.upper(),
It's not just .upper(), also - has to replaced with _
> + version=version)
> + lines.append(version_line)
> +
> + targz = targz.replace(
> + version,
> + '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
> + targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
> + name=pkg_name.upper(),
> + filename=targz)
> + lines.append(targz_line)
> +
> + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> + name=pkg_name.upper(),
> + url=url['url'].replace(url['filename'], '')))
> + if 'sourceforge' in site_line:
> + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> + name=pkg_name.upper(),
> + url=url['url']))
> + lines.append(site_line)
> + return lines
> +
> +
> +def create_mk_setup(pkg_name, tar_folder):
> + """
> + Create the line refering to the setup method of the package of the
> + <package_name>.mk file
> +
> + There are two things you can use to make an installer
> + for a python package: distutils or setuptools
> + distutils comes with python but does not support dependancies.
dependencies
> + distutils is mostly still there for backward support.
> + setuptools is what smart people use,
> + but it is not shipped with python :(
> +
> + Keyword Arguments:
> + pkg_name -- name of the package
> + tar_folder -- name of the folder where the setup.py can be found
> + """
> + lines = []
> + setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
> + name=pkg_name.upper(),
> + method=setup_info(tar_folder)['method'])
> + lines.append(setup_type_line)
> + return lines
> +
> +
> +def create_mk_license(pkg_name, license_name, package_location):
> + """
> + Create the lines referring to the package's license informations of the
> + <package_name>.mk file
> +
> + The license's files are found by searching the package for files named
> + license or license.txt (case insensitive).
> + If more than one license file is found, the user is asked to select which
> + ones he wants to use.
> +
> + Keyword Arguments:
> + pkg_name -- name of the package
> + license_name -- name of the license
> + package_location -- where to look for the licenses
> + """
> + lines = []
> + license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
> + name=pkg_name.upper(),
> + license=license_name)
> + lines.append(license_line)
> + print('WARNING: License has been set to "{license}",'
> + ' please change it manually if necessary'.format(
> + license=license_name))
> + filenames = ['LICENSE', 'LICENSE.TXT']
> + license_files = list(find_file_upper_case(filenames, package_location))
> + license_files = [license.replace(package_location, '')[1:]
> + for license in license_files]
> + if len(license_files) > 1:
> + print('More than one file found for license: ')
No need to go interactive here: just make a space-separated list of license
files. You could spew a warning if it is more than one, but it's anyway going to
be verified manually.
> + for index, item in enumerate(license_files):
> + print('\t{index})'.format(index=index), item)
> + license_choices = raw_input(
> + 'specify file numbers separated by spaces(default 0): ')
> + license_choices = [int(choice)
> + for choice in license_choices.split(' ')
> + if choice.isdigit() and int(choice) in
> + range(len(license_files))]
> + if len(license_choices) == 0:
> + license_choices = [0]
> + license_files = [file
> + for index, file in enumerate(license_files)
> + if index in license_choices]
> + elif len(license_files) == 0:
> + print('WARNING: No license file found,'
> + ' please specify it manually afterward')
> +
> + license_file_line = ('PYTHON_{name}_LICENSE_FILES ='
This line should not be added if len(license_files) == 0, but instead there
should be something like:
# No license file found
> + ' {files}\n'.format(
> + name=pkg_name.upper(),
> + files=' '.join(license_files)))
> + license_file_line = license_file_line.replace(' \n', '\n')
If len(license_files) > 0 then this will not be needed.
> + lines.append(license_file_line)
> + return lines
> +
> +
> +def create_mk_requirements(pkg_name, pkg_req):
> + """
> + Create the lines referring to the dependencies of the of the
> + <package_name>.mk file
> +
> + Keyword Arguments:
> + pkg_name -- name of the package
> + pkg_req -- dependencies of the package
> + """
> + lines = []
> + python_pkg_req = ['python-{name}'.format(name=pkg)
> + for pkg in pkg_req]
> + dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
> + ' {reqs}\n'.format(
> + name=pkg_name.upper(),
> + reqs=' '.join(python_pkg_req)))
> + lines.append(dependencies_line)
> + return lines
> +
> +
> +def create_config_mk(pkg_name, version, license, url, targz,
> + tar_folder, pkg_req, package_location):
create_config_mk -> create_package_mk
> + """
> + Create the lines corresponding to the <package_name>.mk file
> +
> + Keyword Arguments:
> + pkg_name -- name of the package
> + version -- version of the package
> + license -- name of the package's license
> + url -- where to download the package
> + targz -- name of the archive when downloaded
> + tar_folder -- name of the folder where the setup.py can be found
> + pkg_req -- dependencies of the package
> + package_location -- path to the extracted package
> + """
> + lines = create_mk_header(pkg_name)
> + lines += create_mk_download_info(pkg_name, version, targz, url)
> + lines += create_mk_setup(pkg_name, tar_folder)
> + lines += create_mk_license(pkg_name, license, package_location)
> + if pkg_req:
> + lines += create_mk_requirements(pkg_name, pkg_req)
> +
> + lines.append('\n')
> + lines.append('$(eval $(python-package))')
> + lines.append('\n')
> +
> + return lines
> +
> +
> +def create_hash_file(url, digest, hash_function='sha356'):
^^^^^^sha256
> + """
> + Create the lines corresponding to the <package_name>.hash files
> +
> + Keyword Arguments:
> + url -- metadata 'url' from the pypi json api
> + digest -- digest made from the downladed archive
> + hash_function -- algorythm used for hashing
algorithm
> + """
> + lines = []
> + commented_line = '# {method} calculated by scanpypi\n'.format(
I think there would typically be one md5 that comes from pypi, and one sha256
that is calculated locally. So the comment is not correct.
> + method=hash_function)
> + lines.append(commented_line)
> + hash_line = '{method}\t{digest} {filename}\n'.format(
> + method=hash_function,
> + digest=digest,
> + filename=url['filename'])
> + lines.append(hash_line)
> + return lines
> +
> +
> +def create_config_in(pkg_name, pkg_req, package):
> + """
> + Creates the Config.in file of a package
> +
> + pkg_name -- name of the package
> + pkg_req -- dependencies of the package
> + package -- metadata of the package from pypi
> + """
> + lines = []
> + config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
> + name=pkg_name.upper())
> + lines.append(config_line)
> + python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
Why? Are all pypi packages python2-only? Remember, the python2||python3
condition is already in package/Config.in.
> + lines.append(python_line)
> +
> + bool_line = '\tbool "python-{name}"\n'.format(name=pkg_name)
> + lines.append(bool_line)
> + if pkg_req:
> + for dep in pkg_req:
> + dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
> + req=dep.upper())
> + lines.append(dep_line)
> +
> + lines.append('\thelp\n')
> +
> + help_lines = package['info']['summary'].split('\n')
The help_lines should also be wrapped, so call textwrap.wrap() on it.
> + help_lines.append('')
> + help_lines.append(package['info']['home_page'])
> + help_lines = ['\t {line}\n'.format(line=line)
> + for line in help_lines]
> + lines += help_lines
> + return lines
> +
> +
> +if __name__ == "__main__":
> +
> + # Building the parser
> + parser = argparse.ArgumentParser(
> + description="Creates buildroot packages from the metadata of "
> + "an existing pypi(pip) packages and include it "
> + "in menuconfig")
> + parser.add_argument("packages",
> + help="list of packages to be made",
> + nargs='+')
> + parser.add_argument("-o", "--output",
> + help="""
> + Output directory for packages
> + """,
> + default='.')
Since it will be called from the buildroot top dir, this should default to
./packages I think.
> +
> + args = parser.parse_args()
> + packages = list(set(args.packages))
> +
> + # tmp_path is where we'll extract the files later
> + tmp_prefix = 'scanpypi-'
> + pkg_folder = args.output
> + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
There should be a big try: block around here that removes the tmpdir
unconditionally. Clearly that makes debugging harder, but it's easy to remove
the rmtree from the script when you want to debug.
> +
> + for real_pkg_name in packages:
> + pkg_name = pkg_buildroot_name(real_pkg_name)
> + print('buildroot package name for {}:'.format(real_pkg_name),
> + pkg_name)
> + # First we download the package
> + # Most of the info we need can only be found inside the package
> + print('Package:', pkg_name)
> + print('Fetching package', real_pkg_name)
> + pkg_json, url = fetch_package_info(real_pkg_name)
> + if not pkg_json:
> + continue
> +
> + pkg_dir = pkg_folder + '/python-' + pkg_name
> + package = json.loads(pkg_json)
> + used_url = ''
> + print('Downloading package {pkg}...'.format(
> + pkg=package['info']['name']))
> + download, targz, used_url, as_file = download_package(package)
> + version = package['info']['version']
> +
> + if not download:
> + print('Error downloading package :', pkg_name)
> + continue
> +
> + sha256_digest = hashlib.sha256(as_file.read()).hexdigest()
> +
> + # extract the tarball
> + tar_folder, tmp_extract = extract_package(pkg_name, as_file, tmp_path)
> +
> + # Loading the package install info from the package
> + sys.path.append(tmp_extract)
> + print(tmp_extract)
This print is not necessary or useful I think.
> + import setup
> + setup = reload(setup)
A comment explaining why the reload is necessary would be useful.
Also, it seems that some packages import other modules from their source
directory. So instead of appending to sys.path, I think it's better to
temporarily chdir to the tmp_extract directory. Try for instance
json-schema-validator
> + sys.path.remove(tmp_extract)
> +
> + pkg_req = None
> + # Package requierement are an argument of the setup function
requirements
> + if 'install_requires' in setup_info(tar_folder):
> + pkg_req = get_requirements(tar_folder)
> + # We could stop here
> + # or ask the user if he still wants to continue
> +
> + # Buildroot python packages require 3 files
> + # The first is the mk file
> + # See:
> + # http://buildroot.uclibc.org/downloads/manual/manual.html
Indentation is wrong.
But I don't think this comment is very useful.
> + print('Checking if package {name} already exists...'.format(
> + name=pkg_dir))
> + try:
> + os.makedirs(pkg_dir)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('Error: Package {name} already exists'.format(name=pkg_dir))
> + del_pkg = raw_input(
> + 'Do you want to delete existing package ? [y/N]')
> + if del_pkg.lower() == 'y':
> + shutil.rmtree(pkg_dir)
> + os.makedirs(pkg_dir)
> + else:
> + continue
> + pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
> + path_to_mk = '/'.join([pkg_dir, pkg_mk])
> + print('Creating {file}...'.format(file=path_to_mk))
> + config_mk_lines = create_config_mk(pkg_name, version,
> + package['info']['license'],
> + used_url, targz, tar_folder,
> + pkg_req, tmp_extract)
> + with open(path_to_mk, 'w') as mk_file:
> + mk_file.writelines(config_mk_lines)
I think it's more appropriate to do the file writing inside the function.
> +
> + # The second file we make is the hash file
> + # It consists of hashes of the package tarball
> + # http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
> + pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
> + path_to_hash = '/'.join([pkg_dir, pkg_hash])
> + print('Creating {filename}...'.format(filename=path_to_hash))
> + hash_lines = create_hash_file(used_url, sha256_digest)
The pypi md5 should be included as well.
Regards,
Arnout
> + with open(path_to_hash, 'w') as hash_file:
> + hash_file.writelines(hash_lines)
> +
> + # The Config.in is the last file we create
> + # It is used by buildroot's menuconfig, gconfig, xconfig or nconfig
> + # it is used to displayspackage info and to select requirements
> + # http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
> + path_to_config = '/'.join([pkg_dir, 'Config.in'])
> + print('Creating {file}...'.format(file=path_to_config))
> + config_in_lines = create_config_in(pkg_name, pkg_req, package)
> + with open(path_to_config, 'w') as config_file:
> + config_file.writelines(config_in_lines)
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-15 10:06 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
@ 2015-06-21 12:44 ` Arnout Vandecappelle
2015-06-23 16:14 ` Denis Thulin
2015-06-21 12:48 ` Arnout Vandecappelle
2015-06-28 14:14 ` Thomas Petazzoni
2 siblings, 1 reply; 15+ messages in thread
From: Arnout Vandecappelle @ 2015-06-21 12:44 UTC (permalink / raw)
To: buildroot
On 06/15/15 12:06, Denis THULIN wrote:
> A generic test automation framework written in python.
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
[snip]
> diff --git a/package/python-robotframework/python-robotframework.hash b/package/python-robotframework/python-robotframework.hash
> new file mode 100644
> index 0000000..7d9ec46
> --- /dev/null
> +++ b/package/python-robotframework/python-robotframework.hash
> @@ -0,0 +1,2 @@
> +# sha356 calculated by scanpypi
> +sha356 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 robotframework-2.9a2.tar.gz
Doesn't this give an error? I couldn't check for myself since the file has
vanished from pypi...
> diff --git a/package/python-robotframework/python-robotframework.mk b/package/python-robotframework/python-robotframework.mk
> new file mode 100644
> index 0000000..437d5f4
> --- /dev/null
> +++ b/package/python-robotframework/python-robotframework.mk
> @@ -0,0 +1,14 @@
> +################################################################################
> +#
> +# python-robotframework
> +#
> +################################################################################
> +
> +PYTHON_ROBOTFRAMEWORK_VERSION = 2.9a2
> +PYTHON_ROBOTFRAMEWORK_SOURCE = robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
> +PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework/
> +PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = distutils
> +PYTHON_ROBOTFRAMEWORK_LICENSE = Apache License 2.0
Apache-2.0
Regards,
Arnout
> +PYTHON_ROBOTFRAMEWORK_LICENSE_FILES = LICENSE.txt
> +
> +$(eval $(python-package))
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-15 10:06 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
2015-06-21 12:44 ` Arnout Vandecappelle
@ 2015-06-21 12:48 ` Arnout Vandecappelle
2015-06-28 14:14 ` Thomas Petazzoni
2 siblings, 0 replies; 15+ messages in thread
From: Arnout Vandecappelle @ 2015-06-21 12:48 UTC (permalink / raw)
To: buildroot
On 06/15/15 12:06, Denis THULIN wrote:
> +PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework/
There shouldn't be a trailing / here, so something must be done about that in
the scanpypi script.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] python-magic: new package
2015-06-15 10:06 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
@ 2015-06-21 12:52 ` Arnout Vandecappelle
2015-06-28 20:39 ` Thomas Petazzoni
1 sibling, 0 replies; 15+ messages in thread
From: Arnout Vandecappelle @ 2015-06-21 12:52 UTC (permalink / raw)
To: buildroot
On 06/15/15 12:06, Denis THULIN wrote:
> +PYTHON_MAGIC_LICENSE = PSF
Up to now, we've always used it in full: Python software foundation license but
perhaps it's indeed better to abbreviate it.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-21 12:44 ` Arnout Vandecappelle
@ 2015-06-23 16:14 ` Denis Thulin
0 siblings, 0 replies; 15+ messages in thread
From: Denis Thulin @ 2015-06-23 16:14 UTC (permalink / raw)
To: buildroot
Hi Arnout,
On 06/21/15 14:44, Arnout Vandecappelle wrote:
> On 06/15/15 12:06, Denis THULIN wrote:
> > A generic test automation framework written in python.
> >
> > Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> [snip]
> > diff --git
> > a/package/python-robotframework/python-robotframework.hash
> > b/package/python-robotframework/python-robotframework.hash
> > new file mode 100644
> > index 0000000..7d9ec46
> > --- /dev/null
> > +++ b/package/python-robotframework/python-robotframework.hash
> > @@ -0,0 +1,2 @@
> > +# sha356 calculated by scanpypi
> > +sha356
> > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> > robotframework-2.9a2.tar.gz
>
> Doesn't this give an error? I couldn't check for myself since the
> file has
> vanished from pypi...
The file is still on pypi :
https://pypi.python.org/packages/source/r/robotframework/robotframework-2.9a2.tar.gz
But sha356 does not exist as far as I know and the sha256 was wrong anyway
because I accidently deleted a line from scanpypi.py
>
> > diff --git a/package/python-robotframework/python-robotframework.mk
> > b/package/python-robotframework/python-robotframework.mk
> > new file mode 100644
> > index 0000000..437d5f4
> > --- /dev/null
> > +++ b/package/python-robotframework/python-robotframework.mk
> > @@ -0,0 +1,14 @@
> > +################################################################################
> > +#
> > +# python-robotframework
> > +#
> > +################################################################################
> > +
> > +PYTHON_ROBOTFRAMEWORK_VERSION = 2.9a2
> > +PYTHON_ROBOTFRAMEWORK_SOURCE =
> > robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
> > +PYTHON_ROBOTFRAMEWORK_SITE =
> > https://pypi.python.org/packages/source/r/robotframework/
> > +PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = distutils
> > +PYTHON_ROBOTFRAMEWORK_LICENSE = Apache License 2.0
>
> Apache-2.0
>
I will correct that.
Regards,
Denis
>
> Regards,
> Arnout
>
> > +PYTHON_ROBOTFRAMEWORK_LICENSE_FILES = LICENSE.txt
> > +
> > +$(eval $(python-package))
> >
>
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR
> Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-15 10:06 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
2015-06-21 12:44 ` Arnout Vandecappelle
2015-06-21 12:48 ` Arnout Vandecappelle
@ 2015-06-28 14:14 ` Thomas Petazzoni
2015-06-29 8:49 ` Denis Thulin
2 siblings, 1 reply; 15+ messages in thread
From: Thomas Petazzoni @ 2015-06-28 14:14 UTC (permalink / raw)
To: buildroot
Dear Denis THULIN,
On Mon, 15 Jun 2015 12:06:04 +0200, Denis THULIN wrote:
> diff --git a/package/python-robotframework/Config.in b/package/python-robotframework/Config.in
> new file mode 100644
> index 0000000..fe95eea
> --- /dev/null
> +++ b/package/python-robotframework/Config.in
> @@ -0,0 +1,9 @@
> +config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
> + depends on BR2_PACKAGE_PYTHON
So it only works with Python 2. What about using
https://pypi.python.org/pypi/robotframework-python3 instead, which
seems to support both Python 2 and Python 3 ? Or maybe this is not an
"official" version ?
I'll mark your patch as Changes Requested in patchwork. Can you resend
an updated version that takes into account the comments from Arnout?
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] python-magic: new package
2015-06-15 10:06 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
2015-06-21 12:52 ` Arnout Vandecappelle
@ 2015-06-28 20:39 ` Thomas Petazzoni
1 sibling, 0 replies; 15+ messages in thread
From: Thomas Petazzoni @ 2015-06-28 20:39 UTC (permalink / raw)
To: buildroot
Denis,
On Mon, 15 Jun 2015 12:06:05 +0200, Denis THULIN wrote:
> diff --git a/package/python-magic/Config.in b/package/python-magic/Config.in
> new file mode 100644
> index 0000000..e6c06b9
> --- /dev/null
> +++ b/package/python-magic/Config.in
> @@ -0,0 +1,7 @@
> +config BR2_PACKAGE_PYTHON_MAGIC
> + depends on BR2_PACKAGE_PYTHON
According to https://pypi.python.org/pypi/python-magic and my own
tests, it seems that python-magic works with Python 3, so this
dependency is not needed.
However, you're missing the runtime dependency on libmagic, which is
provided in Buildroot with the file package. Indeed python-magic is
just a small wrapper around the libmagic C library: all what
python-magic does is open/use libmagic through the ctypes module.
So, you need something like:
> + bool "python-magic"
# libmagic is needed at runtime
select BR2_PACKAGE_FILE
However, even with that, python-magic unfortunately does not work: it
uses ctypes.util.find_library() to find libmagic, and this function
doesn't seem to behave properly in a cross-compilation context (it
tries to use gcc/objdump on the target, which are obviously not
available). So maybe patching Python is needed here.
> diff --git a/package/python-magic/python-magic.hash b/package/python-magic/python-magic.hash
> new file mode 100644
> index 0000000..36ffd02
> --- /dev/null
> +++ b/package/python-magic/python-magic.hash
> @@ -0,0 +1,2 @@
> +# sha356 calculated by scanpypi
> +sha356 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 python-magic-0.4.6.tar.gz
This hash didn't work for me. And Pypi provides a md5. So your .hash
file should be:
# md5 from https://pypi.python.org/pypi/python-magic
md5 07e7a0fea78dd81ed609414c3484df58 python-magic-0.4.6.tar.gz
# sha256 calculated by scanpypi
sha256 903d3d3c676e2b1244892954e2bbbe27871a633385a9bfe81f1a81a7032df2fe python-magic-0.4.6.tar.gz
> diff --git a/package/python-magic/python-magic.mk b/package/python-magic/python-magic.mk
> new file mode 100644
> index 0000000..fad3056
> --- /dev/null
> +++ b/package/python-magic/python-magic.mk
> @@ -0,0 +1,14 @@
> +################################################################################
> +#
> +# python-magic
> +#
> +################################################################################
> +
> +PYTHON_MAGIC_VERSION = 0.4.6
> +PYTHON_MAGIC_SOURCE = python-magic-$(PYTHON_MAGIC_VERSION).tar.gz
> +PYTHON_MAGIC_SITE = https://pypi.python.org/packages/source/p/python-magic/
> +PYTHON_MAGIC_SETUP_TYPE = setuptools
> +PYTHON_MAGIC_LICENSE = PSF
> +PYTHON_MAGIC_LICENSE_FILES =
Empty variables are quite useless, so you get rid of
PYTHON_MAGIC_LICENSE_FILES entirely.
I'll mark your patch as Changes Requested in patchwork, so could you
resend an updated version with the above issues fixed?
Thanks a lot!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-28 14:14 ` Thomas Petazzoni
@ 2015-06-29 8:49 ` Denis Thulin
2015-06-29 12:41 ` Thomas Petazzoni
0 siblings, 1 reply; 15+ messages in thread
From: Denis Thulin @ 2015-06-29 8:49 UTC (permalink / raw)
To: buildroot
Dear Thomas,
On Mon, 28 Jun 2015 16:14:04 +0200, Denis THULIN wrote:
> Dear Denis THULIN,
>
> On Mon, 15 Jun 2015 12:06:04 +0200, Denis THULIN wrote:
>
> > diff --git a/package/python-robotframework/Config.in
> > b/package/python-robotframework/Config.in
> > new file mode 100644
> > index 0000000..fe95eea
> > --- /dev/null
> > +++ b/package/python-robotframework/Config.in
> > @@ -0,0 +1,9 @@
> > +config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
> > + depends on BR2_PACKAGE_PYTHON
>
> So it only works with Python 2. What about using
> https://pypi.python.org/pypi/robotframework-python3 instead, which
> seems to support both Python 2 and Python 3 ? Or maybe this is not an
> "official" version ?
According to the Robot Framework install instructions, the Python 3 port is an
unofficial one.
see: https://github.com/robotframework/robotframework/blob/master/INSTALL.rst#introduction
It is also not up to date as the python 3 port is 3 versions late.
I could make a different package for it though.
>
> I'll mark your patch as Changes Requested in patchwork. Can you
> resend
> an updated version that takes into account the comments from Arnout?
I'm working on it, but as it is my test package for scanpypi, I might not submit
it before the end of the week.
Regards,
>
> Thanks,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] scanpypi.py: new utility
2015-06-21 12:29 ` Arnout Vandecappelle
@ 2015-06-29 11:49 ` Denis Thulin
0 siblings, 0 replies; 15+ messages in thread
From: Denis Thulin @ 2015-06-29 11:49 UTC (permalink / raw)
To: buildroot
Hi Arnout,
On 06/21/15 14:29, Arnout Vandecappelle wrote:
> On 06/15/15 12:06, Denis THULIN wrote:
> > Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> > ---
> > v0: initial commit
> > python-pacakage-generator.py is an utility for automatically
> > generating a
> > python package. It fetches packages info from
> > http://pypi.python.org and
> > generates corresponding packages files.
>
> This should go above your Sob so there's an actual commit message.
>
> >
> > v1:
> > - renamed python-package-generator to scanpypi
> > - split the huge script into a lot of functions
> > - fixed mistakes and small bugs
> >
> > I did not know where to put the script so I put it in
> > support/scripts.
> > I have updated the python-package section of the manual as well.
>
> We generally make the update to the manual a separate patch, but
> this is OK for
> me as well.
>
> >
> > Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
> > ---
> > docs/manual/adding-packages-python.txt | 36 ++
> > support/scripts/scanpypi.py | 607
> > +++++++++++++++++++++++++++++++++
> > 2 files changed, 643 insertions(+)
> > create mode 100755 support/scripts/scanpypi.py
> >
> > diff --git a/docs/manual/adding-packages-python.txt
> > b/docs/manual/adding-packages-python.txt
> > index f81d625..647cb67 100644
> > --- a/docs/manual/adding-packages-python.txt
> > +++ b/docs/manual/adding-packages-python.txt
> > @@ -7,6 +7,42 @@ This infrastructure applies to Python packages
> > that use the standard
> > Python setuptools mechanism as their build system, generally
> > recognizable by the usage of a +setup.py+ script.
> >
> > +[[scanpypi]]
> > +
> > +==== generating a +python-package+ from a pypi repository
>
> Capitalization: Generating
ok
>
> > +
> > +You may want to use the +scanpypi.py+ located in
> > ++support/script+ to generate a package from an existing pypi(pip)
> > package.
> > +
> > +you can find the list of existing pypi package here:
> > (https://pypi.python.org).
>
> You
>
> The parenthesis are redundant.
>
True, I will change that
>
> > +
> > +Please keep in mind that you most likely need
> > +to manually check the package for any mistakes
> > +as there are things that cannot be guessed by the generator (e.g.
> > +dependencies on any of the python core modules
> > +such as BR2_PACKAGE_PYTHON_ZLIB).
>
> Could you rewrap this paragraph? And remove the end-of-line spaces.
I thought I had done it already. I will correct that.
>
> > +
> > +When at the root of your buildroot directory just do :
> > +
> > +-----------------------
> > +./support/script/scanpypi.py foo bar -o package
> > +-----------------------
> > +
> > +This will generate packages +python-foo+ and +python-bar+ in the
> > package
> > +folder if they exist on https://pypi.python.org.
> > +
> > +You will need to manually write the path to the package inside
> > +the +package/Config.in+ file:
>
> : -> .
>
> But perhaps reformulate:
>
> You need to manually add the package to the +package/Config.in+ file.
>
> It would also be better if this sentence was part of the paragraph a
> couple of
> lines higher, were you say that the package has to be checked
> manually. And the
> following sentence should be part of the same paragraph.
Ok
>
> > +
> > +Find the +external python modules+ menu and insert your package
> > inside.
> > +Keep in mind that the items inside a menu should be in
> > alphabetical order.
> > +
> > +Option +-h+ wil list the options available
> > +
> > +-----------------------
> > +./support/script/scanpypi.py -h
> > +-----------------------
> > +
> > [[python-package-tutorial]]
> >
> > ==== +python-package+ tutorial
> > diff --git a/support/scripts/scanpypi.py
> > b/support/scripts/scanpypi.py
> > new file mode 100755
> > index 0000000..953f8d2
> > --- /dev/null
> > +++ b/support/scripts/scanpypi.py
> > @@ -0,0 +1,607 @@
> > +#!/usr/bin/python2
> > +"""
> > +Utility for building buildroot packages for existing pypi packages
> > +
> > +Any package built by scanpypi should be manually checked for
> > +errors.
> > +"""
> > +from __future__ import print_function
> > +import argparse
> > +import json
> > +import urllib2
> > +import sys
> > +import os
> > +import shutil
> > +import StringIO
> > +import tarfile
> > +import errno
> > +import hashlib
> > +import re
> > +import magic
> > +import tempfile
> > +from functools import wraps
> > +
> > +
> > +# private global
> > +_calls = {}
> > +
> > +
> > +def setup_info(pkg_name):
> > + """Get a package info from _calls
> > +
> > + Keyword arguments:
> > + pkg_name -- the name of the package
> > + """
> > + return _calls[pkg_name]
> > +
> > +
> > +def setup_decorator(func, method):
> > + """
> > + Decorator for distutils.core.setup and setuptools.setup.
> > + Puts the args of setup as a dict inside global private dict
> > _calls.
> > + Add key 'method' which should be either 'setuptools' or
> > 'distutils'.
> > +
> > + Keyword arguments:
> > + func -- either setuptools.setup or distutils.core.setup
> > + method -- either 'setuptools' or 'distutils'
> > + """
> > +
> > + @wraps(func)
> > + def closure(*args, **kwargs):
> > + _calls[kwargs['name']] = kwargs
> > + _calls[kwargs['name']]['method'] = method
> > + return closure
> > +
> > +
> > +# monkey patch
> > +import setuptools
> > +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> > +import distutils
> > +distutils.core.setup = setup_decorator(setuptools.setup,
> > 'distutils')
> > +
> > +
> > +def find_file_upper_case(filenames, path='./'):
> > + """
> > + List generator:
> > + Recursively find files that matches one of the specified
> > filenames.
> > + Returns absolute path
> > +
> > + Keyword arguments:
> > + filenames -- List of filenames to be found
> > + path -- Path to the directory to search
> > + """
> > + for root, dirs, files in os.walk(path):
> > + for file in files:
> > + if file.upper() in filenames:
> > + yield (os.path.join(root, file))
> > +
> > +
> > +def pkg_buildroot_name(pkg_name):
> > + """
> > + Returns name to avoid troublesome characters.
> > + Remove all non alphanumeric characters except -
> > + Also lowers the name
> > +
> > + Keyword arguments:
> > + pkg_name -- String to rename
> > + """
> > + name = re.sub('[^\w-]', '', pkg_name.lower())
> > + name = re.sub('^python-', '', name)
> > + return name
> > +
> > +
> > +def find_setup(package_name, version, archive):
> > + """
> > + Search for setup.py file in an archive and returns True if
> > found
> > + Used for finding the correct path to the setup.py
> > +
> > + Keyword arguments:
> > + package_name -- base name of the package to search (e.g.
> > Flask)
> > + version -- version of the package to search (e.g. 0.8.1)
> > + archive -- tar archive to search in
> > + """
> > + try:
> > + archive.getmember('{name}-{version}/setup.py'.format(
> > + name=package_name,
> > + version=version))
> > + except KeyError:
> > + return False
> > + else:
> > + return True
> > +
> > +
> > +def fetch_package_info(pkg_name):
> > + """
> > + Fetch a package's metadata for the python package index
> > +
> > + Keyword arguments:
> > + pkg_name -- the name of the package
> > + """
> > + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> > + pkg=pkg_name)
> > + print('URL:', url)
> > + try:
> > + pkg_json = urllib2.urlopen(url).read().decode()
> > + except (urllib2.HTTPError) as error:
>
> I don't think these parenthesis are needed?
No they are not.
>
> > + print('ERROR:', error.getcode(), error.msg,
> > file=sys.stderr)
> > + print('ERROR: Could not find package {pkg}.\n'
> > + 'Check syntax inside the python package index:\n'
> > + 'https://pypi.python.org/pypi/
> > '.format(pkg=pkg_name))
> > + return None, None
> > + except urllib2.URLError:
> > + print('ERROR: Could not find package {pkg}.\n'
> > + 'Check syntax inside the python package index:\n'
> > + 'https://pypi.python.org/pypi/
> > '.format(pkg=pkg_name))
> > + return None, None
> > +
> > + else:
> > + return pkg_json, url
> > +
> > +
> > +def download_package(package):
> > + """
> > + Download a package using metadata from pypi
> > +
> > + Keyword arguments:
> > + package -- a dictionary containing info from the pypi json api
> > + """
> > + try:
> > + targz = package['urls'][0]['filename']
> > + except IndexError:
> > + print(
> > + 'Non conventional package, ',
> > + 'please check manually after creation')
>
> Is it even worthwhile to support this case?
Well I ran into one of those case while testing scanpipy.pi, I don't know
how many package are made this way though.
>
> > + download_url = package['info']['download_url']
> > + try:
> > + download = urllib2.urlopen(download_url)
> > + except urllib2.HTTPError:
>
> Shouldn't we print an error message here? Or does urrlib already do
> that?
No it should be printed, but not here. It should be printed when I test if the download was successful.
>
> > + targz = None
> > + download = None
> > + as_file = None
> > + used_url = None
> > + else:
> > + used_url = {'url': download_url}
> > + as_file = StringIO.StringIO(download.read())
>
> Actually, you use the download mostly as a string, it's only used as
> a file in
> tarfile.open(). So I'd put the StringIO call there, and keep it as a
> string
> everywhere else.
>
> Also, perhaps it's worthwhile to make a class for the package
> information
> instead of passing around tuples? I'm not entirely sure if that
> really makes
> things simpler, but it's something to consider.
It is possible to do it, but it makes things harder when I have to consider dependencies
as I need the function that checks dependencies to be aware of the whole list of package being made.
I could probably do everything in a class except that part though.
>
> > + as_file.seek(0)
> > + extension = 'tar.gz'
> > + if 'gzip' not in magic.from_buffer(as_file.read()):
> > + extension = 'tar.bz2'
>
> This part I really don't like. Can't we just get the extension from
> the URL?
I will do that.
>
> > + targz = '{name}-{version}.{extension}'.format(
> > + name=package['info']['name'],
> > + version=package['info']['version'],
> > extension=extension)
> > + as_file.seek(0)
> > + used_url['filename'] = targz
> > + else:
> > + for download_url in package['urls']:
> > + try:
> > + download = urllib2.urlopen(download_url['url'])
> > + except urllib2.HTTPError:
> > + targz = None
> > + download = None
> > + as_file = None
> > + used_url = None
> > + else:
> > + used_url = download_url
> > + as_file = StringIO.StringIO(download.read())
> > + md5_sum = hashlib.md5(as_file.read()).hexdigest()
> > + if md5_sum == download_url['md5_digest']:
> > + break
> > + targz = used_url['filename']
> > + return download, targz, used_url, as_file
> > +
> > +
> > +def extract_package(pkg_name, as_file, tmp_path):
> > + """
> > + Create folders used for extracting a package as file object
> > and extract it
> > +
> > + pkg_name -- name of the package to be extracted
> > + as_file -- file object to extract
> > + tmp_path -- folder where you want the package to be extracted
> > + """
> > + as_file.seek(0)
> > + as_tarfile = tarfile.open(fileobj=as_file)
>
> Better use a with clause here.
True
>
> > + tmp_pkg = '/'.join([tmp_path, pkg_name])
>
> os.path.join is better I think.
I agree.
>
> > + try:
> > + os.makedirs(tmp_pkg)
> > + except OSError as exception:
> > + if exception.errno != errno.EEXIST:
> > + print("ERROR: ", exception.message, file=sys.stderr)
> > + return None, None
> > + print('WARNING:', exception.message, file=sys.stderr)
> > + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> > + shutil.rmtree(tmp_pkg)
> > + os.makedirs(tmp_pkg)
> > + version = package['info']['version']
>
> This accidentally works because the package variable is set in the
> global
> scope, but I think it's better to pass this as an argument to the
> function. Or
> better yet, make a class :-)
Oh, I had not seen that.
I'm seriously considering doing a class now :)
>
> > + tar_folder = package['info']['name']
> > + if not find_setup(tar_folder, version, as_tarfile):
> > + return None, None
>
> Since the tarball will anyway be extracted below, perhaps it's
> easier to do
> this find_setup on the extracted directory.
This is code from a version where I only extracted the setup.py file,
I will change it.
>
> > + as_tarfile.extractall(tmp_pkg)
> > + as_tarfile.close()
> > + as_file.close()
> > + tmp_extract = '{folder}/{name}-{version}'.format(
> > + folder=tmp_pkg,
> > + name=tar_folder,
> > + version=package['info']['version'])
> > + return tar_folder, tmp_extract
> > +
> > +
> > +def get_requirements(package_name):
> > + """
> > + Retrieve dependencies of from a metadata found in the setup.py
> > script of
> ^^^^^^^^^
> dependencies from the metadata
>
> > + a pypi package.
> > +
> > + Keyword Arguments:
> > + package_name -- name of the package found in the pypi metadata
> > of the
> > + package.
> > + """
> > + pkg_req = setup_info(package_name)['install_requires']
> > + pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
> > + for req in pkg_req]
> > + pkg_req = map(pkg_buildroot_name, pkg_req)
> > + req_not_found = [
> > + pkg for pkg in pkg_req
> > + if 'python-{name}'.format(name=pkg)
> > + not in os.listdir(pkg_folder)
> > + ]
> > + req_not_found = [pkg for pkg in req_not_found
> > + if pkg not in packages]
> > + if (req_not_found) != 0:
>
> Just "if req_not_found:" - it's a list, so comparing to 0 is weird.
True
>
> > + print(
> > + 'Error: could not find packages \'{packages}\''
> > + 'required by {current_package}'.format(
> > + packages=", ".join(req_not_found),
> > + current_package=pkg_name))
>
> scancpan instead adds the dependencies to the list of packages to
> create. We
> could do something like this here as well. But that can be done in a
> follow-up
> patch.
Well It should be just one line to add.
>
> > + return pkg_req
> > +
> > +
> > +def create_mk_header(pkg_name):
> > + """
> > + Create the header of the <package_name>.mk file
> > +
> > + Keyword arguments:
> > + pkg_name -- name of the package
> > + """
> > + header = ['#' * 80 + '\n']
> > + header.append('#\n')
> > + header.append('# python-{name}\n'.format(name=pkg_name))
> > + header.append('#\n')
> > + header.append('#' * 80 + '\n')
> > + header.append('\n')
> > + return header
>
> I would return a string here instead of a list, so it can all be
> concatenated
> easily in the calling function.
As I have a list of lines in the calling function and uses writelines to write the file,
I think having lists of lines everywhere should be the correct choice.
>
> > +
> > +
> > +def create_mk_download_info(pkg_name, version, targz, url):
> > + """
> > + Create the lines refering to the download information of the
> > + <package_name>.mk file
> > +
> > + Keyword arguments:
> > + pkg_name -- name of the package
> > + version -- version of the package
> > + targz -- name of the archive corresponding to the package
> > + url -- url to be used for downloading the package
> > + """
> > + lines = []
> > + version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
> > + name=pkg_name.upper(),
>
> It's not just .upper(), also - has to replaced with _
I will change that
>
> > + version=version)
> > + lines.append(version_line)
> > +
> > + targz = targz.replace(
> > + version,
> > + '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
> > + targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
> > + name=pkg_name.upper(),
> > + filename=targz)
> > + lines.append(targz_line)
> > +
> > + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> > + name=pkg_name.upper(),
> > + url=url['url'].replace(url['filename'], '')))
> > + if 'sourceforge' in site_line:
> > + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> > + name=pkg_name.upper(),
> > + url=url['url']))
> > + lines.append(site_line)
> > + return lines
> > +
> > +
> > +def create_mk_setup(pkg_name, tar_folder):
> > + """
> > + Create the line refering to the setup method of the package of
> > the
> > + <package_name>.mk file
> > +
> > + There are two things you can use to make an installer
> > + for a python package: distutils or setuptools
> > + distutils comes with python but does not support dependancies.
>
> dependencies
>
> > + distutils is mostly still there for backward support.
> > + setuptools is what smart people use,
> > + but it is not shipped with python :(
> > +
> > + Keyword Arguments:
> > + pkg_name -- name of the package
> > + tar_folder -- name of the folder where the setup.py can be
> > found
> > + """
> > + lines = []
> > + setup_type_line = 'PYTHON_{name}_SETUP_TYPE =
> > {method}\n'.format(
> > + name=pkg_name.upper(),
> > + method=setup_info(tar_folder)['method'])
> > + lines.append(setup_type_line)
> > + return lines
> > +
> > +
> > +def create_mk_license(pkg_name, license_name, package_location):
> > + """
> > + Create the lines referring to the package's license
> > informations of the
> > + <package_name>.mk file
> > +
> > + The license's files are found by searching the package for
> > files named
> > + license or license.txt (case insensitive).
> > + If more than one license file is found, the user is asked to
> > select which
> > + ones he wants to use.
> > +
> > + Keyword Arguments:
> > + pkg_name -- name of the package
> > + license_name -- name of the license
> > + package_location -- where to look for the licenses
> > + """
> > + lines = []
> > + license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
> > + name=pkg_name.upper(),
> > + license=license_name)
> > + lines.append(license_line)
> > + print('WARNING: License has been set to "{license}",'
> > + ' please change it manually if necessary'.format(
> > + license=license_name))
> > + filenames = ['LICENSE', 'LICENSE.TXT']
> > + license_files = list(find_file_upper_case(filenames,
> > package_location))
> > + license_files = [license.replace(package_location, '')[1:]
> > + for license in license_files]
> > + if len(license_files) > 1:
> > + print('More than one file found for license: ')
>
> No need to go interactive here: just make a space-separated list of
> license
> files. You could spew a warning if it is more than one, but it's
> anyway going to
> be verified manually.
>
> > + for index, item in enumerate(license_files):
> > + print('\t{index})'.format(index=index), item)
> > + license_choices = raw_input(
> > + 'specify file numbers separated by spaces(default 0):
> > ')
> > + license_choices = [int(choice)
> > + for choice in license_choices.split('
> > ')
> > + if choice.isdigit() and int(choice) in
> > + range(len(license_files))]
> > + if len(license_choices) == 0:
> > + license_choices = [0]
> > + license_files = [file
> > + for index, file in
> > enumerate(license_files)
> > + if index in license_choices]
> > + elif len(license_files) == 0:
> > + print('WARNING: No license file found,'
> > + ' please specify it manually afterward')
> > +
> > + license_file_line = ('PYTHON_{name}_LICENSE_FILES ='
>
> This line should not be added if len(license_files) == 0, but
> instead there
> should be something like:
>
> # No license file found
I wasn't sure about that, thanks :)
>
> > + ' {files}\n'.format(
> > + name=pkg_name.upper(),
> > + files=' '.join(license_files)))
> > + license_file_line = license_file_line.replace(' \n', '\n')
>
> If len(license_files) > 0 then this will not be needed.
>
> > + lines.append(license_file_line)
> > + return lines
> > +
> > +
> > +def create_mk_requirements(pkg_name, pkg_req):
> > + """
> > + Create the lines referring to the dependencies of the of the
> > + <package_name>.mk file
> > +
> > + Keyword Arguments:
> > + pkg_name -- name of the package
> > + pkg_req -- dependencies of the package
> > + """
> > + lines = []
> > + python_pkg_req = ['python-{name}'.format(name=pkg)
> > + for pkg in pkg_req]
> > + dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
> > + ' {reqs}\n'.format(
> > + name=pkg_name.upper(),
> > + reqs=' '.join(python_pkg_req)))
> > + lines.append(dependencies_line)
> > + return lines
> > +
> > +
> > +def create_config_mk(pkg_name, version, license, url, targz,
> > + tar_folder, pkg_req, package_location):
>
> create_config_mk -> create_package_mk
ok
>
> > + """
> > + Create the lines corresponding to the <package_name>.mk file
> > +
> > + Keyword Arguments:
> > + pkg_name -- name of the package
> > + version -- version of the package
> > + license -- name of the package's license
> > + url -- where to download the package
> > + targz -- name of the archive when downloaded
> > + tar_folder -- name of the folder where the setup.py can be
> > found
> > + pkg_req -- dependencies of the package
> > + package_location -- path to the extracted package
> > + """
> > + lines = create_mk_header(pkg_name)
> > + lines += create_mk_download_info(pkg_name, version, targz,
> > url)
> > + lines += create_mk_setup(pkg_name, tar_folder)
> > + lines += create_mk_license(pkg_name, license,
> > package_location)
> > + if pkg_req:
> > + lines += create_mk_requirements(pkg_name, pkg_req)
> > +
> > + lines.append('\n')
> > + lines.append('$(eval $(python-package))')
> > + lines.append('\n')
> > +
> > + return lines
> > +
> > +
> > +def create_hash_file(url, digest, hash_function='sha356'):
> ^^^^^^sha256
>
> > + """
> > + Create the lines corresponding to the <package_name>.hash
> > files
> > +
> > + Keyword Arguments:
> > + url -- metadata 'url' from the pypi json api
> > + digest -- digest made from the downladed archive
> > + hash_function -- algorythm used for hashing
>
> algorithm
>
> > + """
> > + lines = []
> > + commented_line = '# {method} calculated by scanpypi\n'.format(
>
> I think there would typically be one md5 that comes from pypi, and
> one sha256
> that is calculated locally. So the comment is not correct.
>
> > + method=hash_function)
> > + lines.append(commented_line)
> > + hash_line = '{method}\t{digest} {filename}\n'.format(
> > + method=hash_function,
> > + digest=digest,
> > + filename=url['filename'])
> > + lines.append(hash_line)
> > + return lines
> > +
> > +
> > +def create_config_in(pkg_name, pkg_req, package):
> > + """
> > + Creates the Config.in file of a package
> > +
> > + pkg_name -- name of the package
> > + pkg_req -- dependencies of the package
> > + package -- metadata of the package from pypi
> > + """
> > + lines = []
> > + config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
> > + name=pkg_name.upper())
> > + lines.append(config_line)
> > + python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
>
> Why? Are all pypi packages python2-only? Remember, the
> python2||python3
> condition is already in package/Config.in.
True.
>
> > + lines.append(python_line)
> > +
> > + bool_line = '\tbool "python-{name}"\n'.format(name=pkg_name)
> > + lines.append(bool_line)
> > + if pkg_req:
> > + for dep in pkg_req:
> > + dep_line = '\tselect
> > BR2_PACKAGE_PYTHON_{req}\n'.format(
> > + req=dep.upper())
> > + lines.append(dep_line)
> > +
> > + lines.append('\thelp\n')
> > +
> > + help_lines = package['info']['summary'].split('\n')
>
> The help_lines should also be wrapped, so call textwrap.wrap() on
> it.
ok
>
> > + help_lines.append('')
> > + help_lines.append(package['info']['home_page'])
> > + help_lines = ['\t {line}\n'.format(line=line)
> > + for line in help_lines]
> > + lines += help_lines
> > + return lines
> > +
> > +
> > +if __name__ == "__main__":
> > +
> > + # Building the parser
> > + parser = argparse.ArgumentParser(
> > + description="Creates buildroot packages from the metadata
> > of "
> > + "an existing pypi(pip) packages and include it
> > "
> > + "in menuconfig")
> > + parser.add_argument("packages",
> > + help="list of packages to be made",
> > + nargs='+')
> > + parser.add_argument("-o", "--output",
> > + help="""
> > + Output directory for packages
> > + """,
> > + default='.')
>
> Since it will be called from the buildroot top dir, this should
> default to
> ./packages I think.
Yes, it should
>
> > +
> > + args = parser.parse_args()
> > + packages = list(set(args.packages))
> > +
> > + # tmp_path is where we'll extract the files later
> > + tmp_prefix = 'scanpypi-'
> > + pkg_folder = args.output
> > + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
>
> There should be a big try: block around here that removes the tmpdir
> unconditionally. Clearly that makes debugging harder, but it's easy
> to remove
> the rmtree from the script when you want to debug.
Ok, I will do that
>
> > +
> > + for real_pkg_name in packages:
> > + pkg_name = pkg_buildroot_name(real_pkg_name)
> > + print('buildroot package name for
> > {}:'.format(real_pkg_name),
> > + pkg_name)
> > + # First we download the package
> > + # Most of the info we need can only be found inside the
> > package
> > + print('Package:', pkg_name)
> > + print('Fetching package', real_pkg_name)
> > + pkg_json, url = fetch_package_info(real_pkg_name)
> > + if not pkg_json:
> > + continue
> > +
> > + pkg_dir = pkg_folder + '/python-' + pkg_name
> > + package = json.loads(pkg_json)
> > + used_url = ''
> > + print('Downloading package {pkg}...'.format(
> > + pkg=package['info']['name']))
> > + download, targz, used_url, as_file =
> > download_package(package)
> > + version = package['info']['version']
> > +
> > + if not download:
> > + print('Error downloading package :', pkg_name)
> > + continue
> > +
> > + sha256_digest = hashlib.sha256(as_file.read()).hexdigest()
> > +
> > + # extract the tarball
> > + tar_folder, tmp_extract = extract_package(pkg_name,
> > as_file, tmp_path)
> > +
> > + # Loading the package install info from the package
> > + sys.path.append(tmp_extract)
> > + print(tmp_extract)
>
> This print is not necessary or useful I think.
Thats a debugging print I forgot to remove
>
> > + import setup
> > + setup = reload(setup)
>
> A comment explaining why the reload is necessary would be useful.
>
> Also, it seems that some packages import other modules from their
> source
> directory. So instead of appending to sys.path, I think it's better
> to
> temporarily chdir to the tmp_extract directory. Try for instance
> json-schema-validator
>
> > + sys.path.remove(tmp_extract)
> > +
> > + pkg_req = None
> > + # Package requierement are an argument of the setup
> > function
>
> requirements
>
> > + if 'install_requires' in setup_info(tar_folder):
> > + pkg_req = get_requirements(tar_folder)
> > + # We could stop here
> > + # or ask the user if he still wants to continue
> > +
> > + # Buildroot python packages require 3 files
> > + # The first is the mk file
> > + # See:
> > + #
> > http://buildroot.uclibc.org/downloads/manual/manual.html
>
> Indentation is wrong.
>
> But I don't think this comment is very useful.
>
> > + print('Checking if package {name} already
> > exists...'.format(
> > + name=pkg_dir))
> > + try:
> > + os.makedirs(pkg_dir)
> > + except OSError as exception:
> > + if exception.errno != errno.EEXIST:
> > + print("ERROR: ", exception.message,
> > file=sys.stderr)
> > + continue
> > + print('Error: Package {name} already
> > exists'.format(name=pkg_dir))
> > + del_pkg = raw_input(
> > + 'Do you want to delete existing package ? [y/N]')
> > + if del_pkg.lower() == 'y':
> > + shutil.rmtree(pkg_dir)
> > + os.makedirs(pkg_dir)
> > + else:
> > + continue
> > + pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
> > + path_to_mk = '/'.join([pkg_dir, pkg_mk])
> > + print('Creating {file}...'.format(file=path_to_mk))
> > + config_mk_lines = create_config_mk(pkg_name, version,
> > +
> > package['info']['license'],
> > + used_url, targz,
> > tar_folder,
> > + pkg_req, tmp_extract)
> > + with open(path_to_mk, 'w') as mk_file:
> > + mk_file.writelines(config_mk_lines)
>
> I think it's more appropriate to do the file writing inside the
> function.
Ok, I wasn't sure about thin. thanks :)
>
> > +
> > + # The second file we make is the hash file
> > + # It consists of hashes of the package tarball
> > + #
> > http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
> > + pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
> > + path_to_hash = '/'.join([pkg_dir, pkg_hash])
> > + print('Creating
> > {filename}...'.format(filename=path_to_hash))
> > + hash_lines = create_hash_file(used_url, sha256_digest)
>
> The pypi md5 should be included as well.
>
Ok.
Regards,
Denis
>
> Regards,
> Arnout
>
> > + with open(path_to_hash, 'w') as hash_file:
> > + hash_file.writelines(hash_lines)
> > +
> > + # The Config.in is the last file we create
> > + # It is used by buildroot's menuconfig, gconfig, xconfig
> > or nconfig
> > + # it is used to displayspackage info and to select
> > requirements
> > + #
> > http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
> > + path_to_config = '/'.join([pkg_dir, 'Config.in'])
> > + print('Creating {file}...'.format(file=path_to_config))
> > + config_in_lines = create_config_in(pkg_name, pkg_req,
> > package)
> > + with open(path_to_config, 'w') as config_file:
> > + config_file.writelines(config_in_lines)
> >
>
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR
> Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-29 8:49 ` Denis Thulin
@ 2015-06-29 12:41 ` Thomas Petazzoni
0 siblings, 0 replies; 15+ messages in thread
From: Thomas Petazzoni @ 2015-06-29 12:41 UTC (permalink / raw)
To: buildroot
Denis,
On Mon, 29 Jun 2015 10:49:24 +0200 (CEST), Denis Thulin wrote:
> > So it only works with Python 2. What about using
> > https://pypi.python.org/pypi/robotframework-python3 instead, which
> > seems to support both Python 2 and Python 3 ? Or maybe this is not an
> > "official" version ?
>
> According to the Robot Framework install instructions, the Python 3 port is an
> unofficial one.
> see: https://github.com/robotframework/robotframework/blob/master/INSTALL.rst#introduction
>
> It is also not up to date as the python 3 port is 3 versions late.
Ok. Can you indicate this maybe in the Config.in help text?
> I could make a different package for it though.
No, no need to do it if you don't have the need for it.
> > I'll mark your patch as Changes Requested in patchwork. Can you
> > resend
> > an updated version that takes into account the comments from Arnout?
>
> I'm working on it, but as it is my test package for scanpypi, I might not submit
> it before the end of the week.
No problem.
Thanks a lot for working on this scanpypi script!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] scanpypi.py: new utility
2015-06-15 10:06 ` [Buildroot] [PATCH 1/3] scanpypi.py: new utility Denis THULIN
2015-06-21 12:29 ` Arnout Vandecappelle
@ 2015-06-29 12:42 ` Thomas Petazzoni
1 sibling, 0 replies; 15+ messages in thread
From: Thomas Petazzoni @ 2015-06-29 12:42 UTC (permalink / raw)
To: buildroot
Denis,
On Mon, 15 Jun 2015 12:06:03 +0200, Denis THULIN wrote:
> docs/manual/adding-packages-python.txt | 36 ++
> support/scripts/scanpypi.py | 607 +++++++++++++++++++++++++++++++++
Minor nit: please call the script just scanpypi, without extension.
Like scancpan.
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2015-06-29 12:42 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-15 10:06 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
2015-06-15 10:06 ` [Buildroot] [PATCH 1/3] scanpypi.py: new utility Denis THULIN
2015-06-21 12:29 ` Arnout Vandecappelle
2015-06-29 11:49 ` Denis Thulin
2015-06-29 12:42 ` Thomas Petazzoni
2015-06-15 10:06 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
2015-06-21 12:44 ` Arnout Vandecappelle
2015-06-23 16:14 ` Denis Thulin
2015-06-21 12:48 ` Arnout Vandecappelle
2015-06-28 14:14 ` Thomas Petazzoni
2015-06-29 8:49 ` Denis Thulin
2015-06-29 12:41 ` Thomas Petazzoni
2015-06-15 10:06 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
2015-06-21 12:52 ` Arnout Vandecappelle
2015-06-28 20:39 ` Thomas Petazzoni
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.