From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (NAM11-CO1-obe.outbound.protection.outlook.com [40.107.220.62]) by mx.groups.io with SMTP id smtpd.web08.34392.1605532677322627316 for ; Mon, 16 Nov 2020 05:17:57 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriversystems.onmicrosoft.com header.s=selector2-windriversystems-onmicrosoft-com header.b=TZFAFVcY; spf=pass (domain: windriver.com, ip: 40.107.220.62, mailfrom: david.reyna@windriver.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DEXrVe6mmDMb5ayWwI9vDibZxZqZ9NB2yWSMOn6PnKib4sm7G59mNKULqkSNxWQ8s0ClCP4yD1Mv7nFvOG5JnTCZ77b7hqVZ9nMHqWCPMUQktn6ZlC2TbF/MpsYiHTVWXGgNKTkzBDko33tnxzP8BR9W3u9VFB5z4boXPlP9/EA1kmxm60Pd47WKNNMxC9MNJ04TM3CzypkKGHGqDfqnB2iP6olEpJ1dfZYjXq2ldoK0GwOK+7mY852sDTWCCpfL0qBIPgxmfQjAvEUpDeeVtkgKzWQlnIbG1AbqgOIAbOOm/yMMesHpUJ3Kr0Zvn1BODZqIXnjDK+c0v+Mxn3sd8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+qOf414Xy5VfRiuHMCdRPD1RV/XhEsFBOj1KPGJDyN8=; b=Xum+nPB9LyqNBvTdqfKsbE7eiDmK2qj0P120/6KJVC+CbyKAEvkiPUtAb7GbTlJnBqX7ESWN+B+Eo4ldOFUbEBxa90M03gJYSboQy1rFI245RUDZjwp67wrJPecOL8Noa1+6NiF4jcE6LJ+w2q54NHqXQ9cPBsTPMCzsv0OySWhfAEx98roilI7AeRu3dJK3dmSV1cxNlR86l2Hb22ralDFcy/57YA/igw2kOiukxecDdCniC+h20Tk84AQzHY1QyAn6UkUx9F6K/QpQLsTJZJX5eIuSSq4CgO+XWXypOADorm+E7a+YF0mBNAZJfrkBTKv1jnb7fB8h0ieUKi9rsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriversystems.onmicrosoft.com; s=selector2-windriversystems-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+qOf414Xy5VfRiuHMCdRPD1RV/XhEsFBOj1KPGJDyN8=; b=TZFAFVcY0JeLg6FGEELP/2xcM1pR38V5U/U/jqRpQ0Dbt5d1+HLicRdqB6KZ9GgmdmegxdzJ0w3Ryv+89VTSeB2VtLHQp8ZYgN1tjWmcIfZFfp47ySwf2i+ulPX8WjKGl1Q7ffi0prYZs05zQj1o3ZpoOtNVfb1qAObGX0Lmjlk= Received: from BY5PR11MB4167.namprd11.prod.outlook.com (2603:10b6:a03:185::28) by BY5PR11MB3925.namprd11.prod.outlook.com (2603:10b6:a03:182::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.28; Mon, 16 Nov 2020 13:17:53 +0000 Received: from BY5PR11MB4167.namprd11.prod.outlook.com ([fe80::c0ab:919b:9e0:4b3a]) by BY5PR11MB4167.namprd11.prod.outlook.com ([fe80::c0ab:919b:9e0:4b3a%5]) with mapi id 15.20.3564.025; Mon, 16 Nov 2020 13:17:53 +0000 From: "Reyna, David" To: "Sebert, Holger.ext" , "toaster@lists.yoctoproject.org" Subject: Re: Database erros due to UTF-8 filenames Thread-Topic: Database erros due to UTF-8 filenames Thread-Index: AQHWvBej5tbm93Zq906JGmCT2/S+E6nKvDUA Date: Mon, 16 Nov 2020 13:17:53 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: karlstorz.com; dkim=none (message not signed) header.d=none;karlstorz.com; dmarc=none action=none header.from=windriver.com; x-originating-ip: [71.202.37.249] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8c4b2d64-9534-4de2-ca71-08d88a3206a7 x-ms-traffictypediagnostic: BY5PR11MB3925: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mfYrRDMvBVVW6GANUQqlI4YBjxYfmP+A81V6qi3/18Uj5DvEtSweZBQtCBwmRN4qc/KEZqAQYLje7kytPka4j7QB/0btwvsW0yfsq6tni63bvgEyVdeNS9oWTz4qPzlTKOK82x/CG5NgMnQo4iqEBajCApKg17VRBsaSiCnPclAV8ZVRKIjTJMxnIvqUP2r4iOdaTs8FVbIkNqYykWLMyzcWFOQK4e/bJmyJCuSne8mF/HuZMKEuR3hItqptNgpUsiTMFljjBvsyrHWbqwICeQuX02qGLdRKkDeZzQl+xEQMP6KC4Fn314bLQd7nyG1/ti1EFq0O5E2Q4M72OMhuQZkV5xfS+k/KHUHZ5Ff3r8px4D9TXG0dzcec6PDSK0Ix x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB4167.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(136003)(396003)(376002)(346002)(39840400004)(71200400001)(86362001)(52536014)(53546011)(33656002)(8676002)(6506007)(316002)(66574015)(83380400001)(66476007)(66946007)(66556008)(66446008)(64756008)(5660300002)(76116006)(7696005)(478600001)(9686003)(186003)(55016002)(26005)(2906002)(8936002)(110136005)(130980200001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata: 5spcfxIfjU1MjZcLvpwjJYHJSDlL6idr7PLtjU1op6F/LWNuGkbT9RGmmItTzjY7pbc+NeCEORHC83KmxEA+acRSeYaTl+/LhMRgvLIBG+/ZiDRcZO0eLsTSds6EV687D+sF2XeuklSOraT/FtjNUPDWdtA97OARbZwBYg4s3zbX42L1lguw781k7jkDhR32+o4e0+Ir6pRFG/K/WIZsRmd6CsJEAynYCFNOSZlFYjY1hIiz0y8tI+jFuqxNh86b66/HYZi36ByDA8wJlkjk0SiFv946HGIOj2C8M3PsDkujLbxYmlTtnk2n7bmiCruKv4zLETQr9xq7895X1fAT29yybGr/HORq+GgDnPO/KwASr16DFAqfc7ld3EE4kjvpDOnzBFFRKxU20B+sB5DpM+kJ4Rv8h9deJlDGlJoXfMwuqbTgNhJYOJ6i3GsHu73sjG3DuingI2nuhDfIDn2zz1iTlnxb6aSV3iKChsd6US5JApKCzbo1ce+DeIOeIi7vTkypleWDfjM84lpaRF5o7of91Rj1b/DcMQS6yZysdu6k45lcCFKU3bENDveSaf9GkHpKzUwC84L2COPkI0a39Z1OOs1d1IPFEGk8QtrE2xKHD5gdD4K7hYVKxUHPx/110WtxMrQMHeEhdj4bufS+ygfyuz2tld8/3qmcJGVNUaRjZ8kKm7pTU5716Q7/dNNh9OQNraPZx85DJYjFNoCv+wdz/zk0CDFsB2rhgup3v6wFm/83esUL6O76mLfQcjAuTAUs/kBNJd3O9U9jr7h6z7K1i3oEZaZlTZJRwStAiF4cPSiFruaQFQlFZysaCIiormyJCCif/LlAklRKtH3Sn8GvSWBsZ45eHfmLCyZor9he9JycatOiyTIAFcvTtAJsiULszBArCmx9/dSyHC+TMw== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB4167.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8c4b2d64-9534-4de2-ca71-08d88a3206a7 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Nov 2020 13:17:53.5893 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 2G3FhV2IpR8CU7U92kTWqkq5hXr8dHYlIn4lyzAlzHQDSiwVi0+iv7fiwBDBOFvfayLzI6HrJk713CWN2W5/4gWYk/KdxNQKV4ToNoMuu3Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB3925 Content-Language: en-US Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable Hi Holger, This is an interesting problem. I will investigate. We should see if there are any other localization fields that might have to= support UTF-8 strings. Certainly all local path names will need to be supp= orted. I am also curious on how the local time zone support is working for you. David -----Original Message----- From: toaster@lists.yoctoproject.org On Be= half Of Sebert, Holger.ext Sent: Monday, November 16, 2020 4:57 AM To: toaster@lists.yoctoproject.org Subject: [Toaster] Database erros due to UTF-8 filenames Hi, I've setup Toaster and a MySQL docker container, all running on Ubuntu 16.0= 4. I am encountering the following database error, when building my Yocto proj= ect: ERROR: (1366, "Incorrect string value: '\\xC5\\x91tan\\xC3...' for column = 'path' at row 1") Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py= ", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.7/dist-packages/django/db/backends/mysql/ba= se.py", line 71, in execute return self.cursor.execute(query, args) File "/usr/local/lib/python3.7/dist-packages/MySQLdb/cursors.py", line 2= 06, in execute res =3D self._query(query) File "/usr/local/lib/python3.7/dist-packages/MySQLdb/cursors.py", line 3= 19, in _query db.query(q) File "/usr/local/lib/python3.7/dist-packages/MySQLdb/connections.py", li= ne 260, in query _mysql.connection.query(self, query) MySQLdb._exceptions.OperationalError: (1366, "Incorrect string value: '\\x= C5\\x91tan\\xC3...' for column 'path' at row 1") The query that raised this error looks as follows: INSERT INTO `orm_target_file` (`target_id`, `path`, `size`, `inodetype`, `permission`, `owner`, `group`, `directory_id`, `sym_target_id`) VALUES (19, '/usr/share/ca-certificates/mozilla/NetLock_Arany_=3DClass_Gold=3D_F\xc5\= x91tan\xc3\xbas\xc3\xadtv\xc3\xa1ny.crt', 1476, 1, 'rw-r--r--', 'root', 'root', NULL, NULL) The file causing this error has the following UTF-8 encoded filename: NetLock_Arany_=3DClass_Gold=3D_F=F5tan=FAs=EDtv=E1ny.crt When looking into the database I found out that the column `path` of table `orm_target_file` has the following properties: CHARACTER_SET_NAME: latin1 COLLATION_NAME: latin1_swedish_ci Apperently, the column `path` is not ready for UTF-8 strings. I can fix tha= t manually by doing the following mysql command using the `mysql` tool: ALTER TABLE orm_target_file CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; This change makes the database error disappear. I would like to fix that directly in Toasters's `orm/models.py`. I found th= e following definition in class `Target_File`: path =3D models.FilePathField() It seems like I need to pass some clever options to `FilePathField`, but wh= ich? My own research in that direction has brought up nothing useful so far. My questions are thus: * How can I parametrize `FilePathField` to properly handle UTF-8 encoded filenames in the underlying database? * How should a correspondig migration file look like in `orm/migrations`? Thanks! Best, Holger