From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB64C433FE for ; Fri, 15 Oct 2021 13:02:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7517960F9F for ; Fri, 15 Oct 2021 13:02:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7517960F9F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35910 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mbMqs-0008Kx-Fd for qemu-devel@archiver.kernel.org; Fri, 15 Oct 2021 09:02:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59338) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mbMo6-0006Hv-PP for qemu-devel@nongnu.org; Fri, 15 Oct 2021 08:59:30 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:23681) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mbMo4-0007ow-Ia for qemu-devel@nongnu.org; Fri, 15 Oct 2021 08:59:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634302767; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mWKUxNUGl2oeLuBY/ljbYe4O8kdZCwjFjo4MJYYDqZQ=; b=d8blL0i5V3MTQDGnzcqbtiNlNgYfnC5yPbxumWGlwdbw5mfrjpjAX2ZNBTye5/6Qi6wsta ov1tpanYTMqph39RsOgbancATfJnAjQ++kHvQAj8b/vVvnIMsPhstp/mUeLfOyxSFxkOc9 wgencGJ1q/lUbRudDKgQEm6+TD2mgQg= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-554-IZ30gyy8NL6c7zSLm6r10Q-1; Fri, 15 Oct 2021 08:59:26 -0400 X-MC-Unique: IZ30gyy8NL6c7zSLm6r10Q-1 Received: by mail-ed1-f70.google.com with SMTP id u23-20020a50a417000000b003db23c7e5e2so8169361edb.8 for ; Fri, 15 Oct 2021 05:59:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mWKUxNUGl2oeLuBY/ljbYe4O8kdZCwjFjo4MJYYDqZQ=; b=vr/Tp38X+pWDwQiTg8BRGBWgZH3kZDklCeOhMPh85AHyaUQodzDgb+QvydlQejhCml AiEMGJXoOJwWFpsgQwKSJylPw8Vc6ND2Pt3TaKOHstNHMGRnaYn1emPt+AHeT+fyYcmj THt37bMVqf7iXllxNxeXTrVseiknRpP+GZXvQ4Z1bBG9TgCApC/M7rWUlRSgLCITMkkj +MUCIfm7GyuDF8B2BhdKWVyyy4kqWxckORJyHbGtm/3poocbPz+Dgkhr6xlw2neVLwwl H0L1vriljvs2Yt9ONw963YQf5EbuPOhzsJxhUislV2oZJre8mqWWqPJH6Yc2GNyxWYW+ WgUA== X-Gm-Message-State: AOAM532XhDGFlmjJU4nAag+QiI/OnqptuZgwyKB8Cg4qLJyLGQ4mhOCQ oMy5cD3LAGBN9yYSlAxAKY29YSa8gDoQ1SYRIuGjuDrUamzXTid9uCs4iKgZT38FJdZdNie2eXW EYUA3lq4lm1+sjEw= X-Received: by 2002:a17:906:2346:: with SMTP id m6mr6668307eja.512.1634302764722; Fri, 15 Oct 2021 05:59:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy75CcpOd/V/8F/KFhRAFFlXIZY0vHMRLUXTr3qovj3jx0FQND+UZbK+Sgxj5/F4/Ybja9Z9Q== X-Received: by 2002:a17:906:2346:: with SMTP id m6mr6668280eja.512.1634302764480; Fri, 15 Oct 2021 05:59:24 -0700 (PDT) Received: from localhost (nat-pool-brq-t.redhat.com. [213.175.37.10]) by smtp.gmail.com with ESMTPSA id p8sm4090658ejo.2.2021.10.15.05.59.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Oct 2021 05:59:23 -0700 (PDT) Date: Fri, 15 Oct 2021 14:59:22 +0200 From: Igor Mammedov To: Bin Meng Subject: Re: [PATCH v2] hw/riscv: virt: bugfix the memory-backend-file command is invalid Message-ID: <20211015145922.5a82e31a@redhat.com> In-Reply-To: References: <20211012014501.24996-1-limingwang@huawei.com> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=imammedo@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=170.10.133.124; envelope-from=imammedo@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.049, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:RISC-V" , "limingwang \(A\)" , Bin Meng , "qemu-devel@nongnu.org Developers" , Jiangyifei , "Wanghaibin \(D\)" , Alistair Francis , "Fanliang \(EulerOS\)" , Palmer Dabbelt , "Wubin \(H\)" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Fri, 15 Oct 2021 17:25:01 +0800 Bin Meng wrote: > On Fri, Oct 15, 2021 at 4:52 PM limingwang (A) wrote: > > > > > > On Wed, Oct 13, 2021 at 22:41 PM Bin Meng wrote: > > > > > > On Tue, Oct 12, 2021 at 9:46 AM MingWang Li wrote: > > > > > > > > From: Mingwang Li > > > > > > > > When I start the VM with the following command: > > > > $ ./qemu-system-riscv64 -M virt,accel=kvm -m 4096M -cpu host -nographic \ > > > > -name guest=riscv-guset \ > > > > -smp 2 \ > > > > -bios none \ > > > > -kernel ./Image \ > > > > -drive file=./guest.img,format=raw,id=hd0 \ > > > > -device virtio-blk-device,drive=hd0 \ > > > > -append "root=/dev/vda rw console=ttyS0 earlycon=sbi" \ > > > > -object > > > memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on \ > > > > -numa node,memdev=mem -mem-prealloc \ > > > > -chardev socket,id=char0,path=/mnt/vhost-net0 \ > > > > -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ > > > > -device > > > > virtio-net-pci,mac=52:54:00:00:00:01,netdev=mynet1,mrg_rxbuf=on,csum=o > > > > n,guest_csum=on,guest_ecn=on \ > > > > > > > > Then, QEMU displays the following error information: > > > > qemu-system-riscv64: Failed initializing vhost-user memory map, > > > > consider using -object memory-backend-file share=on > > > > > > I see your command line parameters already contain "-object memory-backend-file > > > share=on", so this error message is not accurate. > > > > QEMU uses this command to alloc fd in the "memory_region_init_ram_from_file" function > > and assigns the value of fd to mr->ram_block-fd. If the QEMU uses the default memory to > > initialize the system, the QEMU cannot obtain the fd in the "vhost_user_mem_section_filter" > > function when initializing the vhost-user. As a result, an error is reported in the "vhost_user_fill_set_mem_table_msg" > > function. > > > > Because of the above bug, even if "-object memory-backend-file share=on" is added to the command line, > > the QEMU still reports an error. > > Yes, what I meant is that QEMU should not report such inaccurate > messages because of some random codes elsewhere. > > With current message, it suggested user use "-object > memory-backend-file share=on" in the command line, but it is already > used. So this is a false alarm. The "bug" is somewhere else. bug is in using memory_region_init_ram(), which can't possibly handle vhost-user, and can't work as expected with '-numa node,memdev' options. Before main ram infrastructure was converted to memdev, one should have used memory_region_allocate_system_memory() for allocating main RAM, so numa usecase was broken from the start. Later it old API was dropped in favor of more flexible/generic MachineState::ram approach (see commits 68a86dc15ccd..f0530f14c7c35d). Modulo commit message, patch looks good to me and does what every machine should do. (I though that I've converted every existing to generalized MachineState::ram but it looks like riscv was missed). So we can model commit message after bd457782b3b0a, and also add that the patch fixes broken -numa node,memdev case, which never properly worked. It also opens possibility to use vhost-user/virtiosf with main RAM if main RAM is provided explicitly via machine.memory-backend option with shared memory backend. Btw: is there other riscv machines that allocate RAM directly? (if yes, those should be fixed as well, a patch per machine) > > This patch can solve this bug. > > > > > Should this message be altered to mention things like what this patch does? > > > > Thanks, I will rewrite the message in next version. > > > > > > > qemu-system-riscv64: vhost_set_mem_table failed: Interrupted system > > > > call (4) > > > > qemu-system-riscv64: unable to start vhost net: 4: falling back on > > > > userspace virtio > > > > > > > > Note that, before starting the kvm-acceled QEMU VM, following > > > > temporarily unaccepted QEMU patches should be used: > > > > https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02516.html > > > > > > > > This error was made bacause default main_mem is used to be registered > > > > > > typo: because > > > > > Thanks. > > > > > > as the system memory, other memory cannot be initialized. Therefore, > > > > the system memory should be initialized to the machine->ram, which > > > > consists of the default main_mem and other possible memory required by > > > > applications, such as shared hugepage memory in DPDK. > > > > Also, the mc->defaul_ram_id should be set to the default main_mem, > > > > such as "riscv_virt_board.ram" for the virt machine. > > > > > > > > > > How about changing the commit title to: "Use machine->ram as the system > > > memory" ?? > > > > > > > I think it is just a bugfix. > > > > But the current codes run perfectly okay so far. This patch adds an > additional use case for the KVM scenario where current codes cannot > handle. > > Regards, > Bin > From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1mbMoB-0006Sn-BN for mharc-qemu-riscv@gnu.org; Fri, 15 Oct 2021 08:59:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59350) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mbMo9-0006Ls-0L for qemu-riscv@nongnu.org; Fri, 15 Oct 2021 08:59:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:25769) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mbMo7-0007pe-1w for qemu-riscv@nongnu.org; Fri, 15 Oct 2021 08:59:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634302769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mWKUxNUGl2oeLuBY/ljbYe4O8kdZCwjFjo4MJYYDqZQ=; b=fnwXRiVIpJ8u4ZD+hG2YZoWAVcmSFFW3Xx9of/hyNjYGM1ySIfMYtmronghSYprEL5QYcN 4PKxFSyjjr7nAJvGmCPFutGkHi1sJa9g/wFePbVfq+JUw7dXkFxgaX1Tp45Z3Jr00YijOX FJ1JFY+kM8gfmGB60FLFpneY2+2KXQY= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-201-zx4UmJsmNSKOe_m42u4tUg-1; Fri, 15 Oct 2021 08:59:26 -0400 X-MC-Unique: zx4UmJsmNSKOe_m42u4tUg-1 Received: by mail-ed1-f71.google.com with SMTP id r11-20020aa7cfcb000000b003d4fbd652b9so8139402edy.14 for ; Fri, 15 Oct 2021 05:59:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mWKUxNUGl2oeLuBY/ljbYe4O8kdZCwjFjo4MJYYDqZQ=; b=239RSInOiF0kb+HEhk102FDgQvsHnVjGCULfix3AbgJJbJXV0QfkEy93xI8En/QHXT ByVQLERearWpT8F+I8f2WVHbhhz7IzdJk0y2N/vizjbx1K4JQ5oUU6PbN0mKcZhJu7Xu AXMMFBkdXNIzX/KT0+DOVFIm2MPPnL1ZnOhcRN5J2RqryEffMa4LQVt7HHOz64k84s/o JH3bCqptGrvkMjbulqrDKp2oDLb7gDWwKNQ0yEGrusITNLwY4D2at6iIBxEA8fFQbK68 rcCsfoNyBotccdYGw0VQu6qsCAa0PoQNRpRI/jC0udebyicWHk4IN8vzC805+FROvX9e /+iA== X-Gm-Message-State: AOAM533Vre1335fCc2JqWq+laQNED6jlbJ/s+c1BqunOAEAw8dlZ7E3q c6ftqR+d930hZf8mnpf8Dbwxmt6fTrFfroQ6iKCIdjr9wNXmkRzKYiFnemc6P/+xSxHmWWgv7jG ho6qDCqiB8BDNOiM= X-Received: by 2002:a17:906:2346:: with SMTP id m6mr6668315eja.512.1634302764784; Fri, 15 Oct 2021 05:59:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy75CcpOd/V/8F/KFhRAFFlXIZY0vHMRLUXTr3qovj3jx0FQND+UZbK+Sgxj5/F4/Ybja9Z9Q== X-Received: by 2002:a17:906:2346:: with SMTP id m6mr6668280eja.512.1634302764480; Fri, 15 Oct 2021 05:59:24 -0700 (PDT) Received: from localhost (nat-pool-brq-t.redhat.com. [213.175.37.10]) by smtp.gmail.com with ESMTPSA id p8sm4090658ejo.2.2021.10.15.05.59.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Oct 2021 05:59:23 -0700 (PDT) Date: Fri, 15 Oct 2021 14:59:22 +0200 From: Igor Mammedov To: Bin Meng Cc: "limingwang (A)" , "open list:RISC-V" , Bin Meng , "qemu-devel@nongnu.org Developers" , Jiangyifei , Alistair Francis , "Wanghaibin \(D\)" , "Fanliang \(EulerOS\)" , Palmer Dabbelt , "Wubin \(H\)" Subject: Re: [PATCH v2] hw/riscv: virt: bugfix the memory-backend-file command is invalid Message-ID: <20211015145922.5a82e31a@redhat.com> In-Reply-To: References: <20211012014501.24996-1-limingwang@huawei.com> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=imammedo@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=216.205.24.124; envelope-from=imammedo@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.049, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-riscv@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Oct 2021 12:59:33 -0000 On Fri, 15 Oct 2021 17:25:01 +0800 Bin Meng wrote: > On Fri, Oct 15, 2021 at 4:52 PM limingwang (A) wrote: > > > > > > On Wed, Oct 13, 2021 at 22:41 PM Bin Meng wrote: > > > > > > On Tue, Oct 12, 2021 at 9:46 AM MingWang Li wrote: > > > > > > > > From: Mingwang Li > > > > > > > > When I start the VM with the following command: > > > > $ ./qemu-system-riscv64 -M virt,accel=kvm -m 4096M -cpu host -nographic \ > > > > -name guest=riscv-guset \ > > > > -smp 2 \ > > > > -bios none \ > > > > -kernel ./Image \ > > > > -drive file=./guest.img,format=raw,id=hd0 \ > > > > -device virtio-blk-device,drive=hd0 \ > > > > -append "root=/dev/vda rw console=ttyS0 earlycon=sbi" \ > > > > -object > > > memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on \ > > > > -numa node,memdev=mem -mem-prealloc \ > > > > -chardev socket,id=char0,path=/mnt/vhost-net0 \ > > > > -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ > > > > -device > > > > virtio-net-pci,mac=52:54:00:00:00:01,netdev=mynet1,mrg_rxbuf=on,csum=o > > > > n,guest_csum=on,guest_ecn=on \ > > > > > > > > Then, QEMU displays the following error information: > > > > qemu-system-riscv64: Failed initializing vhost-user memory map, > > > > consider using -object memory-backend-file share=on > > > > > > I see your command line parameters already contain "-object memory-backend-file > > > share=on", so this error message is not accurate. > > > > QEMU uses this command to alloc fd in the "memory_region_init_ram_from_file" function > > and assigns the value of fd to mr->ram_block-fd. If the QEMU uses the default memory to > > initialize the system, the QEMU cannot obtain the fd in the "vhost_user_mem_section_filter" > > function when initializing the vhost-user. As a result, an error is reported in the "vhost_user_fill_set_mem_table_msg" > > function. > > > > Because of the above bug, even if "-object memory-backend-file share=on" is added to the command line, > > the QEMU still reports an error. > > Yes, what I meant is that QEMU should not report such inaccurate > messages because of some random codes elsewhere. > > With current message, it suggested user use "-object > memory-backend-file share=on" in the command line, but it is already > used. So this is a false alarm. The "bug" is somewhere else. bug is in using memory_region_init_ram(), which can't possibly handle vhost-user, and can't work as expected with '-numa node,memdev' options. Before main ram infrastructure was converted to memdev, one should have used memory_region_allocate_system_memory() for allocating main RAM, so numa usecase was broken from the start. Later it old API was dropped in favor of more flexible/generic MachineState::ram approach (see commits 68a86dc15ccd..f0530f14c7c35d). Modulo commit message, patch looks good to me and does what every machine should do. (I though that I've converted every existing to generalized MachineState::ram but it looks like riscv was missed). So we can model commit message after bd457782b3b0a, and also add that the patch fixes broken -numa node,memdev case, which never properly worked. It also opens possibility to use vhost-user/virtiosf with main RAM if main RAM is provided explicitly via machine.memory-backend option with shared memory backend. Btw: is there other riscv machines that allocate RAM directly? (if yes, those should be fixed as well, a patch per machine) > > This patch can solve this bug. > > > > > Should this message be altered to mention things like what this patch does? > > > > Thanks, I will rewrite the message in next version. > > > > > > > qemu-system-riscv64: vhost_set_mem_table failed: Interrupted system > > > > call (4) > > > > qemu-system-riscv64: unable to start vhost net: 4: falling back on > > > > userspace virtio > > > > > > > > Note that, before starting the kvm-acceled QEMU VM, following > > > > temporarily unaccepted QEMU patches should be used: > > > > https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02516.html > > > > > > > > This error was made bacause default main_mem is used to be registered > > > > > > typo: because > > > > > Thanks. > > > > > > as the system memory, other memory cannot be initialized. Therefore, > > > > the system memory should be initialized to the machine->ram, which > > > > consists of the default main_mem and other possible memory required by > > > > applications, such as shared hugepage memory in DPDK. > > > > Also, the mc->defaul_ram_id should be set to the default main_mem, > > > > such as "riscv_virt_board.ram" for the virt machine. > > > > > > > > > > How about changing the commit title to: "Use machine->ram as the system > > > memory" ?? > > > > > > > I think it is just a bugfix. > > > > But the current codes run perfectly okay so far. This patch adds an > additional use case for the KVM scenario where current codes cannot > handle. > > Regards, > Bin >